SCOM: SP1 Edition of the Cisco Management Pack , v1.0.2.6

I have completed the first version of the Cisco Management Pack for SP1 compatibility.  The monitoring in the management pack is identical to the R2 version of the MP, described most recently here.      Due to the dependence on the WMI SNMP provider for object discovery, there are inevitably some scalability limitations intrinsic to the SP1 edition of this MP, but I haven’t done enough full-scale testing to ascertain those limitations as of yet.   Additionally, deployment of this management pack requires some additional steps.   These steps are detailed below (taken from the MP documentation):

Prerequisites

This management pack utilizes the WMI SNMP provider to perform discovery of SNMP objects.  In order to use this management pack, the following steps must be completed on each server that will function as a proxy agent for SNMP-monitored Cisco devices.

Install the SNMP protocol and WMI SNMP provider

• To install these components, access Add/Remove Programs in the Control Panel, and select Add/Remove Windows Components.  Under Management and Monitoring select: Simple Network Management Protocol and WMI SNMP Provider

The following MIBs must be exported to MOF files with smi2smir.exe and imported with MOFComp.exe:  CISCO-ENVMON-MIB and CISCO-MEMORY-POOL-MIB.  

• The MIBs and a batch file to perform these steps can be found in the /Setup directory included with this MP distribution.   Run the RegCiscoMibs.cmd file and check the output log: register.log to confirm that the mibs were successfully compiled and imported.

Recommended Proxy Agent Configuration

If WMI receives too many requests in a short time, it may suspend processing of requests for a period of time.   This can impact the ability of this management pack to discover Cisco SNMP objects in a timely fashion (WMI SNMP is only used by this MP for discovery, and not object monitoring).   In order to minimize the chances of this situation occurring, the object discoveries in this MP should not be scheduled too aggressively.   Additionally, it is recommended that if a large number of Cisco SNMP devices are to be monitored, they should be distributed across multiple proxy agents for load-balancing of the WMI SNMP requests. 

It is highly recommended that all agents that will function as proxy agents for SNMP devices have the Windows Script Host version 5.7 installed.  Version 5.7 is far more efficient than previous versions and reduces the resource utilization by the cscript.exe process dramatically.

It is also highly recommended that all agents that will function as proxy agents for SNMP devices have the hotfix: KB96163 applied.  This hotfix resolves instability with SNMP monitoring in Operations Manager 2007 SP1. http://support.microsoft.com/default.aspx/kb/961363

I’m interested to hear how this SP1 edition of the Cisco Management Pack functions in different environments, so any feedback is most certainl welcome.  I will continue to post updates to this site so be sure to check back regularly.   For more information about the scripts utilized in discoveries in this edition of the MP, this post should sum it up.

SCOM: Updates to the Cisco Management Pack (R2) v1.0.2.6

I’m hoping to finish up the SP1 version of the Cisco Management Pack pretty soon, but I’ve modified the R2 version to include several new changes.  The current version: 1.0.2.6 can be downloaded here. 

The changes in this version are:

• Added three new containment classes:  Cisco Device Chassis, Cisco Device System Components, and Cisco Device Interfaces.   These classes contain monitored objects to add an additional level of hierarchical organization.
• Added discovery of the IFAlias property for Interfaces
• Added discovery of the Hostname (OLD-CISCO-MIB) and Chassis description for the Cisco Device class.
• Updated the properties displayed by default in the Device and Interface views
• Added a rule to clean up unused XML temporary files once a day.   Several of the monitors utilize temporary XML files written to the %TEMP% path.  In the previous version, old files would be left on the file system if a previously monitored object was removed.  This rule will remove those temporary files.
• Modified discovery intervals for some objects for more balanced timing.
• Added four new monitors for switches that implement the CISCO-STACK-MIB.  The monitors are targeted at the Cisco Device Chassis class and include
  • Fan Alarm
  • Temperature Alarm
  • Minor Alarm
  • Major Alarm

With the new containment classes, the diagram view looks a lot better:

SCOM: Updates to the Net-SNMP Management Packs. v1.0.1.30

I’ve made a few minor updates to the Net-SNMP Management Packs, available at the same download location.   The changes in 1.0.1.30 are:

• Implemented new class:  Net-SNMP Monitored Process Instance, along with rules to collect and monitor process instance CPU and memory use.  Reference the MP documentation in the zip file (and the preceding post) for more information.
• Updated all alerts to include the hosting device name.
• Added a rule to periodically purge old temporary files used by the Interface Utilization data source.

Any future updates will be posted to this blog, and thanks to everyone who has commented on these management packs. 

Some screenshots of the new process instance monitoring capabilities:

SCOM: UNIX/Linux Process Monitoring in the Net-SNMP MP In Detail

Regardless of the operating system, monitoring of the availability and resource utilization of individual processes is a pretty standard requirement.  Between WMI and PerfMon counters, this is easy on Windows systems, but doing the same on UNIX/Linux systems can be a little bit more complicated.   In Operations Manager 2007 (R2) environments, there are three general approaches (excluding third party products) that can be utilized to monitor individual processes on UNIX and Linux systems:

1. An agent-based solution using the R2 Cross Platform agents
2. A purely SNMP solution using tables in the HOST-RESOURCES MIB
3. An extended SNMP solution using the proc or exec directives in the Net-SNMP agent’s snmpd.conf file

I think it’s fair to say that in most cases (and when it is supported), the R2 Cross Platform agent is the best and most robust approach.   However, it’s almost an inevitability in medium and large enterprises that there will be some UNIX or Linux servers or appliances running distributions not supported by the R2 agents.  In these cases, or if there is another compelling reason not to deploy agent software to the device, SNMP may be the best or only option.   The pure SNMP option is probably the most universally applicable approach, but introduces a number of challenges, which I will discuss in this post.  The third option brings a great degree of flexibility (particularly with the exec directive, which can return the result of an on-demand shell script to an SNMP OID) but requires decentralized configuration. 

The approach that I took in the Net-SNMP Management Pack is a hybrid of the pure SNMP and extended SNMP options.  The latest version of the MP (which I will be posting soon) supports process resource utilization through the HOST-RESOURCES MIB tables in addition to process availability monitoring facilitated by identifying the monitored processes with the proc directive in snmpd.conf.  And as described in the previous post about the MP, if ultimate flexibility is needed, the Extensible Object capability with the exec directive is still supported.  

UNIX/Linux SNMP Process Monitoring In-Depth

Read more of this post

SCOM: Net-SNMP Management Packs for UNIX/Linux Monitoring, Version 1.0.1

I have completed version 1.0.1 of the Net-SNMP management packs for OpsMgr 2007 R2 , and I thought I’d go ahead and share them.  At this point, my testing of the management packs has been on a small development network with Sun and CentOS servers, but so far everything is looking pretty good.   These management packs should provide a pretty good set of monitoring for UNIX and Linux servers that run the Net-SNMP agent, which includes Solaris, most Linux distributions, and even VMWare ESX and Checkpoint Secure Platform. 

As I had discussed in my last post, there are two management packs in this set.  The Net-SNMP Library management pack defines the classes and performs discoveries.   This can be imported by itself to facilitate completely custom monitoring or to be referenced in other management packs.   The Net-SNMP Monitoring management pack implements a pretty standard set of monitors for UNIX/Linux server performance and availability monitoring and supports the Exec, Proc, and File directives of the Net-SNMP agent configuration. 

The management packs can be downloaded here.  They are released under the GNU Public License and can be used, modified, and distributed freely, as long as the attribution remains intact. 

Some screenshots:

Read more of this post

SCOM: A Net-SNMP Management Pack for UNIX Monitoring, Part I – Desiging the MP

The Net-SNMP agent is a cross-platform SNMP agent that is often bundled with UNIX (e.g. Solaris) and Linux distributions (including VMWare and Checkpoint Splat).   The agent implements some relatively standard MIBs, such as the UC Davis (UCD-SNMP-MIB) and HOST-RESOURCES MIBs, exposing a good selection of operating system metrics for monitoring.  Furthermore, the agent is remarkably extensible in a few different ways.  The agent can be extended with “dlMod” module extensions to integrate additional SNMP components, such as the VMWare or HP Insight Management Agent modules.   Additionally, the Net-SNMP agent can be configured to proxy for another SNMP agent listening on a different port, which is how the Sun SunFire Management Agent is typically configured.   Lastly, the Net-SNMP agent provides the ability to define processes, files, extensible objects, load average objects, and disks for specific monitoring.  For each of these, directives are configured in snmpd.conf with parameters and thresholds, and an SNMP table is exposed that will return the monitored value and status value to an SNMP poll, on-demand.   I had previously  posted an article on using the ext (Extensible Object/Arbitrary Extension) capabilities of the Net-SNMP agent to utilize shell scripts for monitoring of Sun hardware.   With this degree of flexibility, it’s easy to understand why the Net-SNMP agent is so widely utilized.

Design Approach for the Net-SNMP Management Pack

Building on the lessons learned from developing the Cisco Management Pack, I wanted to create a management pack with similar capabilities for monitoring of Net-SNMP agents, followed by a set of management packs for monitoring of vendor-specific hardware (e.g. HP Proliant servers running Linux).   After some thought, I decided that it would be advantageous to split the Net-SNMP management pack into two separate management packs, one that defines the base classes and performs discovery of the classes and one for the actual monitoring.   The reason for this is that I wanted a base management pack without any actual monitoring functions so that it could easily (and more flexibly) be reused for future hardware monitoring management packs.   To illustrate this point, if the native UNIX agent capabilities of R2 are being used to monitor OS-level objects, there is still a need to discover the Net-SNMP agent and create base classes for use in SNMP-based hardware monitoring, but in this case, monitoring OS-level objects with SNMP and the OpsMgr agent would be redundant.  With the multiple MP approach, just the base Net-SNMP management pack could be imported along with the hardware monitoring management packs, skipping the Net-SNMP monitoring management pack altogether.  Additionally, I wanted to leave the monitoring management pack unsealed for easy customization, and MP references can only be made to sealed management packs.  In this way, the Net-SNMP Library MP will be sealed, but the monitoring MP can remain unsealed.

The Net-SNMP Library Management Pack

Read more of this post

SCOM: Advanced SNMP Monitoring Part IV: Making the Cisco MP SP1 Compatible

As previously mentioned, the Cisco Management Pack that I have been describing in the past several posts is specific to OpsMgr R2, due to its reliance on the capability of the System.SnmpProbe’s function to return multiple items in an SNMP walk, which is a feature introduced in R2.   My intention was to design the management pack so that it could easily be made SP1 compatible by substituting the discovery methodology, but keeping the classes and monitoring objects intact.   I am still doing more testing to finalize the SP1 version, but in this post, I am going to introduce the discovery methods that I have implemented for SP1 compatibility.

In general, when dealing with SNMP objects stored in tables, options in SCOM 2007 SP1 are a bit limited.  If the table objects are static, rules could be configured targeting specific OID’s, but this is unlikely to be a common scenario.   Another option (and a quite good one) would be to segregate monitoring duties between SCOM (for servers and applications) and a task-specific SNMP monitoring tool like SolarWinds or JalaSoft for SNMP device monitoring.   But for the purposes of this management pack, the option I pursued involves wrapping an external SNMP provider in WSH scripts to perform the discovery.   I chose to use the WMI provider because of its easy use in VBscripts and object-oriented capabilities, but something like the Net-SNMP binaries could be called with ShellExec methods in a vbs script to accomplish the same goal.  

Before I get into the use of the WMI SNMP provider in this MP, there is some background information that should be discussed.  Firstly, when the WMI SNMP provider is installed, it includes support only for the RFC-1213 MIB.   In order to use the WMI SNMP provider for OID’s not in the RFC-1213 MIB, the MIB files must be converted to MOF files with smi2smir.exe and then imported with mofcomp.exe.    This can get a bit tricky, largely depending on how well constructed the vendor’s MIB is and the dependencies of the MIB.   Once these steps are completed, the WMI SNMP provider exposes SNMP objects as collections that can be accessed natively in VBscripts.   However, the WMI provider is not terribly scalable and when overloaded with requests, WMI will cease to function in a timely fashion.   I had done some testing with using the WMI SNMP provider for rules and monitors, but decided against this approach due to the ease with which the provider can be overloaded when monitoring a handful of objects with reasonably aggressive frequencies.   On the other hand, discoveries are not as time-sensitive as monitors/rules and don’t need to run as frequently, so by using the WMI SNMP provider for discoveries, and the System.SnmpProbe provider for monitors and rules, this hybrid approach should remain pretty scalable. 

Implementation Overview

Read more of this post

SCOM: Updates to the Cisco Management Pack

I have uploaded version 1.01.27 of the Cisco Management Pack.   This minor update includes a few fixes and some new features added in response to some helpful comments from readers who have tried this MP out.  The changes in this version are:

1. Group Population rules have been fixed so that they can be edited in the GUI
2. Performance data collection has been modified to not collect data when the interface is in a partially discovered state.  This should prevent duplication of performance counters for interfaces going forward.   Because the Interface discovery occurs in two parts: 1) discovering the interfaces and 2) discovering the interface properties, I added a property to the Interface class named: Discovered.  When the Interface discovery runs (which just discovers the index), the discovered property will be set to False.  Once the Interface Property discovery runs, this property will be set to True.   The performance collection rules have been modified to only collect data if this property is set to true. 
3. Index discovery can now be filtered with a RegEx expression for a device via an override (create the override on the Discover Cisco Interfaces discovery).   By default, the RegEx expression:  (.|..|…) will be used to discover interfaces, which will discover all interfaces with indexes with 1, 2, or 3 digits in length.  To filter by interface index, the expression can be modified to:  (1|2|3|10|15) where the numbers are the index values for the interfaces to be discovered, separated by pipe characters.   The override for this filter can be targeted at all Cisco devices, a group, or specific devices.
4. I have added a new property to the Interface class named:  “Speed – Override.”  This property allows for manually defining a speed value for the interface to be used in utilization calculations.  There are two scenarios where this is useful: 1) sometimes the value reported in the RFC-1213 ifTable for an interface’s speed isn’t accurate, and 2) if an Ethernet port is the last monitored hop before a lower bandwidth wide-area connection (i.e. the vendor equipment is not monitored), it can be useful to monitor the Ethernet port as if its speed were equivalent to the wide-area connection bandwidth.   For example, if a switch is the last monitored device before connecting to a vendor-managed MetroE connection with a maximum bandwidth of 500Mbit, monitoring utilization on the gigabit switch port will be more accurate if the speed were overridden to 500Mbit.  This property can be set with an override on the Discover Cisco Interface Properties object discovery, by setting the OverrideIFSpeed value to the desired speed, in bits per second.

SCOM: Advanced SNMP Monitoring Part III: The Completed Cisco Management Pack

Note:  This management pack has been replaced by the more feature-rich xSNMP suite.

I still have some more full-scale testing to complete, but I have finished the first version of the Cisco monitoring management pack that I have described in the last few posts.   This version is R2 only, but I hope to have an SP1 version of it complete soon. 

Read more of this post