SCOM: Advanced SNMP Monitoring Part I: Discovering Cisco Devices and Hosted Classes

When limited to editing within the Authoring pane of the Operations Console in OpsMgr, SNMP monitoring options are relatively limited – only SNMP GET requests are supported and data cannot be manipulated before calculating health state.   However, with some pretty heavy authoring, very complex SNMP monitoring becomes completely viable, particularly in R2.   In the next several posts, I will be describing some methods to perform advanced SNMP discovery and monitoring in OpsMgr 2007, such as discovering hosting relationships for entities like network interfaces, as well as passing Snmp data to script probes in order to perform state change monitoring and mathematical operations.  When all is said and done, I will post my management packs for SNMP monitoring of Cisco, Checkpoint, and Net-SNMP devices.   In this post, I will describe the methods I used for custom discovery of Cisco SNMP devices and their hosted entities (interfaces, power supplies, etc).

Cisco SNMP Management Pack: Designing the Classes and Groups

In order to facilitate monitoring of the device (e.g. CPU, Memory, up/down status), individual interfaces (e.g. status, utilization, resets), and Cisco EnvMon objects (e.g. Power Supplies, Temperature, etc), each of these entities are best represented as a unique class with hosting relationships.   For the objects stored in tables, there are two feasible options for discovery.  In SCOM 2007 R2, the ‘walkreturnmultipleitems’ attribute of the System.SnmpProbe probe action can be set in order to walk an SNMP table and return each snmpdata item individually.   In SCOM 2007 SP1, this option is not available, requiring the use of a script discovery with an external snmp probe (such as WMI) in order to retrieve the discovery items.  In this post, I will describe the R2 method, and I will address the script option using the WMI SNMP provider in a later post.  In either case, the class design will be similar.

For monitoring of Cisco SNMP devices, I created five classes:

• CiscoSNMP.Class.CiscoDevice
• CiscoSNMP.Class.CiscoDevice.Interface
• CiscoSNMP.Class.CiscoDevice.PowerSupply
• CiscoSNMP.Class.CiscoDevice.Fan
• CiscoSNMP.Class.CiscoDevice.TemperatureSensor

For the four hardware entities, I created a hosting relationship so that they are hosted by the CiscoDevice class.   The classes, properties, and relationships are represented on this diagram.

Additionally, in order to target monitor overrides for specific interface types, I created two groups:

• CiscoSNMP.Group.CiscoSerialInterfaces
• CiscoSNMP.Group.CiscoEthernetInterfaces

A containment relationship was added for each of these groups to define that they contain CiscoDevice.Interface objects.

Discovering the Base Device Class

The base CiscoDevice class can be discovered by looking for the string: 1.3.6.1.4.1.9. in the device’s SysObjectID (.1.3.6.1.2.1.1.2.0 in the RFC-1213 MIB).   This discovery can be done in OpsMgr with a composite data source combing a scheduler, an SNMP Probe, and a FilteredClassSnapShotDataMapper.  An example of this can be found here:  (note, there are a few things that need to be tweaked to get this example to work).  Additionally, Raphael Burri has created a sealed management pack that provides this functionality in a reusable form.  I opted to go the composite data source route similar based on the methodology in the TechNet example.

The data source modules:

The SnmpProbe config:

And the Mapper Configuration:

Discovering the Interfaces:

As I mentioned previously, the “walkreturnmultipleitems” flag can be set along with the “walk” setting in the System.SnmpProbe configuration in order to return multiple SnmpData objects to the discovery mapper.  Thus, to discover the interfaces, an SnmpProbe can be configured to walk the column of ifIndexes (.1.3.6.1.2.1.2.2.1.1) in the RFC-1213 ifTable.   It’s worth noting that when an SNMP walk is used for the SnmpProbe, an ExpressionFilter should be added to filter the results by OID to make sure that what is being returned is what is expected.   Because a walk is a sequence of GET NEXT requests, if no values are returned at the expected OID, the walk may move on to another set of OID’s, returning unexpected values. 

So this discovery data source has a module composition like this:

 

The SnmpProbe configuration:

The filter (to confirm the values are from the expected OID’s):

And a screenshot of the mapper configuration (the expression filter here is just confirming that a valid value has been returned for the Index):

When combined with the appropriate discovery rule (targeted to CiscoDevices, and set to discover both the Interface class and the hosting relationship), the device’s interfaces will be created as objects of the CiscoDevice.Interface class with the relationship we want.  From my experimentation, it seems that returning multiple SnmpVarBinds when the “walkreturnmultipleitems” property is set to true is not a supported configuration.   So, to discover other relevant interface properties that will be used in monitoring (such as interface type and speed), a second discovery data source and discovery rule is required, this time targeting the individual interfaces (I have the interface discovery scheduled for every 24 hours, and the interface property discovery scheduled for every 30 minutes, using slightly offset synchronization times so that the property discovery will execute a few minutes after the daily interface discovery).

In this second discovery, I am passing the interface index from the discovery rule to the data source in order to define the appropriate SnmpVarBinds.   For example, the interface description in the ifTable has an OID of .1.3.6.1.2.1.2.2.1.2.X, where X is the index of the interface.  So by passing the index that was discovered previously, the SnmpVarBinds OID can be defined as .1.3.6.1.2.1.2.2.1.2.$Config/Index$.  

Discovering the Cisco EnvMon Objects:

This is where the discoveries start to get a bit tricky.   I had planned on discovering the Cisco EnvMon objects (power supplies, fans, etc) with the exact approach as the interfaces.   However, in my testing, it seems that Cisco devices don’t actually populate the index field in the relevant EnvMon tables, which makes discovering the index a bit more complicated.  In order to target SNMP GET requests to the individual objects, the index must be known, so I had to implement a work around.  In order to discover these objects, I added a key property to the class definition named OID.  The initial SnmpProbe discover populates only the object name (snmpvarbind[1]\value) and the OID (snmpvarbind[1]\OID) properties.   A second, more frequent discovery is then used to pass the OID value (from the class property) to a property bag script probe which calculates the index value (the last digit(s) in the OID string), and then updates the Index property for the class object.   In this case, the secondary discovery doesn’t actually poll the device.

Note: In the initial version of this post, I described a method I was using involving passing SnmpVarBinds from an SnmpProbe (set to retrieve multiple items) to a property bag script probe and then a mapper, to allow the script probe to calculate the index value from the SnmpVarBind’s OID.  At the time, this appeared to be a viable method, but once I expanded my testing to SNMP objects that had multiple instances of the objects being discovered, I realized the flaw in this method.  When an SnmpProbe is set to return multiple items, it passes the multiple instances of SnmpData objects to the mapper concurrently.  This is comparable to creating multiple discovery instances in a single script and returning them concurrently in one discovery data object and the multiple instances are processed as a single snapshot.   However, when the SnmpProbe passes the the arguments to a property bag script probe, the SnmpData object is then handled serially by the mapper. The mapper interprets the single discovery data instance as a snapshot and deletes any previously discovered objects for that target.

Discovering the Interface Groups

Discovering the interface groups (Ethernet and serial) can be done with a straightforward group populator targeted at the RMS. 

And the Results

Viewing one of the devices in a diagram view:

The list of discovered interfaces:

Viewing the members of one of the groups:

Stay tuned for the next posts on monitoring these objects as well as the link to download the management pack.