SCOM: Advanced SNMP Monitoring Part I: Discovering Cisco Devices and Hosted Classes

When limited to editing within the Authoring pane of the Operations Console in OpsMgr, SNMP monitoring options are relatively limited – only SNMP GET requests are supported and data cannot be manipulated before calculating health state.   However, with some pretty heavy authoring, very complex SNMP monitoring becomes completely viable, particularly in R2.   In the next several posts, I will be describing some methods to perform advanced SNMP discovery and monitoring in OpsMgr 2007, such as discovering hosting relationships for entities like network interfaces, as well as passing Snmp data to script probes in order to perform state change monitoring and mathematical operations.  When all is said and done, I will post my management packs for SNMP monitoring of Cisco, Checkpoint, and Net-SNMP devices.   In this post, I will describe the methods I used for custom discovery of Cisco SNMP devices and their hosted entities (interfaces, power supplies, etc).

Cisco SNMP Management Pack: Designing the Classes and Groups

In order to facilitate monitoring of the device (e.g. CPU, Memory, up/down status), individual interfaces (e.g. status, utilization, resets), and Cisco EnvMon objects (e.g. Power Supplies, Temperature, etc), each of these entities are best represented as a unique class with hosting relationships.   For the objects stored in tables, there are two feasible options for discovery.  In SCOM 2007 R2, the ‘walkreturnmultipleitems’ attribute of the System.SnmpProbe probe action can be set in order to walk an SNMP table and return each snmpdata item individually.   In SCOM 2007 SP1, this option is not available, requiring the use of a script discovery with an external snmp probe (such as WMI) in order to retrieve the discovery items.  In this post, I will describe the R2 method, and I will address the script option using the WMI SNMP provider in a later post.  In either case, the class design will be similar.

For monitoring of Cisco SNMP devices, I created five classes:

  • CiscoSNMP.Class.CiscoDevice
  • CiscoSNMP.Class.CiscoDevice.Interface
  • CiscoSNMP.Class.CiscoDevice.PowerSupply
  • CiscoSNMP.Class.CiscoDevice.Fan
  • CiscoSNMP.Class.CiscoDevice.TemperatureSensor

For the four hardware entities, I created a hosting relationship so that they are hosted by the CiscoDevice class.   The classes, properties, and relationships are represented on this diagram.

Additionally, in order to target monitor overrides for specific interface types, I created two groups:

  • CiscoSNMP.Group.CiscoSerialInterfaces
  • CiscoSNMP.Group.CiscoEthernetInterfaces

A containment relationship was added for each of these groups to define that they contain CiscoDevice.Interface objects.

Discovering the Base Device Class

The base CiscoDevice class can be discovered by looking for the string: 1.3.6.1.4.1.9. in the device’s SysObjectID (.1.3.6.1.2.1.1.2.0 in the RFC-1213 MIB).   This discovery can be done in OpsMgr with a composite data source combing a scheduler, an SNMP Probe, and a FilteredClassSnapShotDataMapper.  An example of this can be found here:  (note, there are a few things that need to be tweaked to get this example to work).  Additionally, Raphael Burri has created a sealed management pack that provides this functionality in a reusable form.  I opted to go the composite data source route similar based on the methodology in the TechNet example.

The data source modules:

The SnmpProbe config:

And the Mapper Configuration:

Discovering the Interfaces:

As I mentioned previously, the “walkreturnmultipleitems” flag can be set along with the “walk” setting in the System.SnmpProbe configuration in order to return multiple SnmpData objects to the discovery mapper.  Thus, to discover the interfaces, an SnmpProbe can be configured to walk the column of ifIndexes (.1.3.6.1.2.1.2.2.1.1) in the RFC-1213 ifTable.   It’s worth noting that when an SNMP walk is used for the SnmpProbe, an ExpressionFilter should be added to filter the results by OID to make sure that what is being returned is what is expected.   Because a walk is a sequence of GET NEXT requests, if no values are returned at the expected OID, the walk may move on to another set of OID’s, returning unexpected values. 

So this discovery data source has a module composition like this:

 

The SnmpProbe configuration:

The filter (to confirm the values are from the expected OID’s):

And a screenshot of the mapper configuration (the expression filter here is just confirming that a valid value has been returned for the Index):

When combined with the appropriate discovery rule (targeted to CiscoDevices, and set to discover both the Interface class and the hosting relationship), the device’s interfaces will be created as objects of the CiscoDevice.Interface class with the relationship we want.  From my experimentation, it seems that returning multiple SnmpVarBinds when the “walkreturnmultipleitems” property is set to true is not a supported configuration.   So, to discover other relevant interface properties that will be used in monitoring (such as interface type and speed), a second discovery data source and discovery rule is required, this time targeting the individual interfaces (I have the interface discovery scheduled for every 24 hours, and the interface property discovery scheduled for every 30 minutes, using slightly offset synchronization times so that the property discovery will execute a few minutes after the daily interface discovery).

In this second discovery, I am passing the interface index from the discovery rule to the data source in order to define the appropriate SnmpVarBinds.   For example, the interface description in the ifTable has an OID of .1.3.6.1.2.1.2.2.1.2.X, where X is the index of the interface.  So by passing the index that was discovered previously, the SnmpVarBinds OID can be defined as .1.3.6.1.2.1.2.2.1.2.$Config/Index$.  

Discovering the Cisco EnvMon Objects:

This is where the discoveries start to get a bit tricky.   I had planned on discovering the Cisco EnvMon objects (power supplies, fans, etc) with the exact approach as the interfaces.   However, in my testing, it seems that Cisco devices don’t actually populate the index field in the relevant EnvMon tables, which makes discovering the index a bit more complicated.  In order to target SNMP GET requests to the individual objects, the index must be known, so I had to implement a work around.  In order to discover these objects, I added a key property to the class definition named OID.  The initial SnmpProbe discover populates only the object name (snmpvarbind[1]\value) and the OID (snmpvarbind[1]\OID) properties.   A second, more frequent discovery is then used to pass the OID value (from the class property) to a property bag script probe which calculates the index value (the last digit(s) in the OID string), and then updates the Index property for the class object.   In this case, the secondary discovery doesn’t actually poll the device.

Note: In the initial version of this post, I described a method I was using involving passing SnmpVarBinds from an SnmpProbe (set to retrieve multiple items) to a property bag script probe and then a mapper, to allow the script probe to calculate the index value from the SnmpVarBind’s OID.  At the time, this appeared to be a viable method, but once I expanded my testing to SNMP objects that had multiple instances of the objects being discovered, I realized the flaw in this method.  When an SnmpProbe is set to return multiple items, it passes the multiple instances of SnmpData objects to the mapper concurrently.  This is comparable to creating multiple discovery instances in a single script and returning them concurrently in one discovery data object and the multiple instances are processed as a single snapshot.   However, when the SnmpProbe passes the the arguments to a property bag script probe, the SnmpData object is then handled serially by the mapper. The mapper interprets the single discovery data instance as a snapshot and deletes any previously discovered objects for that target.

Discovering the Interface Groups

Discovering the interface groups (Ethernet and serial) can be done with a straightforward group populator targeted at the RMS. 

And the Results

Viewing one of the devices in a diagram view:

The list of discovered interfaces:

Viewing the members of one of the groups:

Stay tuned for the next posts on monitoring these objects as well as the link to download the management pack.

Advertisements

About Kristopher Bash
Kris is a Senior Program Manager at Microsoft, working on UNIX and Linux management features in Microsoft System Center. Prior to joining Microsoft, Kris worked in systems management, server administration, and IT operations for nearly 15 years.

8 Responses to SCOM: Advanced SNMP Monitoring Part I: Discovering Cisco Devices and Hosted Classes

  1. Dan Nichols says:

    Hi Kris,

    Firstly thank you for sharing your efforts through what is a very well written and content rich blog. I hope you are able to continue with this degree of momentum!

    When I saw your Cisco SNMP MP I thought I would try to create a HP network device version using the same logic with only the necessary changes.

    I started with a new blank MP and started from the bottom defining the base classes and discovery logic etc. Whilst I was recreating a HP specific version of your discovery data source modules I thought to myself would it be possible to re-use some standard modules and simply substitute the vendor specific values. So I created a core/library MP that contains a device discovery DSM and defined additional parameters that would be substituted. Then I created a second MP and defined a device class and a device discovery that uses the discovery DSM in the core MP. In the configuration of the device discovery I populated the values to substitute in the DSM, e.g. the class that is being discovered and the OID value to match in a discovery. I tested this configuration and the substitution works as I hoped with successful discovery of the defined class instances.

    I think it would be good to create a series of modular SNMP MP’s to make it easier for others to contribute MP’s for other vendors. What do you think to this approach? Can you see any pitfalls? Would you be interested in taking the MP you have written in this direction?

    Keep up the good work.
    Dan.

    • Kristopher Bash says:

      Thank you for your comments. The modular approach does sound interesting, and I’d be interested in any approach that fosters community development. In terms of SNMP-monitoring, there is a whole lot out there to be monitored, so a collaborative development process would certainly be a noble goal. I suppose there is some balance to be found in terms of determining those classes and subcomponents that are reusable, and those that are vendor or product specific. Would you care to share more about your approach – the general architecture, what classes are defined, etc? The Cisco MP was a learning process for me, implementing some capabilities of the OpsMgr SNMP implementation that were new to me as well as tackling some relatively difficult monitoring implementations. Cisco device monitoring was a great starting point for this as Cisco devices provide a pretty consistent and wholly contained silo of hardware and OS objects, but a modular approach could certainly have its benefits. I’m sure many revisions will be made to that MP in the future, and if a modular approach looks to be more viable, I certainly have no opposition to pursuing that direction.

      Thanks again for your comments, and please feel free to share whatever details about your development efforts that you are comfortable with sharing.

      • Dan Nichols says:

        Hi Kris,

        I too am exploring the capabilities of OpsMgr with SNMP and your Cisco MP has broadened my understanding. I am not a developer but have authored MPs to provide bespoke monitoring requirements for my customers.

        When I get some more time I will continue to develop the MPs that I have started and see how viable my approach looks, I have only really scratched the surface and it may turn out to be a no-thru-road that I am headed down!

        I would hope that your work will encourage some MP or SNMP experts to speak out and perhaps suggest how if/how this approach could best be applied.

    • Kristopher Bash says:

      It seems to me that there would be a few ways to approach this with more modularity. One way would be to just create a sealed management pack which contains reusable data source definitions (which would have instance settings and snmpvarbinds passed to them as configuration items) to be reused in other MPs. Another option would be to go a bit more complex and define universal classes (e.g. power supply, chassis, network interface) much like some of the objects in the System libraries, and attempt to expose flexible discoveries. And lastly, another approach would be to identify the nearly universal SNMP tables (ifTable, hrStorage, etc) that would be used widely by many SNMP implementations and put them in a universal SNMP MP and have vendor/implementation specific monitoring in different MPs.

      Do one of these methods align with what you had in mind? What do you think is the best approach?

  2. Alexey says:

    Hello!

    We use this MP for SCOM.
    But this MP dont discover network interface whith type 131 (tunnel) 😦
    Can you give some reomendation how we can do it?

    • Kirill says:

      Alexey, do you resolve this issue? Kris, it’s very actual problem to monitor tunnel interfaces on cisco devices. Could you give some recomedation?

  3. Amro says:

    How can i move the switch to the cisco container?
    I can see my switch in Network Devices but i cant see the switch in Cisco container.
    Can you help me please?

  4. Arek says:

    Hi,

    I have created my own MP based on this article but I’m strugling with second discovery to get Interfaces property. I created a probe with $Config/Index$ as variable but I don’t know how to configure mapper in this case to discover correct IfDescription for each object in the class.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: