SCOM: Advanced SNMP Monitoring Part II: Designing the SNMP Monitors

For the Cisco Management Pack, a handful of different approaches were required when designing the monitors and rules and their supporting workflows.  These approaches can generally be placed into three categories:

• Simple SNMP GET operations
• Operations that require data manipulation
• Operations that require access to previously collected values  

For example, monitoring of an interface’s operational status only requires current SNMP GET requests (to retrieve the ifOperStatus and ifAdminStatus values) from the SNMP ifTable.  Condition detection for the monitor type definition can handle the SNMP values as they are presented with no further manipulation.   However, to monitor a value such as the percent free bytes on a Cisco memory pool, data manipulation is required.  In this example, a percentage value is not exposed via SNMP, but free and available values are.  So, to calculate a percentage, the free and available bytes values must be summed and then the free value divided by that sum.  And lastly, in some cases, a value from a previous poll must be referenced to detect the desired condition, such as monitoring for an increase in the collisions reported for a Cisco interface.  In this example, the locIfCollisions values can be retrieved from the locIfTable, but this is a continually augmenting counter, which is difficult to monitor.   By recording the value from the immediately previous poll, and comparing it with the current poll, a delta value can be established for that polling cycle to determine the number of collisions recorded in a single polling cycle. 

Simple SNMP GET Operations Workflow

The management pack entity workflow for what I am describing as simple SNMP GET operations starts with a data source.  The data source combines a scheduler, an snmp probe, and a filter to check for a successful retrieval of the value (e.g. checking that the value – as a string – does not equal a blank string).  This data source can then be used by rules (combing a mapper and write actions) to record historical performance data, or by monitor types (adding condition detections to determine state) and ultimately monitors (to map the monitor type states to operational states and alerts). 

A logical diagram of this workflow:

Operations that Require Data Manipulation

The System.SnmpProbe probe action allows a lot of flexibility in data retrieval, but not in post-retrieval manipulation.   So I combined this probe action with a script probe action that returns a property bag in order to create data sources that could retrieve SNMP values and then manipulate the data into more useful values.  In this case, the data source combines a scheduler, an snmp probe, a filter to check for successful value retrieval, and then a script property bag probe.   This data source that returns a property bag can then be utilized by rules or monitor types (and monitors).

A diagram of this workflow:

And an example script (to calculate memory pool percent utilization, with the free and available values retrieved by the SnmpProbe passed as arguments)

Dim oAPI, oBag, oArgs
Set oAPI = CreateObject(“MOM.ScriptAPI”)
Set oBag = oAPI.CreatePropertyBag()
set oArgs = wscript.arguments

If oArgs.Count = 2 then
 nMemUsed = oArgs(0)
 nMemFree = oArgs(1)
 nMemTotal = cdbl(nMemUsed) + cdbl(nMemFree)
 nMemPctUsed = cdbl(nMemUsed) / nMemtotal * 100
 nMemPctUsed = formatNumber(nMemPctUsed,2)
Else
 nMemPctUsed = 0
 nMemUsed = 0
 nMemFree = 0
End if

Call oBag.AddValue(“MemoryUsed”, nMemUsed)
Call oBag.AddValue(“MemoryFree”, nMemTotal)
Call oBag.AddValue(“MemoryPctUsed”, nMemPctUsed)
Call oAPI.Return(oBag)
 
Set obag = nothing
set oargs = nothing
set oAPI = nothing

Operations that Require Access To Previously Collected Values

The combination of an SnmpProbe and script property bag probe can also be used to create a data source that necessitates the use of previously collected values.  To accomplish this, I configured the script for the script probe to write values to an XML file stored in the %TEMP% path.  Subsequent polls then retrieve the value(s) from this XML file, perform their comparisons, update the XML file with current values, and output a property bag.   For the monitors I am working with, I only required access to one previous polling cycle’s data, but this method could be used for keeping numerous increments of data by modifying the scripts to maintain X number of values in the XML file.   This has proven to be a very viable method for monitoring changes between the current value and previous value.

The workflow in this case is the same as the previous example, but an example script looks like this (this script accepts the locIfCollisions value, the device IP, and interface index as arguments and compares the locIfCollisions to the previous value to determine a delta value):

set oArgs = wscript.arguments
Set oAPI = CreateObject(“MOM.ScriptAPI”)
Set oBag = oAPI.CreatePropertyBag()
set oFSO = CreateObject(“Scripting.FileSystemObject”)

if oArgs.Count < 3 Then
   Wscript.Quit -1
End If

Main

Sub Main
 nCollisions = oArgs(0)
 sDeviceIP = oArgs(1)
 nIndex = oArgs(2) 
 
 ‘Check for output directory 
  Dim oShell, sTemp
  Set oShell = CreateObject( “WScript.Shell” )
  sTemp=oShell.ExpandEnvironmentStrings(“%TEMP%”)
  Set oShell = nothing
   strOutPath =””
  strOutPath = sTEmp & “\CustomSNMP_TempFiles\”
 
  if not oFSO.FolderExists(strOutPath) then
   oFSO.CreateFolder(strOutPath)
  end if
 
 nLastIfCollisions = nIfCollisions
 nDelta = 0
 
 ‘If output file exists, load and read its values
  strOutputFile = strOutPath & “IFCollisions_” & sDeviceIP & “_” & nIndex & “.xml”

  if oFSO.FileExists(strOutputFile) then
  ‘Load XML output from last poll
   Set oXMLDoc = CreateObject(“MSXML2.DOMDocument.3.0”)
   oXMLDoc.async = False
   If Not oXMLDoc.load(strOutputFile) Then
    Set oErr = oXMLDoc.parseError
    sErrMsg = “XML Parsing Error: ” & oErr.reason
    err.raise 999, , sErrMsg
   Else
    dtLastPoll = oXMLDoc.selectsinglenode(“.//timestamp”).text
    nLastIfCollisions = oXMLDoc.selectsinglenode(“.//ifcollisions”).text
    dtLastPoll = cDate(dtLastPoll)
    nLastIfCollisions = cdbl(nLastIFCollisions)
  
   End if
   set oXMLDoc = nothing
 
   ‘Ignore old value if it’s too old
   if dateDiff(“n”,dtLastPoll,now) < 60 and cdbl(nLastIfCollisions) > 0 then
    nDelta = nCollisions – nLastIfCollisions
   end if 
 
  end if
 
 ‘Write Current values to XML File
  Set oXMLDoc = CreateObject(“MSXML2.DOMDocument.3.0”)
  oXMLDoc.async = False
 
  set objRoot = oXmlDoc.CreateElement(“locifcollisions”)
  oxmlDoc.AppendChild(objRoot)
 
  set objTimeStamp = oXmlDoc.CreateElement(“timestamp”)
  objTimeStamp.text = now()
  objRoot.AppendChild(objTimeStamp)
 
  set objLastValue = oXmlDoc.CreateElement(“ifcollisions”)
  objLastValue.text = nCollisions
  objRoot.AppendChild(objLastValue)
 
  oXMLDoc.save strOutputFile
  set oXMLDoc = nothing
 
 ‘Return value 
  Call oBag.AddValue(“IFCollisions”,nDelta)
  Call oAPI.Return(oBag)
End Sub

set ofso = nothing
set oAPI = nothing
set obag = nothing
set oxmldoc = nothing

These methods allow for just about any degree of functionality when working with SNMP data and I’ve found combing an SnmpProbe with a ScriptProbe to be a more reliable and scalable combination than using encapsulating an unmanaged SNMP provider (i.e. WMI) in a monolithic script probe action.