Using the Shell Command Templates

• Targeting:  The only acceptable targets for the UNIX/Linux Shell Command templates are UNIX/Linux Computer, or classes based on it.  So UNIX/Linux Computer, Linux Computer, Red Hat Enterprise Linux Computer, and Red Hat Enterprise Linux 5 Computer (for example) are all valid targets. Operating System, Logical Disk, Local Application, and Computer Role types are not valid targets.  Best practice for rules and monitors is to target the computer object but disable the rule/monitor by default, and use an override to enable it for a group. The Shell Command templates do not support object discovery. If you want to discover and target custom class types (like a UNIX/Linux Computer Role or Local Application), traditional MP Authoring (XML) can be used.
• Shell Command Input:  As the templates use the ExecuteShellCommand method of the agent’s script provider with a WSMan Invoke module, the command is sent to the agent on each execution. This requires that the shell command is a single line. Simple shell scripts can be condensed into a “one-liner” by replacing line breaks with semi-colons. If you want to use a complex or long script, the best approach is to distribute it as a shell script to a common path on monitored agents, then use the shell command templates to invoke the script by providing the path, such as:  /opt/scripts/myscript.sh
• Synchronous Execution & Command Performance:  The agent’s script provider executes scripts synchronously. For optimal server and agent performance and monitor reliability, it is important to make sure that shell commands execute quickly. If you want to use a long-running command or script in a monitor or rule, it is advisable to configure the script to execute periodically on the agent (e.g. with cron) outputting to a file, and then use the shell command monitor or rule to parse the output file. Unless all parameters for the rule or monitor (e.g. scheduler, shell command, timeout, etc) are identical, rules and monitors created by the Shell Command templates will not cookdown.
• Handling Complex Output:  The agent’s script provider returns StdOut, StdErr, and ReturnCode for the shell command execution. Using the templates, it is not possible to post-process this output to split the output into multiple data items, so it is best to consider StdOut from the shell command as representative of one value.  You can always call the same shell command and use the pipeline to parse with grep, awk, sed, etc to filter the output to a single output value. If your shell command returns structured data representing multiple data points, you still have the option of authoring outside of the Operations Console and using a PowerShell script to parse the output.
• Mind the Quotes and Special Characters:  The agent’s script provider executes the shell command using the default shell (for the user context of the specified Run as Account), and wraps the actual command in quotes.  So the command executed on the agent host will be something like: sh –c “my shell command” or bash –c “my shell command.” This means that double-quotes inside the shell command need special handling. If you can’t use single-quotes instead of double-quotes, escape the double-quotes with a backslash. As Management Packs are XML-based, some characters like < and /> can be problematic. It is not possible to embed CDATA blocks in the template wizards, so if you can’t avoid use of < and /> in the command and they prove to be problematic, it is advisable to distribute the script to a common file system location on the monitored agents and use the Shell Command templates to invoke the script on the file system.
• sudo Eevation:  If the Run As account specified via the selected Run As profile in the Shell Command templates is configured for sudo elevation, sudoers will need to be configured to authorize the user to execute the command.  For example, adding the following line through visudo will allow the user monuser to execute the command service named restart as input in the Shell Command task template (assuming the default shell of /bin/sh):
  monuser ALL=(root) NOPASSWD: /bin/sh -c service named restart
• Expression Filters
  Valid values for Parameter Names in the template Expression Filters are:
  • //*[local-name()=”StdErr”]  for StdErr
  • //*[local-name()=”StdOut”] for StdOut
  • //*[local-name()=”ReturnCode”]for ReturnCodeThe Expression Filters in the templates are pre-configured for common scenarios. However, some commands used in monitoring will return values to evaluate on StdOut while other cases will mandate evaluation of StdErr.  For example, if you wanted to run a command to check a connection to a MySQL server, any errors would be output to StdErr. Thusly, the monitor’s expression filter would need to look for the presence of the string ERROR in the StdErr output.Additionally, the default Expression Filters evaluate that ReturnCode = 0 for rules and monitors. This default behavior drops the workflow if the shell command itself fails.  However, if you want to generate an alert if a command fails, the ReturnCode may be 1 in the error state. In this case, the Expression Filter could be configured to go unhealthy when ReturnCode = 1, or StdErr could be evaluated for a string, and the ReturnCode evaluation removed.As in all OpsMgr templates, the Expression Filter dialogs assume values are strings, which can be problematic when evaluating numeric data.  Options for addressing this issue are
  • Export the MP after creating the rule or monitor, and edit the Type attribute in the Expression XM
  • Use Regular Expressions in the Expression Filter instead of greater than/less than evaluations
  • Perform numeric evaluation operations on the agent in the Shell Command and output a simple health state (OK or ERROR) from the shell command itself.And lastly, be mindful of the logical outcome of Expression Filters. For monitors, any potential value from the shell script should positively evaluate to one of the health states (Error or Healthy for two state monitors, or Error, Warning, or Healthy for three state monitors). Particularly, be mindful of AND and OR groupings on Error and Healthy states.  For example, a monitor that requires condition 1 AND condition 2 to be true to be in an Error state should require that only condition 1 OR condition 2 not be true to be Healthy.

Tips, Tricks, and Best Practices

• When creating shell command rules and monitors, test out the commands in a task first!  Shell Command tasks are very simple to create and provide immediate feedback.  Syntactical or authorization issues will be most quickly identified through this method.
• Target “Computer” classes, but disable the rules and monitors by default.  Override the rule and monitor for a group to enable it.
• Only use short-running shell commands in rules and monitors
  • Long running commands or scripts can be scheduled with cron and output to file, using the Shell Command templates to parse the output file
• Avoid double quotes as well as < and /> characters if possible.
  • Complex scripts can be deployed to the file system of the monitored agents and invoked by the Shell Command templates
• For optimal performance, use the largest schedule interval that makes sense (15 minutes is preferred for most cases)
• Optimize the shell commands (using pipeline formatting with grep, awk, sed, etc)  to return a single, easy to parse value
  • For monitors, returning values like: OK, WARNING, ERROR are best
  • For performance rules, the value evaluated must be a single numeric value
• To troubleshoot sudo issues, tail /var/log/messages and look for the exact command that failed – to cross reference with sudoers configuration.
• Be aware what the shell command returns for StdOut, StdErr, and ReturnCode in healthy and unhealthy scenarios. This is critical information when configuring the Expression Filters.
• Have fun!

The new templates are:

Monitors

• UNIX/Linux Shell Command Three State Monitor
• UNIX/Linux Shell Command Two State Monitor

Rules

• UNIX/Linux Shell Command (Alert)
• UNIX/Linux Shell Command (Performance)

Tasks

• Run a UNIX/Linux Shell Command

Note: For the OpsMgr 2012 Release Candidate, the Shell Command Template MP needs to be downloaded and imported.  In the final release, it will be imported by default.

Underneath the covers, all of these templates use the ExecuteShellCommand method of the agent’s script provider with the WSMan Invoke module. This method executes the command and outputs StdOut, StdErr, and ReturnCode. The command can be a path to a simple command, a command or script existing on the managed computer, or a “one-liner” script (a shell script condensed to one line with pipes and semi-colons).  The templates also allow you to select whether to run with the nonprivileged action account, or the privileged account (which also supports sudo elevation).

If you’ve done this kind of UNIX/Linux authoring in 2007 R2, you will quickly see how much easier and faster this can be done in 2012.

To show the use of these templates, I have put together an MP authoring walkthrough for monitoring BIND DNS servers on Linux. This entire MP will be created in the Operations Console, with no XML editing!

Walkthrough: Monitoring BIND on Linux

Before starting on the MP implementation, let’s have a look at what we might want to monitor to check health of a BIND DNS server.  Some basic metrics would be:

• Daemon status:   if the named daemon is not running, it’s a pretty good indication that BIND is not healthy!
• DNS resolution success:  a good way to validate that BIND is functioning properly is to check that the server can resolve a hostname.  This could be the FQDN of a local host or an Internet URL (if we wanted to confirm that Internet DNS resolution was functioning properly).  This can be done easily with nslookup on the BIND server itself:
  • nslookup lx12.contoso.com 127.0.0.1
  • nslookup www.microsoft.com 127.0.0.1
• Performance metrics:   we can use the shell command templates to collect or monitor numeric performance data as well.  For a BIND server, one performance metric we could collect is the time to resolve a name (in seconds):
  /usr/bin/time -f %e nslookup lx12.contoso.com 127.0.0.1 > /dev/null
  

Setting up the Management Pack:

• The first step is to create a new Management Pack.  In the Administration pane of the Operations Console, right-click on left-hand menu and choose Create Management Pack. Input a name and description and complete the wizard:
  
• We will use a group to control targeting of the rules and monitors in the MP, so the next step is to create a group. In the Authoring pane, right click on Groups and choose: Create a new Group.  Provide a name and description, and select the target Management Pack (the one created in the previous step) and click Next.
  
• In the Explicit Members, add the Bind hosts (filter by UNIX/Linux Computer or Linux Computer objects).  Note:  membership can be controlled for this group at any point in the future, so not all BIND servers have to be added at this point.
  
• No action is needed for the Dynamic Members, Subgroups, or Excluded Members dialogs, so click next through these to complete the group creation wizard.
• With the MP and group created, it’s on to the actual monitoring.

Creating a BIND Restart Task

The Run a UNIX/Linux Shell Command task wizard is the simplest of the shell command templates, so it is a good place to start. These steps will result in a task that restarts the BIND daemon on a Linux computer from the Operations Console.

• In the Authoring pane of the console, expand Management Pack Objects and right click Tasks. Select Create a New Task.
• Select Run a UNIX/Linux Shell Command from the Agent Tasks list, and select the target Management Pack (created in a previous step).  Click Next.
  
• Input a Name for the task (Restart BIND Daemon), provide a Description, and select the Target (Linux Computer). Click Next.
  
• The command to restart the BIND daemon is: service named restart.  Type this into the Shell Command entry pane.  Restarting a daemon is a privileged operation, so select the UNIX/Linux Privileged Account Run As profile.  The default timeout of 120 seconds should be sufficient, so click Create.
  
• That’s all it takes to create a shell command task.  When clicking on a Linux Computer instance in the monitoring view, the Restart BIND Daemon task will be listed in the right-hand task pane and can be run directly from the console.

Monitoring the BIND Daemon

• In the Authoring pane of the console, right-click Management Pack Templates and choose Add Monitoring Wizard.
• Select UNIX/Linux Process Monitoring and click Next
  
• Input a Name for the monitor, a Description, and select the target Management Pack (created previously):
  
• This is a pretty simple process monitor, so simply input named for the Process name, click Select a group and select the BIND servers group created previously.  An Alert severity of Error is appropriate and process argument filtering is not required, so click Next.
  
• On the Settings page, accept the defaults by clicking Next. This will generate an alert if named is not running.
  
• Click Create to complete the wizard.

Creating the Name Resolution Monitor

As described above, the command nslookup can be used to check name resolution health from the DNS server (specifying the local host as the server in the second argument).  For example: nslookup lx12.contoso.com 127.0.0.1.  The actual output of this command looks like:

lx11:/var/lib/named # nslookup lx12.contoso.com 127.0.0.1
Server:         127.0.0.1
Address:        127.0.0.1#53

Name:   lx12.contoso.com
Address: 192.168.1.76

To make this command more monitor friendly, we can do a bit of pipeline parsing:
nslookup lx12.contoso.com 127.0.0.1|egrep ‘^Name:.*lx12.contoso.com’|wc -l

This shell command will return a value of 1 if the line: Name:   lx12.contoso.com is found in StdOut and a value of 0 otherwise.  Thusly, a value of 1 means that the name resolution attempt succeeded, and a value of 0 means that it failed.

The steps to create a monitor using this command are:

• In the Authoring pane of the console, expand Management Pack Objects and right click Monitors. Select Create a Monitor and Unit Monitor.
• Expand Scripting, then Generic and select UNIX/Linux Shell Command Two State Monitor and select the target Management Pack (created previously). Click Next.
  
• Input a Name, Description, and Target (Linux Computer) for the monitor. Select a Parent monitor (Availability) and uncheck Monitor is enabled.  Click Next.
  
• Configure a schedule interval.  For performance optimization, this should be as large of a value as reasonable.  10 or 15 minutes should be sufficient for most purposes. Click Next
  
• Input the Shell Command (replacing lx12.contoso.com with the hostname to resolve):
  
  nslookup lx12.contoso.com 127.0.0.1|egrep ‘^Name:.*lx12.contoso.com’|wc –l
  
  The UNIX/Linux Action Account Run As profile is appropriate for this command, and 120 seconds is a sufficient value for the Timeout. Click Next.
  
• The next page of the wizard is for configuring the Error Expression. If the conditions defined in this expression are matched, the monitor will go to an error state. The Expression Filter dialog is preloaded with the following values://*[local-name()="StdOut"]  Contains  <input value>
  //*[local-name()="ReturnCode"] Equals  0With the shell command used in this example, the error state should be triggered when StdOut does not equal 1, so we can simply modify the first line to that effect.  This results in an error condition that is triggered when StdOut does not equal 1 and the nslookup command executed successfully (ReturnCode equals 0).
  
• After clicking Next, the Healthy Expression dialog is displayed. As a StdOut value of 1 indicates a successful nslookup operation using the provided shell command, simply set the first line to: //*[local-name()="StdOut"]  Equals 1 and click Next.
  
• In the Configure Health dialog, we can choose whether we want the error state to map to a Critical or Warning event by changing the Health State drop down.   In this example, I will set the Health State to Warning.
  
• The next dialog is for alert configuration. Check Generate alerts for this monitor and select an appropriate Priority and Severity (Match monitor’s health).  Edit the Alert name if appropriate and provide an Alert description.Standard $Target$ variables can be embedded in the Alert description by clicking […].  The syntax to include data from the shell command execution is:StdOut:  $Data/Context///*[local-name()=”StdOut”]$
  StdErr:  $Data/Context///*[local-name()=”StdErr”]$
  ReturnCode: $Data/Context///*[local-name()=”ReturnCode”]$In this example, I used the following description:The BIND DNS server: $Target/Property[Type="MicrosoftUnixLibrary7320040!Microsoft.Unix.Computer"]/NetworkName$ failed a name resolution test.StdErr: $Data/Context///*[local-name()="StdErr"]$Click Create to complete the monitor creation.
  
• As this monitor targets all Linux Computers, we created it without enabling it by default. We can enable it for the group of BIND servers with an override. In the Authoring pane, expand Management Pack Objects, and click on Monitors. In the top-right of the Monitors pane, click Change Scope, and check Linux Computers from the Scope Management Pack Objects dialog.  Find the monitor that was just created (BIND Name Resolution Check).
  
• Right-click the monitor, click Overrides, then Override the Monitor, then For a Group.  Select the BIND servers group created previously and click OK.
  
• Override the Enabled property to equal True and click OK
  

Creating the Name Resolution Time Performance Rule

• In the Authoring pane of the console, right-click Rules, and select Create a new rule.
• Under Collection Rules, Probe Based, select UNIX/Linux Shell Command (Performance) and select the target Management Pack (previously created).  Click Next.
  
• Input the Name and Description for the rule. Select the Target (Linux Computer) and uncheck Rule is enabled. Click Next.
  
• Configure a schedule interval.  For performance optimization, this should be as large of a value as reasonable.  10 or 15 minutes should be sufficient for most purposes. Click Next.
  
• On the Shell Command Details page, input the Shell Command:
  /usr/bin/time -f %e nslookup lx12.contoso.com 127.0.0.1 > /dev/null
  This command will return the time in seconds (to StdErr) that it took to complete the name resolution lookup. This is a non-privileged operation, so the UNIX/Linux Action Account is sufficient for the Run As profile.  Click Next.
  
• The next page provides the opportunity to filter the output before mapping to performance data.  Performance data mapping can only occur if the value is a valid double value, so the default expression syntax uses a RegExp to validate that StdOut is a numeric value, and also filters that the ReturnCode = 0, indicating a successful execution.  While the default configuration is valid for most scenarios, the time command used in this shell command actually outputs its value to StdErr.  So in this case, the first line of the filter should be modified to use a Parameter Name of //*[local-name()=”StdErr”]. Click Next.
  
• Configure the Performance Mapper.  Object, Counter, and Instance are arbitrary values that will be used to identify the performance metric in performance views and reports. The default Value of $Data///*[local-name()=”StdOut”]$ is the variable syntax for the returned StdOut, which is appropriate for most cases. Again, this needs to be modified because the time command used in this example outputs to StdErr.  The StdErr variable is: $Data///*[local-name()='StdErr']$. Click Create.
  
• As this rule targets all Linux Computers, we created it without enabling it by default. We can enable it for the group of BIND servers with an override. In the Authoring pane, expand Management Pack Objects, and click on Rules. In the top-right of the Rules pane, click Change Scope, and check Linux Computers from the Scope Management Pack Objects dialog.  Find the rules that was just created (BIND Name Resolution Test Time in Seconds).
  
• Right-click the rule, click Overrides, then Override the Monitor, then For a Group.  Select the BIND servers group created previously and click OK.
  
• Override the Enabled property to equal True and click OK.
  

Creating a Dashboard View

With the monitoring pieces now in place, we can move on to creating a dashboard view to show health and performance.

• In the Monitoring pane of the console, find the folder for the BIND management pack. Right-click the folder and choose New, then Dashboard View
  
• Select Grid Layout and click Next.
  
• Input a Name and Description and click Next.
  
• Select 3 Cells and pick a layout. Click Next then Create.
  
• We now have an empty dashboard view.

• Click Click to add widget in the top-left rectangle. Select State Widget and click Next.
  
• Provide a Name and Description and click Next.
  
• Under Groups and objects click Add. In the search dialog, find the BIND servers group, add it to the selection list and click OK.
  
• Configure the display and filtering preferences and complete the widget wizard.

Adding the Performance Widget

• Click Click to add widget in the top-right rectangle. Select Performance Widget and click Next.
  
• Input a Name and Description and click Next.
  
• Under Select a group or object click […] and find the BIND servers group.  Select the group and click OK.
  
• Under Select performance counters click Add. Select the Object (BIND Server), Counter (Name Resolution Time (s)), and select (All) for Instance.  Click Add to add the performance counter and click OK.
  
  Note: only performance counters that have been collected will show up in this list. If you created the rule in the past few minutes and don’t see it listed yet, wait a little bit longer.
• Select the Time Range and Display Options and complete the wizard.

Adding the Alert Widget

• Click Click to add widget in the bottom rectangle. Select Alert Widget and click Next.
  
•  Input a Name and Description and click Next.
  
• Under Select a group or object click […], find and selecd the BIND servers group and click OK.
  
• Specify the criteria.  Be sure to filter for only the New resolution state if you just want to see active alerts. Click Next.
  
• Configure Display Preferences and complete the wizard.

The end result is a dashboard showing health and performance of monitored BIND servers:

Summary

As you can see from this walkthrough, the new template wizards for UNIX/Linux monitoring in OpsMgr 2012 are a significant improvement and make custom monitoring far easier to implement!

In Part III of this series, I walked through creation of data sources, a discovery, and a rule for discovering dynamically-named log files and implementing an alert-generating rule for log file monitoring.  In this post, I will continue to expand this Management Pack to implement performance collection rules, using WSMan Invoke methods to collect numerical performance data from a shell command. 

Using Shell Commands to Collect Performance Data

Whether it is system performance data from the /proc or /sys file systems, or application performance metrics in other locations, performance data for UNIX and Linux systems can often be found in flat files.   In this example Management Pack, I wanted to demonstrate using a WSMan Invoke module with the script provider to gather a numeric value from a file and publish the data as performance data.   In many cases, this would be slightly more complex than is represented in this example (e.g. if the performance metric value should be the delta between data points in the file over time), but this example should provide the framework for using the contents of a file to drive performance collection rules.   The root of these workflows is a shell command using the cat command to parse the file, which could be piped to grep, awk, and sed to filter for specific lines and columns.  

Additionally, if the performance data (e.g. hardware temperature or fan speed, current application user or connection count) that you are looking for is not stored in a file, but available in the output of a utility command, the same method could be used by using the utility command instead of cat.

Collecting Performance Data from a File

In this example, the MyApp application stores three performance metrics in flat files in the subdirectory ./perf.   I have built three rules that cat these files, and map the values to performance data.  The three rules are functionally identical, so I will only describe one of them.

Performance Collection Rule:  MyApp.Monitoring.Rule.CollectMyAppMem

This rule collects the value from the ./perf/mem file in the application directory, which represents the current memory used by the application in KB.   The rule targets the MyApp.Monitoring.MyApp application class.

Rule Data Source:

The rule uses the MyApp.Monitoring.ShellCommandMonitoring data source, described in Post I of this series, with the configuration:

<Interval>300</Interval>
<TargetSystem>
$Target/Host/Property[Type="MicrosoftUnixLibrary!
Microsoft.Unix.Computer"]/NetworkName$</TargetSystem>
<ShellCommand>
cat $Target/Property[Type="MyApp.Monitoring.MyApp"]
  /InstallPath$/perf/mem
</ShellCommand>
<Timeout>120</Timeout>

Notice that the ShellCommand is our cat command:  cat $Target/Property[Type="MyApp.Monitoring.MyApp"]/InstallPath$/perf/mem

Rule Condition Detection:

A System.Performance.DataGenericMapper is used as the condition detection module to map the StdOut to performance data, with the configuration:

<ObjectName>
$Target/Property[Type="MyApp.Monitoring.MyApp"]
  /Name$
</ObjectName>
<CounterName>Memory used (KB)</CounterName>
<InstanceName>Total</InstanceName>
<Value>
  $Data///*[local-name()="StdOut"]$
</Value>

Write Actions:

The rule defines two write actions:  Microsoft.SystemCenter.CollectPerformanceData and Microsoft.SystemCenter.DataWarehouse.PublishPerformanceData to collect the data and publish it to the DW.   These require no configuration.

The end result is that the scheduled rule grabs the value from the text file, maps it to performance data, collects and publishes the performance data to the OM and DW databases.  This mechanism can be used for nearly any numerical performance metric that is accessible (in a timely fashion) from a shell command pipeline or script.

Stay tuned for more in this series….

In Part II of this series, I walked through creation of data sources, a discovery, a monitor type, and a monitor for customized ”Process Count” monitoring for discovered instances of a ”Service” class. In this post, I will continue to build on this example MP to implement dynamic log file discovery and monitoring.

Dynamic Log File Discovery and Monitoring

Log file monitoring of a single known log file can be easily implemented with the Microsoft.Unix.SCXLog modules, but in some cases, the full path to a log file isn’t static.   For example, if an application maintains multiple log files in a directory, the file name portion of the log file path may not be known ahead of time.    To handle this monitoring scenario, we can implement dynamic log file discovery – using a shell command execution, and then pass the full path of the log file to the SCXLog module for standard log file monitoring. This requires a new class instance, a discovery data source, a discovery rule, and a rule that actually implements the log file monitoring.

Defining the Log File Class and Hosting Relationship

Firstly, a new custom class is required to represent the log file objects.   Instances of this class will be discovered by the discovery rule.  

Definition 

• ID:  MyApp.Monitoring.LogFile
• Base Class:  Microsoft.Unix.ApplicationComponent
• Name:  MyApp Log file

Properties

• Name (String)
• Path (String) – Key

The properties for the log file class represent the file name and full path.   The full path is assured to be unique, so I have specified that as the key property of the class.

The log file class needs to be hosted by the MyApp class, to maintain the relationship between the log files and the application.  

Discovery Data Source:  MyApp.Monitoring.DataSource.DiscoverLogFiles

This data source will use the MyApp.Monitoring.DataSource.ShellCommandDiscovery probe action to find files in a given directory that match a pattern.   The output from this command execution will then be passed to a Microsoft.Windows.PowerShellDiscoveryProbe.   The reason that I am using a PowerShellDiscoveryProbe is that the listing of matched files will be returned as a single data item, the StdOut from the command.   Using a PowerShellDiscoveryProbe provides an easy way to split each line from the output and discover an instance per line. 

Configuration Parameters:

• Interval (integer):  Scheduler interval in seconds – overridable
• TargetSystem (string):  UNIX/Linux agent computer to execute the discovery
• Appname (string):   The name of the application object (which is the key property for the hosting class instance)
• LogFileNamePattern (string): The pattern that will be used in the grep operation to identify log files to discovery
• LogFilepath (string):  The path to search for log files at (via an ls command)

Member Modules:

The first member modules is a MyApp.Monitoring.DataSource.ShellCommandDiscovery probe action, that executes the following command:
ls $Config/LogFilepath$ |grep $Config/LogFileNamePattern$.  This simply enumerates the contents of the specified directory path, and pipes the results to grep, to match a specified pattern, which could be a string match or regular expression.

Module Configuration:

<Interval>$Config/Interval$</Interval>
<TargetSystem>$Config/TargetSystem$</TargetSystem>
<ShellCommand>ls $Config/LogFilepath$
   |grep $Config/LogFileNamePattern$</ShellCommand>
<Timeout>120</Timeout>

The output of this shell command then needs to be parsed so that each valid line in the output is discovered as an instance of a log file object.   This is most easily done with a PowerShellDiscoveryProbe:

param ([string]$CmdOutput,[string]$AppName,[string]$LogFilePath, [string] $TargetSystem,[string] $SourceID,[string]$ManagedEntityID)

$api = New-Object -comObject ‘Mom.ScriptAPI’
$discoveryData = $api.CreateDiscoveryData(0, $SourceID, $ManagedEntityID)

if ($CmdOutput -ne $null){
        $CmdOutput = $CmdOutput.Replace([Environment]::newline,” “)
 [array]$arList = $CmdOutput.Split(” “)
 $arList | ForEach-Object{
  [string]$sFile = $_
 if([int]$sFile.Length -ge [int]1){
  $SFilePath = $LogFilePath + “/” + $sFile
  $oInst = $discoveryData.CreateClassInstance(“$MPElement[Name='MyApp.Monitoring.Logfile']$”)
  $oInst.AddProperty(“$MPElement[Name='MyApp.Monitoring.Logfile']/Name$”, $sFile)
  $oInst.AddProperty(“$MPElement[Name='System!System.Entity']/DisplayName$”, $sFile)
  $oInst.AddProperty(“$MPElement[Name='MyApp.Monitoring.Logfile']/Path$”, $sFilePath)
  $oInst.AddProperty(“$MPElement[Name='MyApp.Monitoring.MyApp']/Name$”, $AppName)
  $oInst.AddProperty(“$MPElement[Name='MicrosoftUnixLibrary!Microsoft.Unix.Computer']/PrincipalName$”, $TargetSystem)

  $discoveryData.AddInstance($oInst)
 }     
}
 $discoveryData
}

Remove-variable api
Remove-variable discoveryData

The PowerShell script loads the Mom.ScriptAPI, creates a Discovery Data instance, and then walks through each line of the ouptut.   If the file name is a valid string (not empty), a class instance is created for the MyApp.Monitoring.Logfile class, and the path and file name properties are set.   The PrincipalName property of the Microsoft.Unix.Computer object, and the AppName property of the MyApp.Monitoring.MyApp class ares included in the DiscoveryData, so that the discovery mapping process can map the hosting relationships. 

Parameters are passed from the module configuration to the script using the Parameters XML fragment in the module configuration:

<Parameters>
<Parameter>
 <Name>TargetSystem</Name>
 <Value>$Config/TargetSystem$</Value>
 </Parameter>
<Parameter>
 <Name>AppName</Name>
 <Value>$Config/Appname$</Value>
 </Parameter>
<Parameter>
 <Name>LogFilePath</Name>
 <Value>$Config/LogFilepath$</Value>
 </Parameter>
<Parameter>
 <Name>CmdOutput</Name>
 <Value>
 $Data///*[local-name()="StdOut"]$
</Value>
 </Parameter>
<Parameter>
 <Name>ManagedEntityID</Name>
 <Value>$Target/Id$</Value>
 </Parameter>
<Parameter>
 <Name>SourceID</Name>
 <Value>$MPElement$</Value>
 </Parameter>
 </Parameters>

This data source can then be used to discover log files matching a pattern, in a specified directory.  

Discovery Rule:  MyApp.Monitoring.Discovery.LogFile

This discovery will discover dynamically-named log files, in a specified path, using a regular expression to filter by file name.   It discovers instances of the MyApp.Monitoring.LogFile class, and uses the MyApp.Monitoring.DataSource.DiscoverLogFiles data source.  The discovery targets  MyApp.Monitoring.MyApp

Data Source Configuration:

• <Interval>14400</Interval>
• <TargetSystem>$Target/Host/Property[Type="MicrosoftUnixLibrary!Microsoft.Unix.Computer"]/PrincipalName$</TargetSystem>
• <Appname>$Target/Property[Type="MyApp.Monitoring.MyApp"]/Name$</Appname>
• <LogFileNamePattern>‘^log[0-9]+’</LogFileNamePattern>
• <LogFilepath>$Target/Property[Type="MyApp.Monitoring.MyApp"]/InstallPath$/logs</LogFilepath>

The two parameters to note are the LogFilepath (which is defined as the application path discovered for the MyApp application, appended with “/logs”) and the LogFileNamePattern (which is a regular expression – ‘^log[0-9]+’ – that will match log files named:  logxxx, where xxx is a number).  

Monitoring the Discovered Log Files

Log File Monitoring Rule:   MyApp.Monitoring.Rule.AlertOnLogError

Now that the dynamically-named log files will be discovered, we need a rule to alert when an error is found in one of the logs.   The rule will target all instances of the MyApp.Monitoring.LogFile class, so that when a new log file instance is discovered, it is automatically monitored.  The rule uses the MicrosoftUnixLibrary!Microsoft.Unix.SCXLog.Privileged.Datasource (assuming the log files require privileged credentials to access).

Data source configuration:

• <Host>$Target/Host/Host/Property[Type="MicrosoftUnixLibrary!Microsoft.Unix.Computer"]/NetworkName$</Host>
• <LogFile>$Target/Property[Type="MyApp.Monitoring.Logfile"]/Path$</LogFile> <RegExpFilter>^.*(e|E)rror.*$</RegExpFilter>

The discovered path to the logfile instance is input as the LogFile parameter value, and a Regular Exprssion (^.*(e|E)rror.*$) is defined to match any log entries with the string:  error or Error in the message.  

Condition Detection configuration:

A System!System.Event.GenericDataMapper condition detection is then configured to map the data to EventData, for consumption by OpsMgr.  The configuration of this module is:

• <EventOriginId>$MPElement$</EventOriginId>
• <PublisherId>$MPElement$</PublisherId>
• <PublisherName>MyApp</PublisherName>
• <Channel>Application</Channel>
• <LoggingComputer>$Target/Host/Host/Property[Type="MicrosoftUnixLibrary!Microsoft.Unix.Computer"]/NetworkName$</LoggingComputer>
• <EventNumber>8001</EventNumber>
• <EventCategory>0</EventCategory>
• <EventLevel>1</EventLevel>
• <UserName/>
•  <Params/>
•  </ConditionDetection>

Write Actions:

In this rule, I have configured two write actions, for collecting the event, and generating an alert.  The CollectEvent (SC!Microsoft.SystemCenter.CollectEvent) module requires no additional configuration, and the alert can be configured to provide details about the logged error message:

Stay tuned for more in this series…

Advanced Service Monitoring

1. Define a custom service class, and discover an instance of this class for each service to monitor, configure monitor types and monitors targeting this class
2. Create a monitor for each service to monitor, targeting the custom application class

Both methods are completely viable, and in most cases, it is appropriate to take the simpler approach and target the custom monitors to the application, providing static inputs into the monitor.   There are some cases where discovering a class instance for the service makes sense though.  Facilitating dynamic discovery of services or thresholds (read from a config file), using the service class in a Distributed Application model in OpsMgr, or maintaining logical seperation (in terms of monitoring) between the application and its subsystems are all scenarios that would benefit from discovering the monitored services as class instances.   For the purpose of illustration, I will discover the daemon to monitor in this example Management Pack as a class instance.

Class Definition

Class:  MyApp.Monitoring.Service

Definition

• ID:  MyApp.Monitoring.Service
• Base Class:  Microsoft.Unix.ApplicationComponent
• Name:  MyApp Service

Properties

• Name (String) – Key
• MinRunning (Integer)
• MaxRunning (Integer)

Discovery

Then we can define the data source to discover a service.   In this case, we know the name of the service and the value of the properties, so we don’t need to actually poll the agent to return data.   We can simply combine a Discovery Scheduler with a Discovery Data Mapper module to implement the data source.  However, we want to be able to override the values of MinRunning and MaxRunning, so these will need to be exposed as overridable configuration parameters.

Therefore, I’ve chosen to implement this data source in two parts.   The first data source, will simply combine a System.Discovery.Scheduler module and a System.Discovery.ClassSnapshotDataMapper module.   This data source will accept Interval, ClassId and InstanceSettings parameters as inputs.  The second data source will reference the first data source, but implement parameters for Service Name, MinRunning, and MaxRunning.    By breaking this into two data sources, the first data source can be used for other simple discoveries.

Discovery Data Source:  MyApp.Monitoring.DataSource.DiscoverObject

This is the data source that simply combines a scheduler and a discovery data mapper.  It requires that the MapperSchema be added to the Configuration:

<Configuration>
<IncludeSchemaTypes>
<SchemaType>
 System!System.Discovery.MapperSchema
</SchemaType>
</IncludeSchemaTypes>
…

Configuration Parameters:

• Interval (integer):  Scheduler interval in seconds
• ClassId (string):  ID of the Class to discover
• InstanceSettings (SettingsType):  Discovery Instance Settings

Member Modules:

The first member module is the System.Discovery.Scheduler module, with the configuration:

<Scheduler>
<SimpleReccuringSchedule>
<Interval>$Config/Interval$</Interval>
<SyncTime/>
</SimpleReccuringSchedule>
<ExcludeDates/>
</Scheduler>

This is followed by a System.Discovery.ClassSnapshotDataMappermodule, with the configuration:

<ClassId>$Config/ClassId$</ClassId>
<InstanceSettings>
 $Config/InstanceSettings$
</InstanceSettings>

So this data source accepts the arbitrary Instance Settings and Class Id and maps the inputs to Discovery Data.

Discovery Data Source:  MyApp.Monitoring.DataSource.DiscoverService

This data source uses the MyApp.Monitoring.DataSource.DiscoverObject data source that we just created, but supports overridable inputs for the MinRunning and MaxRunning service class properties, by embedding the $Config/$ variables in the Instance Settings definition.

Configuration Parameters:

• Interval (integer):  Scheduler interval in seconds – overridable
• TargetSystem (string):  UNIX/Linux agent computer to execute the discovery
• AppName (string):   The name of the application object (which is the key property for the hosting class instance)
• ServiceName (string): The name of the service to discover
• MinRunning (integer):  The minimum threshold of running processes expected – overridable
• MaxRunning (integer):  The maximum threshold of running processes expected – overridable

Member Modules

This data source only has one member module:  MyApp.Monitoring.DataSource.DiscoverObject, with the configuration:

<Interval>$Config/Interval$</Interval>
<ClassId>
 $MPElement[Name="MyApp.Monitoring.Service"]$
</ClassId>
<InstanceSettings>
<Settings>
<Setting>
<Name>
 $MPElement[Name='MyApp.Monitoring.MyApp']/Name$
</Name>
<Value>$Config/AppName$</Value>
</Setting> 
<Setting>
<Name>
 $MPElement[Name='MyApp.Monitoring.Service']/Name$
</Name>
<Value>$Config/ServiceName$</Value>
</Setting>
<Setting>
<Name>
 $MPElement[Name='MyApp.Monitoring.Service']/MinRunning$
</Name>
<Value>$Config/MinProcesses$</Value>
</Setting>
<Setting>
<Name>
 $MPElement[Name='MyApp.Monitoring.Service']/MaxRunning$
</Name>
<Value>$Config/MaxProcesses$</Value>
</Setting>
<Setting>
<Name>
 $MPElement[Name='MicrosoftUnixLibrary!
  Microsoft.Unix.Computer']/PrincipalName$
</Name>
<Value>$Config/TargetSystem$</Value>
</Setting>
<Setting>
<Name>
$MPElement[Name='System!System.Entity']/DisplayName$
</Name>
<Value>$Config/ServiceName$</Value>
</Setting>
</Settings>
</InstanceSettings>

Note that the key properties for the hosting class instances (MyApp.Monitoring.MyApp, and Microsoft.Unix.Computer) are included in the Instance Settings so that the relationships can be mapped.  With the data sources in place, the next step is to configure the Discovery Rule (one for each service to discover).

Discovery Rule:  MyApp.Monitoring.Discovery.MyAppDService

This rule discovers the MyAppD daemon, using the data source just created.  It is targeted to instances of the MyApp class, and provides the name of the service and process count thresholds.

Data Source Configuration:

<Interval>28800</Interval>
<TargetSystem>
 $Target/Host/Property[Type="MicrosoftUnixLibrary!
  Microsoft.Unix.Computer"]/PrincipalName$
</TargetSystem>
<AppName>
 $Target/Property[Type="MyApp.Monitoring.MyApp"]/Name$
</AppName>
<ServiceName>myappd</ServiceName>
<MinProcesses>1</MinProcesses>
<MaxProcesses>3</MaxProcesses>

Once the MP is imported and the dicsovery has run, we can see the discovered service in the Discovered Inventory view in the Ops Console:

Building the Monitor Type

In this example, we are going to monitor the process/daemon and generate an alert if the count of running instances is outside of the threshold range.   To do this, we can use the Shell Command monitoring data source to execute a  ps h -e |grep <process name> grep -v grep |wc –l command string.  This calls the ps command, with switches to suppress the header row, and show all processes.   Grep is used to find the process name, and exclude the grep process itself.  Finally, the results are piped to wc –l to return a line count.   The result of this command string is that the running count of processes matching the process name is returned as a numeric value.

Monitor Type: MyApp.Monitoring.MonitorType.ProcessCount

This monitor type will use the MyApp.Monitoring.DataSource.ShellCommandMonitoring data source, to call the ps command string described above.  A set of condition detection modules are used to determine if the process count is below or above the minimum and maximum thresholds.

Configuration Parameters

• Interval (integer):  Scheduler interval in seconds – overridable
• TargetSystem (string):  UNIX/Linux agent computer to monitor
• ServiceName (string): The name of the service to monitor
• MinRunning (integer):  The minimum threshold of running processes expected
• MaxRunning (integer):  The maximum threshold of running processes expected

Health States

• ProcessCountOK
• ProcessCountNotOK

Member Modules

The data source for this monitor type is: MyApp.Monitoring.DataSource.ShellCommandMonitoring, with the configuration:
<Interval>$Config/Interval$</Interval>
<TargetSystem>
 $Config/TargetSystem$
</TargetSystem>
<ShellCommand>
 ps h -e |grep $Config/ProcessName$|grep -v grep |wc –l
</ShellCommand>
<Timeout>120</Timeout>

Two configuration detection modules are required to detect the OK and NotOK states, comparing the returned StdOut to the thresholds:

<ConditionDetection ID="CDProcessCountOK"
  TypeID="System!System.ExpressionFilter">
<Expression>
<And>
<Expression>
<SimpleExpression>
<ValueExpression>
 <XPathQuery Type="Double">
 //*[local-name()="StdOut"]
</XPathQuery>
</ValueExpression>
<Operator>GreaterEqual</Operator>
<ValueExpression>
<Value Type="Double">$Config/MinCount$</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
<Expression>
<SimpleExpression>
<ValueExpression>
<XPathQuery Type="Double">
 //*[local-name()="StdOut"]
</XPathQuery>
</ValueExpression>
<Operator>LessEqual</Operator>
<ValueExpression>
<Value Type="Double">$Config/MaxCount$</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
</And>
</Expression>
</ConditionDetection>

And

<ConditionDetection ID="CDProcessCountNotOK"
  TypeID="System!System.ExpressionFilter">
<Expression>
<Or>
<Expression>
<SimpleExpression>
<ValueExpression>
<XPathQuery Type="Double">
 //*[local-name()="StdOut"]
</XPathQuery>
</ValueExpression>
<Operator>Greater</Operator>
<ValueExpression>
<Value Type="Double">$Config/MaxCount$</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
<Expression>
<SimpleExpression>
<ValueExpression>
<XPathQuery Type="Double">
 //*[local-name()="StdOut"]
</XPathQuery>
</ValueExpression>
<Operator>Less</Operator>
<ValueExpression>
<Value Type="Double">$Config/MinCount$</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
</Or>
</Expression>
</ConditionDetection>

Regular Detections

The regular detections just need to be configured to map the Condition Detection member modules to the defined Health States:

<RegularDetections>
<RegularDetection
  MonitorTypeStateID="ProcessCountOK">
<Node ID="CDProcessCountOK">
<Node ID="DS1"/>
</Node>
</RegularDetection>
<RegularDetection
  MonitorTypeStateID="ProcessCountNotOK">
<Node ID="CDProcessCountNotOK">
<Node ID="DS1"/>
</Node>
</RegularDetection>
</RegularDetections>

In summary, this monitor type will execute our ps command, for a given service name, and compare the count of running/matched process instances  to the defined thresholds (defined during service discovery).

Building the Custom Process Monitor

In this step, we will create a monitor, of the type that we just created: MyApp.Monitoring.MonitorType.ProcessCount

<Configuration>
<Interval>180</Interval>
<TargetSystem>
 $Target/Host/Host/Property[Type="MicrosoftUnixLibrary!
     Microsoft.Unix.Computer"]/NetworkName$
</TargetSystem>
<ProcessName>
 $Target/Property[Type="MyApp.Monitoring.Service"]/Name$
</ProcessName>
<MinCount>
 $Target/Property[Type="MyApp.Monitoring.Service"]/MinRunning$
</MinCount>
<MaxCount>
 $Target/Property[Type="MyApp.Monitoring.Service"]/MaxRunning$
</MaxCount>
</Configuration>

The MyApp Service: $Target/Property[Type="MyApp.Monitoring.Service"]/Name$, currently has too many or too few processes running.

Current process count:  $Data/Context///*[local-name()="StdOut"]$.

Expected range:
$Target/Property[Type="MyApp.Monitoring.Service"]/MinRunning$ – $Target/Property[Type="MyApp.Monitoring.Service"]/MaxRunning$.

Stay tuned for more in this series…

The OpsMgr UNIX and Linux monitoring implementation can be extended through MP authoring to implement robust system and application monitoring for UNIX/Linux servers.   The most direct mechanism of extension comes in the form of the script provider, accessed with WSMan Invoke modules.   The WSMan Invoke modules support three methods of invoking actions:

• ExecuteCommand – execute a command (e.g. a script already on the file system ) and return the results
• ExecuteShellCommand – execute a command through sh (with pipeline support) and return the results
• ExecuteScript  – download and execute an embedded script and return the results

Of these three methods, I prefer to use ExecuteShellCommand in most cases, as it allows for the use of complex one-liner shell commands, embedded in the MP.

In a series of posts, I will describe the creation of an example Management Pack for monitoring an application, featuring dynamic application discovery, discovery of multiple log files, and advanced monitoring implementations.

Example Application Details

The example MP described in these blog posts implements monitoring for a hypothetical application (MyApp).  The application involves a daemon, a set of log files, and application performance counters where the metrics are accessible as the contents of files.

Part I – Discovering an Application

Setting up the MP

I am a big fan of the R2 Authoring Console and will be using it to create this example MP.   The first step then is to create a new MP in the Authoring Console (ID:  MyApp.Monitoring).    Once the MP is created and saved, references are needed.   References I am adding are:

• Microsoft.Unix.Library – contains UNIX/Linux classes and modules
• Microsoft.SystemCenter.DataWarehouse.Library – required for publishing performance data to the DW
• System.Image.Library – contains icon images referenced in class definition

Configuring the Base Composite Modules

Most monitoring and discovery workflows used in this MP example will involve the execution of a shell command through the Microsoft.Unix.WSMan.Invoke.ProbeAction ExecuteShellCommand method.   Therefore, setting up a new composite probe action and a couple of base data sources will make further module implementation much simpler.

Probe Action:  MyApp.Monitoring.ProbeAction.ShellCommand

This probe action simply wraps the Microsoft.Unix.WSMan.Invoke.ProbeAction in a way that makes it easier to use for shell command execution.

Configuration Parameters:

• TargetSystem (string)  – the UNIX/Linux agent computer that will execute the command
• ShellCommand (string) – the shell command to execute
• Timeout (integer) – the command timeout, in seconds

Member Modules:

This probe action will only include one member module:  Microsoft.Unix.WSMan.Invoke.ProbeAction.  The configuration of the module looks like:

<TargetSystem>$Config/TargetSystem$</TargetSystem>
<Uri>

http://schemas.microsoft.com/wbem/wscim/1/cim-schema/2/

 SCX_OperatingSystem?__cimnamespace=root/scx
</Uri>
<Selector/>
<InvokeAction>ExecuteShellCommand</InvokeAction>
<Input>
<p:ExecuteShellCommand_INPUT xmlns:p="http://schemas.
 microsoft.com/wbem/wscim/1/cim-schema/2/SCX_OperatingSystem">
<p:command>$Config/ShellCommand$</p:command>
<p:timeout>$Config/Timeout$</p:timeout>
</p:ExecuteShellCommand_INPUT>
</Input>

Note that the configuration parameters for ShellCommand and Timeout are embedded in the “Input” parameter XML fragment.

Data Source: MyApp.Monitoring.DataSource.ShellCommandMonitoring

The probe action created in the previous step will be used in a number of monitoring workflows, so creating a data source that combines the probe action with a scheduler will make monitoring workflow configuration simpler.

Configuration Parameters:

• Interval (integer) – timer interval in seconds
• TargetSystem (string)  – passed to the probe action module
• ShellCommand (string) – passed to the probe action module
• Timeout (integer) – passed to the probe action module

Member Modules:

The first module in this composite data source is a System.Scheduler module, with a basic configuration:

<Scheduler>
<SimpleReccuringSchedule>
<Interval>$Config/Interval$</Interval>
<SyncTime/>
</SimpleReccuringSchedule>
<ExcludeDates/>
</Scheduler>

This scheduler module is followed by the ShellComand probe action, with the configuration:

<TargetSystem>$Config/TargetSystem$</TargetSystem>
<ShellCommand>$Config/ShellCommand$</ShellCommand>
<Timeout>$Config/Timeout$</Timeout>

Data Source: MyApp.Monitoring.DataSource.ShellCommandDiscovery

This data source is identical to the MyApp.Monitoring.DataSource.ShellCommandMonitoring data source, except it features a System.Discovery.Scheduler scheduler module instead of a System.Scheduler module.  Discovery workflows must use a Discovery scheduler in order to prevent problems with cookdown (i.e. discovery workflows ‘cooking-down’ along with monitoring workflows).   So this data source combines the ShellCommand probe action with a Discovery scheduler.

Additionally, I have added a System.ExpressionFilter module to the ShellCommandDiscovery data source to prevent the discovery workflows from continuing if an error is encountered (either WSMan error or ShellCommand StdErr output).   The ExpressionFilter configuration is:

<Expression>
<And>
<Expression>
<RegExExpression>
<ValueExpression>
 <XPathQuery>//*[local-name()="StdErr"]</XPathQuery>
</ValueExpression>
 <Operator>DoesNotMatchRegularExpression</Operator>
 <Pattern>^.+</Pattern>
 </RegExExpression>
 </Expression>
<Expression>
<Not><Expression><Exists>
<ValueExpression>
 <XPathQuery>WsManData/ErrorCode</XPathQuery>
</ValueExpression>
</Exists> </Expression> </Not>
</Expression>
</And>
</Expression>

Class Definition

With the base composite modules configured, we can move on to the actual application discovery, starting with the definition of the application class.  The Microsoft.Unix.LocalApplication is an ideal base class for the application class, and already has a hosting relationship defined so that Microsoft.Unix.Computer hosts Microsoft.Unix.LocalApplication.

Class Definition:

• ID:  MyApp.Monitoring.MyApp
• Base Class:  Microsoft.Unix.LocalApplication
• Name:  MyApp

Properties:

• Name (String) – Key
• InstallPath (String)

Discovering the Application

When discovering a Windows application in OpsMgr, we would typically use a registry probe.  In UNIX/Linux application discovery, probing the existence or contents of a file is typically the most effective way to identify the instance of an application.   In this example, the MyApp application has a configuration file named: myapp.conf in the installation path of /opt/myapp.  Thus, a shell command:  ls /opt/myapp/myapp.conf | wc –l would return a 1 if the file exists, or 0 if the file doesn’t exist.   We can use this logic in the application discovery.

Discovery Data Source: MyApp.Monitoring.DataSource.MyAppDiscovery

To implement this application discovery, I will create a new data source, building on the ShellCommandDiscovery data source previously created.

Configuration Parameters:

• Interval (integer):  Scheduler interval in seconds
• TargetSystem (string):  UNIX/Linux agent computer to execute the discovery
• AppPath (string):  File system path to the application, overrideable

Member Modules:

The first module in this data source is the ShellCommandDiscovery data source (MyApp.Monitoring.DataSource.ShellCommandDiscovery).   This will be used to query the file system for the application configuration file, to determine if the app is present.  The configuration for this module is:

<Interval>$Config/Interval$</Interval>
<TargetSystem>$Config/TargetSystem$</TargetSystem>
<ShellCommand>
 ls $Config/AppPath$/myapp.conf | wc -l
</ShellCommand>
<Timeout>120</Timeout>

In order to return discovery data from this data source, a Discovery Mapper module is required.   In this case, I will use a System.Discovery.FilteredClassSnapshotDataMapper module to filter the results (only return discovery data if the app is found).   An output of 1 from the shell command indicates that the configuration file was found, while an output of 0 indicates that the configuration file was not found.  The configuration for the FilteredClassSnapShotDataMapperModule would be:

<Expression>
<SimpleExpression>
<ValueExpression>
 <XPathQuery Type="String">
  //*[local-name()="StdOut"]
 </XPathQuery>
</ValueExpression>
<Operator>Equal</Operator>
<ValueExpression>
 <Value Type="String">1</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
<ClassId>
 $MPElement[Name='MyApp.Monitoring.MyApp']$
</ClassId>
<InstanceSettings>
<Settings>
<Setting>
<Name>
$MPElement[Name='MyApp.Monitoring.MyApp']/Name$
</Name>
<Value>MyApp</Value>
</Setting>
<Setting>
<Name>
 $MPElement[Name='MyApp.Monitoring.MyApp']/InstallPath$
</Name>
<Value>$Config/AppPath$</Value>
</Setting>
<Setting>
<Name>
 $MPElement[Name='System!System.Entity']/DisplayName$
</Name>
<Value>MyApp - $Config/TargetSystem$</Value>
</Setting>
<Setting>
<Name>
 $MPElement[Name='MicrosoftUnixLibrary!
 Microsoft.Unix.Computer']/PrincipalName$
</Name>
<Value>$Config/TargetSystem$</Value>
</Setting>
</Settings>
</InstanceSettings>

Note that the Key Property (PrincipalName) for the hosting Microsoft.Unix.Computer object is included in the DiscoveryData.

Discovery Rule:  MyApp.Monitoring.Discovery.MyApp

With the discovery data source implemented, finishing the discovery just requires creation of the discovery rule, which targets Microsoft.Unix.Computer objects.

Data Source Configuration:

• Interval:  14400
• TargetSystem: $Target/Property[Type="MicrosoftUnixLibrary!Microsoft.Unix.Computer"]/PrincipalName$
• AppPath: /opt/myapp

This is a good time to create the MP’s presentation folder and a state view targeting MyApp objects.   Once the discovery has executed, discovered instances of the app can be viewed in the state view or Discovered Inventory view:

Stay tuned for more posts on completing this example Management Pack

• xSNMP for APC Reports
• xSNMP for APC NetBotz Reports
• xSNMP for Brocade Reports
• xsNMP for Check Point Secure Platform Reports
• xSNMP for Cisco Reports
• xSNMP for Data Domain Reports
• xSNMP for Dell PowerEdge Reports
• xSNMP for HP ProCurve Reports
• xSNMP for IBM AIX Reports
• xSNMP for Juniper Networks Reports
• xSNMP for Juniper-NetScreen Reports
• xSNMP for NetApp Reports
• xSNMP for Net-SNMP Reports
• xSNMP for SonicWALL Reports

Requirements are:   OpsMgr 2007 R2, xSNMP suite, OpsMgr reporting implementation.   Like the xSNMP suite, the reports MP’s are licensed with the GNU-GPL and unsealed versions are provided.

New Management Packs:

• xSNMP for APC NetBotz Management Pack
• xSNMP for IBM AIX
• xSNMP for Juniper Networks Management Pack
• xSNMP for SonicWALL Management Pack

New Features:

• Added support for monitoring of Net-SNMP Extend objects (xSNMP for Net-SNMP Management Pack)
• Added a three state monitor for Net-SNMP Exec objects(xSNMP for Net-SNMP Management Pack)
• New Cisco Firewall Subsystem monitoring for PIX and ASA firewalls
• Added new data sources to the xSNMP MP to decrease data source redundancy
• Individual inbound/outbound speeds can now be set through the Speed Override discovery in the xSNMP MP
• All views are now publically accessible

Issues Resolved:

• Fixed an uncommon issue with network interface utilization calculations
  that could result in invalid values being calculated if a previous poll returned null data
• Updated the list of Device OID’s for APC UPS devices to include missing UPS models
• Fixed an issue with an incorrect OID specified for HP Proliant SCSI (IDA) storage health monitoring
• Fixed an issue with the Cisco Default Gateway Changed alert-generating rule

To illustrate an oversight resulting in this error, the following ExpressionFilter configuration would result in a workflow failure:

<SimpleExpression>
   <ValueExpression>
        <XPathQuery>ErrorCode</XPathQuery>
   </ValueExpression>
   <Operator>Equal</Operator>
   <ValueExpression>
       <XPathQuery>1</XPathQuery>
    </ValueExpression>
</SimpleExpression>

The correct implementation of this Expression Filter module would be:

<SimpleExpression>
   <ValueExpression>
        <XPathQuery>ErrorCode</XPathQuery>
   </ValueExpression>
   <Operator>Equal</Operator>
   <ValueExpression>
       <Value>1</Value>
    </ValueExpression>
</SimpleExpression>

Unfortunately, this type of error is not caught by the MPBPA, most likely due to the difficulty in differentiating between a valid XPathQuery value and string value.  To perform some basic error checking to identify these errors when authoring, I have written a simple PowerShell script that analyzes the XML of an unsealed Management Pack and reports errors and potential errors with ValueExpression configuration in ExpressionFilter modules.  

The logic of the script assumes the following:

• Any ValueExpression configured as an XPathQuery expression and having a value that starts with a dollar sign ($) can be assumed to be a configuration error
• Any ValueExpression configured as an XPathQuery expression and having a value that does not start with a forward slash or the strings:  Property or ErrorCode might be misconfigured

The script accepts the path to the Management Pack XML file as the single input parameter and then searches the XML for RegExExpression, SimpleExpression, and DayTimeExpression XML nodes.  These nodes are evaluated for mismatches on XPathQuery expressions and mismatches are reported to the console.

The output of the script looks like:

PS D:\Development\SCOM> ./checkexpressionfilters.ps1 -MPFIle:”d:\development\scom\xsnmp\1.1.Dev\xSNMP.AIX.xml”
Evaluating d:\development\scom\xsnmp\1.1.Dev\xSNMP.AIX.xml 

Error – Found mismatched XPathQuery value with XML:
<XPathQuery>$Target/Property[Type="xSNMP.AIX.Processor"]/Name$</XPathQuery> 

Error – Found mismatched XPathQuery value with XML:
<XPathQuery>$Target/Property[Type="xSNMP.AIX.Processor"]/Name$</XPathQuery> 

Errors found: 2
Warnings found: 0

The script can be downloaded here.

The body of the script is:

Param([string]$MPFile)

[xml]$XMLData = Get-Content $MPFile
write-host “Evaluating $MPFile”
write-host  “”
 
$ErrorCount=0
$WarningCount=0
$Nodes = $XMLData.SelectNodes(“//RegExExpression”)
$Nodes| ForEach-Object{
 $XPathExpressions = $_.SelectNodes(“.//XPathQuery”)
 $XPathExpressions | ForEach-Object{
  $ExpressionText = $_.InnerText
  if([string]$ExpressionText.Substring(0,1) -eq [string]“$”){
   [String]$OuterXML = $_.OuterXML
   $ErrorCount += 1
   $sMesg = “Error – Found mismatched XPathQuery value with XML: ” `
   + [environment]::NewLine + $OuterXML + [environment]::NewLine
   write-host $sMesg
  }
  
  if(($ExpressionText.Substring(0,1) -ne “/”) `
   -and ($ExpressionText.Substring(0,8) -ne “Property”) `
   -and ($ExpressionText.Substring(0,1) -ne “$”) `
   -and ($ExpressionText.Substring(0,9) -ne “ErrorCode”)){
   [String]$OuterXML = $_.OuterXML
   $WarningCount += 1
   $sMesg = “Warning – Found mismatched XPathQuery value with XML: ” `
   + [environment]::NewLine + $OuterXML + [environment]::NewLine
   write-host $sMesg
  }
 }
}

$Nodes = $XMLData.SelectNodes(“//SimpleExpression”)
$Nodes| ForEach-Object{
 $XPathExpressions = $_.SelectNodes(“.//XPathQuery”)
 $XPathExpressions | ForEach-Object{
  $ExpressionText = $_.InnerText
  if([string]$ExpressionText.Substring(0,1) -eq [string]“$”){
   [String]$OuterXML = $_.OuterXML
   $ErrorCount += 1
   $sMesg = “Error – Found mismatched XPathQuery value with XML: ” `
   + [environment]::NewLine + $OuterXML + [environment]::NewLine
   write-host $sMesg
  }
  
  if(($ExpressionText.Substring(0,1) -ne “/”) `
   -and ($ExpressionText.Substring(0,8) -ne “Property”) `
   -and ($ExpressionText.Substring(0,1) -ne “$”) `
   -and ($ExpressionText.Substring(0,9) -ne “ErrorCode”)){
   [String]$OuterXML = $_.OuterXML
   $WarningCount += 1
   $sMesg = “Warning – Found mismatched XPathQuery value with XML: ” `
   + [environment]::NewLine + $OuterXML + [environment]::NewLine
   write-host $sMesg
  }
 }
}

$Nodes = $XMLData.SelectNodes(“//DayTimeExpression”)
$Nodes| ForEach-Object{
 $XPathExpressions = $_.SelectNodes(“.//XPathQuery”)
 $XPathExpressions | ForEach-Object{
  $ExpressionText = $_.InnerText
  if([string]$ExpressionText.Substring(0,1) -eq [string]“$”){
   [String]$OuterXML = $_.OuterXML
   $ErrorCount += 1
   $sMesg = “Error – Found mismatched XPathQuery value with XML: ” `
   + [environment]::NewLine + $OuterXML + [environment]::NewLine
   write-host $sMesg
  }
  
  if(($ExpressionText.Substring(0,1) -ne “/”) `
   -and ($ExpressionText.Substring(0,8) -ne “Property”) `
   -and ($ExpressionText.Substring(0,1) -ne “$”) `
   -and ($ExpressionText.Substring(0,9) -ne “ErrorCode”)){
   [String]$OuterXML = $_.OuterXML
   $WarningCount += 1
   $sMesg = “Warning – Found mismatched XPathQuery value with XML: ” `
   + [environment]::NewLine + $OuterXML + [environment]::NewLine
   write-host $sMesg
  }
 }
}
Write-Host “Errors found: $ErrorCount”
Write-Host “Warnings found: $WarningCount”

• xSNMP  for Juniper Networks
• xSNMP for SonicWALL
• xSNMP for APC NetBotz

We’re still finishing up testing and fine-tuning on the Oracle Unix/Linux MP as well.

These updates will be described in more detail here and available for download on manage-X.net soon.