Using the Shell Command Templates

• Targeting:  The only acceptable targets for the UNIX/Linux Shell Command templates are UNIX/Linux Computer, or classes based on it.  So UNIX/Linux Computer, Linux Computer, Red Hat Enterprise Linux Computer, and Red Hat Enterprise Linux 5 Computer (for example) are all valid targets. Operating System, Logical Disk, Local Application, and Computer Role types are not valid targets.  Best practice for rules and monitors is to target the computer object but disable the rule/monitor by default, and use an override to enable it for a group. The Shell Command templates do not support object discovery. If you want to discover and target custom class types (like a UNIX/Linux Computer Role or Local Application), traditional MP Authoring (XML) can be used.
• Shell Command Input:  As the templates use the ExecuteShellCommand method of the agent’s script provider with a WSMan Invoke module, the command is sent to the agent on each execution. This requires that the shell command is a single line. Simple shell scripts can be condensed into a “one-liner” by replacing line breaks with semi-colons. If you want to use a complex or long script, the best approach is to distribute it as a shell script to a common path on monitored agents, then use the shell command templates to invoke the script by providing the path, such as:  /opt/scripts/myscript.sh
• Synchronous Execution & Command Performance:  The agent’s script provider executes scripts synchronously. For optimal server and agent performance and monitor reliability, it is important to make sure that shell commands execute quickly. If you want to use a long-running command or script in a monitor or rule, it is advisable to configure the script to execute periodically on the agent (e.g. with cron) outputting to a file, and then use the shell command monitor or rule to parse the output file. Unless all parameters for the rule or monitor (e.g. scheduler, shell command, timeout, etc) are identical, rules and monitors created by the Shell Command templates will not cookdown.
• Handling Complex Output:  The agent’s script provider returns StdOut, StdErr, and ReturnCode for the shell command execution. Using the templates, it is not possible to post-process this output to split the output into multiple data items, so it is best to consider StdOut from the shell command as representative of one value.  You can always call the same shell command and use the pipeline to parse with grep, awk, sed, etc to filter the output to a single output value. If your shell command returns structured data representing multiple data points, you still have the option of authoring outside of the Operations Console and using a PowerShell script to parse the output.
• Mind the Quotes and Special Characters:  The agent’s script provider executes the shell command using the default shell (for the user context of the specified Run as Account), and wraps the actual command in quotes.  So the command executed on the agent host will be something like: sh –c “my shell command” or bash –c “my shell command.” This means that double-quotes inside the shell command need special handling. If you can’t use single-quotes instead of double-quotes, escape the double-quotes with a backslash. As Management Packs are XML-based, some characters like < and /> can be problematic. It is not possible to embed CDATA blocks in the template wizards, so if you can’t avoid use of < and /> in the command and they prove to be problematic, it is advisable to distribute the script to a common file system location on the monitored agents and use the Shell Command templates to invoke the script on the file system.
• sudo Eevation:  If the Run As account specified via the selected Run As profile in the Shell Command templates is configured for sudo elevation, sudoers will need to be configured to authorize the user to execute the command.  For example, adding the following line through visudo will allow the user monuser to execute the command service named restart as input in the Shell Command task template (assuming the default shell of /bin/sh):
  monuser ALL=(root) NOPASSWD: /bin/sh -c service named restart
• Expression Filters
  Valid values for Parameter Names in the template Expression Filters are:
  • //*[local-name()=”StdErr”]  for StdErr
  • //*[local-name()=”StdOut”] for StdOut
  • //*[local-name()=”ReturnCode”]for ReturnCodeThe Expression Filters in the templates are pre-configured for common scenarios. However, some commands used in monitoring will return values to evaluate on StdOut while other cases will mandate evaluation of StdErr.  For example, if you wanted to run a command to check a connection to a MySQL server, any errors would be output to StdErr. Thusly, the monitor’s expression filter would need to look for the presence of the string ERROR in the StdErr output.Additionally, the default Expression Filters evaluate that ReturnCode = 0 for rules and monitors. This default behavior drops the workflow if the shell command itself fails.  However, if you want to generate an alert if a command fails, the ReturnCode may be 1 in the error state. In this case, the Expression Filter could be configured to go unhealthy when ReturnCode = 1, or StdErr could be evaluated for a string, and the ReturnCode evaluation removed.As in all OpsMgr templates, the Expression Filter dialogs assume values are strings, which can be problematic when evaluating numeric data.  Options for addressing this issue are
  • Export the MP after creating the rule or monitor, and edit the Type attribute in the Expression XM
  • Use Regular Expressions in the Expression Filter instead of greater than/less than evaluations
  • Perform numeric evaluation operations on the agent in the Shell Command and output a simple health state (OK or ERROR) from the shell command itself.And lastly, be mindful of the logical outcome of Expression Filters. For monitors, any potential value from the shell script should positively evaluate to one of the health states (Error or Healthy for two state monitors, or Error, Warning, or Healthy for three state monitors). Particularly, be mindful of AND and OR groupings on Error and Healthy states.  For example, a monitor that requires condition 1 AND condition 2 to be true to be in an Error state should require that only condition 1 OR condition 2 not be true to be Healthy.

Tips, Tricks, and Best Practices

• When creating shell command rules and monitors, test out the commands in a task first!  Shell Command tasks are very simple to create and provide immediate feedback.  Syntactical or authorization issues will be most quickly identified through this method.
• Target “Computer” classes, but disable the rules and monitors by default.  Override the rule and monitor for a group to enable it.
• Only use short-running shell commands in rules and monitors
  • Long running commands or scripts can be scheduled with cron and output to file, using the Shell Command templates to parse the output file
• Avoid double quotes as well as < and /> characters if possible.
  • Complex scripts can be deployed to the file system of the monitored agents and invoked by the Shell Command templates
• For optimal performance, use the largest schedule interval that makes sense (15 minutes is preferred for most cases)
• Optimize the shell commands (using pipeline formatting with grep, awk, sed, etc)  to return a single, easy to parse value
  • For monitors, returning values like: OK, WARNING, ERROR are best
  • For performance rules, the value evaluated must be a single numeric value
• To troubleshoot sudo issues, tail /var/log/messages and look for the exact command that failed – to cross reference with sudoers configuration.
• Be aware what the shell command returns for StdOut, StdErr, and ReturnCode in healthy and unhealthy scenarios. This is critical information when configuring the Expression Filters.
• Have fun!

The new templates are:

Monitors

• UNIX/Linux Shell Command Three State Monitor
• UNIX/Linux Shell Command Two State Monitor

Rules

• UNIX/Linux Shell Command (Alert)
• UNIX/Linux Shell Command (Performance)

Tasks

• Run a UNIX/Linux Shell Command

Note: For the OpsMgr 2012 Release Candidate, the Shell Command Template MP needs to be downloaded and imported.  In the final release, it will be imported by default.

Underneath the covers, all of these templates use the ExecuteShellCommand method of the agent’s script provider with the WSMan Invoke module. This method executes the command and outputs StdOut, StdErr, and ReturnCode. The command can be a path to a simple command, a command or script existing on the managed computer, or a “one-liner” script (a shell script condensed to one line with pipes and semi-colons).  The templates also allow you to select whether to run with the nonprivileged action account, or the privileged account (which also supports sudo elevation).

If you’ve done this kind of UNIX/Linux authoring in 2007 R2, you will quickly see how much easier and faster this can be done in 2012.

To show the use of these templates, I have put together an MP authoring walkthrough for monitoring BIND DNS servers on Linux. This entire MP will be created in the Operations Console, with no XML editing!

Walkthrough: Monitoring BIND on Linux

Before starting on the MP implementation, let’s have a look at what we might want to monitor to check health of a BIND DNS server.  Some basic metrics would be:

• Daemon status:   if the named daemon is not running, it’s a pretty good indication that BIND is not healthy!
• DNS resolution success:  a good way to validate that BIND is functioning properly is to check that the server can resolve a hostname.  This could be the FQDN of a local host or an Internet URL (if we wanted to confirm that Internet DNS resolution was functioning properly).  This can be done easily with nslookup on the BIND server itself:
  • nslookup lx12.contoso.com 127.0.0.1
  • nslookup www.microsoft.com 127.0.0.1
• Performance metrics:   we can use the shell command templates to collect or monitor numeric performance data as well.  For a BIND server, one performance metric we could collect is the time to resolve a name (in seconds):
  /usr/bin/time -f %e nslookup lx12.contoso.com 127.0.0.1 > /dev/null
  

Setting up the Management Pack:

• The first step is to create a new Management Pack.  In the Administration pane of the Operations Console, right-click on left-hand menu and choose Create Management Pack. Input a name and description and complete the wizard:
  
• We will use a group to control targeting of the rules and monitors in the MP, so the next step is to create a group. In the Authoring pane, right click on Groups and choose: Create a new Group.  Provide a name and description, and select the target Management Pack (the one created in the previous step) and click Next.
  
• In the Explicit Members, add the Bind hosts (filter by UNIX/Linux Computer or Linux Computer objects).  Note:  membership can be controlled for this group at any point in the future, so not all BIND servers have to be added at this point.
  
• No action is needed for the Dynamic Members, Subgroups, or Excluded Members dialogs, so click next through these to complete the group creation wizard.
• With the MP and group created, it’s on to the actual monitoring.

Creating a BIND Restart Task

The Run a UNIX/Linux Shell Command task wizard is the simplest of the shell command templates, so it is a good place to start. These steps will result in a task that restarts the BIND daemon on a Linux computer from the Operations Console.

• In the Authoring pane of the console, expand Management Pack Objects and right click Tasks. Select Create a New Task.
• Select Run a UNIX/Linux Shell Command from the Agent Tasks list, and select the target Management Pack (created in a previous step).  Click Next.
  
• Input a Name for the task (Restart BIND Daemon), provide a Description, and select the Target (Linux Computer). Click Next.
  
• The command to restart the BIND daemon is: service named restart.  Type this into the Shell Command entry pane.  Restarting a daemon is a privileged operation, so select the UNIX/Linux Privileged Account Run As profile.  The default timeout of 120 seconds should be sufficient, so click Create.
  
• That’s all it takes to create a shell command task.  When clicking on a Linux Computer instance in the monitoring view, the Restart BIND Daemon task will be listed in the right-hand task pane and can be run directly from the console.

Monitoring the BIND Daemon

• In the Authoring pane of the console, right-click Management Pack Templates and choose Add Monitoring Wizard.
• Select UNIX/Linux Process Monitoring and click Next
  
• Input a Name for the monitor, a Description, and select the target Management Pack (created previously):
  
• This is a pretty simple process monitor, so simply input named for the Process name, click Select a group and select the BIND servers group created previously.  An Alert severity of Error is appropriate and process argument filtering is not required, so click Next.
  
• On the Settings page, accept the defaults by clicking Next. This will generate an alert if named is not running.
  
• Click Create to complete the wizard.

Creating the Name Resolution Monitor

As described above, the command nslookup can be used to check name resolution health from the DNS server (specifying the local host as the server in the second argument).  For example: nslookup lx12.contoso.com 127.0.0.1.  The actual output of this command looks like:

lx11:/var/lib/named # nslookup lx12.contoso.com 127.0.0.1
Server:         127.0.0.1
Address:        127.0.0.1#53

Name:   lx12.contoso.com
Address: 192.168.1.76

To make this command more monitor friendly, we can do a bit of pipeline parsing:
nslookup lx12.contoso.com 127.0.0.1|egrep ‘^Name:.*lx12.contoso.com’|wc -l

This shell command will return a value of 1 if the line: Name:   lx12.contoso.com is found in StdOut and a value of 0 otherwise.  Thusly, a value of 1 means that the name resolution attempt succeeded, and a value of 0 means that it failed.

The steps to create a monitor using this command are:

• In the Authoring pane of the console, expand Management Pack Objects and right click Monitors. Select Create a Monitor and Unit Monitor.
• Expand Scripting, then Generic and select UNIX/Linux Shell Command Two State Monitor and select the target Management Pack (created previously). Click Next.
  
• Input a Name, Description, and Target (Linux Computer) for the monitor. Select a Parent monitor (Availability) and uncheck Monitor is enabled.  Click Next.
  
• Configure a schedule interval.  For performance optimization, this should be as large of a value as reasonable.  10 or 15 minutes should be sufficient for most purposes. Click Next
  
• Input the Shell Command (replacing lx12.contoso.com with the hostname to resolve):
  
  nslookup lx12.contoso.com 127.0.0.1|egrep ‘^Name:.*lx12.contoso.com’|wc –l
  
  The UNIX/Linux Action Account Run As profile is appropriate for this command, and 120 seconds is a sufficient value for the Timeout. Click Next.
  
• The next page of the wizard is for configuring the Error Expression. If the conditions defined in this expression are matched, the monitor will go to an error state. The Expression Filter dialog is preloaded with the following values://*[local-name()="StdOut"]  Contains  <input value>
  //*[local-name()="ReturnCode"] Equals  0With the shell command used in this example, the error state should be triggered when StdOut does not equal 1, so we can simply modify the first line to that effect.  This results in an error condition that is triggered when StdOut does not equal 1 and the nslookup command executed successfully (ReturnCode equals 0).
  
• After clicking Next, the Healthy Expression dialog is displayed. As a StdOut value of 1 indicates a successful nslookup operation using the provided shell command, simply set the first line to: //*[local-name()="StdOut"]  Equals 1 and click Next.
  
• In the Configure Health dialog, we can choose whether we want the error state to map to a Critical or Warning event by changing the Health State drop down.   In this example, I will set the Health State to Warning.
  
• The next dialog is for alert configuration. Check Generate alerts for this monitor and select an appropriate Priority and Severity (Match monitor’s health).  Edit the Alert name if appropriate and provide an Alert description.Standard $Target$ variables can be embedded in the Alert description by clicking […].  The syntax to include data from the shell command execution is:StdOut:  $Data/Context///*[local-name()=”StdOut”]$
  StdErr:  $Data/Context///*[local-name()=”StdErr”]$
  ReturnCode: $Data/Context///*[local-name()=”ReturnCode”]$In this example, I used the following description:The BIND DNS server: $Target/Property[Type="MicrosoftUnixLibrary7320040!Microsoft.Unix.Computer"]/NetworkName$ failed a name resolution test.StdErr: $Data/Context///*[local-name()="StdErr"]$Click Create to complete the monitor creation.
  
• As this monitor targets all Linux Computers, we created it without enabling it by default. We can enable it for the group of BIND servers with an override. In the Authoring pane, expand Management Pack Objects, and click on Monitors. In the top-right of the Monitors pane, click Change Scope, and check Linux Computers from the Scope Management Pack Objects dialog.  Find the monitor that was just created (BIND Name Resolution Check).
  
• Right-click the monitor, click Overrides, then Override the Monitor, then For a Group.  Select the BIND servers group created previously and click OK.
  
• Override the Enabled property to equal True and click OK
  

Creating the Name Resolution Time Performance Rule

• In the Authoring pane of the console, right-click Rules, and select Create a new rule.
• Under Collection Rules, Probe Based, select UNIX/Linux Shell Command (Performance) and select the target Management Pack (previously created).  Click Next.
  
• Input the Name and Description for the rule. Select the Target (Linux Computer) and uncheck Rule is enabled. Click Next.
  
• Configure a schedule interval.  For performance optimization, this should be as large of a value as reasonable.  10 or 15 minutes should be sufficient for most purposes. Click Next.
  
• On the Shell Command Details page, input the Shell Command:
  /usr/bin/time -f %e nslookup lx12.contoso.com 127.0.0.1 > /dev/null
  This command will return the time in seconds (to StdErr) that it took to complete the name resolution lookup. This is a non-privileged operation, so the UNIX/Linux Action Account is sufficient for the Run As profile.  Click Next.
  
• The next page provides the opportunity to filter the output before mapping to performance data.  Performance data mapping can only occur if the value is a valid double value, so the default expression syntax uses a RegExp to validate that StdOut is a numeric value, and also filters that the ReturnCode = 0, indicating a successful execution.  While the default configuration is valid for most scenarios, the time command used in this shell command actually outputs its value to StdErr.  So in this case, the first line of the filter should be modified to use a Parameter Name of //*[local-name()=”StdErr”]. Click Next.
  
• Configure the Performance Mapper.  Object, Counter, and Instance are arbitrary values that will be used to identify the performance metric in performance views and reports. The default Value of $Data///*[local-name()=”StdOut”]$ is the variable syntax for the returned StdOut, which is appropriate for most cases. Again, this needs to be modified because the time command used in this example outputs to StdErr.  The StdErr variable is: $Data///*[local-name()='StdErr']$. Click Create.
  
• As this rule targets all Linux Computers, we created it without enabling it by default. We can enable it for the group of BIND servers with an override. In the Authoring pane, expand Management Pack Objects, and click on Rules. In the top-right of the Rules pane, click Change Scope, and check Linux Computers from the Scope Management Pack Objects dialog.  Find the rules that was just created (BIND Name Resolution Test Time in Seconds).
  
• Right-click the rule, click Overrides, then Override the Monitor, then For a Group.  Select the BIND servers group created previously and click OK.
  
• Override the Enabled property to equal True and click OK.
  

Creating a Dashboard View

With the monitoring pieces now in place, we can move on to creating a dashboard view to show health and performance.

• In the Monitoring pane of the console, find the folder for the BIND management pack. Right-click the folder and choose New, then Dashboard View
  
• Select Grid Layout and click Next.
  
• Input a Name and Description and click Next.
  
• Select 3 Cells and pick a layout. Click Next then Create.
  
• We now have an empty dashboard view.

• Click Click to add widget in the top-left rectangle. Select State Widget and click Next.
  
• Provide a Name and Description and click Next.
  
• Under Groups and objects click Add. In the search dialog, find the BIND servers group, add it to the selection list and click OK.
  
• Configure the display and filtering preferences and complete the widget wizard.

Adding the Performance Widget

• Click Click to add widget in the top-right rectangle. Select Performance Widget and click Next.
  
• Input a Name and Description and click Next.
  
• Under Select a group or object click […] and find the BIND servers group.  Select the group and click OK.
  
• Under Select performance counters click Add. Select the Object (BIND Server), Counter (Name Resolution Time (s)), and select (All) for Instance.  Click Add to add the performance counter and click OK.
  
  Note: only performance counters that have been collected will show up in this list. If you created the rule in the past few minutes and don’t see it listed yet, wait a little bit longer.
• Select the Time Range and Display Options and complete the wizard.

Adding the Alert Widget

• Click Click to add widget in the bottom rectangle. Select Alert Widget and click Next.
  
•  Input a Name and Description and click Next.
  
• Under Select a group or object click […], find and selecd the BIND servers group and click OK.
  
• Specify the criteria.  Be sure to filter for only the New resolution state if you just want to see active alerts. Click Next.
  
• Configure Display Preferences and complete the wizard.

The end result is a dashboard showing health and performance of monitored BIND servers:

Summary

As you can see from this walkthrough, the new template wizards for UNIX/Linux monitoring in OpsMgr 2012 are a significant improvement and make custom monitoring far easier to implement!

UNIX/Linux Process Monitoring Template

Operations Manager 2007 R2 included the Unix Service Monitoring template for custom monitoring of daemons on UNIX and Linux agents.   This template has been replaced in the System Center 2012 release of Operations Manager with the far more capable UNIX/Linux Process Monitoring template.   The new UNIX/Linux Process Monitoring template allows more flexibility in process/daemon monitoring, including the ability to monitor for minimum and maximum process count thresholds, and the ability to filter processes on arguments in addition to the process name. For this example, I will walk through the use the UNIX/Linux Process Monitoring template to monitor a Tomcat daemon.

The UNIX/Linux Process Monitoring template is accessible in the Authoring Pane of the Operations Console.   It can be launched with the “Add Monitoring Wizard” task under the Management Pack Templates view.

Input a name, description, and select the target management pack (where the template-created MP elements will be saved).

The next page is the details page, where process information is input.   If you already know the process name, you can just type that in and select the group to target.  Alternatively, you can click Select a Process to connect to an agent and list the currently running processes.

In this case, I will connect to an agent and enumerate the running processes.  The Tomcat daemon is a Java process, so I select the process named: java.

In the bottom field of the details dialog, the running processes on the agent that match the selected name are listed.   This particular agent is also running WebSphere, and its process also has the name: java, which is why two java processes are listed in this field.   As I only want to monitor the Tomcat daemon with this particular monitor, I don’t want the WebSphere process to affect the monitoring.   While the two processes share the same name, they have different process arguments.  The Regular expression to filter process arguments field allows me to input a regular expression that will be evaluated against the process arguments.   Only processes that match the regular expression will be included in this monitor.   So, I can input a regular expression of ^.+tomcat.+ and only processes with a name of “java” that have the string: tomcat in their arguments list will be evaluated by the monitor.   The wizard filters the list of running processes on the fly to show what will be evaluated with a given regular expression.

A few notes on this functionality:

• The regular expression filtering by arguments is an optional function.  If you just want to monitor processes by name, you can leave this field blank.
• The regular expression is evaluated against all arguments as one long string.  That is, the arguments (argv[0] – argv[<max>]) are concatenated with a  space separator, and the regular expression is evaluated against this concatenated string.
• The regular expression filtering is performed on the Windows-side (Health Service), not on the UNIX/Linux agent. Thus, .NET regular expressions are used (this differs from the log file monitoring template where the regular expression matching is performed on the UNIX/Linux agent).
• If you know the regular expression that you want to use, and you are targeting a group, you can just select a group, and type the process name and regular expression into the field without connecting to an agent.   However, if you want to test the regular expression in the wizard, you need to use the Select a process function to connect to an agent and list running processes that match the selected name.   Once you are satisfied with the arguments filter, you can retarget the monitor to a group by clicking Select a group.    The filtering by arguments functionality works whether you are targeting a single agent or a group, but testing it requires connecting to an agent to list the running processes.

On the next page, optional minimum and maximum thresholds can be set for the acceptable count of processes that match the name and optional arguments filter.  For example, to monitor that at least one instance of a process is running, set a minimum threshold of 1, to alert if any instances of a process are running, set a maximum threshold of 0.  To monitor that no less than 5 and no more than 10 instances of a process is running, set a minimum threshold of 5 and a maximum of 10.   Note:  the process count is calculated after the optional arguments filter is applied.   So in this example, the WebSphere “java” process will not be included in the process count, because the ^.+tomcat.+ filter was applied to the arguments.

All done:

When I kill the Tomcat daemon, this is what the alert looks like:

Note:

Despite similar functionality to the Unix Service Template from 2007 R2, the new UNIX/Linux Process Monitoring template is an entirely new template. If you are upgrading from 2007 R2 to 2012, both templates will be available. To make use of the new functionality in the process template, you will need to create new instances to replace the previous instances created by the Unix Service Template.

Performance and Cookdown Considerations

Each unique process name that is specified in a process template will involve one WSMan query to the agent per polling interval.  Multiple template instances that inspect a single process name but implement unique regular expression filtering will cookdown into a single WSMan query to the agent per polling interval. For example, if ten process template instances target a single host and all monitor a process named java with distinct process argument filters (RegExp), only one WSMan query will be sent to the agent per polling interval.  If ten process template instances target a single host and each monitors a unique process name (processA, processB, etc.), ten WSMan queries will be sent to the agent per polling interval.

In cases where a large number of unique process names need to be monitored, process argument filtering is not required, and the monitored agents do not have a large number of running processes (with sizable parameter strings), custom MP authoring can be used to create a cookdown-optimized monitor type that enumerates SCX_UnixProcess with the filter: Select Name, Parameters from SCX_UnixProcess. This would allow all of the process monitors to cookdown to a single WSMan query assuming all other configuration parameters (such as Interval) are aligned.

The vulnerability is described in bulletin MS12-006, and more information can be found in the KB article.  While we continue to investigate options for resolving this issue, there are two viable workarounds (which must be applied to all Mgmt Servers and Gateways that manage UNIX/Linux agents):

1. Uninstall the update KB2585542 
2. Make a registry modification to disable the SecureChannel changes implemented in the update

Note: the registry modification described here and in the KB article effectively disables the security fix that the update implements, so the modified system is subject to the same vulnerability as an unpatched system.

Modifying the registry to disable the SecureChannel changes:

• A “FixIt” package is available in the KB article under the Known Issues section that can be used to disable the security update
• Alternatively, you can add the 32bit DWORD value:
  HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control
  \SecurityProviders\SCHANNEL\
   SendExtraRecord = 2

These changes take effect immediately and do not require a reboot.

UNIX and Linux agent monitoring in Operations Manager requires certificates to secure the SSL communication channel between the Management Servers and agents.  In this post, I will provide some background information on this communication and the certificates, as well as describe an optional approach to replace the default Operations Manager certificate infrastructure with your organization’s Public Key Infrastructure.

The Protocols

The Operations Manager UNIX/Linux agent is a very lightweight agent implementation, comprising a CIM Object Manager (OpenPegasus) and CIM Providers.   Unlike Operations Manager Windows agents, the UNIX/Linux agent doesn’t have a health service, and doesn’t run workflows locally.  Rather, the Management Server (or Gateway) that manages the agent runs the workflows and remotely connects to the UNIX/Linux agent to retrieve current data.  

There are two protocols involved in the communication between the Management Server and the UNIX/Linux agent:  ssh and WS-Management.   

Ssh is used purely for agent maintenance activities, and is not used for any monitoring.   Operations like agent installation, uninstallation, upgrade, or agent daemon restart (through a recovery task) are executed over ssh.    Ssh facilitates the transfer of files and execution of remote commands for these operations when the agent daemon is unavailable.  

WS-Management (or WSMan) is the core protocol used in UNIX/Linux monitoring.   WSMan is a SOAP-based protocol for cross-platform management.   All monitoring operations (e.g. enumerating CIM providers for data on file systems, memory, etc, execution of commands/scripts for monitoring, executing log file reads for monitoring) are implemented over WSMan.   As WSMan is a web service protocol, the OpenPegasus-based CIMOM functions as a secure web server (user credentials are authenticated through PAM).  This is where the agent certificate comes in to play.

The Certificate

The UNIX/Linux agent certificate is quite simply used to secure the WSMan connection using SSL and provide authentication for the remote agent host.   The requirements for this certificate are:

• The certificate is a server authentication certificate (Enhanced Key Usage: 1.3.6.1.5.5.7.3.1)
• The CN of the certificate matches the FQDN that the Management Server uses to connect to the agent
• The certificate is signed by a trusted authority (and can be checked for revocation)

When the Operations Manager UNIX/Linux agent is installed, it generates a certificate (using openssl) at the path:  /etc/opt/microsoft/ssl.  The file name of the certificate is scx-host-<hostname>.pem and the corresponding private key is named scx-key.pem.   The agent actually looks for the certificate at /etc/opt/microsoft/scx/ssl/scx.pem, which is initially configured as a symbolic link pointing to scx-host-<hostname>.pem.

Upon initial agent installation, the certificate is not signed, and is not usable for securing the WSMan SSL communication.

Note:  when initially creating the certificate, the agent attempts to determine the agent hostname for use as the CN value of the certificate.   In cases where the DNS name known to the local host does not match the FQDN that OpsMgr will use to communicate with the agent, additional steps are required to establish a valid certificate.  More information can be found here: http://technet.microsoft.com/en-us/library/dd891009.aspx

Certificates and Management Servers

When a Management Server discovers a UNIX or Linux agent, the server uses its certificate to sign the agent certificate, acting like a standalone Certificate Authority.  In the discovery process, this actually involves securely transferring the certificate from the agent to the Management Server, signing it, copying it back to the agent, and restarting the agent daemon.  

In order to move an agent between Management Servers, the new Management Server must trust the certificate that was used to sign the agent’s certificate.  This becomes particularly important in the 2012 version of Operations Manager, where agents will move automatically between the Management Servers that are members of the Resource Pool managing the agent.  For more information on the procedure to trust a server’s certificate from another server, review this document: http://technet.microsoft.com/en-us/library/hh287152.aspx.

Using a PKI Instead of Management Servers for Signing

Because the certificates used for securing the agent SSL channel are not proprietary, a separate Public Key Infrastructure can be used to manage the agent certificates, if the PKI option is appealing for your organization.  While this requires some additional resources in the environment (a Certificate Authority) and customization, there are a few benefits to using a PKI: 

• Certificate policies are controlled by the PKI and customizable to meet your organization’s security requirements
• Migrations of agents between Management Servers (within or between Resource Pools) can be done without exporting/importing Management Server certificates – simplifying the provisioning of Management Servers.
• More options exist for automation of agent deployment and certificate signing

The procedure to use a PKI instead of Management Server signed certificates varies with different requirements and environments, but I will describe the steps required for one example approach.  This example assumes that the Certificate Authority is a Windows 2008 Certificate Authority. 

Prerequisites:

1. Configure the certificate template on the Certificate Authority – you can use the “Web Server” template or a copy of it – configure options and permissions, publish the template.
2. Import the CA certificate from the signing CA  to the trusted authorities list on every management serverthat will manage the UNIX/Linux agents:
   1. certutil -f -config “<CAHostname>\<CAName>” -ca.cert <CACertFile>
   2. certutil -addstore Root <CACertFile>

Per-Agent steps:

1. Install the agent – this can be done through the OpsMgr Discovery Wizard, manually, or with another package distribution tool.  If you use the OpsMgr Discovery Wizard to install the agent, the agent will generate a certificate that is signed by the management server, but this can be replaced with your PKI CA signed certificate.
2. Generate a cert signing request – either create a new private key with OpenSSL or use the private key generated during the agent install
1. a.      Command to generate a CSR using the key generated during agent install:
   openssl req -new -key /etc/opt/microsoft/scx/ssl/scx-key.pem  -subj /CN=<FQDN of agent host> -text -out <OutputPath>
2. 3.       Copy the CSR back to a Windows machine
3. 4.       Submit the CSR to the CA – this command assumes auto-enrollment is enabled and authorized:
1. certreq.exe -submit -config <CAHostName>\<CAName> -attrib “CertificateTemplate:<TemplateName>” <CSR FileName> <OutputCertName>
2. Copy the signed cert back to the UNIX/Linux agent using  a secure copy method.  If auto-enrollment was used in step 4, the value for <OutputCertName>  specifies the file name of the signed certificate to copy to the agent.
3. Update the symbolic link: /etc/opt/microsoft/scx/ssl/scx.pem to point to your new signed certificate
4. Restart the agent:  /opt/microsoft/scx/bin/tools/scxadmin –restart
5. Discover the agent using the Operations Console or PowerShell Cmdlet

Automation and Customization Opportunities

All of the per-agent steps described above can be executed from a command line, meaning that this procedure can be automated through scripting.  Using a script on a Windows server, the UNIX/Linux commands and file copying actions can be executed with SSH utilities like PuTTY’s plink and pscp.  For really robust automation capabilities, all of the steps can be implemented in a PowerShell script – I like the plink.exe integration example described on this blog: http://www.christowles.com/2011/06/how-to-ssh-from-powershell-using.html.

Aside from the primary benefits of automating these steps in terms of reducing manual interactions, other customization opportunities are exposed with using this scripting approach.  For example, if your DNS infrastructure and UNIX/Linux agent hostnames don’t neatly correlate, you could modify step 2 of the per-agent steps to also generate a new certificate with openssl using the desired FQDN as the certificate’s CN (http://technet.microsoft.com/en-us/library/dd891009.aspx).  Alternatively, if you are using Operations Manager 2007 R2 and want to implement agent deployment and certificate signing using sudo elevation instead of root credentials, the UNIX/Linux host commands in the per-agent steps could be prepended with the sudo command (this functionality is built into the 2012 version of Operations Manager).

Operations Manager 2012 Beta

http://www.microsoft.com/systemcenter/en/us/om-vnext-beta.aspx

Operations Manager 2012 Beta is available.   Exciting new features for UNIX/Linux monitoring include:

• Discovery Wizard:  All new Discovery Wizard, making it easier to deploy and discover UNIX and Linux agents
• Sudo support:  Privileged operations (monitoring and agent maintenance) can now be performed without root privileges by using a non-privileged credential and sudo elevation.
• SSH Key support:  Agent maintenance operations (via SSH) can be authenticated with an SSH key instead of password
• High Availability for UNIX/Linux agent monitoring:   Resource Pools implement highly-available UNIX/Linux monitoring with automatic failover and load distribution.
• RHEL 6 and AIX 7.1 support

Operations Manager 2007 R2 CU5

http://support.microsoft.com/kb/2495674

Operations Manager 2007 R2 Cumulative Update 5 is also now available.   Fixes for UNIX/Linux monitoring include:

• Performance data for LVM managed volumes not available
• Process monitoring does not keep name if run by using symbolic link
• AIX with large number of running processes crashes with bad alloc

CU5 also implements RHEL 6 support for Operations Manager 2007 R2.

In Part III of this series, I walked through creation of data sources, a discovery, and a rule for discovering dynamically-named log files and implementing an alert-generating rule for log file monitoring.  In this post, I will continue to expand this Management Pack to implement performance collection rules, using WSMan Invoke methods to collect numerical performance data from a shell command. 

Using Shell Commands to Collect Performance Data

Whether it is system performance data from the /proc or /sys file systems, or application performance metrics in other locations, performance data for UNIX and Linux systems can often be found in flat files.   In this example Management Pack, I wanted to demonstrate using a WSMan Invoke module with the script provider to gather a numeric value from a file and publish the data as performance data.   In many cases, this would be slightly more complex than is represented in this example (e.g. if the performance metric value should be the delta between data points in the file over time), but this example should provide the framework for using the contents of a file to drive performance collection rules.   The root of these workflows is a shell command using the cat command to parse the file, which could be piped to grep, awk, and sed to filter for specific lines and columns.  

Additionally, if the performance data (e.g. hardware temperature or fan speed, current application user or connection count) that you are looking for is not stored in a file, but available in the output of a utility command, the same method could be used by using the utility command instead of cat.

Collecting Performance Data from a File

In this example, the MyApp application stores three performance metrics in flat files in the subdirectory ./perf.   I have built three rules that cat these files, and map the values to performance data.  The three rules are functionally identical, so I will only describe one of them.

Performance Collection Rule:  MyApp.Monitoring.Rule.CollectMyAppMem

This rule collects the value from the ./perf/mem file in the application directory, which represents the current memory used by the application in KB.   The rule targets the MyApp.Monitoring.MyApp application class.

Rule Data Source:

The rule uses the MyApp.Monitoring.ShellCommandMonitoring data source, described in Post I of this series, with the configuration:

<Interval>300</Interval>
<TargetSystem>
$Target/Host/Property[Type="MicrosoftUnixLibrary!
Microsoft.Unix.Computer"]/NetworkName$</TargetSystem>
<ShellCommand>
cat $Target/Property[Type="MyApp.Monitoring.MyApp"]
  /InstallPath$/perf/mem
</ShellCommand>
<Timeout>120</Timeout>

Notice that the ShellCommand is our cat command:  cat $Target/Property[Type="MyApp.Monitoring.MyApp"]/InstallPath$/perf/mem

Rule Condition Detection:

A System.Performance.DataGenericMapper is used as the condition detection module to map the StdOut to performance data, with the configuration:

<ObjectName>
$Target/Property[Type="MyApp.Monitoring.MyApp"]
  /Name$
</ObjectName>
<CounterName>Memory used (KB)</CounterName>
<InstanceName>Total</InstanceName>
<Value>
  $Data///*[local-name()="StdOut"]$
</Value>

Write Actions:

The rule defines two write actions:  Microsoft.SystemCenter.CollectPerformanceData and Microsoft.SystemCenter.DataWarehouse.PublishPerformanceData to collect the data and publish it to the DW.   These require no configuration.

The end result is that the scheduled rule grabs the value from the text file, maps it to performance data, collects and publishes the performance data to the OM and DW databases.  This mechanism can be used for nearly any numerical performance metric that is accessible (in a timely fashion) from a shell command pipeline or script.

Stay tuned for more in this series….

In Part II of this series, I walked through creation of data sources, a discovery, a monitor type, and a monitor for customized ”Process Count” monitoring for discovered instances of a ”Service” class. In this post, I will continue to build on this example MP to implement dynamic log file discovery and monitoring.

Dynamic Log File Discovery and Monitoring

Log file monitoring of a single known log file can be easily implemented with the Microsoft.Unix.SCXLog modules, but in some cases, the full path to a log file isn’t static.   For example, if an application maintains multiple log files in a directory, the file name portion of the log file path may not be known ahead of time.    To handle this monitoring scenario, we can implement dynamic log file discovery – using a shell command execution, and then pass the full path of the log file to the SCXLog module for standard log file monitoring. This requires a new class instance, a discovery data source, a discovery rule, and a rule that actually implements the log file monitoring.

Defining the Log File Class and Hosting Relationship

Firstly, a new custom class is required to represent the log file objects.   Instances of this class will be discovered by the discovery rule.  

Definition 

• ID:  MyApp.Monitoring.LogFile
• Base Class:  Microsoft.Unix.ApplicationComponent
• Name:  MyApp Log file

Properties

• Name (String)
• Path (String) – Key

The properties for the log file class represent the file name and full path.   The full path is assured to be unique, so I have specified that as the key property of the class.

The log file class needs to be hosted by the MyApp class, to maintain the relationship between the log files and the application.  

Discovery Data Source:  MyApp.Monitoring.DataSource.DiscoverLogFiles

This data source will use the MyApp.Monitoring.DataSource.ShellCommandDiscovery probe action to find files in a given directory that match a pattern.   The output from this command execution will then be passed to a Microsoft.Windows.PowerShellDiscoveryProbe.   The reason that I am using a PowerShellDiscoveryProbe is that the listing of matched files will be returned as a single data item, the StdOut from the command.   Using a PowerShellDiscoveryProbe provides an easy way to split each line from the output and discover an instance per line. 

Configuration Parameters:

• Interval (integer):  Scheduler interval in seconds – overridable
• TargetSystem (string):  UNIX/Linux agent computer to execute the discovery
• Appname (string):   The name of the application object (which is the key property for the hosting class instance)
• LogFileNamePattern (string): The pattern that will be used in the grep operation to identify log files to discovery
• LogFilepath (string):  The path to search for log files at (via an ls command)

Member Modules:

The first member modules is a MyApp.Monitoring.DataSource.ShellCommandDiscovery probe action, that executes the following command:
ls $Config/LogFilepath$ |grep $Config/LogFileNamePattern$.  This simply enumerates the contents of the specified directory path, and pipes the results to grep, to match a specified pattern, which could be a string match or regular expression.

Module Configuration:

<Interval>$Config/Interval$</Interval>
<TargetSystem>$Config/TargetSystem$</TargetSystem>
<ShellCommand>ls $Config/LogFilepath$
   |grep $Config/LogFileNamePattern$</ShellCommand>
<Timeout>120</Timeout>

The output of this shell command then needs to be parsed so that each valid line in the output is discovered as an instance of a log file object.   This is most easily done with a PowerShellDiscoveryProbe:

param ([string]$CmdOutput,[string]$AppName,[string]$LogFilePath, [string] $TargetSystem,[string] $SourceID,[string]$ManagedEntityID)

$api = New-Object -comObject ‘Mom.ScriptAPI’
$discoveryData = $api.CreateDiscoveryData(0, $SourceID, $ManagedEntityID)

if ($CmdOutput -ne $null){
        $CmdOutput = $CmdOutput.Replace([Environment]::newline,” “)
 [array]$arList = $CmdOutput.Split(” “)
 $arList | ForEach-Object{
  [string]$sFile = $_
 if([int]$sFile.Length -ge [int]1){
  $SFilePath = $LogFilePath + “/” + $sFile
  $oInst = $discoveryData.CreateClassInstance(“$MPElement[Name='MyApp.Monitoring.Logfile']$”)
  $oInst.AddProperty(“$MPElement[Name='MyApp.Monitoring.Logfile']/Name$”, $sFile)
  $oInst.AddProperty(“$MPElement[Name='System!System.Entity']/DisplayName$”, $sFile)
  $oInst.AddProperty(“$MPElement[Name='MyApp.Monitoring.Logfile']/Path$”, $sFilePath)
  $oInst.AddProperty(“$MPElement[Name='MyApp.Monitoring.MyApp']/Name$”, $AppName)
  $oInst.AddProperty(“$MPElement[Name='MicrosoftUnixLibrary!Microsoft.Unix.Computer']/PrincipalName$”, $TargetSystem)

  $discoveryData.AddInstance($oInst)
 }     
}
 $discoveryData
}

Remove-variable api
Remove-variable discoveryData

The PowerShell script loads the Mom.ScriptAPI, creates a Discovery Data instance, and then walks through each line of the ouptut.   If the file name is a valid string (not empty), a class instance is created for the MyApp.Monitoring.Logfile class, and the path and file name properties are set.   The PrincipalName property of the Microsoft.Unix.Computer object, and the AppName property of the MyApp.Monitoring.MyApp class ares included in the DiscoveryData, so that the discovery mapping process can map the hosting relationships. 

Parameters are passed from the module configuration to the script using the Parameters XML fragment in the module configuration:

<Parameters>
<Parameter>
 <Name>TargetSystem</Name>
 <Value>$Config/TargetSystem$</Value>
 </Parameter>
<Parameter>
 <Name>AppName</Name>
 <Value>$Config/Appname$</Value>
 </Parameter>
<Parameter>
 <Name>LogFilePath</Name>
 <Value>$Config/LogFilepath$</Value>
 </Parameter>
<Parameter>
 <Name>CmdOutput</Name>
 <Value>
 $Data///*[local-name()="StdOut"]$
</Value>
 </Parameter>
<Parameter>
 <Name>ManagedEntityID</Name>
 <Value>$Target/Id$</Value>
 </Parameter>
<Parameter>
 <Name>SourceID</Name>
 <Value>$MPElement$</Value>
 </Parameter>
 </Parameters>

This data source can then be used to discover log files matching a pattern, in a specified directory.  

Discovery Rule:  MyApp.Monitoring.Discovery.LogFile

This discovery will discover dynamically-named log files, in a specified path, using a regular expression to filter by file name.   It discovers instances of the MyApp.Monitoring.LogFile class, and uses the MyApp.Monitoring.DataSource.DiscoverLogFiles data source.  The discovery targets  MyApp.Monitoring.MyApp

Data Source Configuration:

• <Interval>14400</Interval>
• <TargetSystem>$Target/Host/Property[Type="MicrosoftUnixLibrary!Microsoft.Unix.Computer"]/PrincipalName$</TargetSystem>
• <Appname>$Target/Property[Type="MyApp.Monitoring.MyApp"]/Name$</Appname>
• <LogFileNamePattern>‘^log[0-9]+’</LogFileNamePattern>
• <LogFilepath>$Target/Property[Type="MyApp.Monitoring.MyApp"]/InstallPath$/logs</LogFilepath>

The two parameters to note are the LogFilepath (which is defined as the application path discovered for the MyApp application, appended with “/logs”) and the LogFileNamePattern (which is a regular expression – ‘^log[0-9]+’ – that will match log files named:  logxxx, where xxx is a number).  

Monitoring the Discovered Log Files

Log File Monitoring Rule:   MyApp.Monitoring.Rule.AlertOnLogError

Now that the dynamically-named log files will be discovered, we need a rule to alert when an error is found in one of the logs.   The rule will target all instances of the MyApp.Monitoring.LogFile class, so that when a new log file instance is discovered, it is automatically monitored.  The rule uses the MicrosoftUnixLibrary!Microsoft.Unix.SCXLog.Privileged.Datasource (assuming the log files require privileged credentials to access).

Data source configuration:

• <Host>$Target/Host/Host/Property[Type="MicrosoftUnixLibrary!Microsoft.Unix.Computer"]/NetworkName$</Host>
• <LogFile>$Target/Property[Type="MyApp.Monitoring.Logfile"]/Path$</LogFile> <RegExpFilter>^.*(e|E)rror.*$</RegExpFilter>

The discovered path to the logfile instance is input as the LogFile parameter value, and a Regular Exprssion (^.*(e|E)rror.*$) is defined to match any log entries with the string:  error or Error in the message.  

Condition Detection configuration:

A System!System.Event.GenericDataMapper condition detection is then configured to map the data to EventData, for consumption by OpsMgr.  The configuration of this module is:

• <EventOriginId>$MPElement$</EventOriginId>
• <PublisherId>$MPElement$</PublisherId>
• <PublisherName>MyApp</PublisherName>
• <Channel>Application</Channel>
• <LoggingComputer>$Target/Host/Host/Property[Type="MicrosoftUnixLibrary!Microsoft.Unix.Computer"]/NetworkName$</LoggingComputer>
• <EventNumber>8001</EventNumber>
• <EventCategory>0</EventCategory>
• <EventLevel>1</EventLevel>
• <UserName/>
•  <Params/>
•  </ConditionDetection>

Write Actions:

In this rule, I have configured two write actions, for collecting the event, and generating an alert.  The CollectEvent (SC!Microsoft.SystemCenter.CollectEvent) module requires no additional configuration, and the alert can be configured to provide details about the logged error message:

Stay tuned for more in this series…

Advanced Service Monitoring

1. Define a custom service class, and discover an instance of this class for each service to monitor, configure monitor types and monitors targeting this class
2. Create a monitor for each service to monitor, targeting the custom application class

Both methods are completely viable, and in most cases, it is appropriate to take the simpler approach and target the custom monitors to the application, providing static inputs into the monitor.   There are some cases where discovering a class instance for the service makes sense though.  Facilitating dynamic discovery of services or thresholds (read from a config file), using the service class in a Distributed Application model in OpsMgr, or maintaining logical seperation (in terms of monitoring) between the application and its subsystems are all scenarios that would benefit from discovering the monitored services as class instances.   For the purpose of illustration, I will discover the daemon to monitor in this example Management Pack as a class instance.

Class Definition

Class:  MyApp.Monitoring.Service

Definition

• ID:  MyApp.Monitoring.Service
• Base Class:  Microsoft.Unix.ApplicationComponent
• Name:  MyApp Service

Properties

• Name (String) – Key
• MinRunning (Integer)
• MaxRunning (Integer)

Discovery

Then we can define the data source to discover a service.   In this case, we know the name of the service and the value of the properties, so we don’t need to actually poll the agent to return data.   We can simply combine a Discovery Scheduler with a Discovery Data Mapper module to implement the data source.  However, we want to be able to override the values of MinRunning and MaxRunning, so these will need to be exposed as overridable configuration parameters.

Therefore, I’ve chosen to implement this data source in two parts.   The first data source, will simply combine a System.Discovery.Scheduler module and a System.Discovery.ClassSnapshotDataMapper module.   This data source will accept Interval, ClassId and InstanceSettings parameters as inputs.  The second data source will reference the first data source, but implement parameters for Service Name, MinRunning, and MaxRunning.    By breaking this into two data sources, the first data source can be used for other simple discoveries.

Discovery Data Source:  MyApp.Monitoring.DataSource.DiscoverObject

This is the data source that simply combines a scheduler and a discovery data mapper.  It requires that the MapperSchema be added to the Configuration:

<Configuration>
<IncludeSchemaTypes>
<SchemaType>
 System!System.Discovery.MapperSchema
</SchemaType>
</IncludeSchemaTypes>
…

Configuration Parameters:

• Interval (integer):  Scheduler interval in seconds
• ClassId (string):  ID of the Class to discover
• InstanceSettings (SettingsType):  Discovery Instance Settings

Member Modules:

The first member module is the System.Discovery.Scheduler module, with the configuration:

<Scheduler>
<SimpleReccuringSchedule>
<Interval>$Config/Interval$</Interval>
<SyncTime/>
</SimpleReccuringSchedule>
<ExcludeDates/>
</Scheduler>

This is followed by a System.Discovery.ClassSnapshotDataMappermodule, with the configuration:

<ClassId>$Config/ClassId$</ClassId>
<InstanceSettings>
 $Config/InstanceSettings$
</InstanceSettings>

So this data source accepts the arbitrary Instance Settings and Class Id and maps the inputs to Discovery Data.

Discovery Data Source:  MyApp.Monitoring.DataSource.DiscoverService

This data source uses the MyApp.Monitoring.DataSource.DiscoverObject data source that we just created, but supports overridable inputs for the MinRunning and MaxRunning service class properties, by embedding the $Config/$ variables in the Instance Settings definition.

Configuration Parameters:

• Interval (integer):  Scheduler interval in seconds – overridable
• TargetSystem (string):  UNIX/Linux agent computer to execute the discovery
• AppName (string):   The name of the application object (which is the key property for the hosting class instance)
• ServiceName (string): The name of the service to discover
• MinRunning (integer):  The minimum threshold of running processes expected – overridable
• MaxRunning (integer):  The maximum threshold of running processes expected – overridable

Member Modules

This data source only has one member module:  MyApp.Monitoring.DataSource.DiscoverObject, with the configuration:

<Interval>$Config/Interval$</Interval>
<ClassId>
 $MPElement[Name="MyApp.Monitoring.Service"]$
</ClassId>
<InstanceSettings>
<Settings>
<Setting>
<Name>
 $MPElement[Name='MyApp.Monitoring.MyApp']/Name$
</Name>
<Value>$Config/AppName$</Value>
</Setting> 
<Setting>
<Name>
 $MPElement[Name='MyApp.Monitoring.Service']/Name$
</Name>
<Value>$Config/ServiceName$</Value>
</Setting>
<Setting>
<Name>
 $MPElement[Name='MyApp.Monitoring.Service']/MinRunning$
</Name>
<Value>$Config/MinProcesses$</Value>
</Setting>
<Setting>
<Name>
 $MPElement[Name='MyApp.Monitoring.Service']/MaxRunning$
</Name>
<Value>$Config/MaxProcesses$</Value>
</Setting>
<Setting>
<Name>
 $MPElement[Name='MicrosoftUnixLibrary!
  Microsoft.Unix.Computer']/PrincipalName$
</Name>
<Value>$Config/TargetSystem$</Value>
</Setting>
<Setting>
<Name>
$MPElement[Name='System!System.Entity']/DisplayName$
</Name>
<Value>$Config/ServiceName$</Value>
</Setting>
</Settings>
</InstanceSettings>

Note that the key properties for the hosting class instances (MyApp.Monitoring.MyApp, and Microsoft.Unix.Computer) are included in the Instance Settings so that the relationships can be mapped.  With the data sources in place, the next step is to configure the Discovery Rule (one for each service to discover).

Discovery Rule:  MyApp.Monitoring.Discovery.MyAppDService

This rule discovers the MyAppD daemon, using the data source just created.  It is targeted to instances of the MyApp class, and provides the name of the service and process count thresholds.

Data Source Configuration:

<Interval>28800</Interval>
<TargetSystem>
 $Target/Host/Property[Type="MicrosoftUnixLibrary!
  Microsoft.Unix.Computer"]/PrincipalName$
</TargetSystem>
<AppName>
 $Target/Property[Type="MyApp.Monitoring.MyApp"]/Name$
</AppName>
<ServiceName>myappd</ServiceName>
<MinProcesses>1</MinProcesses>
<MaxProcesses>3</MaxProcesses>

Once the MP is imported and the dicsovery has run, we can see the discovered service in the Discovered Inventory view in the Ops Console:

Building the Monitor Type

In this example, we are going to monitor the process/daemon and generate an alert if the count of running instances is outside of the threshold range.   To do this, we can use the Shell Command monitoring data source to execute a  ps h -e |grep <process name> grep -v grep |wc –l command string.  This calls the ps command, with switches to suppress the header row, and show all processes.   Grep is used to find the process name, and exclude the grep process itself.  Finally, the results are piped to wc –l to return a line count.   The result of this command string is that the running count of processes matching the process name is returned as a numeric value.

Monitor Type: MyApp.Monitoring.MonitorType.ProcessCount

This monitor type will use the MyApp.Monitoring.DataSource.ShellCommandMonitoring data source, to call the ps command string described above.  A set of condition detection modules are used to determine if the process count is below or above the minimum and maximum thresholds.

Configuration Parameters

• Interval (integer):  Scheduler interval in seconds – overridable
• TargetSystem (string):  UNIX/Linux agent computer to monitor
• ServiceName (string): The name of the service to monitor
• MinRunning (integer):  The minimum threshold of running processes expected
• MaxRunning (integer):  The maximum threshold of running processes expected

Health States

• ProcessCountOK
• ProcessCountNotOK

Member Modules

The data source for this monitor type is: MyApp.Monitoring.DataSource.ShellCommandMonitoring, with the configuration:
<Interval>$Config/Interval$</Interval>
<TargetSystem>
 $Config/TargetSystem$
</TargetSystem>
<ShellCommand>
 ps h -e |grep $Config/ProcessName$|grep -v grep |wc –l
</ShellCommand>
<Timeout>120</Timeout>

Two configuration detection modules are required to detect the OK and NotOK states, comparing the returned StdOut to the thresholds:

<ConditionDetection ID="CDProcessCountOK"
  TypeID="System!System.ExpressionFilter">
<Expression>
<And>
<Expression>
<SimpleExpression>
<ValueExpression>
 <XPathQuery Type="Double">
 //*[local-name()="StdOut"]
</XPathQuery>
</ValueExpression>
<Operator>GreaterEqual</Operator>
<ValueExpression>
<Value Type="Double">$Config/MinCount$</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
<Expression>
<SimpleExpression>
<ValueExpression>
<XPathQuery Type="Double">
 //*[local-name()="StdOut"]
</XPathQuery>
</ValueExpression>
<Operator>LessEqual</Operator>
<ValueExpression>
<Value Type="Double">$Config/MaxCount$</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
</And>
</Expression>
</ConditionDetection>

And

<ConditionDetection ID="CDProcessCountNotOK"
  TypeID="System!System.ExpressionFilter">
<Expression>
<Or>
<Expression>
<SimpleExpression>
<ValueExpression>
<XPathQuery Type="Double">
 //*[local-name()="StdOut"]
</XPathQuery>
</ValueExpression>
<Operator>Greater</Operator>
<ValueExpression>
<Value Type="Double">$Config/MaxCount$</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
<Expression>
<SimpleExpression>
<ValueExpression>
<XPathQuery Type="Double">
 //*[local-name()="StdOut"]
</XPathQuery>
</ValueExpression>
<Operator>Less</Operator>
<ValueExpression>
<Value Type="Double">$Config/MinCount$</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
</Or>
</Expression>
</ConditionDetection>

Regular Detections

The regular detections just need to be configured to map the Condition Detection member modules to the defined Health States:

<RegularDetections>
<RegularDetection
  MonitorTypeStateID="ProcessCountOK">
<Node ID="CDProcessCountOK">
<Node ID="DS1"/>
</Node>
</RegularDetection>
<RegularDetection
  MonitorTypeStateID="ProcessCountNotOK">
<Node ID="CDProcessCountNotOK">
<Node ID="DS1"/>
</Node>
</RegularDetection>
</RegularDetections>

In summary, this monitor type will execute our ps command, for a given service name, and compare the count of running/matched process instances  to the defined thresholds (defined during service discovery).

Building the Custom Process Monitor

In this step, we will create a monitor, of the type that we just created: MyApp.Monitoring.MonitorType.ProcessCount

<Configuration>
<Interval>180</Interval>
<TargetSystem>
 $Target/Host/Host/Property[Type="MicrosoftUnixLibrary!
     Microsoft.Unix.Computer"]/NetworkName$
</TargetSystem>
<ProcessName>
 $Target/Property[Type="MyApp.Monitoring.Service"]/Name$
</ProcessName>
<MinCount>
 $Target/Property[Type="MyApp.Monitoring.Service"]/MinRunning$
</MinCount>
<MaxCount>
 $Target/Property[Type="MyApp.Monitoring.Service"]/MaxRunning$
</MaxCount>
</Configuration>

The MyApp Service: $Target/Property[Type="MyApp.Monitoring.Service"]/Name$, currently has too many or too few processes running.

Current process count:  $Data/Context///*[local-name()="StdOut"]$.

Expected range:
$Target/Property[Type="MyApp.Monitoring.Service"]/MinRunning$ – $Target/Property[Type="MyApp.Monitoring.Service"]/MaxRunning$.

Stay tuned for more in this series…

The OpsMgr UNIX and Linux monitoring implementation can be extended through MP authoring to implement robust system and application monitoring for UNIX/Linux servers.   The most direct mechanism of extension comes in the form of the script provider, accessed with WSMan Invoke modules.   The WSMan Invoke modules support three methods of invoking actions:

• ExecuteCommand – execute a command (e.g. a script already on the file system ) and return the results
• ExecuteShellCommand – execute a command through sh (with pipeline support) and return the results
• ExecuteScript  – download and execute an embedded script and return the results

Of these three methods, I prefer to use ExecuteShellCommand in most cases, as it allows for the use of complex one-liner shell commands, embedded in the MP.

In a series of posts, I will describe the creation of an example Management Pack for monitoring an application, featuring dynamic application discovery, discovery of multiple log files, and advanced monitoring implementations.

Example Application Details

The example MP described in these blog posts implements monitoring for a hypothetical application (MyApp).  The application involves a daemon, a set of log files, and application performance counters where the metrics are accessible as the contents of files.

Part I – Discovering an Application

Setting up the MP

I am a big fan of the R2 Authoring Console and will be using it to create this example MP.   The first step then is to create a new MP in the Authoring Console (ID:  MyApp.Monitoring).    Once the MP is created and saved, references are needed.   References I am adding are:

• Microsoft.Unix.Library – contains UNIX/Linux classes and modules
• Microsoft.SystemCenter.DataWarehouse.Library – required for publishing performance data to the DW
• System.Image.Library – contains icon images referenced in class definition

Configuring the Base Composite Modules

Most monitoring and discovery workflows used in this MP example will involve the execution of a shell command through the Microsoft.Unix.WSMan.Invoke.ProbeAction ExecuteShellCommand method.   Therefore, setting up a new composite probe action and a couple of base data sources will make further module implementation much simpler.

Probe Action:  MyApp.Monitoring.ProbeAction.ShellCommand

This probe action simply wraps the Microsoft.Unix.WSMan.Invoke.ProbeAction in a way that makes it easier to use for shell command execution.

Configuration Parameters:

• TargetSystem (string)  – the UNIX/Linux agent computer that will execute the command
• ShellCommand (string) – the shell command to execute
• Timeout (integer) – the command timeout, in seconds

Member Modules:

This probe action will only include one member module:  Microsoft.Unix.WSMan.Invoke.ProbeAction.  The configuration of the module looks like:

<TargetSystem>$Config/TargetSystem$</TargetSystem>
<Uri>

http://schemas.microsoft.com/wbem/wscim/1/cim-schema/2/

 SCX_OperatingSystem?__cimnamespace=root/scx
</Uri>
<Selector/>
<InvokeAction>ExecuteShellCommand</InvokeAction>
<Input>
<p:ExecuteShellCommand_INPUT xmlns:p="http://schemas.
 microsoft.com/wbem/wscim/1/cim-schema/2/SCX_OperatingSystem">
<p:command>$Config/ShellCommand$</p:command>
<p:timeout>$Config/Timeout$</p:timeout>
</p:ExecuteShellCommand_INPUT>
</Input>

Note that the configuration parameters for ShellCommand and Timeout are embedded in the “Input” parameter XML fragment.

Data Source: MyApp.Monitoring.DataSource.ShellCommandMonitoring

The probe action created in the previous step will be used in a number of monitoring workflows, so creating a data source that combines the probe action with a scheduler will make monitoring workflow configuration simpler.

Configuration Parameters:

• Interval (integer) – timer interval in seconds
• TargetSystem (string)  – passed to the probe action module
• ShellCommand (string) – passed to the probe action module
• Timeout (integer) – passed to the probe action module

Member Modules:

The first module in this composite data source is a System.Scheduler module, with a basic configuration:

<Scheduler>
<SimpleReccuringSchedule>
<Interval>$Config/Interval$</Interval>
<SyncTime/>
</SimpleReccuringSchedule>
<ExcludeDates/>
</Scheduler>

This scheduler module is followed by the ShellComand probe action, with the configuration:

<TargetSystem>$Config/TargetSystem$</TargetSystem>
<ShellCommand>$Config/ShellCommand$</ShellCommand>
<Timeout>$Config/Timeout$</Timeout>

Data Source: MyApp.Monitoring.DataSource.ShellCommandDiscovery

This data source is identical to the MyApp.Monitoring.DataSource.ShellCommandMonitoring data source, except it features a System.Discovery.Scheduler scheduler module instead of a System.Scheduler module.  Discovery workflows must use a Discovery scheduler in order to prevent problems with cookdown (i.e. discovery workflows ‘cooking-down’ along with monitoring workflows).   So this data source combines the ShellCommand probe action with a Discovery scheduler.

Additionally, I have added a System.ExpressionFilter module to the ShellCommandDiscovery data source to prevent the discovery workflows from continuing if an error is encountered (either WSMan error or ShellCommand StdErr output).   The ExpressionFilter configuration is:

<Expression>
<And>
<Expression>
<RegExExpression>
<ValueExpression>
 <XPathQuery>//*[local-name()="StdErr"]</XPathQuery>
</ValueExpression>
 <Operator>DoesNotMatchRegularExpression</Operator>
 <Pattern>^.+</Pattern>
 </RegExExpression>
 </Expression>
<Expression>
<Not><Expression><Exists>
<ValueExpression>
 <XPathQuery>WsManData/ErrorCode</XPathQuery>
</ValueExpression>
</Exists> </Expression> </Not>
</Expression>
</And>
</Expression>

Class Definition

With the base composite modules configured, we can move on to the actual application discovery, starting with the definition of the application class.  The Microsoft.Unix.LocalApplication is an ideal base class for the application class, and already has a hosting relationship defined so that Microsoft.Unix.Computer hosts Microsoft.Unix.LocalApplication.

Class Definition:

• ID:  MyApp.Monitoring.MyApp
• Base Class:  Microsoft.Unix.LocalApplication
• Name:  MyApp

Properties:

• Name (String) – Key
• InstallPath (String)

Discovering the Application

When discovering a Windows application in OpsMgr, we would typically use a registry probe.  In UNIX/Linux application discovery, probing the existence or contents of a file is typically the most effective way to identify the instance of an application.   In this example, the MyApp application has a configuration file named: myapp.conf in the installation path of /opt/myapp.  Thus, a shell command:  ls /opt/myapp/myapp.conf | wc –l would return a 1 if the file exists, or 0 if the file doesn’t exist.   We can use this logic in the application discovery.

Discovery Data Source: MyApp.Monitoring.DataSource.MyAppDiscovery

To implement this application discovery, I will create a new data source, building on the ShellCommandDiscovery data source previously created.

Configuration Parameters:

• Interval (integer):  Scheduler interval in seconds
• TargetSystem (string):  UNIX/Linux agent computer to execute the discovery
• AppPath (string):  File system path to the application, overrideable

Member Modules:

The first module in this data source is the ShellCommandDiscovery data source (MyApp.Monitoring.DataSource.ShellCommandDiscovery).   This will be used to query the file system for the application configuration file, to determine if the app is present.  The configuration for this module is:

<Interval>$Config/Interval$</Interval>
<TargetSystem>$Config/TargetSystem$</TargetSystem>
<ShellCommand>
 ls $Config/AppPath$/myapp.conf | wc -l
</ShellCommand>
<Timeout>120</Timeout>

In order to return discovery data from this data source, a Discovery Mapper module is required.   In this case, I will use a System.Discovery.FilteredClassSnapshotDataMapper module to filter the results (only return discovery data if the app is found).   An output of 1 from the shell command indicates that the configuration file was found, while an output of 0 indicates that the configuration file was not found.  The configuration for the FilteredClassSnapShotDataMapperModule would be:

<Expression>
<SimpleExpression>
<ValueExpression>
 <XPathQuery Type="String">
  //*[local-name()="StdOut"]
 </XPathQuery>
</ValueExpression>
<Operator>Equal</Operator>
<ValueExpression>
 <Value Type="String">1</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
<ClassId>
 $MPElement[Name='MyApp.Monitoring.MyApp']$
</ClassId>
<InstanceSettings>
<Settings>
<Setting>
<Name>
$MPElement[Name='MyApp.Monitoring.MyApp']/Name$
</Name>
<Value>MyApp</Value>
</Setting>
<Setting>
<Name>
 $MPElement[Name='MyApp.Monitoring.MyApp']/InstallPath$
</Name>
<Value>$Config/AppPath$</Value>
</Setting>
<Setting>
<Name>
 $MPElement[Name='System!System.Entity']/DisplayName$
</Name>
<Value>MyApp - $Config/TargetSystem$</Value>
</Setting>
<Setting>
<Name>
 $MPElement[Name='MicrosoftUnixLibrary!
 Microsoft.Unix.Computer']/PrincipalName$
</Name>
<Value>$Config/TargetSystem$</Value>
</Setting>
</Settings>
</InstanceSettings>

Note that the Key Property (PrincipalName) for the hosting Microsoft.Unix.Computer object is included in the DiscoveryData.

Discovery Rule:  MyApp.Monitoring.Discovery.MyApp

With the discovery data source implemented, finishing the discovery just requires creation of the discovery rule, which targets Microsoft.Unix.Computer objects.

Data Source Configuration:

• Interval:  14400
• TargetSystem: $Target/Property[Type="MicrosoftUnixLibrary!Microsoft.Unix.Computer"]/PrincipalName$
• AppPath: /opt/myapp

This is a good time to create the MP’s presentation folder and a state view targeting MyApp objects.   Once the discovery has executed, discovered instances of the app can be viewed in the state view or Discovered Inventory view:

Stay tuned for more posts on completing this example Management Pack