UNIX/Linux Process Monitoring Template

Operations Manager 2007 R2 included the Unix Service Monitoring template for custom monitoring of daemons on UNIX and Linux agents.   This template has been replaced in the System Center 2012 release of Operations Manager with the far more capable UNIX/Linux Process Monitoring template.   The new UNIX/Linux Process Monitoring template allows more flexibility in process/daemon monitoring, including the ability to monitor for minimum and maximum process count thresholds, and the ability to filter processes on arguments in addition to the process name. For this example, I will walk through the use the UNIX/Linux Process Monitoring template to monitor a Tomcat daemon.

The UNIX/Linux Process Monitoring template is accessible in the Authoring Pane of the Operations Console.   It can be launched with the “Add Monitoring Wizard” task under the Management Pack Templates view.

Input a name, description, and select the target management pack (where the template-created MP elements will be saved).

The next page is the details page, where process information is input.   If you already know the process name, you can just type that in and select the group to target.  Alternatively, you can click Select a Process to connect to an agent and list the currently running processes.

In this case, I will connect to an agent and enumerate the running processes.  The Tomcat daemon is a Java process, so I select the process named: java.

In the bottom field of the details dialog, the running processes on the agent that match the selected name are listed.   This particular agent is also running WebSphere, and its process also has the name: java, which is why two java processes are listed in this field.   As I only want to monitor the Tomcat daemon with this particular monitor, I don’t want the WebSphere process to affect the monitoring.   While the two processes share the same name, they have different process arguments.  The Regular expression to filter process arguments field allows me to input a regular expression that will be evaluated against the process arguments.   Only processes that match the regular expression will be included in this monitor.   So, I can input a regular expression of ^.+tomcat.+ and only processes with a name of “java” that have the string: tomcat in their arguments list will be evaluated by the monitor.   The wizard filters the list of running processes on the fly to show what will be evaluated with a given regular expression.

A few notes on this functionality:

• The regular expression filtering by arguments is an optional function.  If you just want to monitor processes by name, you can leave this field blank.
• The regular expression is evaluated against all arguments as one long string.  That is, the arguments (argv[0] – argv[<max>]) are concatenated with a  space separator, and the regular expression is evaluated against this concatenated string.
• The regular expression filtering is performed on the Windows-side (Health Service), not on the UNIX/Linux agent. Thus, .NET regular expressions are used (this differs from the log file monitoring template where the regular expression matching is performed on the UNIX/Linux agent).
• If you know the regular expression that you want to use, and you are targeting a group, you can just select a group, and type the process name and regular expression into the field without connecting to an agent.   However, if you want to test the regular expression in the wizard, you need to use the Select a process function to connect to an agent and list running processes that match the selected name.   Once you are satisfied with the arguments filter, you can retarget the monitor to a group by clicking Select a group.    The filtering by arguments functionality works whether you are targeting a single agent or a group, but testing it requires connecting to an agent to list the running processes.

On the next page, optional minimum and maximum thresholds can be set for the acceptable count of processes that match the name and optional arguments filter.  For example, to monitor that at least one instance of a process is running, set a minimum threshold of 1, to alert if any instances of a process are running, set a maximum threshold of 0.  To monitor that no less than 5 and no more than 10 instances of a process is running, set a minimum threshold of 5 and a maximum of 10.   Note:  the process count is calculated after the optional arguments filter is applied.   So in this example, the WebSphere “java” process will not be included in the process count, because the ^.+tomcat.+ filter was applied to the arguments.

All done:

When I kill the Tomcat daemon, this is what the alert looks like:

Note:

Despite similar functionality to the Unix Service Template from 2007 R2, the new UNIX/Linux Process Monitoring template is an entirely new template. If you are upgrading from 2007 R2 to 2012, both templates will be available. To make use of the new functionality in the process template, you will need to create new instances to replace the previous instances created by the Unix Service Template.

Performance and Cookdown Considerations

Each unique process name that is specified in a process template will involve one WSMan query to the agent per polling interval.  Multiple template instances that inspect a single process name but implement unique regular expression filtering will cookdown into a single WSMan query to the agent per polling interval. For example, if ten process template instances target a single host and all monitor a process named java with distinct process argument filters (RegExp), only one WSMan query will be sent to the agent per polling interval.  If ten process template instances target a single host and each monitors a unique process name (processA, processB, etc.), ten WSMan queries will be sent to the agent per polling interval.

In cases where a large number of unique process names need to be monitored, process argument filtering is not required, and the monitored agents do not have a large number of running processes (with sizable parameter strings), custom MP authoring can be used to create a cookdown-optimized monitor type that enumerates SCX_UnixProcess with the filter: Select Name, Parameters from SCX_UnixProcess. This would allow all of the process monitors to cookdown to a single WSMan query assuming all other configuration parameters (such as Interval) are aligned.

The vulnerability is described in bulletin MS12-006, and more information can be found in the KB article.  While we continue to investigate options for resolving this issue, there are two viable workarounds (which must be applied to all Mgmt Servers and Gateways that manage UNIX/Linux agents):

1. Uninstall the update KB2585542 
2. Make a registry modification to disable the SecureChannel changes implemented in the update

Note: the registry modification described here and in the KB article effectively disables the security fix that the update implements, so the modified system is subject to the same vulnerability as an unpatched system.

Modifying the registry to disable the SecureChannel changes:

• A “FixIt” package is available in the KB article under the Known Issues section that can be used to disable the security update
• Alternatively, you can add the 32bit DWORD value:
  HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control
  \SecurityProviders\SCHANNEL\
   SendExtraRecord = 2

These changes take effect immediately and do not require a reboot.

UNIX and Linux agent monitoring in Operations Manager requires certificates to secure the SSL communication channel between the Management Servers and agents.  In this post, I will provide some background information on this communication and the certificates, as well as describe an optional approach to replace the default Operations Manager certificate infrastructure with your organization’s Public Key Infrastructure.

The Protocols

The Operations Manager UNIX/Linux agent is a very lightweight agent implementation, comprising a CIM Object Manager (OpenPegasus) and CIM Providers.   Unlike Operations Manager Windows agents, the UNIX/Linux agent doesn’t have a health service, and doesn’t run workflows locally.  Rather, the Management Server (or Gateway) that manages the agent runs the workflows and remotely connects to the UNIX/Linux agent to retrieve current data.  

There are two protocols involved in the communication between the Management Server and the UNIX/Linux agent:  ssh and WS-Management.   

Ssh is used purely for agent maintenance activities, and is not used for any monitoring.   Operations like agent installation, uninstallation, upgrade, or agent daemon restart (through a recovery task) are executed over ssh.    Ssh facilitates the transfer of files and execution of remote commands for these operations when the agent daemon is unavailable.  

WS-Management (or WSMan) is the core protocol used in UNIX/Linux monitoring.   WSMan is a SOAP-based protocol for cross-platform management.   All monitoring operations (e.g. enumerating CIM providers for data on file systems, memory, etc, execution of commands/scripts for monitoring, executing log file reads for monitoring) are implemented over WSMan.   As WSMan is a web service protocol, the OpenPegasus-based CIMOM functions as a secure web server (user credentials are authenticated through PAM).  This is where the agent certificate comes in to play.

The Certificate

The UNIX/Linux agent certificate is quite simply used to secure the WSMan connection using SSL and provide authentication for the remote agent host.   The requirements for this certificate are:

• The certificate is a server authentication certificate (Enhanced Key Usage: 1.3.6.1.5.5.7.3.1)
• The CN of the certificate matches the FQDN that the Management Server uses to connect to the agent
• The certificate is signed by a trusted authority (and can be checked for revocation)

When the Operations Manager UNIX/Linux agent is installed, it generates a certificate (using openssl) at the path:  /etc/opt/microsoft/ssl.  The file name of the certificate is scx-host-<hostname>.pem and the corresponding private key is named scx-key.pem.   The agent actually looks for the certificate at /etc/opt/microsoft/scx/ssl/scx.pem, which is initially configured as a symbolic link pointing to scx-host-<hostname>.pem.

Upon initial agent installation, the certificate is not signed, and is not usable for securing the WSMan SSL communication.

Note:  when initially creating the certificate, the agent attempts to determine the agent hostname for use as the CN value of the certificate.   In cases where the DNS name known to the local host does not match the FQDN that OpsMgr will use to communicate with the agent, additional steps are required to establish a valid certificate.  More information can be found here: http://technet.microsoft.com/en-us/library/dd891009.aspx

Certificates and Management Servers

When a Management Server discovers a UNIX or Linux agent, the server uses its certificate to sign the agent certificate, acting like a standalone Certificate Authority.  In the discovery process, this actually involves securely transferring the certificate from the agent to the Management Server, signing it, copying it back to the agent, and restarting the agent daemon.  

In order to move an agent between Management Servers, the new Management Server must trust the certificate that was used to sign the agent’s certificate.  This becomes particularly important in the 2012 version of Operations Manager, where agents will move automatically between the Management Servers that are members of the Resource Pool managing the agent.  For more information on the procedure to trust a server’s certificate from another server, review this document: http://technet.microsoft.com/en-us/library/hh287152.aspx.

Using a PKI Instead of Management Servers for Signing

Because the certificates used for securing the agent SSL channel are not proprietary, a separate Public Key Infrastructure can be used to manage the agent certificates, if the PKI option is appealing for your organization.  While this requires some additional resources in the environment (a Certificate Authority) and customization, there are a few benefits to using a PKI: 

• Certificate policies are controlled by the PKI and customizable to meet your organization’s security requirements
• Migrations of agents between Management Servers (within or between Resource Pools) can be done without exporting/importing Management Server certificates – simplifying the provisioning of Management Servers.
• More options exist for automation of agent deployment and certificate signing

The procedure to use a PKI instead of Management Server signed certificates varies with different requirements and environments, but I will describe the steps required for one example approach.  This example assumes that the Certificate Authority is a Windows 2008 Certificate Authority. 

Prerequisites:

1. Configure the certificate template on the Certificate Authority – you can use the “Web Server” template or a copy of it – configure options and permissions, publish the template.
2. Import the CA certificate from the signing CA  to the trusted authorities list on every management serverthat will manage the UNIX/Linux agents:
   1. certutil -f -config “<CAHostname>\<CAName>” -ca.cert <CACertFile>
   2. certutil -addstore Root <CACertFile>

Per-Agent steps:

1. Install the agent – this can be done through the OpsMgr Discovery Wizard, manually, or with another package distribution tool.  If you use the OpsMgr Discovery Wizard to install the agent, the agent will generate a certificate that is signed by the management server, but this can be replaced with your PKI CA signed certificate.
2. Generate a cert signing request – either create a new private key with OpenSSL or use the private key generated during the agent install
1. a.      Command to generate a CSR using the key generated during agent install:
   openssl req -new -key /etc/opt/microsoft/scx/ssl/scx-key.pem  -subj /CN=<FQDN of agent host> -text -out <OutputPath>
2. 3.       Copy the CSR back to a Windows machine
3. 4.       Submit the CSR to the CA – this command assumes auto-enrollment is enabled and authorized:
1. certreq.exe -submit -config <CAHostName>\<CAName> -attrib “CertificateTemplate:<TemplateName>” <CSR FileName> <OutputCertName>
2. Copy the signed cert back to the UNIX/Linux agent using  a secure copy method.  If auto-enrollment was used in step 4, the value for <OutputCertName>  specifies the file name of the signed certificate to copy to the agent.
3. Update the symbolic link: /etc/opt/microsoft/scx/ssl/scx.pem to point to your new signed certificate
4. Restart the agent:  /opt/microsoft/scx/bin/tools/scxadmin –restart
5. Discover the agent using the Operations Console or PowerShell Cmdlet

Automation and Customization Opportunities

All of the per-agent steps described above can be executed from a command line, meaning that this procedure can be automated through scripting.  Using a script on a Windows server, the UNIX/Linux commands and file copying actions can be executed with SSH utilities like PuTTY’s plink and pscp.  For really robust automation capabilities, all of the steps can be implemented in a PowerShell script – I like the plink.exe integration example described on this blog: http://www.christowles.com/2011/06/how-to-ssh-from-powershell-using.html.

Aside from the primary benefits of automating these steps in terms of reducing manual interactions, other customization opportunities are exposed with using this scripting approach.  For example, if your DNS infrastructure and UNIX/Linux agent hostnames don’t neatly correlate, you could modify step 2 of the per-agent steps to also generate a new certificate with openssl using the desired FQDN as the certificate’s CN (http://technet.microsoft.com/en-us/library/dd891009.aspx).  Alternatively, if you are using Operations Manager 2007 R2 and want to implement agent deployment and certificate signing using sudo elevation instead of root credentials, the UNIX/Linux host commands in the per-agent steps could be prepended with the sudo command (this functionality is built into the 2012 version of Operations Manager).

Operations Manager 2012 Beta

http://www.microsoft.com/systemcenter/en/us/om-vnext-beta.aspx

Operations Manager 2012 Beta is available.   Exciting new features for UNIX/Linux monitoring include:

• Discovery Wizard:  All new Discovery Wizard, making it easier to deploy and discover UNIX and Linux agents
• Sudo support:  Privileged operations (monitoring and agent maintenance) can now be performed without root privileges by using a non-privileged credential and sudo elevation.
• SSH Key support:  Agent maintenance operations (via SSH) can be authenticated with an SSH key instead of password
• High Availability for UNIX/Linux agent monitoring:   Resource Pools implement highly-available UNIX/Linux monitoring with automatic failover and load distribution.
• RHEL 6 and AIX 7.1 support

Operations Manager 2007 R2 CU5

http://support.microsoft.com/kb/2495674

Operations Manager 2007 R2 Cumulative Update 5 is also now available.   Fixes for UNIX/Linux monitoring include:

• Performance data for LVM managed volumes not available
• Process monitoring does not keep name if run by using symbolic link
• AIX with large number of running processes crashes with bad alloc

CU5 also implements RHEL 6 support for Operations Manager 2007 R2.

In Part III of this series, I walked through creation of data sources, a discovery, and a rule for discovering dynamically-named log files and implementing an alert-generating rule for log file monitoring.  In this post, I will continue to expand this Management Pack to implement performance collection rules, using WSMan Invoke methods to collect numerical performance data from a shell command. 

Using Shell Commands to Collect Performance Data

Whether it is system performance data from the /proc or /sys file systems, or application performance metrics in other locations, performance data for UNIX and Linux systems can often be found in flat files.   In this example Management Pack, I wanted to demonstrate using a WSMan Invoke module with the script provider to gather a numeric value from a file and publish the data as performance data.   In many cases, this would be slightly more complex than is represented in this example (e.g. if the performance metric value should be the delta between data points in the file over time), but this example should provide the framework for using the contents of a file to drive performance collection rules.   The root of these workflows is a shell command using the cat command to parse the file, which could be piped to grep, awk, and sed to filter for specific lines and columns.  

Additionally, if the performance data (e.g. hardware temperature or fan speed, current application user or connection count) that you are looking for is not stored in a file, but available in the output of a utility command, the same method could be used by using the utility command instead of cat.

Collecting Performance Data from a File

In this example, the MyApp application stores three performance metrics in flat files in the subdirectory ./perf.   I have built three rules that cat these files, and map the values to performance data.  The three rules are functionally identical, so I will only describe one of them.

Performance Collection Rule:  MyApp.Monitoring.Rule.CollectMyAppMem

This rule collects the value from the ./perf/mem file in the application directory, which represents the current memory used by the application in KB.   The rule targets the MyApp.Monitoring.MyApp application class.

Rule Data Source:

The rule uses the MyApp.Monitoring.ShellCommandMonitoring data source, described in Post I of this series, with the configuration:

<Interval>300</Interval>
<TargetSystem>
$Target/Host/Property[Type="MicrosoftUnixLibrary!
Microsoft.Unix.Computer"]/NetworkName$</TargetSystem>
<ShellCommand>
cat $Target/Property[Type="MyApp.Monitoring.MyApp"]
  /InstallPath$/perf/mem
</ShellCommand>
<Timeout>120</Timeout>

Notice that the ShellCommand is our cat command:  cat $Target/Property[Type="MyApp.Monitoring.MyApp"]/InstallPath$/perf/mem

Rule Condition Detection:

A System.Performance.DataGenericMapper is used as the condition detection module to map the StdOut to performance data, with the configuration:

<ObjectName>
$Target/Property[Type="MyApp.Monitoring.MyApp"]
  /Name$
</ObjectName>
<CounterName>Memory used (KB)</CounterName>
<InstanceName>Total</InstanceName>
<Value>
  $Data///*[local-name()="StdOut"]$
</Value>

Write Actions:

The rule defines two write actions:  Microsoft.SystemCenter.CollectPerformanceData and Microsoft.SystemCenter.DataWarehouse.PublishPerformanceData to collect the data and publish it to the DW.   These require no configuration.

The end result is that the scheduled rule grabs the value from the text file, maps it to performance data, collects and publishes the performance data to the OM and DW databases.  This mechanism can be used for nearly any numerical performance metric that is accessible (in a timely fashion) from a shell command pipeline or script.

Stay tuned for more in this series….

In Part II of this series, I walked through creation of data sources, a discovery, a monitor type, and a monitor for customized ”Process Count” monitoring for discovered instances of a ”Service” class. In this post, I will continue to build on this example MP to implement dynamic log file discovery and monitoring.

Dynamic Log File Discovery and Monitoring

Log file monitoring of a single known log file can be easily implemented with the Microsoft.Unix.SCXLog modules, but in some cases, the full path to a log file isn’t static.   For example, if an application maintains multiple log files in a directory, the file name portion of the log file path may not be known ahead of time.    To handle this monitoring scenario, we can implement dynamic log file discovery – using a shell command execution, and then pass the full path of the log file to the SCXLog module for standard log file monitoring. This requires a new class instance, a discovery data source, a discovery rule, and a rule that actually implements the log file monitoring.

Defining the Log File Class and Hosting Relationship

Firstly, a new custom class is required to represent the log file objects.   Instances of this class will be discovered by the discovery rule.  

Definition 

• ID:  MyApp.Monitoring.LogFile
• Base Class:  Microsoft.Unix.ApplicationComponent
• Name:  MyApp Log file

Properties

• Name (String)
• Path (String) – Key

The properties for the log file class represent the file name and full path.   The full path is assured to be unique, so I have specified that as the key property of the class.

The log file class needs to be hosted by the MyApp class, to maintain the relationship between the log files and the application.  

Discovery Data Source:  MyApp.Monitoring.DataSource.DiscoverLogFiles

This data source will use the MyApp.Monitoring.DataSource.ShellCommandDiscovery probe action to find files in a given directory that match a pattern.   The output from this command execution will then be passed to a Microsoft.Windows.PowerShellDiscoveryProbe.   The reason that I am using a PowerShellDiscoveryProbe is that the listing of matched files will be returned as a single data item, the StdOut from the command.   Using a PowerShellDiscoveryProbe provides an easy way to split each line from the output and discover an instance per line. 

Configuration Parameters:

• Interval (integer):  Scheduler interval in seconds – overridable
• TargetSystem (string):  UNIX/Linux agent computer to execute the discovery
• Appname (string):   The name of the application object (which is the key property for the hosting class instance)
• LogFileNamePattern (string): The pattern that will be used in the grep operation to identify log files to discovery
• LogFilepath (string):  The path to search for log files at (via an ls command)

Member Modules:

The first member modules is a MyApp.Monitoring.DataSource.ShellCommandDiscovery probe action, that executes the following command:
ls $Config/LogFilepath$ |grep $Config/LogFileNamePattern$.  This simply enumerates the contents of the specified directory path, and pipes the results to grep, to match a specified pattern, which could be a string match or regular expression.

Module Configuration:

<Interval>$Config/Interval$</Interval>
<TargetSystem>$Config/TargetSystem$</TargetSystem>
<ShellCommand>ls $Config/LogFilepath$
   |grep $Config/LogFileNamePattern$</ShellCommand>
<Timeout>120</Timeout>

The output of this shell command then needs to be parsed so that each valid line in the output is discovered as an instance of a log file object.   This is most easily done with a PowerShellDiscoveryProbe:

param ([string]$CmdOutput,[string]$AppName,[string]$LogFilePath, [string] $TargetSystem,[string] $SourceID,[string]$ManagedEntityID)

$api = New-Object -comObject ‘Mom.ScriptAPI’
$discoveryData = $api.CreateDiscoveryData(0, $SourceID, $ManagedEntityID)

if ($CmdOutput -ne $null){
        $CmdOutput = $CmdOutput.Replace([Environment]::newline,” “)
 [array]$arList = $CmdOutput.Split(” “)
 $arList | ForEach-Object{
  [string]$sFile = $_
 if([int]$sFile.Length -ge [int]1){
  $SFilePath = $LogFilePath + “/” + $sFile
  $oInst = $discoveryData.CreateClassInstance(“$MPElement[Name='MyApp.Monitoring.Logfile']$”)
  $oInst.AddProperty(“$MPElement[Name='MyApp.Monitoring.Logfile']/Name$”, $sFile)
  $oInst.AddProperty(“$MPElement[Name='System!System.Entity']/DisplayName$”, $sFile)
  $oInst.AddProperty(“$MPElement[Name='MyApp.Monitoring.Logfile']/Path$”, $sFilePath)
  $oInst.AddProperty(“$MPElement[Name='MyApp.Monitoring.MyApp']/Name$”, $AppName)
  $oInst.AddProperty(“$MPElement[Name='MicrosoftUnixLibrary!Microsoft.Unix.Computer']/PrincipalName$”, $TargetSystem)

  $discoveryData.AddInstance($oInst)
 }     
}
 $discoveryData
}

Remove-variable api
Remove-variable discoveryData

The PowerShell script loads the Mom.ScriptAPI, creates a Discovery Data instance, and then walks through each line of the ouptut.   If the file name is a valid string (not empty), a class instance is created for the MyApp.Monitoring.Logfile class, and the path and file name properties are set.   The PrincipalName property of the Microsoft.Unix.Computer object, and the AppName property of the MyApp.Monitoring.MyApp class ares included in the DiscoveryData, so that the discovery mapping process can map the hosting relationships. 

Parameters are passed from the module configuration to the script using the Parameters XML fragment in the module configuration:

<Parameters>
<Parameter>
 <Name>TargetSystem</Name>
 <Value>$Config/TargetSystem$</Value>
 </Parameter>
<Parameter>
 <Name>AppName</Name>
 <Value>$Config/Appname$</Value>
 </Parameter>
<Parameter>
 <Name>LogFilePath</Name>
 <Value>$Config/LogFilepath$</Value>
 </Parameter>
<Parameter>
 <Name>CmdOutput</Name>
 <Value>
 $Data///*[local-name()="StdOut"]$
</Value>
 </Parameter>
<Parameter>
 <Name>ManagedEntityID</Name>
 <Value>$Target/Id$</Value>
 </Parameter>
<Parameter>
 <Name>SourceID</Name>
 <Value>$MPElement$</Value>
 </Parameter>
 </Parameters>

This data source can then be used to discover log files matching a pattern, in a specified directory.  

Discovery Rule:  MyApp.Monitoring.Discovery.LogFile

This discovery will discover dynamically-named log files, in a specified path, using a regular expression to filter by file name.   It discovers instances of the MyApp.Monitoring.LogFile class, and uses the MyApp.Monitoring.DataSource.DiscoverLogFiles data source.  The discovery targets  MyApp.Monitoring.MyApp

Data Source Configuration:

• <Interval>14400</Interval>
• <TargetSystem>$Target/Host/Property[Type="MicrosoftUnixLibrary!Microsoft.Unix.Computer"]/PrincipalName$</TargetSystem>
• <Appname>$Target/Property[Type="MyApp.Monitoring.MyApp"]/Name$</Appname>
• <LogFileNamePattern>‘^log[0-9]+’</LogFileNamePattern>
• <LogFilepath>$Target/Property[Type="MyApp.Monitoring.MyApp"]/InstallPath$/logs</LogFilepath>

The two parameters to note are the LogFilepath (which is defined as the application path discovered for the MyApp application, appended with “/logs”) and the LogFileNamePattern (which is a regular expression – ‘^log[0-9]+’ – that will match log files named:  logxxx, where xxx is a number).  

Monitoring the Discovered Log Files

Log File Monitoring Rule:   MyApp.Monitoring.Rule.AlertOnLogError

Now that the dynamically-named log files will be discovered, we need a rule to alert when an error is found in one of the logs.   The rule will target all instances of the MyApp.Monitoring.LogFile class, so that when a new log file instance is discovered, it is automatically monitored.  The rule uses the MicrosoftUnixLibrary!Microsoft.Unix.SCXLog.Privileged.Datasource (assuming the log files require privileged credentials to access).

Data source configuration:

• <Host>$Target/Host/Host/Property[Type="MicrosoftUnixLibrary!Microsoft.Unix.Computer"]/NetworkName$</Host>
• <LogFile>$Target/Property[Type="MyApp.Monitoring.Logfile"]/Path$</LogFile> <RegExpFilter>^.*(e|E)rror.*$</RegExpFilter>

The discovered path to the logfile instance is input as the LogFile parameter value, and a Regular Exprssion (^.*(e|E)rror.*$) is defined to match any log entries with the string:  error or Error in the message.  

Condition Detection configuration:

A System!System.Event.GenericDataMapper condition detection is then configured to map the data to EventData, for consumption by OpsMgr.  The configuration of this module is:

• <EventOriginId>$MPElement$</EventOriginId>
• <PublisherId>$MPElement$</PublisherId>
• <PublisherName>MyApp</PublisherName>
• <Channel>Application</Channel>
• <LoggingComputer>$Target/Host/Host/Property[Type="MicrosoftUnixLibrary!Microsoft.Unix.Computer"]/NetworkName$</LoggingComputer>
• <EventNumber>8001</EventNumber>
• <EventCategory>0</EventCategory>
• <EventLevel>1</EventLevel>
• <UserName/>
•  <Params/>
•  </ConditionDetection>

Write Actions:

In this rule, I have configured two write actions, for collecting the event, and generating an alert.  The CollectEvent (SC!Microsoft.SystemCenter.CollectEvent) module requires no additional configuration, and the alert can be configured to provide details about the logged error message:

Stay tuned for more in this series…

Advanced Service Monitoring

1. Define a custom service class, and discover an instance of this class for each service to monitor, configure monitor types and monitors targeting this class
2. Create a monitor for each service to monitor, targeting the custom application class

Both methods are completely viable, and in most cases, it is appropriate to take the simpler approach and target the custom monitors to the application, providing static inputs into the monitor.   There are some cases where discovering a class instance for the service makes sense though.  Facilitating dynamic discovery of services or thresholds (read from a config file), using the service class in a Distributed Application model in OpsMgr, or maintaining logical seperation (in terms of monitoring) between the application and its subsystems are all scenarios that would benefit from discovering the monitored services as class instances.   For the purpose of illustration, I will discover the daemon to monitor in this example Management Pack as a class instance.

Class Definition

Class:  MyApp.Monitoring.Service

Definition

• ID:  MyApp.Monitoring.Service
• Base Class:  Microsoft.Unix.ApplicationComponent
• Name:  MyApp Service

Properties

• Name (String) – Key
• MinRunning (Integer)
• MaxRunning (Integer)

Discovery

Then we can define the data source to discover a service.   In this case, we know the name of the service and the value of the properties, so we don’t need to actually poll the agent to return data.   We can simply combine a Discovery Scheduler with a Discovery Data Mapper module to implement the data source.  However, we want to be able to override the values of MinRunning and MaxRunning, so these will need to be exposed as overridable configuration parameters.

Therefore, I’ve chosen to implement this data source in two parts.   The first data source, will simply combine a System.Discovery.Scheduler module and a System.Discovery.ClassSnapshotDataMapper module.   This data source will accept Interval, ClassId and InstanceSettings parameters as inputs.  The second data source will reference the first data source, but implement parameters for Service Name, MinRunning, and MaxRunning.    By breaking this into two data sources, the first data source can be used for other simple discoveries.

Discovery Data Source:  MyApp.Monitoring.DataSource.DiscoverObject

This is the data source that simply combines a scheduler and a discovery data mapper.  It requires that the MapperSchema be added to the Configuration:

<Configuration>
<IncludeSchemaTypes>
<SchemaType>
 System!System.Discovery.MapperSchema
</SchemaType>
</IncludeSchemaTypes>
…

Configuration Parameters:

• Interval (integer):  Scheduler interval in seconds
• ClassId (string):  ID of the Class to discover
• InstanceSettings (SettingsType):  Discovery Instance Settings

Member Modules:

The first member module is the System.Discovery.Scheduler module, with the configuration:

<Scheduler>
<SimpleReccuringSchedule>
<Interval>$Config/Interval$</Interval>
<SyncTime/>
</SimpleReccuringSchedule>
<ExcludeDates/>
</Scheduler>

This is followed by a System.Discovery.ClassSnapshotDataMappermodule, with the configuration:

<ClassId>$Config/ClassId$</ClassId>
<InstanceSettings>
 $Config/InstanceSettings$
</InstanceSettings>

So this data source accepts the arbitrary Instance Settings and Class Id and maps the inputs to Discovery Data.

Discovery Data Source:  MyApp.Monitoring.DataSource.DiscoverService

This data source uses the MyApp.Monitoring.DataSource.DiscoverObject data source that we just created, but supports overridable inputs for the MinRunning and MaxRunning service class properties, by embedding the $Config/$ variables in the Instance Settings definition.

Configuration Parameters:

• Interval (integer):  Scheduler interval in seconds – overridable
• TargetSystem (string):  UNIX/Linux agent computer to execute the discovery
• AppName (string):   The name of the application object (which is the key property for the hosting class instance)
• ServiceName (string): The name of the service to discover
• MinRunning (integer):  The minimum threshold of running processes expected – overridable
• MaxRunning (integer):  The maximum threshold of running processes expected – overridable

Member Modules

This data source only has one member module:  MyApp.Monitoring.DataSource.DiscoverObject, with the configuration:

<Interval>$Config/Interval$</Interval>
<ClassId>
 $MPElement[Name="MyApp.Monitoring.Service"]$
</ClassId>
<InstanceSettings>
<Settings>
<Setting>
<Name>
 $MPElement[Name='MyApp.Monitoring.MyApp']/Name$
</Name>
<Value>$Config/AppName$</Value>
</Setting> 
<Setting>
<Name>
 $MPElement[Name='MyApp.Monitoring.Service']/Name$
</Name>
<Value>$Config/ServiceName$</Value>
</Setting>
<Setting>
<Name>
 $MPElement[Name='MyApp.Monitoring.Service']/MinRunning$
</Name>
<Value>$Config/MinProcesses$</Value>
</Setting>
<Setting>
<Name>
 $MPElement[Name='MyApp.Monitoring.Service']/MaxRunning$
</Name>
<Value>$Config/MaxProcesses$</Value>
</Setting>
<Setting>
<Name>
 $MPElement[Name='MicrosoftUnixLibrary!
  Microsoft.Unix.Computer']/PrincipalName$
</Name>
<Value>$Config/TargetSystem$</Value>
</Setting>
<Setting>
<Name>
$MPElement[Name='System!System.Entity']/DisplayName$
</Name>
<Value>$Config/ServiceName$</Value>
</Setting>
</Settings>
</InstanceSettings>

Note that the key properties for the hosting class instances (MyApp.Monitoring.MyApp, and Microsoft.Unix.Computer) are included in the Instance Settings so that the relationships can be mapped.  With the data sources in place, the next step is to configure the Discovery Rule (one for each service to discover).

Discovery Rule:  MyApp.Monitoring.Discovery.MyAppDService

This rule discovers the MyAppD daemon, using the data source just created.  It is targeted to instances of the MyApp class, and provides the name of the service and process count thresholds.

Data Source Configuration:

<Interval>28800</Interval>
<TargetSystem>
 $Target/Host/Property[Type="MicrosoftUnixLibrary!
  Microsoft.Unix.Computer"]/PrincipalName$
</TargetSystem>
<AppName>
 $Target/Property[Type="MyApp.Monitoring.MyApp"]/Name$
</AppName>
<ServiceName>myappd</ServiceName>
<MinProcesses>1</MinProcesses>
<MaxProcesses>3</MaxProcesses>

Once the MP is imported and the dicsovery has run, we can see the discovered service in the Discovered Inventory view in the Ops Console:

Building the Monitor Type

In this example, we are going to monitor the process/daemon and generate an alert if the count of running instances is outside of the threshold range.   To do this, we can use the Shell Command monitoring data source to execute a  ps h -e |grep <process name> grep -v grep |wc –l command string.  This calls the ps command, with switches to suppress the header row, and show all processes.   Grep is used to find the process name, and exclude the grep process itself.  Finally, the results are piped to wc –l to return a line count.   The result of this command string is that the running count of processes matching the process name is returned as a numeric value.

Monitor Type: MyApp.Monitoring.MonitorType.ProcessCount

This monitor type will use the MyApp.Monitoring.DataSource.ShellCommandMonitoring data source, to call the ps command string described above.  A set of condition detection modules are used to determine if the process count is below or above the minimum and maximum thresholds.

Configuration Parameters

• Interval (integer):  Scheduler interval in seconds – overridable
• TargetSystem (string):  UNIX/Linux agent computer to monitor
• ServiceName (string): The name of the service to monitor
• MinRunning (integer):  The minimum threshold of running processes expected
• MaxRunning (integer):  The maximum threshold of running processes expected

Health States

• ProcessCountOK
• ProcessCountNotOK

Member Modules

The data source for this monitor type is: MyApp.Monitoring.DataSource.ShellCommandMonitoring, with the configuration:
<Interval>$Config/Interval$</Interval>
<TargetSystem>
 $Config/TargetSystem$
</TargetSystem>
<ShellCommand>
 ps h -e |grep $Config/ProcessName$|grep -v grep |wc –l
</ShellCommand>
<Timeout>120</Timeout>

Two configuration detection modules are required to detect the OK and NotOK states, comparing the returned StdOut to the thresholds:

<ConditionDetection ID="CDProcessCountOK"
  TypeID="System!System.ExpressionFilter">
<Expression>
<And>
<Expression>
<SimpleExpression>
<ValueExpression>
 <XPathQuery Type="Double">
 //*[local-name()="StdOut"]
</XPathQuery>
</ValueExpression>
<Operator>GreaterEqual</Operator>
<ValueExpression>
<Value Type="Double">$Config/MinCount$</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
<Expression>
<SimpleExpression>
<ValueExpression>
<XPathQuery Type="Double">
 //*[local-name()="StdOut"]
</XPathQuery>
</ValueExpression>
<Operator>LessEqual</Operator>
<ValueExpression>
<Value Type="Double">$Config/MaxCount$</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
</And>
</Expression>
</ConditionDetection>

And

<ConditionDetection ID="CDProcessCountNotOK"
  TypeID="System!System.ExpressionFilter">
<Expression>
<Or>
<Expression>
<SimpleExpression>
<ValueExpression>
<XPathQuery Type="Double">
 //*[local-name()="StdOut"]
</XPathQuery>
</ValueExpression>
<Operator>Greater</Operator>
<ValueExpression>
<Value Type="Double">$Config/MaxCount$</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
<Expression>
<SimpleExpression>
<ValueExpression>
<XPathQuery Type="Double">
 //*[local-name()="StdOut"]
</XPathQuery>
</ValueExpression>
<Operator>Less</Operator>
<ValueExpression>
<Value Type="Double">$Config/MinCount$</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
</Or>
</Expression>
</ConditionDetection>

Regular Detections

The regular detections just need to be configured to map the Condition Detection member modules to the defined Health States:

<RegularDetections>
<RegularDetection
  MonitorTypeStateID="ProcessCountOK">
<Node ID="CDProcessCountOK">
<Node ID="DS1"/>
</Node>
</RegularDetection>
<RegularDetection
  MonitorTypeStateID="ProcessCountNotOK">
<Node ID="CDProcessCountNotOK">
<Node ID="DS1"/>
</Node>
</RegularDetection>
</RegularDetections>

In summary, this monitor type will execute our ps command, for a given service name, and compare the count of running/matched process instances  to the defined thresholds (defined during service discovery).

Building the Custom Process Monitor

In this step, we will create a monitor, of the type that we just created: MyApp.Monitoring.MonitorType.ProcessCount

<Configuration>
<Interval>180</Interval>
<TargetSystem>
 $Target/Host/Host/Property[Type="MicrosoftUnixLibrary!
     Microsoft.Unix.Computer"]/NetworkName$
</TargetSystem>
<ProcessName>
 $Target/Property[Type="MyApp.Monitoring.Service"]/Name$
</ProcessName>
<MinCount>
 $Target/Property[Type="MyApp.Monitoring.Service"]/MinRunning$
</MinCount>
<MaxCount>
 $Target/Property[Type="MyApp.Monitoring.Service"]/MaxRunning$
</MaxCount>
</Configuration>

The MyApp Service: $Target/Property[Type="MyApp.Monitoring.Service"]/Name$, currently has too many or too few processes running.

Current process count:  $Data/Context///*[local-name()="StdOut"]$.

Expected range:
$Target/Property[Type="MyApp.Monitoring.Service"]/MinRunning$ – $Target/Property[Type="MyApp.Monitoring.Service"]/MaxRunning$.

Stay tuned for more in this series…

The OpsMgr UNIX and Linux monitoring implementation can be extended through MP authoring to implement robust system and application monitoring for UNIX/Linux servers.   The most direct mechanism of extension comes in the form of the script provider, accessed with WSMan Invoke modules.   The WSMan Invoke modules support three methods of invoking actions:

• ExecuteCommand – execute a command (e.g. a script already on the file system ) and return the results
• ExecuteShellCommand – execute a command through sh (with pipeline support) and return the results
• ExecuteScript  – download and execute an embedded script and return the results

Of these three methods, I prefer to use ExecuteShellCommand in most cases, as it allows for the use of complex one-liner shell commands, embedded in the MP.

In a series of posts, I will describe the creation of an example Management Pack for monitoring an application, featuring dynamic application discovery, discovery of multiple log files, and advanced monitoring implementations.

Example Application Details

The example MP described in these blog posts implements monitoring for a hypothetical application (MyApp).  The application involves a daemon, a set of log files, and application performance counters where the metrics are accessible as the contents of files.

Part I – Discovering an Application

Setting up the MP

I am a big fan of the R2 Authoring Console and will be using it to create this example MP.   The first step then is to create a new MP in the Authoring Console (ID:  MyApp.Monitoring).    Once the MP is created and saved, references are needed.   References I am adding are:

• Microsoft.Unix.Library – contains UNIX/Linux classes and modules
• Microsoft.SystemCenter.DataWarehouse.Library – required for publishing performance data to the DW
• System.Image.Library – contains icon images referenced in class definition

Configuring the Base Composite Modules

Most monitoring and discovery workflows used in this MP example will involve the execution of a shell command through the Microsoft.Unix.WSMan.Invoke.ProbeAction ExecuteShellCommand method.   Therefore, setting up a new composite probe action and a couple of base data sources will make further module implementation much simpler.

Probe Action:  MyApp.Monitoring.ProbeAction.ShellCommand

This probe action simply wraps the Microsoft.Unix.WSMan.Invoke.ProbeAction in a way that makes it easier to use for shell command execution.

Configuration Parameters:

• TargetSystem (string)  – the UNIX/Linux agent computer that will execute the command
• ShellCommand (string) – the shell command to execute
• Timeout (integer) – the command timeout, in seconds

Member Modules:

This probe action will only include one member module:  Microsoft.Unix.WSMan.Invoke.ProbeAction.  The configuration of the module looks like:

<TargetSystem>$Config/TargetSystem$</TargetSystem>
<Uri>

http://schemas.microsoft.com/wbem/wscim/1/cim-schema/2/

 SCX_OperatingSystem?__cimnamespace=root/scx
</Uri>
<Selector/>
<InvokeAction>ExecuteShellCommand</InvokeAction>
<Input>
<p:ExecuteShellCommand_INPUT xmlns:p="http://schemas.
 microsoft.com/wbem/wscim/1/cim-schema/2/SCX_OperatingSystem">
<p:command>$Config/ShellCommand$</p:command>
<p:timeout>$Config/Timeout$</p:timeout>
</p:ExecuteShellCommand_INPUT>
</Input>

Note that the configuration parameters for ShellCommand and Timeout are embedded in the “Input” parameter XML fragment.

Data Source: MyApp.Monitoring.DataSource.ShellCommandMonitoring

The probe action created in the previous step will be used in a number of monitoring workflows, so creating a data source that combines the probe action with a scheduler will make monitoring workflow configuration simpler.

Configuration Parameters:

• Interval (integer) – timer interval in seconds
• TargetSystem (string)  – passed to the probe action module
• ShellCommand (string) – passed to the probe action module
• Timeout (integer) – passed to the probe action module

Member Modules:

The first module in this composite data source is a System.Scheduler module, with a basic configuration:

<Scheduler>
<SimpleReccuringSchedule>
<Interval>$Config/Interval$</Interval>
<SyncTime/>
</SimpleReccuringSchedule>
<ExcludeDates/>
</Scheduler>

This scheduler module is followed by the ShellComand probe action, with the configuration:

<TargetSystem>$Config/TargetSystem$</TargetSystem>
<ShellCommand>$Config/ShellCommand$</ShellCommand>
<Timeout>$Config/Timeout$</Timeout>

Data Source: MyApp.Monitoring.DataSource.ShellCommandDiscovery

This data source is identical to the MyApp.Monitoring.DataSource.ShellCommandMonitoring data source, except it features a System.Discovery.Scheduler scheduler module instead of a System.Scheduler module.  Discovery workflows must use a Discovery scheduler in order to prevent problems with cookdown (i.e. discovery workflows ‘cooking-down’ along with monitoring workflows).   So this data source combines the ShellCommand probe action with a Discovery scheduler.

Additionally, I have added a System.ExpressionFilter module to the ShellCommandDiscovery data source to prevent the discovery workflows from continuing if an error is encountered (either WSMan error or ShellCommand StdErr output).   The ExpressionFilter configuration is:

<Expression>
<And>
<Expression>
<RegExExpression>
<ValueExpression>
 <XPathQuery>//*[local-name()="StdErr"]</XPathQuery>
</ValueExpression>
 <Operator>DoesNotMatchRegularExpression</Operator>
 <Pattern>^.+</Pattern>
 </RegExExpression>
 </Expression>
<Expression>
<Not><Expression><Exists>
<ValueExpression>
 <XPathQuery>WsManData/ErrorCode</XPathQuery>
</ValueExpression>
</Exists> </Expression> </Not>
</Expression>
</And>
</Expression>

Class Definition

With the base composite modules configured, we can move on to the actual application discovery, starting with the definition of the application class.  The Microsoft.Unix.LocalApplication is an ideal base class for the application class, and already has a hosting relationship defined so that Microsoft.Unix.Computer hosts Microsoft.Unix.LocalApplication.

Class Definition:

• ID:  MyApp.Monitoring.MyApp
• Base Class:  Microsoft.Unix.LocalApplication
• Name:  MyApp

Properties:

• Name (String) – Key
• InstallPath (String)

Discovering the Application

When discovering a Windows application in OpsMgr, we would typically use a registry probe.  In UNIX/Linux application discovery, probing the existence or contents of a file is typically the most effective way to identify the instance of an application.   In this example, the MyApp application has a configuration file named: myapp.conf in the installation path of /opt/myapp.  Thus, a shell command:  ls /opt/myapp/myapp.conf | wc –l would return a 1 if the file exists, or 0 if the file doesn’t exist.   We can use this logic in the application discovery.

Discovery Data Source: MyApp.Monitoring.DataSource.MyAppDiscovery

To implement this application discovery, I will create a new data source, building on the ShellCommandDiscovery data source previously created.

Configuration Parameters:

• Interval (integer):  Scheduler interval in seconds
• TargetSystem (string):  UNIX/Linux agent computer to execute the discovery
• AppPath (string):  File system path to the application, overrideable

Member Modules:

The first module in this data source is the ShellCommandDiscovery data source (MyApp.Monitoring.DataSource.ShellCommandDiscovery).   This will be used to query the file system for the application configuration file, to determine if the app is present.  The configuration for this module is:

<Interval>$Config/Interval$</Interval>
<TargetSystem>$Config/TargetSystem$</TargetSystem>
<ShellCommand>
 ls $Config/AppPath$/myapp.conf | wc -l
</ShellCommand>
<Timeout>120</Timeout>

In order to return discovery data from this data source, a Discovery Mapper module is required.   In this case, I will use a System.Discovery.FilteredClassSnapshotDataMapper module to filter the results (only return discovery data if the app is found).   An output of 1 from the shell command indicates that the configuration file was found, while an output of 0 indicates that the configuration file was not found.  The configuration for the FilteredClassSnapShotDataMapperModule would be:

<Expression>
<SimpleExpression>
<ValueExpression>
 <XPathQuery Type="String">
  //*[local-name()="StdOut"]
 </XPathQuery>
</ValueExpression>
<Operator>Equal</Operator>
<ValueExpression>
 <Value Type="String">1</Value>
</ValueExpression>
</SimpleExpression>
</Expression>
<ClassId>
 $MPElement[Name='MyApp.Monitoring.MyApp']$
</ClassId>
<InstanceSettings>
<Settings>
<Setting>
<Name>
$MPElement[Name='MyApp.Monitoring.MyApp']/Name$
</Name>
<Value>MyApp</Value>
</Setting>
<Setting>
<Name>
 $MPElement[Name='MyApp.Monitoring.MyApp']/InstallPath$
</Name>
<Value>$Config/AppPath$</Value>
</Setting>
<Setting>
<Name>
 $MPElement[Name='System!System.Entity']/DisplayName$
</Name>
<Value>MyApp - $Config/TargetSystem$</Value>
</Setting>
<Setting>
<Name>
 $MPElement[Name='MicrosoftUnixLibrary!
 Microsoft.Unix.Computer']/PrincipalName$
</Name>
<Value>$Config/TargetSystem$</Value>
</Setting>
</Settings>
</InstanceSettings>

Note that the Key Property (PrincipalName) for the hosting Microsoft.Unix.Computer object is included in the DiscoveryData.

Discovery Rule:  MyApp.Monitoring.Discovery.MyApp

With the discovery data source implemented, finishing the discovery just requires creation of the discovery rule, which targets Microsoft.Unix.Computer objects.

Data Source Configuration:

• Interval:  14400
• TargetSystem: $Target/Property[Type="MicrosoftUnixLibrary!Microsoft.Unix.Computer"]/PrincipalName$
• AppPath: /opt/myapp

This is a good time to create the MP’s presentation folder and a state view targeting MyApp objects.   Once the discovery has executed, discovered instances of the app can be viewed in the state view or Discovered Inventory view:

Stay tuned for more posts on completing this example Management Pack

Additionally, I am excited to announce some new developments regarding the xSNMP and Manage-X for Oracle Management Pack projects.   After developing these Management Pack projects as a solo effort for some time, these projects are now being converted to community-driven projects.  Pete Zerger, of SystemCenterCentral.com, has assembled a fantastic team of MP authoring experts, with a remarkable depth of OpsMgr knowledge,  to drive continued development of these management packs.   

The xSNMP and Manage-X for Oracle MP projects are now homed at:  http://xsnmp.codeplex.com.   General download, source code, documentation, and discussion can be found at this CodePlex site.

• xSNMP for APC Reports
• xSNMP for APC NetBotz Reports
• xSNMP for Brocade Reports
• xsNMP for Check Point Secure Platform Reports
• xSNMP for Cisco Reports
• xSNMP for Data Domain Reports
• xSNMP for Dell PowerEdge Reports
• xSNMP for HP ProCurve Reports
• xSNMP for IBM AIX Reports
• xSNMP for Juniper Networks Reports
• xSNMP for Juniper-NetScreen Reports
• xSNMP for NetApp Reports
• xSNMP for Net-SNMP Reports
• xSNMP for SonicWALL Reports

Requirements are:   OpsMgr 2007 R2, xSNMP suite, OpsMgr reporting implementation.   Like the xSNMP suite, the reports MP’s are licensed with the GNU-GPL and unsealed versions are provided.