Operations Manager Cross-Platform Authoring: Invoke Action Monitor

When monitoring UNIX/Linux servers, command execution or script-based monitors can provide a great deal of flexibility in many health-checking applications.   The Operations Manager 2007 R2 cross-platform agent facilitates the execution of shell command lines or executable binaries and scripts with the Microsoft.Unix.WSMan.Invoke.Probe module.   In this post, I will walk through the use of this module in an example monitoring scenario:  monitoring UNIX/Linux systems for the count of defunct/zombie processes.   The management pack described in this post can be downloaded here.

Background

The Microsoft.Unix.WSMan.Invoke.Probe is a nicely wrapped implementation of the module Microsoft.SystemCenter.WSManagement.TimedInvoker from the Microsoft.SystemCenter.WsManagement.Library management pack.  The Microsoft.Unix.WSMan.Invoke.Probe facilitates the execution of commands or processes on the agent with two Invoke Actions:  ExecuteCommand and ExecuteShellCommand.   The ExecuteCommand Invoke Action executes a script or binary executable (along with command line parameters), whereas the ExecuteShellCommand executes a command string in a shell environment.   While similar, a key functional difference between the two is that the ExecuteShellCommand Invoke Action supports command-line pipe operations while the ExecuteCommand Invoke Action does not.   So, any output filtering with awk, sed, or grep (for example) will require the use of the ExecuteShellCommand Invoke Action.  An example of using the ExecuteCommand Invoke Action in a discovery and monitor can be found in the Cross Platform MP Authoring Guide.   However, one advantage of using ‘one-liner’ commands with the ExecuteShellCommand Invoke Action in monitoring scenarios as opposed to calling local scripts with ExecuteCommand is that the need to distribute and maintain scripts to agents is eliminated and the monitoring script is thus embedded in the MP to be managed centrally. 

As for monitoring of defunct processs count, the UNIX ps command can easily be utilized to identify defunct/zombie processes.  With some output manipulation by grep and awk, the command string can be configured to return just the number of defunct processes to StdOut: ps -eo ‘s’ | grep Z | awk ‘END{print NR}’

Turning this command into a functional monitor then just requires a data source to execute the InvokeAction, a monitor type to define the condition detections and health states, and a unit monitor.

Walk Through

 1:  Create a new management pack in the authoring console.  In this case, I am calling the management pack:  UNIX Custom Monitors with the id: UNIX.Custom.

2: Add management pack references. In this example, the referenced class and data source are both made available through a reference to the Microsoft.Unix.Library, so it is the only additional reference that I have specified.   Additional references could be added to the Microsoft.SystemCenter.DataWarehouse.Library MP if performance data were going to be collected, or specific operating system management packs if those classes were required.

3A:  Create the data source. 

3B: Add data source Configuration Parameters.   The parameters that will be required for the Scheduler and Invoke probe modules by this data source are.

  • Interval – Integer (Overridable)
  • TargetSystem – String
  • InvokeAction – String
  • Selector – String
  • Uri – String
  • Input – String

3C: Create the System.Scheduler data source member module, using the Interval configuration parameter to control the interval.

3D: Create the Invoke Probe member module (module type:  Microsoft.Unix.WSMan.Invoke.Probe).

 

3E:  Add an Expression Filter member module to filter errors.

3F:  Link the Data Source member modules in order.

4A:  Create the Monitor Type (note:  there is no need to worry about the runas account in these workflows, as the Microsoft.Unix.WSMan.Invoke.Probe has the standard Cross Platform RunAs accounts already configured.  There is an additional module that provides the same funtionality but utilizes the Privileged RunAs profile).

 
4B:  Define the Health States
 
4C:  Define the configuration parameters that the monitor will provide to the monitor type (interval, targetsystem, threshold – with interval and threshold defined as overridable).
 
4D:  Create the Data Source member module using the freshly created custom data source:
 
The parameters for the probe action in this case are:
<p:command>ps -eo ‘s’ | grep Z | awk ‘END{print NR}’</p:command>
<p:timeout>25</p:timeout>
</p:ExecuteShellCommand_INPUT>
 
4E:  Add the Expression Filter modules for detecting both healthy and unhealthy conditions.
 
 
A sample WSManDataItem looks like:
<DataItem type=”Microsoft.SystemCenter.WSManagement.WSManData“time=”2010-02-04T11:30:41.1785581-05:00“sourceHealthServiceId=”913F69B3-DD39-1779-6080-9377BD649872“>
<WsManData>
<n1:ExecuteShellCommand_OUTPUT xml:lang=” “>
 <n1:ReturnCode>0</n1:ReturnCode>
 <n1:StdOut>0 </n1:StdOut>
 <n1:StdErr/>
 <n1:ReturnValue>true</n1:ReturnValue>
 </n1:ExecuteShellCommand_OUTPUT>
 </WsManData>
 </DataItem>
As evidenced by this example, the ReturnCode, StdOut, and StdErr from the ExectueShellCommand are all returned in the data item.   In this case, the monitor will evaluate the StdOut data, which can be accessed with the XPathQuery //*[local-name()="StdOut"].     
 
4F:  Configure the Monitor Type regular detection:
 
5A:  Create the Unit Monitor, targeting the Microsoft UNIX Computer class.
 
5B:  Select the monitor type and set the interval, targetsystem, and threshold parameters.
 
5C:  Set the health state severities.
 
5D: Configure the alert parameters.
Note:  in the alert description, the syntax to refernce the Invoke probe StdOut value is:  $Data/Context///*[local-name()="StdOut"]$

 By overriding the default threshold  and setting a threshold of zero, the monitor can easily be tested:

 

That’s all it takes to turn a shell command string into a monitor, and the data source can be reused by multiple monitors and rules.  This is a pretty simple example, but the possibilities to use this method for highly customized cross-platform monitoring are substantial.  
About these ads

About Kristopher Bash
Kris is a Senior Program Manager at Microsoft, working on UNIX and Linux management features in Microsoft System Center. Prior to joining Microsoft, Kris worked in systems management, server administration, and IT operations for nearly 15 years.

9 Responses to Operations Manager Cross-Platform Authoring: Invoke Action Monitor

  1. PaulD says:

    hey, nice article.

    do you have any thoughts how to utilize complex WSMan StdOut output in Scom ? Say, produce 3 perf. counters.

    • Kristopher Bash says:

      I will be posting more on this subject soon, as I am currently working on some SCX management packs relying heavily on processing WSMan StdOut data after it is returned, and I intend to blog about the MP’s. I think the best approach in this case would be to follow your WSMan invoker module with a PowerShell Property Bag probe module. Pass the StdOut data item as a parameter to the PowerShell probe, and then parse the StdOut with the PS script and return it as a property bag. This would allow you to deal with multiple instances of one data item in a discovery scenario (just pipe the StdOut data to a PS foreach-object loop), or handle multiple data values simultaneously (each as a distinct property in the property bag).

  2. PaulD says:

    hmmmm, i forgot about posh modules and tried to accomplish this task using vbs/js modules or even thought about writing our own c# probe module to process wsman output.
    nice idea, need to test it

    • Kristopher Bash says:

      If you need examples, I can provide some.

    • KongYing says:

      hey, I have the same question as you asked. I am wondering whether you finally resolve the problem utilizing complex WSMan StdOut output in Scom ? I am really instrested in this issue. Could you share your ideas with me? Thaks a lot!

  3. PaulD says:

    yes, please, it would be great. you should see my email as blog owner.

  4. AFed says:

    I join question about StdOut to PS script parser

  5. Ryan Taylor says:

    I have attempted to use your guide to create a custom MP for ActiveMQ on a Red Hat Server. The first thing I did was create my script to get the StdOut. I have that script working as such.

    winrm invoke ExecuteShellCommand http://schemas.microsoft.com/wbem/wscim/1/cim-schema/2/SCX_OperatingSystem?__cimnamespace=root/scx @{command=”/opt/progress/fuse-message-broker-5.3.0.3/bin/activemq-admin query -QQueue=SearchIndex.Service1.Request | grep QueueSize”;timeout=”60″} -username:rtaylor -password:Hailey13 -auth:basic -r:https://qtvslax1lmsq001.taservs.net:1270/wsman -skipCACheck -encoding:UTF-8

    With a return of

    ReturnCode = 0
    StdOut = QueueSize = 101

    StdErr
    ReturnValue = true

    PERFECT! I figured authoring this with the Authoring Console was going to be easy. No such luck. I added the reference to the Unix.Library and though the reference ID and token match I cannot move on to the next part of the tutorial. ManagementPack not found in the store is the actual message. Your help would be appreciated!

  6. KongYing says:

    nice article!
    I am working on your tutorial as you provided. I also need some examples to figure out utilizing complex WSMan StdOut output in Scom.Could you provide some to me? Thanks a lot!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 46 other followers

%d bloggers like this: