OpsMgr 2012 UNIX/Linux Authoring Templates: More Information on the Shell Command Templates

In my last post, I described the new UNIX/Linux Shell Command templates in OpsMgr 2012, and provided a walkthrough on creating a Management Pack for monitoring BIND DNS servers on Linux. In this post, I will provide more detail on the nuts and bolts of the Shell Command templates, as well as a list of tips & tricks to consider when using the templates.

Using the Shell Command Templates

• Targeting:  The only acceptable targets for the UNIX/Linux Shell Command templates are UNIX/Linux Computer, or classes based on it.  So UNIX/Linux Computer, Linux Computer, Red Hat Enterprise Linux Computer, and Red Hat Enterprise Linux 5 Computer (for example) are all valid targets. Operating System, Logical Disk, Local Application, and Computer Role types are not valid targets.  Best practice for rules and monitors is to target the computer object but disable the rule/monitor by default, and use an override to enable it for a group. The Shell Command templates do not support object discovery. If you want to discover and target custom class types (like a UNIX/Linux Computer Role or Local Application), traditional MP Authoring (XML) can be used.
• Shell Command Input:  As the templates use the ExecuteShellCommand method of the agent’s script provider with a WSMan Invoke module, the command is sent to the agent on each execution. This requires that the shell command is a single line. Simple shell scripts can be condensed into a “one-liner” by replacing line breaks with semi-colons. If you want to use a complex or long script, the best approach is to distribute it as a shell script to a common path on monitored agents, then use the shell command templates to invoke the script by providing the path, such as:  /opt/scripts/myscript.sh
• Synchronous Execution & Command Performance:  The agent’s script provider executes scripts synchronously. For optimal server and agent performance and monitor reliability, it is important to make sure that shell commands execute quickly. If you want to use a long-running command or script in a monitor or rule, it is advisable to configure the script to execute periodically on the agent (e.g. with cron) outputting to a file, and then use the shell command monitor or rule to parse the output file. Unless all parameters for the rule or monitor (e.g. scheduler, shell command, timeout, etc) are identical, rules and monitors created by the Shell Command templates will not cookdown.
• Handling Complex Output:  The agent’s script provider returns StdOut, StdErr, and ReturnCode for the shell command execution. Using the templates, it is not possible to post-process this output to split the output into multiple data items, so it is best to consider StdOut from the shell command as representative of one value.  You can always call the same shell command and use the pipeline to parse with grep, awk, sed, etc to filter the output to a single output value. If your shell command returns structured data representing multiple data points, you still have the option of authoring outside of the Operations Console and using a PowerShell script to parse the output.
• Mind the Quotes and Special Characters:  The agent’s script provider executes the shell command using the default shell (for the user context of the specified Run as Account), and wraps the actual command in quotes.  So the command executed on the agent host will be something like: sh –c “my shell command” or bash –c “my shell command.” This means that double-quotes inside the shell command need special handling. If you can’t use single-quotes instead of double-quotes, escape the double-quotes with a backslash. As Management Packs are XML-based, some characters like < and /> can be problematic. It is not possible to embed CDATA blocks in the template wizards, so if you can’t avoid use of < and /> in the command and they prove to be problematic, it is advisable to distribute the script to a common file system location on the monitored agents and use the Shell Command templates to invoke the script on the file system.
• sudo Eevation:  If the Run As account specified via the selected Run As profile in the Shell Command templates is configured for sudo elevation, sudoers will need to be configured to authorize the user to execute the command.  For example, adding the following line through visudo will allow the user monuser to execute the command service named restart as input in the Shell Command task template (assuming the default shell of /bin/sh):
  monuser ALL=(root) NOPASSWD: /bin/sh -c service named restart
• Expression Filters
  Valid values for Parameter Names in the template Expression Filters are:
  • //*[local-name()=”StdErr”]  for StdErr
  • //*[local-name()=”StdOut”] for StdOut
  • //*[local-name()=”ReturnCode”]for ReturnCodeThe Expression Filters in the templates are pre-configured for common scenarios. However, some commands used in monitoring will return values to evaluate on StdOut while other cases will mandate evaluation of StdErr.  For example, if you wanted to run a command to check a connection to a MySQL server, any errors would be output to StdErr. Thusly, the monitor’s expression filter would need to look for the presence of the string ERROR in the StdErr output.Additionally, the default Expression Filters evaluate that ReturnCode = 0 for rules and monitors. This default behavior drops the workflow if the shell command itself fails.  However, if you want to generate an alert if a command fails, the ReturnCode may be 1 in the error state. In this case, the Expression Filter could be configured to go unhealthy when ReturnCode = 1, or StdErr could be evaluated for a string, and the ReturnCode evaluation removed.As in all OpsMgr templates, the Expression Filter dialogs assume values are strings, which can be problematic when evaluating numeric data.  Options for addressing this issue are
  • Export the MP after creating the rule or monitor, and edit the Type attribute in the Expression XM
  • Use Regular Expressions in the Expression Filter instead of greater than/less than evaluations
  • Perform numeric evaluation operations on the agent in the Shell Command and output a simple health state (OK or ERROR) from the shell command itself.And lastly, be mindful of the logical outcome of Expression Filters. For monitors, any potential value from the shell script should positively evaluate to one of the health states (Error or Healthy for two state monitors, or Error, Warning, or Healthy for three state monitors). Particularly, be mindful of AND and OR groupings on Error and Healthy states.  For example, a monitor that requires condition 1 AND condition 2 to be true to be in an Error state should require that only condition 1 OR condition 2 not be true to be Healthy.

Tips, Tricks, and Best Practices

• When creating shell command rules and monitors, test out the commands in a task first!  Shell Command tasks are very simple to create and provide immediate feedback.  Syntactical or authorization issues will be most quickly identified through this method.
• Target “Computer” classes, but disable the rules and monitors by default.  Override the rule and monitor for a group to enable it.
• Only use short-running shell commands in rules and monitors
  • Long running commands or scripts can be scheduled with cron and output to file, using the Shell Command templates to parse the output file
• Avoid double quotes as well as < and /> characters if possible.
  • Complex scripts can be deployed to the file system of the monitored agents and invoked by the Shell Command templates
• For optimal performance, use the largest schedule interval that makes sense (15 minutes is preferred for most cases)
• Optimize the shell commands (using pipeline formatting with grep, awk, sed, etc)  to return a single, easy to parse value
  • For monitors, returning values like: OK, WARNING, ERROR are best
  • For performance rules, the value evaluated must be a single numeric value
• To troubleshoot sudo issues, tail /var/log/messages and look for the exact command that failed – to cross reference with sudoers configuration.
• Be aware what the shell command returns for StdOut, StdErr, and ReturnCode in healthy and unhealthy scenarios. This is critical information when configuring the Expression Filters.
• Have fun!