OpsMgr: R2 PowerShell Modules Example – DNS Resolution Monitor

While I’m finalizing some work on the Oracle SCX management pack,  I wanted to interrupt that series of posts to make a post walking through the use of one of the OpsMgr R2 PowerShell modules –something that I have been meaning to write up for some time.

The addition of native PowerShell modules was one of the major improvements in OpsMgr R2, and these modules collectively introduce an incredible degree of flexibility and improved efficiency in highly customized script-based monitoring.  There are a total of six PowerShell modules available in R2, and in this post, I will walkthrough an example using the Microsoft.Windows.PowerShellPropertyBagProbe module.   The monitor created in this example is a simple DNS resolution check that can be deployed to Windows agent-managed systems for perspective monitoring of DNS client resolution.

The example management pack can be downloaded here.

Management Pack, Class, and Discovery

Firstly, to build this monitor, a new management pack named DNSClientMonitoring is created. 

Read more of this post

Building an Oracle Management Pack for OpsMgr Cross-Platform Agents, Part 3

Continuing on with the development of a management pack for monitoring Oracle database servers on a UNIX/Linux platform (part 1, part 2), I will describe in this post some of the characteristics and methodology of the class structure and object discoveries. 

Class Structure

To start the class structure, I defined a few arbitrary abstract classes to function as the base classes for Oracle objects:  OracleSCX.OracleSubsystem, OracleSCX.OracleApplication, and OracleSCX.OracleComponent.  Subsequently, I created a set of classes to represent the Oracle objects and components that would be discovered.  This class hierarchy is best illustrated with this image:

As for the object relationships, the discovered objects are organized as follows:

  • Oracle SCX Server
    • Oracle SCX Listeners      
      • Oracle SCX Listener
    • Oracle SCX Instance
      • Oracle SCX TableSpaces
        • Oracle SCX TableSpace
          • Oracle SCX Data File
      • Oracle SCX Processes
        • Oracle SCX Process
      • Oracle SCX Databases
        • Oracle SCX Database
      • Oracle SCX Alert Logs

A diagram view of a discovered Oracle system illustrates these relationships:

Secure References

Read more of this post

Building an Oracle Management Pack for OpsMgr Cross-Platform Agents, Part 2

In the first of my posts regarding development of an UNIX/Linux Cross-Platform management pack for monitoring of Oracle, I described the creation of two custom probe action modules for WSMan Invoke Action and Shell Command Execution operations.  In this post, I will describe the creation of a third custom Probe Action for executing SQL queries locally with sqlplus.  It was a bit of a challenge to get this probe action functioning the way I had intended, but after some trial and error, it does seem to work quite nicely.  The SQL Query probe action is really  just a convenient wrapper for the previously discussed ShellCommand probe action. 

 It accepts as configuration parameters:

Read more of this post

xSNMP Management Pack Suite Version 1.1.0 (Beta)

After weeks of pilot testing, the xSNMP Suite version 1.1.0 is ready for public beta release.  In addition to some bug fixes and feature enhancements, this release includes four completely new management packs:

  • xSNMP Juniper-NetScreen MP:   Implements comprehensive monitoring of NetScreen firewall devices
  • xSNMP NetApp MP:  Implements comprehensive monitoring of NetApp storage devices
  • xSNMP Sun Hardware MP: Implements monitoring of Sun Server hardware, through the SunFire Management Agent, Sun ILOM out-of-band management interface, or the Fujitsu XSCF out-of-band management interface
  • xSNMP Dell PowerEdge MP:  Implements monitoring of hardware for Dell PowerEdge devices via the Dell OpenManage agent or DRAC out-of-band management interface

These management packs are in addition to the other 11 management packs previously included in the suite:

  • xSNMP Management Pack – Implements filtered discovery and monitoring of SNMP devices and interfaces that support the standard RFC1213 MIB, IF-MIB, and EtherLike-MIB.  This management pack is the core of the xSNMP Suite and contains public datasources that are utilized in the other optional management packs.
  • xSNMP Overrides Management Pack – This unsealed management pack can be used as a container for overrides, but also provides preconfigured groups and overrides for easily controlling interface monitoring through groups of network interfaces.
  • xSNMP APC Managment Pack – Implements monitoring for APC Rackmount PDU, UPS, Automatic Transfer Switch, and Environmental Monitor devices.
  • xSNMP Brocade Management Pack – Implements chassis monitoring for Brocade Fibre-Channel switch devices (Fibre-Channel ports are monitored as network interfaces with the xSNMP MP).
  • xSNMP Check Point Secure Platform Management Pack – Implements module health and firewall HA failover monitoring for Check Point Secure Platform firewall devices.
  • xSNMP Cisco Management Pack – Implements additional monitoring for Cisco devices, primarily including chassis hardware moniotring for devices that support the EnvMon MIB, Entity-MB, or Cisco-Stack MIB.
  • xSNMP Data Domain Management Pack – Implements monitoring for the performance, hardware status, and replication status of Data Domain Restorer storage appliances.
  • xSNMP HP ProCurve Management Pack – Implements component health monitoring for HP ProCurve switches and wireless access points.
  • xSNMP HP Proliant Management Pack – Implements hardware health monitoring for SNMP-enabled HP servers that support the Proliant Insight Management Agents.
  • xSNMP Net-SNMP Management Pack  – Implements operating system monitoring for Net-SNMP agent devices, such as UNIX/Linux servers through the UCD and Host-Resources MIBs. 
  • xSNMP Syslog Management Pack – Provides  warning and critical alert generating rules that can be enabled and filtered with overrides to alert on incoming syslog messages from discovered SNMP devices.

Behind the scenes, significant improvements have been made in the class structure and shared data source configuration in the root xSNMP MP.  These changes facilitate better organization and easier future development of dependent management packs.  These changes are probably best illustrated with a screenshot of the improved xSNMP base class hierarchy:

I greatly appreciate the testing carried out by the volunteer pilot testers, as well as all of the feedback from those who have tested the xSNMP MP’s, but I specifically want to thank Chris L. for all of his assistance in every stage of the development of the NetScreen MP, as well as Raphael Burri for all of his help in designing the organizational improvements in the xSNMP MP.  

The updated xSNMP Suite can be downloaded here.

Note:  One impact of the reorganization in the xSNMP MP is that direct upgrade from previous versions is not possible and the xSNMP MP’s must be removed prior to importing the new version.   To assist in this regard, I have provided a PowerShell script which can be used to automate the upgrade process and preserve configured overrides (in most cases).   The prerequisites and procedures for upgrading are covered in detail in the Upgrading_xSNMP_To_1.1.0 document that is included in the download package.

Building an Oracle Management Pack for OpsMgr Cross-Platform Agents, Part 1

Over a series of future posts, I intend to describe my efforts to build an OpsMgr SCX Management Pack for monitoring of Oracle running on UNIX/Linux platforms.   In this initial post, I will describe the first step in building this MP:  creating a custom shell command invoker probe action.  As I had described in this previous post, the Microsoft.Unix.WSMan.Invoke.Probe in the Microsoft Unix Library MP provides a wrapped version of the Microsoft.SystemCenter.WSManagement.Invoker.   However, I wanted a bit more granular control of the invoke action (e.g. modifying the command timeout) and felt it appropriate to create a custom Probe Action to wrap the Microsoft.SystemCenter.WSManagement.Invoker module and decided to create a similar, but custom probe action.

The creation of such a Probe Action is fairly straight-foward:

Read more of this post

Operations Manager Cross-Platform Authoring: Invoke Action Monitor

When monitoring UNIX/Linux servers, command execution or script-based monitors can provide a great deal of flexibility in many health-checking applications.   The Operations Manager 2007 R2 cross-platform agent facilitates the execution of shell command lines or executable binaries and scripts with the Microsoft.Unix.WSMan.Invoke.Probe module.   In this post, I will walk through the use of this module in an example monitoring scenario:  monitoring UNIX/Linux systems for the count of defunct/zombie processes.   The management pack described in this post can be downloaded here.

Background

The Microsoft.Unix.WSMan.Invoke.Probe is a nicely wrapped implementation of the module Microsoft.SystemCenter.WSManagement.TimedInvoker from the Microsoft.SystemCenter.WsManagement.Library management pack.  The Microsoft.Unix.WSMan.Invoke.Probe facilitates the execution of commands or processes on the agent with two Invoke Actions:  ExecuteCommand and ExecuteShellCommand.   The ExecuteCommand Invoke Action executes a script or binary executable (along with command line parameters), whereas the ExecuteShellCommand executes a command string in a shell environment.   While similar, a key functional difference between the two is that the ExecuteShellCommand Invoke Action supports command-line pipe operations while the ExecuteCommand Invoke Action does not.   So, any output filtering with awk, sed, or grep (for example) will require the use of the ExecuteShellCommand Invoke Action.  An example of using the ExecuteCommand Invoke Action in a discovery and monitor can be found in the Cross Platform MP Authoring Guide.   However, one advantage of using ‘one-liner’ commands with the ExecuteShellCommand Invoke Action in monitoring scenarios as opposed to calling local scripts with ExecuteCommand is that the need to distribute and maintain scripts to agents is eliminated and the monitoring script is thus embedded in the MP to be managed centrally. 

As for monitoring of defunct processs count, the UNIX ps command can easily be utilized to identify defunct/zombie processes.  With some output manipulation by grep and awk, the command string can be configured to return just the number of defunct processes to StdOut: ps -eo ‘s’ | grep Z | awk ‘END{print NR}’

Turning this command into a functional monitor then just requires a data source to execute the InvokeAction, a monitor type to define the condition detections and health states, and a unit monitor.

Walk Through

Read more of this post

xSNMP Management Packs – Beta Version 1.0.8 Release

After many weeks of development efforts and testing, I have made the latest beta version of the xSNMP Management Packs available for download.   Before getting into any more detail, I would be remiss if I did not acknowledge the invaluable help provided by some of the volunteers who tested these management packs through all stages of development.  Many thanks in particular to Chris and Davey, who played a huge role in every stage of the development of the xSNMP MP’s.   Many thanks also to Gary and Björn for their great help in testing the MP’s.  

The included documentation covers recommendations for deployment and configuration as well as the details of the management packs.  Additional information about performance considerations in large SNMP monitoring environments can be found in this previous post.   At present, the following management packs are included in this suite, and more are currently in the works.

  • xSNMP Management Pack – Implements filtered discovery and monitoring of SNMP devices and interfaces that support the standard RFC1213 MIB, IF-MIB, and EtherLike-MIB.  This management pack is the core of the xSNMP Suite and contains public datasources that are utilized in the other optional management packs.
  • xSNMP Overrides Management Pack – This unsealed management pack can be used as a container for overrides, but also provides preconfigured groups and overrides for easily controlling interface monitoring through groups of network interfaces.
  • xSNMP APC Managment Pack – Implements monitoring for APC Rackmount PDU, UPS, Automatic Transfer Switch, and Environmental Monitor devices.
  • xSNMP Brocade Management Pack – Implements chassis monitoring for Brocade Fibre-Channel switch devices (Fibre-Channel ports are monitored as network interfaces with the xSNMP MP).
  • xSNMP Check Point Secure Platform Management Pack – Implements module health and firewall HA failover monitoring for Check Point Secure Platform firewall devices.
  • xSNMP Cisco Management Pack – Implements additional monitoring for Cisco devices, primarily including chassis hardware moniotring for devices that support the EnvMon MIB, Entity-MIB, or Cisco-Stack MIB.
  • xSNMP Data Domain Management Pack – Implements monitoring for the performance, hardware status, and replication status of Data Domain Restorer storage appliances.
  • xSNMP HP ProCurve Management Pack – Implements component health monitoring for HP ProCurve switches and wireless access points.
  • xSNMP HP Proliant Management Pack – Implements hardware health monitoring for SNMP-enabled HP servers that support the Proliant Insight Management Agents.
  • xSNMP Net-SNMP Management Pack  – Implements operating system monitoring for Net-SNMP agent devices, such as UNIX/Linux servers through the UCD and Host-Resources MIBs. 
  • xSNMP Syslog Management Pack – Provides  warning and critical alert generating rules that can be enabled and filtered with overrides to alert on incoming syslog messages from discovered SNMP devices.

Feedback is, of course, welcomed.

Some Screenshots of the xSNMP Management Packs

Diagram view of an HP ProCurve device:

Diagram view of a Data Domain Restorer:

Health Explorer view for an APC UPS:

Diagram View for an HP Proliant server:

Performance View for a network interface:

Diagram view for a Brocade Fibre Channel Switch:

Scalability and Performance (Design and Testing) in the xSNMP Management Packs

In this post, I intend to describe some of the challenges in scaling SNMP monitoring in an Operations Manager environment to a large number of monitored objects, as well as my experiences from testing and the approaches that I took to address these challenges with the xSNMP Management Packs.

Background

In spite of the market availability of many task-specific SNMP monitoring applications boasting rich feature sets, I think that a strong case can be made for the use of System Center Operations Manager in this SNMP monitoring role. Using a single product for systems and infrastructure (SNMP) monitoring facilitates unparalleled monitoring integration (e.g. including critical network devices/interfaces or appliances in Distributed Application Models for vital business functions). The rich MP authoring implementation, dynamic discovery capabilities, and object-oriented modeling approach allow a level of depth and flexibility in SNMP monitoring not often found in pure SNMP monitoring tools.

However, Operations Manager is first and foremost a distributed monitoring application, most often depending on agents to run small workloads independently. Inevitably, running centralized monitoring workloads (i.e. SNMP polls) in a distributed monitoring application is going to carry a higher performance load than the same workloads in a task-specific centralized monitoring application that was built from the ground up to handle a very high number of concurrent polls with maximum efficiency. This centralized architecture would likely feature a single scheduler process that distributes execution of polls in an optimized fashion as well as common polling functions implemented in streamlined managed code. With SNMP monitoring in Operations Manager, any optimization of workload scheduling and code optimization more or less falls to the MP author to implement.

While working on the xSNMP Management Packs, I spent a lot of time testing different approaches to maximize efficiency (and thus scalability) in a centralized SNMP monitoring scenario. I’m sure there is always room for continual improvement, but I will try to highlight some of the key points of my experiences in this pursuits.

Designing for Cookdown

Cookdown is one of the most important concepts in MP authoring when considering the performance impact of workflows. A great summary of OpsMgr cookdown can be found here. In effect, the cookdown process looks for modules with identical configurations (including input parameters) and replaces the multiple executions of redundant modules with a single execution. So, if one wanted to monitor and collect historical data on the inbound and outbound percent utilization and Mb/s throughput of an SNMP network interface, a scheduler and SNMP Probe (with VarBinds defined to retrieve the in and out octets counters for the interface) could be configured. As long as each of the rules and monitors provided the same input parameters to these modules for each interface, the scheduler and SNMP probe would only execute once per interval per interface. Taking this a step further, the SNMP probe could be configured to gather all SNMP values for objects to monitor in the IFTable for this interface (e.g. Admin Status, Oper Status, In Errors, Out Errors), and these values could be used in even more rules and monitors. The one big catch here is that the SNMP Probe module stops processing SNMP VarBind’s once it hits an error. So, it’s typically not a good idea to mix SNMP VarBinds for objects that may not be implemented on some agents with OIDS that would be implemented on all agents.

Workflow Scheduling

Read more of this post

Introducing the xSNMP Management Pack Suite

Introduction

Over the past several weeks, I’ve been hard at work on some new SNMP management packs for Operations Manager 2007 R2, to replace the Cisco SNMP MP and extend similar functionality to a wide range of SNMP-enabled devices.   In the next few posts, I hope to describe some of the design and development considerations related to these Management Packs, which I am calling the xSNMP Management Pack Suite.   For this post, I hope to give a basic overview of the development effort and resulting management packs.

As I was working on some feature enhancements to the Cisco SNMP Management Pack, and following some really great discussions with others on potential improvements,  I concluded that a more efficient and effective design could be realized by aligning the management pack structure along the lines of the SNMP standard itself.   To expound on this point, much of the monitoring in the Cisco MP is not specific to Cisco devices, but rather, mostly common to all SNMP devices.   The SNMP standard defines a hierarchical set of standard MIBs, and a hierarchical implementation of vendor-specific MIBS, with consideration to the elimination of  redundancy.   I tried to loosely adapt this model in the xSNMP MP architecture.   The first of the MP’s, and the one that all of the others depend on, is the root xSNMP Management Pack.   This management pack has a few functions:

  1. It performs the base discovery of SNMP devices (the discovery is targeted to the  SNMP Network Device class)
  2. It implements monitoring of the SNMP v1/v2 objects for discovered devices and interfaces
  3.  It provides a set of standardized and reusable data sources for use in dependent management packs

From there, the remaining management packs implement vendor-specific monitoring.   Devices and/or interfaces are discovered for the vendor-specific management packs as derived objects from the xSNMP MP, and most of the discoveries, monitors, and rules utilize the common data sources from the xSNMP MP, which makes the initial and ongoing development for vendor-specific MP’s much more efficient.

Controlling Interface Monitoring

One of the topics frequently commented on with the Cisco SNMP Management Pack, and a subject of much deliberation, was that of selecting network interfaces for monitoring.   Even determining the optimal default interface monitoring behavior (disabled vs. enabled) isn’t a terribly easy decision.  For example, a core network switch in a datacenter may require that nearly all interfaces are monitored, while a user distribution switch may just require some uplink ports to be monitored.   In the end, I decided on an approach that seems to work quite well.   In the xSNMP Management Pack, all interface monitoring is disabled by default.   A second, unsealed management pack, is also provided and includes groups to control interface monitoring (e.g. Fully Monitored, Not Monitored, Status Only Monitored).  Overrides are pre-configured in this MP to enable/disable the appropriate interface rules and monitors for these groups.   So, to enable interface monitoring for all Ethernet interfaces, a dynamic group membership rule can be configured to include objects based with interface type 6, or if critical interfaces are consistently labeled on switches with an Alias, the Interface Alias can be used in rules for group population.  

Organizing Hosted Objects

For each of the management packs,  I tried to take a standardized approach for hierarchical organization of hosted objects and their relationships.   This organization was facilitated primarily through the use of arbitrary classes to contain child objects.   So, rather than discover all interfaces of a device with a single hosting relationship to the parent, an intermediary logical class (named “Interfaces”) is discovered with parent and child hosting relationships.   This approach has three primary benefits: 1) the graphical Diagram View is easier to navigate, 2) the object hierarchy is more neatly organized for devices that may be monitored by multiple MP’s (e.g. a server monitored by three MP’s for SNMP hardware monitoring, O/S monitoring, and application monitoring), and 3) the organization of hosted objects is consistent, even for devices with multiple entities exposed through a single SNMP agent. 

Scalability

With loads of invaluable help from some volunteer beta testers, a great deal of time has been spent testing and investigating performance and scalability for these management packs.  While I will save many of these details for a later post, I can offer a few comments on the topic.   In all but the smallest SNMP-monitoring environments, it’s highly advisable to configure SNMP devices to be monitored by a node other than the RMS.  For larger environments, one or more dedicated Management Servers or Agent Proxies (Operations Manager agents configured to proxy requests for SNMP devices) are preferred for optimal performance.    From our testing with these Management Packs, a dedicated agent proxy can be expected to effectively monitor between 1500-3500 objects, depending on the number of monitors/rules, the intervals configured, and the processing power of the agent proxy.   By object, I am referring to any discovered object that is monitored by SNMP modules, such as devices, interfaces, fans, file systems, power supplies, etc.   So, monitoring a switch infrastructure with 4000-6000 monitored network interfaces should be doable with two dedicated agent proxy systems.  

I intend to write in greater detail about these topics in the coming weeks, and hope to post the first public beta version of these management packs soon.