OpsMgr UNIX/Linux updates

Updates were released this week for UNIX/Linux features multiple versions of OpsMgr, with a good selection of fixes:

CU7 for SCOM 2007 R2 fixes the following issues for UNIX and Linux monitoring:

  • Logical disk performance statistics are not collected for some volume types on Solaris computers.
  • Some Network Adapters on HP-UX computers may not be discovered.
  • Network adapter performance statistics are not collected for HP-UX network adapters.
  • The Solaris 8 and 9 agent may not restart after an ungraceful shutdown.

The HP-UX managements have also been updated for 2007 R2, available in catalog or here.

For 2012 and 2012 SP1, fixes related to UNIX and Linux monitoring are delivered in MP updates. The fixes are described in the KB articles for the Update Rollup releases:

System Center 2012 – Operations Manager

  • When multiple process monitors are used to target the same computer or group, processes may incorrectly monitor some template instances.  Additionally, problems with the monitored processes may not be detected. This issue occurs when each process monitor uses the same name as the process even though different argument filters are used in each process monitor.
  • Logical disk performance statistics are not collected for certain volume kinds of disks on Solaris-based computers.
  • Certain Network Adapters on HP-UX computers may not be discovered.
  • Network adapter performance statistics are not collected for HP-UX Network Adapters.
  • Logical disk performance statistics are not collected for certain kinds of device on Linux-based computers.
  • In Solaris operating systems, memory that is used for the ZFS Adaptive Replacement Cache is considered used memory incorrectly.

UR1 for System Center 2012 SP1 – Operations Manager

  • When multiple process monitors are used to target the same computer or group, processes may incorrectly monitor some template instances.  Additionally, problems with the monitored processes may not be detected. This issue occurs when each process monitor uses the same name as the process even though different argument filters are used in each process monitor.

For both 2012 and SP1, the UNIX/Linux MPs can be found here.

I’d also recommend checking out Bob Cornelisson’s notes on the updates:
http://www.bictt.com/blogs/bictt.php/2013/01/09/ur1-for-system-center-2012

http://www.bictt.com/blogs/bictt.php/2013/01/09/cu7-for-scom-2007-r2

Using PowerShell for automated UNIX/Linux Agent Discovery

PowerShell cmdlets for administration of UNIX/Linux agents were added in the System Center 2012 release of Operations Manager. There is good documentation available on the cmdlet use, but a basic discovery script might look something like this:

$SSHCredential=Get-SCXSSHCredential
$WSCredential=Get-Credential
$Pool = Get-SCOMResourcePool -DisplayName “All Management Servers Resource Pool”
$DiscResult = Invoke-SCXDiscovery -name $HostName -ResourcePool $Pool -WsManCredential $WSCredential -SshCredential $SSHcredential
$DiscResult | Install-scxagent

In this example, the Invoke-SCXDiscovery cmdlet is provided the following parameters:

  • $Hostname – the fqdn of the agent to discover
  • $Pool – the Resource Pool object used to discover and manage the agent, from Get-SCOMResourcePool
  • $WSCredential – a PSCredential object used for WSMan authentication, from Get-Credential
  • $SSHCredential – an ssh Credential object used for ssh authentication, from Get-SCXSSHCredential

If you have tried a PowerShell discovery like this, you’ll know that both Get-Credential and Get-SCXSSHCredential prompt you for credential input and don’t allow specification of passwords as command-line arguments.  This is for good reason, as plain-text scripts are a bad place to store passwords. However, this does have the effect of limiting your ability to truly automate UNIX/Linux agent discovery.  Well, with a bit of extra scripting, you can actually embed your credentials in a script in a fairly secure manner.

This article does a great job explaining how to securely write a password to a file, and then retrieve it from a script.  The steps to do this are:

  1. Logged in as the user that will run the script, create a credential object with:  $Credential = Get-Credential
  2. Write this as a secure string to a file:
    $credential.Password | ConvertFrom-SecureString | Set-Content $env:userprofile\password.txt

Now, this password can be read back into a script (but only if the script is run with the same user that wrote the password to the file), by using the following scriptlet:

$wsmanuser=”monuser”
$wsmanpassword =  Get-Content $env:userprofile\password.txt | ConvertTo-SecureString
$WSCredential = New-Object System.Management.Automation.PSCredential ($wsmanuser, $wsmanpassword)

Using this method, we can securely create the credential object to use as our WSManCredential value without a prompt, but Invoke-SCXDiscovery also needs an ssh Credential.  The ssh Credential is a bit more involved, but can be done in a similar fashion.

A function to create the ssh Credential object, using encrypted passwords stored in files is:

function Get-SCXSSHCredentialFromScript{
[CmdletBinding()]
param
(
[Parameter(Mandatory=$True)]
[string]$UserName,
[string]$PassphraseFile,
[string]$SSHKeyFile,
[string]$SuPasswordFile,
[string]$ElevationType
)

process {
$SSHcredential=””
$scred=””
$SSHcredential = New-Object Microsoft.SystemCenter.CrossPlatform.
ClientLibrary.CredentialManagement.Core.CredentialSet
$scred = New-Object Microsoft.SystemCenter.CrossPlatform.
ClientLibrary.CredentialManagement.Core.PosixHostCredential
$scred.Usage = 2
$scred.PrincipalName = $username

if ($PassphraseFile.Length -gt 0){
$sPassphrase=Get-Content $PassphraseFile | ConvertTo-SecureString
$scred.Passphrase = $sPassphrase
}

if ($SSHKeyFile.Length -gt 0)  {
$scred.KeyFile = $SSHKeyFile
Write-Host “Validating SSH Key”
$scred.ReadAndValidateSshKey()
}

#add posixhost credential to credential set
$SSHcredential.Add($scred)

if ($elevationType.Equals(“su”)) {
$sucred = New-Object Microsoft.SystemCenter.CrossPlatform.
ClientLibrary.CredentialManagement.Core.PosixHostCredential
$sucred.Usage = 32 #su elevation
$sucred.PrincipalName = “root”
$sucred.Passphrase = Get-Content $SUPasswordFile | ConvertTo-SecureString
$SSHcredential.Add($sucred)
}

if ($elevationType.Equals(“sudo”)) {
$sudocred = New-Object Microsoft.SystemCenter.CrossPlatform.
ClientLibrary.CredentialManagement.Core.PosixHostCredential
$sudocred.Usage = 16 #su elevation
$SSHcredential.Add($sudocred)
}
Return $SSHCredential
}
}

With this function defined, it can be used like this:

PS C:\Users\Administrator> $SSHCredential=Get-SCXSSHCredentialFromScript -username:monuser -PassphraseFile:$env:userprofile\password.txt –ElevationType:sudo

PS C:\Users\Administrator> $SSHCredential
SshUserName      : monuser
SshElevationType : sudo
Credentials      : {, }
Count            : 2
IsSSHKey         : False
Usage            : 0

Of course, this needs to be run in an OpsMgr shell, or the script needs to be prefaced with:

Import-Module “C:\Program Files\System Center 2012\Operations Manager\Powershell\OperationsManager\OperationsManager”
New-SCOMManagementGroupConnection localhost;

So, now we have snippets to create WSMan and ssh Credential objects, using an ecrypted password stored in a file.  Building on this, we can define a function to invoke UNIX/Linux discovery and install the agent:

function DiscoverSCXAgents{
[CmdletBinding()]
param
(
[Parameter(Mandatory=$True)]
[string]$Hostname
)
$PassphraseFile=”$env:userprofile\password.txt”
$SSHCredential=Get-SCXSSHCredentialFromScript -username:monuser -PassphraseFile:$PassphraseFile –ElevationType:sudo
$wsmanuser=”monuser”
$wsmanpassword =  Get-Content $PassphraseFile | ConvertTo-SecureString
$WSCredential = New-Object System.Management.Automation.PSCredential ($wsmanuser, $wsmanpassword)
$Pool = Get-SCOMResourcePool -DisplayName “All Management Servers Resource Pool”
Write-Output “Attempting discovery of $Hostname”
$DiscResult = Invoke-SCXDiscovery -name $HostName -ResourcePool $Pool -WsManCredential $WSCredential -SshCredential $SSHcredential
$DiscResult | fl Succeeded, ErrorData
$DiscResult | Install-scxagent
}

Testing out the new function:

DiscoverSCXAgents “lnx-db-007.contoso.com”

Attempting discovery of lnx-db-007.contoso.com

Name            AgentVersion  ManagementPackPlatformIdentifier  Id                                  
—-            ————  ——————————–  —                                  
lnx-db-007.c… 1.4.0-906     Microsoft.Linux.SLES.11           c5944ea1-e4f4-1908-ea15-d5be6ba7d14e

 

And with that, we can use an Orchestrator runbook to call our discovery script and fully automate UNIX/Linux agent discovery.

SC2012 SP1 (Beta) – Operations Manager Adds Support for Additional Linux Distros

The Beta release of System Center 2012 SP1 is now available, and in a series of posts, I wanted to highlight some new features in UNIX/Linux monitoring.

The big announcement is that as of SC 2012 SP1 Beta, we are expanding support to additional Linux distributions. In addition to Red Hat Enterprise Linux, and SUSE Linux Enterprise Server, SP1 Beta adds support for:

  • CentOS 5 and 6
  • Debian 5 and 6
  • Ubuntu 12.04

(Look for this list to expand a bit when SP1 goes to RTM)

We took a bit of a different approach in implementing support for these distros, which I will describe here. 

Read more of this post

OpsMgr 2012 UNIX/Linux Authoring Templates: More Information on the Shell Command Templates

In my last post, I described the new UNIX/Linux Shell Command templates in OpsMgr 2012, and provided a walkthrough on creating a Management Pack for monitoring BIND DNS servers on Linux. In this post, I will provide more detail on the nuts and bolts of the Shell Command templates, as well as a list of tips & tricks to consider when using the templates.

Using the Shell Command Templates

Read more of this post

OpsMgr 2012 UNIX/Linux Authoring Templates: Shell Command

Many of the OpsMgr authoring examples for UNIX/Linux monitoring that I have described on this blog are based on the use of the WSMan Invoke modules to execute shell commands. This is a really powerful mechanism to extend the capabilities of Operations Manager monitoring, and the 2012 version of Operations Manager includes a new set of templates allowing the creation of rules, monitors, and tasks using UNIX/Linux shell commands directly from the Authoring pane of the console.

The new templates are:

Monitors

  • UNIX/Linux Shell Command Three State Monitor
  • UNIX/Linux Shell Command Two State Monitor

Rules

  • UNIX/Linux Shell Command (Alert)
  • UNIX/Linux Shell Command (Performance)

Tasks

  • Run a UNIX/Linux Shell Command

Note: For the OpsMgr 2012 Release Candidate, the Shell Command Template MP needs to be downloaded and imported.  In the final release, it will be imported by default.

Underneath the covers, all of these templates use the ExecuteShellCommand method of the agent’s script provider with the WSMan Invoke module. This method executes the command and outputs StdOut, StdErr, and ReturnCode. The command can be a path to a simple command, a command or script existing on the managed computer, or a “one-liner” script (a shell script condensed to one line with pipes and semi-colons).  The templates also allow you to select whether to run with the nonprivileged action account, or the privileged account (which also supports sudo elevation).

If you’ve done this kind of UNIX/Linux authoring in 2007 R2, you will quickly see how much easier and faster this can be done in 2012.

To show the use of these templates, I have put together an MP authoring walkthrough for monitoring BIND DNS servers on Linux. This entire MP will be created in the Operations Console, with no XML editing!

Walkthrough: Monitoring BIND on Linux

Read more of this post

OpsMgr 2012 UNIX/Linux Authoring Templates: Process Monitoring

In Operations Manager, custom rules and monitors can be used to extensively build on the out-of-the-box Management Pack contents. Unfortunately, this kind of custom authoring for UNIX/Linux monitoring carried a steep learning curve with OpsMgr 2007 R2. However, the 2012 release of Operations Manager has some new features to enable many common UNIX/Linux authoring scenarios using templates, directly from the console.  The first of these new templates I wanted to cover is the new process monitoring template.

UNIX/Linux Process Monitoring Template

Operations Manager 2007 R2 included the Unix Service Monitoring template for custom monitoring of daemons on UNIX and Linux agents.   This template has been replaced in the System Center 2012 release of Operations Manager with the far more capable UNIX/Linux Process Monitoring template.   The new UNIX/Linux Process Monitoring template allows more flexibility in process/daemon monitoring, including the ability to monitor for minimum and maximum process count thresholds, and the ability to filter processes on arguments in addition to the process name. For this example, I will walk through the use the UNIX/Linux Process Monitoring template to monitor a Tomcat daemon.

The UNIX/Linux Process Monitoring template is accessible in the Authoring Pane of the Operations Console.   It can be launched with the “Add Monitoring Wizard” task under the Management Pack Templates view.

Read more of this post

OpsMgr: UNIX/Linux Heartbeat Failures After Applying KB2585542

The OpsMgr UNIX/Linux monitoring team at Microsoft is currently investigating an issue that results in heartbeat failures on Operations Manager UNIX/Linux agents after the security update KB2585542 is applied to a Management Server or Gateway.  This update fixes a vulnerability in SSL/TLS1.0, but appears to cause WS-Management connections to UNIX/Linux agents to fail. 

The vulnerability is described in bulletin MS12-006, and more information can be found in the KB article.  While we continue to investigate options for resolving this issue, there are two viable workarounds (which must be applied to all Mgmt Servers and Gateways that manage UNIX/Linux agents):

  1. Uninstall the update KB2585542 
  2. Make a registry modification to disable the SecureChannel changes implemented in the update

Note: the registry modification described here and in the KB article effectively disables the security fix that the update implements, so the modified system is subject to the same vulnerability as an unpatched system.

Modifying the registry to disable the SecureChannel changes:

  • A “FixIt” package is available in the KB article under the Known Issues section that can be used to disable the security update
  • Alternatively, you can add the 32bit DWORD value:
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control
    \SecurityProviders\SCHANNEL\

     SendExtraRecord = 2

These changes take effect immediately and do not require a reboot.