Using PowerShell for automated UNIX/Linux Agent Discovery

PowerShell cmdlets for administration of UNIX/Linux agents were added in the System Center 2012 release of Operations Manager. There is good documentation available on the cmdlet use, but a basic discovery script might look something like this:

$SSHCredential=Get-SCXSSHCredential
$WSCredential=Get-Credential
$Pool = Get-SCOMResourcePool -DisplayName “All Management Servers Resource Pool”
$DiscResult = Invoke-SCXDiscovery -name $HostName -ResourcePool $Pool -WsManCredential $WSCredential -SshCredential $SSHcredential
$DiscResult | Install-scxagent

In this example, the Invoke-SCXDiscovery cmdlet is provided the following parameters:

  • $Hostname – the fqdn of the agent to discover
  • $Pool – the Resource Pool object used to discover and manage the agent, from Get-SCOMResourcePool
  • $WSCredential – a PSCredential object used for WSMan authentication, from Get-Credential
  • $SSHCredential – an ssh Credential object used for ssh authentication, from Get-SCXSSHCredential

If you have tried a PowerShell discovery like this, you’ll know that both Get-Credential and Get-SCXSSHCredential prompt you for credential input and don’t allow specification of passwords as command-line arguments.  This is for good reason, as plain-text scripts are a bad place to store passwords. However, this does have the effect of limiting your ability to truly automate UNIX/Linux agent discovery.  Well, with a bit of extra scripting, you can actually embed your credentials in a script in a fairly secure manner.

This article does a great job explaining how to securely write a password to a file, and then retrieve it from a script.  The steps to do this are:

  1. Logged in as the user that will run the script, create a credential object with:  $Credential = Get-Credential
  2. Write this as a secure string to a file:
    $credential.Password | ConvertFrom-SecureString | Set-Content $env:userprofile\password.txt

Now, this password can be read back into a script (but only if the script is run with the same user that wrote the password to the file), by using the following scriptlet:

$wsmanuser=”monuser”
$wsmanpassword =  Get-Content $env:userprofile\password.txt | ConvertTo-SecureString
$WSCredential = New-Object System.Management.Automation.PSCredential ($wsmanuser, $wsmanpassword)

Using this method, we can securely create the credential object to use as our WSManCredential value without a prompt, but Invoke-SCXDiscovery also needs an ssh Credential.  The ssh Credential is a bit more involved, but can be done in a similar fashion.

A function to create the ssh Credential object, using encrypted passwords stored in files is:

function Get-SCXSSHCredentialFromScript{
[CmdletBinding()]
param
(
[Parameter(Mandatory=$True)]
[string]$UserName,
[string]$PassphraseFile,
[string]$SSHKeyFile,
[string]$SuPasswordFile,
[string]$ElevationType
)

process {
$SSHcredential=””
$scred=””
$SSHcredential = New-Object Microsoft.SystemCenter.CrossPlatform.
ClientLibrary.CredentialManagement.Core.CredentialSet
$scred = New-Object Microsoft.SystemCenter.CrossPlatform.
ClientLibrary.CredentialManagement.Core.PosixHostCredential
$scred.Usage = 2
$scred.PrincipalName = $username

if ($PassphraseFile.Length -gt 0){
$sPassphrase=Get-Content $PassphraseFile | ConvertTo-SecureString
$scred.Passphrase = $sPassphrase
}

if ($SSHKeyFile.Length -gt 0)  {
$scred.KeyFile = $SSHKeyFile
Write-Host “Validating SSH Key”
$scred.ReadAndValidateSshKey()
}

#add posixhost credential to credential set
$SSHcredential.Add($scred)

if ($elevationType.Equals(“su”)) {
$sucred = New-Object Microsoft.SystemCenter.CrossPlatform.
ClientLibrary.CredentialManagement.Core.PosixHostCredential
$sucred.Usage = 32 #su elevation
$sucred.PrincipalName = “root”
$sucred.Passphrase = Get-Content $SUPasswordFile | ConvertTo-SecureString
$SSHcredential.Add($sucred)
}

if ($elevationType.Equals(“sudo”)) {
$sudocred = New-Object Microsoft.SystemCenter.CrossPlatform.
ClientLibrary.CredentialManagement.Core.PosixHostCredential
$sudocred.Usage = 16 #su elevation
$SSHcredential.Add($sudocred)
}
Return $SSHCredential
}
}

With this function defined, it can be used like this:

PS C:\Users\Administrator> $SSHCredential=Get-SCXSSHCredentialFromScript -username:monuser -PassphraseFile:$env:userprofile\password.txt –ElevationType:sudo

PS C:\Users\Administrator> $SSHCredential
SshUserName      : monuser
SshElevationType : sudo
Credentials      : {, }
Count            : 2
IsSSHKey         : False
Usage            : 0

Of course, this needs to be run in an OpsMgr shell, or the script needs to be prefaced with:

Import-Module “C:\Program Files\System Center 2012\Operations Manager\Powershell\OperationsManager\OperationsManager”
New-SCOMManagementGroupConnection localhost;

So, now we have snippets to create WSMan and ssh Credential objects, using an ecrypted password stored in a file.  Building on this, we can define a function to invoke UNIX/Linux discovery and install the agent:

function DiscoverSCXAgents{
[CmdletBinding()]
param
(
[Parameter(Mandatory=$True)]
[string]$Hostname
)
$PassphraseFile=”$env:userprofile\password.txt”
$SSHCredential=Get-SCXSSHCredentialFromScript -username:monuser -PassphraseFile:$PassphraseFile –ElevationType:sudo
$wsmanuser=”monuser”
$wsmanpassword =  Get-Content $PassphraseFile | ConvertTo-SecureString
$WSCredential = New-Object System.Management.Automation.PSCredential ($wsmanuser, $wsmanpassword)
$Pool = Get-SCOMResourcePool -DisplayName “All Management Servers Resource Pool”
Write-Output “Attempting discovery of $Hostname”
$DiscResult = Invoke-SCXDiscovery -name $HostName -ResourcePool $Pool -WsManCredential $WSCredential -SshCredential $SSHcredential
$DiscResult | fl Succeeded, ErrorData
$DiscResult | Install-scxagent
}

Testing out the new function:

DiscoverSCXAgents “lnx-db-007.contoso.com”

Attempting discovery of lnx-db-007.contoso.com

Name            AgentVersion  ManagementPackPlatformIdentifier  Id                                  
—-            ————  ——————————–  —                                  
lnx-db-007.c… 1.4.0-906     Microsoft.Linux.SLES.11           c5944ea1-e4f4-1908-ea15-d5be6ba7d14e

 

And with that, we can use an Orchestrator runbook to call our discovery script and fully automate UNIX/Linux agent discovery.

Advertisements

SC2012 SP1 (Beta) – Operations Manager Adds Support for Additional Linux Distros

The Beta release of System Center 2012 SP1 is now available, and in a series of posts, I wanted to highlight some new features in UNIX/Linux monitoring.

The big announcement is that as of SC 2012 SP1 Beta, we are expanding support to additional Linux distributions. In addition to Red Hat Enterprise Linux, and SUSE Linux Enterprise Server, SP1 Beta adds support for:

  • CentOS 5 and 6
  • Debian 5 and 6
  • Ubuntu 12.04

(Look for this list to expand a bit when SP1 goes to RTM)

We took a bit of a different approach in implementing support for these distros, which I will describe here. 

Read more of this post

OpsMgr 2012 UNIX/Linux Authoring Templates: More Information on the Shell Command Templates

In my last post, I described the new UNIX/Linux Shell Command templates in OpsMgr 2012, and provided a walkthrough on creating a Management Pack for monitoring BIND DNS servers on Linux. In this post, I will provide more detail on the nuts and bolts of the Shell Command templates, as well as a list of tips & tricks to consider when using the templates.

Using the Shell Command Templates

Read more of this post

OpsMgr 2012 UNIX/Linux Authoring Templates: Shell Command

Many of the OpsMgr authoring examples for UNIX/Linux monitoring that I have described on this blog are based on the use of the WSMan Invoke modules to execute shell commands. This is a really powerful mechanism to extend the capabilities of Operations Manager monitoring, and the 2012 version of Operations Manager includes a new set of templates allowing the creation of rules, monitors, and tasks using UNIX/Linux shell commands directly from the Authoring pane of the console.

The new templates are:

Monitors

  • UNIX/Linux Shell Command Three State Monitor
  • UNIX/Linux Shell Command Two State Monitor

Rules

  • UNIX/Linux Shell Command (Alert)
  • UNIX/Linux Shell Command (Performance)

Tasks

  • Run a UNIX/Linux Shell Command

Note: For the OpsMgr 2012 Release Candidate, the Shell Command Template MP needs to be downloaded and imported.  In the final release, it will be imported by default.

Underneath the covers, all of these templates use the ExecuteShellCommand method of the agent’s script provider with the WSMan Invoke module. This method executes the command and outputs StdOut, StdErr, and ReturnCode. The command can be a path to a simple command, a command or script existing on the managed computer, or a “one-liner” script (a shell script condensed to one line with pipes and semi-colons).  The templates also allow you to select whether to run with the nonprivileged action account, or the privileged account (which also supports sudo elevation).

If you’ve done this kind of UNIX/Linux authoring in 2007 R2, you will quickly see how much easier and faster this can be done in 2012.

To show the use of these templates, I have put together an MP authoring walkthrough for monitoring BIND DNS servers on Linux. This entire MP will be created in the Operations Console, with no XML editing!

Walkthrough: Monitoring BIND on Linux

Read more of this post

OpsMgr 2012 UNIX/Linux Authoring Templates: Process Monitoring

In Operations Manager, custom rules and monitors can be used to extensively build on the out-of-the-box Management Pack contents. Unfortunately, this kind of custom authoring for UNIX/Linux monitoring carried a steep learning curve with OpsMgr 2007 R2. However, the 2012 release of Operations Manager has some new features to enable many common UNIX/Linux authoring scenarios using templates, directly from the console.  The first of these new templates I wanted to cover is the new process monitoring template.

UNIX/Linux Process Monitoring Template

Operations Manager 2007 R2 included the Unix Service Monitoring template for custom monitoring of daemons on UNIX and Linux agents.   This template has been replaced in the System Center 2012 release of Operations Manager with the far more capable UNIX/Linux Process Monitoring template.   The new UNIX/Linux Process Monitoring template allows more flexibility in process/daemon monitoring, including the ability to monitor for minimum and maximum process count thresholds, and the ability to filter processes on arguments in addition to the process name. For this example, I will walk through the use the UNIX/Linux Process Monitoring template to monitor a Tomcat daemon.

The UNIX/Linux Process Monitoring template is accessible in the Authoring Pane of the Operations Console.   It can be launched with the “Add Monitoring Wizard” task under the Management Pack Templates view.

Read more of this post

OpsMgr: UNIX/Linux Heartbeat Failures After Applying KB2585542

The OpsMgr UNIX/Linux monitoring team at Microsoft is currently investigating an issue that results in heartbeat failures on Operations Manager UNIX/Linux agents after the security update KB2585542 is applied to a Management Server or Gateway.  This update fixes a vulnerability in SSL/TLS1.0, but appears to cause WS-Management connections to UNIX/Linux agents to fail. 

The vulnerability is described in bulletin MS12-006, and more information can be found in the KB article.  While we continue to investigate options for resolving this issue, there are two viable workarounds (which must be applied to all Mgmt Servers and Gateways that manage UNIX/Linux agents):

  1. Uninstall the update KB2585542 
  2. Make a registry modification to disable the SecureChannel changes implemented in the update

Note: the registry modification described here and in the KB article effectively disables the security fix that the update implements, so the modified system is subject to the same vulnerability as an unpatched system.

Modifying the registry to disable the SecureChannel changes:

  • A “FixIt” package is available in the KB article under the Known Issues section that can be used to disable the security update
  • Alternatively, you can add the 32bit DWORD value:
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control
    \SecurityProviders\SCHANNEL\

     SendExtraRecord = 2

These changes take effect immediately and do not require a reboot.

Operations Manager UNIX/Linux Agent Certificates (and using a PKI)

Introduction

UNIX and Linux agent monitoring in Operations Manager requires certificates to secure the SSL communication channel between the Management Servers and agents.  In this post, I will provide some background information on this communication and the certificates, as well as describe an optional approach to replace the default Operations Manager certificate infrastructure with your organization’s Public Key Infrastructure.

The Protocols

The Operations Manager UNIX/Linux agent is a very lightweight agent implementation, comprising a CIM Object Manager (OpenPegasus) and CIM Providers.   Unlike Operations Manager Windows agents, the UNIX/Linux agent doesn’t have a health service, and doesn’t run workflows locally.  Rather, the Management Server (or Gateway) that manages the agent runs the workflows and remotely connects to the UNIX/Linux agent to retrieve current data.  

There are two protocols involved in the communication between the Management Server and the UNIX/Linux agent:  ssh and WS-Management.   

Ssh is used purely for agent maintenance activities, and is not used for any monitoring.   Operations like agent installation, uninstallation, upgrade, or agent daemon restart (through a recovery task) are executed over ssh.    Ssh facilitates the transfer of files and execution of remote commands for these operations when the agent daemon is unavailable.  

WS-Management (or WSMan) is the core protocol used in UNIX/Linux monitoring.   WSMan is a SOAP-based protocol for cross-platform management.   All monitoring operations (e.g. enumerating CIM providers for data on file systems, memory, etc, execution of commands/scripts for monitoring, executing log file reads for monitoring) are implemented over WSMan.   As WSMan is a web service protocol, the OpenPegasus-based CIMOM functions as a secure web server (user credentials are authenticated through PAM).  This is where the agent certificate comes in to play.

The Certificate

The UNIX/Linux agent certificate is quite simply used to secure the WSMan connection using SSL and provide authentication for the remote agent host.   The requirements for this certificate are:

  • The certificate is a server authentication certificate (Enhanced Key Usage: 1.3.6.1.5.5.7.3.1)
  • The CN of the certificate matches the FQDN that the Management Server uses to connect to the agent
  • The certificate is signed by a trusted authority (and can be checked for revocation)

When the Operations Manager UNIX/Linux agent is installed, it generates a certificate (using openssl) at the path:  /etc/opt/microsoft/ssl.  The file name of the certificate is scx-host-<hostname>.pem and the corresponding private key is named scx-key.pem.   The agent actually looks for the certificate at /etc/opt/microsoft/scx/ssl/scx.pem, which is initially configured as a symbolic link pointing to scx-host-<hostname>.pem.

Upon initial agent installation, the certificate is not signed, and is not usable for securing the WSMan SSL communication.

Note:  when initially creating the certificate, the agent attempts to determine the agent hostname for use as the CN value of the certificate.   In cases where the DNS name known to the local host does not match the FQDN that OpsMgr will use to communicate with the agent, additional steps are required to establish a valid certificate.  More information can be found here: http://technet.microsoft.com/en-us/library/dd891009.aspx

Certificates and Management Servers

When a Management Server discovers a UNIX or Linux agent, the server uses its certificate to sign the agent certificate, acting like a standalone Certificate Authority.  In the discovery process, this actually involves securely transferring the certificate from the agent to the Management Server, signing it, copying it back to the agent, and restarting the agent daemon.  

In order to move an agent between Management Servers, the new Management Server must trust the certificate that was used to sign the agent’s certificate.  This becomes particularly important in the 2012 version of Operations Manager, where agents will move automatically between the Management Servers that are members of the Resource Pool managing the agent.  For more information on the procedure to trust a server’s certificate from another server, review this document: http://technet.microsoft.com/en-us/library/hh287152.aspx.

Using a PKI Instead of Management Servers for Signing

Because the certificates used for securing the agent SSL channel are not proprietary, a separate Public Key Infrastructure can be used to manage the agent certificates, if the PKI option is appealing for your organization.  While this requires some additional resources in the environment (a Certificate Authority) and customization, there are a few benefits to using a PKI: 

  • Certificate policies are controlled by the PKI and customizable to meet your organization’s security requirements
  • Migrations of agents between Management Servers (within or between Resource Pools) can be done without exporting/importing Management Server certificates – simplifying the provisioning of Management Servers.
  • More options exist for automation of agent deployment and certificate signing

The procedure to use a PKI instead of Management Server signed certificates varies with different requirements and environments, but I will describe the steps required for one example approach.  This example assumes that the Certificate Authority is a Windows 2008 Certificate Authority. 

Prerequisites:

  1. Configure the certificate template on the Certificate Authority – you can use the “Web Server” template or a copy of it – configure options and permissions, publish the template.
  2. Import the CA certificate from the signing CA  to the trusted authorities list on every management serverthat will manage the UNIX/Linux agents:
    1. certutil -f -config “<CAHostname>\<CAName>” -ca.cert <CACertFile>
    2. certutil -addstore Root <CACertFile>

Per-Agent steps:

  1. Install the agent – this can be done through the OpsMgr Discovery Wizard, manually, or with another package distribution tool.  If you use the OpsMgr Discovery Wizard to install the agent, the agent will generate a certificate that is signed by the management server, but this can be replaced with your PKI CA signed certificate.
  2. Generate a cert signing request – either create a new private key with OpenSSL or use the private key generated during the agent install
    1. a.      Command to generate a CSR using the key generated during agent install:
      openssl req -new -key /etc/opt/microsoft/scx/ssl/scx-key.pem  -subj /CN=<FQDN of agent host> -text -out <OutputPath>
    2. 3.       Copy the CSR back to a Windows machine
    3. 4.       Submit the CSR to the CA – this command assumes auto-enrollment is enabled and authorized:
      1. certreq.exe -submit -config <CAHostName>\<CAName> -attrib “CertificateTemplate:<TemplateName>” <CSR FileName> <OutputCertName>
      2. Copy the signed cert back to the UNIX/Linux agent using  a secure copy method.  If auto-enrollment was used in step 4, the value for <OutputCertName>  specifies the file name of the signed certificate to copy to the agent.
      3. Update the symbolic link: /etc/opt/microsoft/scx/ssl/scx.pem to point to your new signed certificate
      4. Restart the agent:  /opt/microsoft/scx/bin/tools/scxadmin –restart
      5. Discover the agent using the Operations Console or PowerShell Cmdlet

Automation and Customization Opportunities

All of the per-agent steps described above can be executed from a command line, meaning that this procedure can be automated through scripting.  Using a script on a Windows server, the UNIX/Linux commands and file copying actions can be executed with SSH utilities like PuTTY’s plink and pscp.  For really robust automation capabilities, all of the steps can be implemented in a PowerShell script – I like the plink.exe integration example described on this blog: http://www.christowles.com/2011/06/how-to-ssh-from-powershell-using.html.

Aside from the primary benefits of automating these steps in terms of reducing manual interactions, other customization opportunities are exposed with using this scripting approach.  For example, if your DNS infrastructure and UNIX/Linux agent hostnames don’t neatly correlate, you could modify step 2 of the per-agent steps to also generate a new certificate with openssl using the desired FQDN as the certificate’s CN (http://technet.microsoft.com/en-us/library/dd891009.aspx).  Alternatively, if you are using Operations Manager 2007 R2 and want to implement agent deployment and certificate signing using sudo elevation instead of root credentials, the UNIX/Linux host commands in the per-agent steps could be prepended with the sudo command (this functionality is built into the 2012 version of Operations Manager).