SCOM: Automatically Starting Maintenance Mode When Servers are Rebooted for Patching
August 15, 2009 69 Comments
One general way to address this ‘false alarm during patching’ issue would be to utilize one of these maintenance mode tools to drop groups of servers into maintenance mode. However, there are some gaps in this approach, in my opinion. Firstly, if wide-scale server maintenance (i.e. patching) is done in a rolling window over a weekend, it may be hard to schedule maintenance mode windows on groups of servers that just cover the time for reboot. If the maintenance mode window is too broad, real problems that surface after the reboot may not be caught in orderly fashion. Additionally, the staff responsible for patch deployments and scheduling may not necessarily be the same staff responsible for SCOM administration. Finally, large-scale patch deployment is enough of a chore itself and adding an additional step to the process may not be welcomed by the staff responsible for patch deployments.
An Automated Solution:
The solution that I wanted was one that would automatically put servers into maintenance mode only when they are being rebooted for patching, and only for the minimum time necessary – all without administrative intervention. If all went according to plan, the only alerts received during patch maintenance windows should be for problems that persisted after the reboot, i.e. the real problems.
The first part of this equation is the detection of patching. How does a monitor differentiate between a reboot for unplanned maintenance or due to a problem versus a reboot due to planned patch maintenance? The answer to this issue lies in the eventlog. WSUS or Windows Automatic Updates should log an Event ID 22 before instantiating a reboot, and other tools like Shavlik NetChk are likely to log a similar event to indicate a pending reboot from patching. If all else fails, a script could be setup in the patch management utility to use the eventcreate.exe (Windows 2003) or logevent.exe (pre-Windows 2003) utilities to log a custom event. To trap these events, create an alert-generating Windows Event log rule with the required parameters to alert on the event that indicates an imminent reboot due to patching. In order to make notification subscriptions easier in SCOM 2007 SP1, we can make use of the “Maintenance” category, so select “Maintenance” as the alert category.
The second part of this solution is the mechanism to trigger the maintenance mode script. In theory, the script could be called client-side in the form of a recovery task, but this would require uniform distributions of powershell as well as the use of runas accounts with appropriate rights in SCOM. A much simpler method is the use of Command Notification channel in SCOM.
With a Command Notification (and notification subscription), the alert can call a powershell script and pass the object id to the script. The powershell script can kick off the maintenance mode for a set period of time.
The main portion of a powershell script to do this would be something like, assuming the Managed Entity Display Name variable is passed to the script as the only argument:
$time = [DateTime]::Now
$nMinutes=20
$sHost=$args[0]
$class=get-monitoringclass | where {$_.displayname -eq “Windows Server”}
$monObj=$class|get-monitoringobject |where {$_.name -like “$sHost*”}
New-MaintenanceWindow -MonitoringObject $monObj -Comment “Planned Maintenace – scripted” -StartTime $time -EndTime $time.AddMinutes($nMinutes)
This example is assuming that 20 minutes after the initial event is logged, the server should be back in proper order.
The sample powershell script can be downloaded here.
Lots more behind the cut:
The solution thus far works great for automatically putting individual nodes into Maintenance Mode when they are rebooted for patching. However, there are a few shortcomings. Firstly, because SCOM will not respond to events logged to the eventlog by a different computer (except for event collection rules with the AgentProxying switch turned on), cluster virtual nodes will not go into maintenance mode if their hosting physical nodes are rebooted. Secondly, if you are using remote synthetic transaction monitors (such as TCP port, OLEDB, or Web Application monitors) alerts will be generated by these monitors when the node is rebooted.
As this solution involves deploying Powershell scripts at a central location, it’s easy to address these conditions with a little bit of Powershell logic.
We can check if the object is a cluster physical node with something like:
#Check if the node is hosting MSCS
$blIsCluster=$monObj | Get-MonitoringObjectProperty | where {$_.displayname -eq “IsHostingMSCS”}
If the node is hosting MSCS, some recursive logic can determine the cluster object:
#The node is hosting MSCS, get the cluster relationships
$sNodeName=$monObj|get-monitoringobjectproperty|where {$_.name -eq “NodeName”}
$classHostClus=get-monitoringclass -name “Microsoft.Windows.Cluster.HostedGroup”$clusterclass=Get-monitoringclass -name “Microsoft.Windows.Cluster”
#Loop through all cluster hosted relationships to determine clusters hosted on this node
foreach ($hostobj in $classHostClus |get-monitoringobject)
{
$sClusNodeName=$hostobj |get-monitoringobjectproperty |where {$_.name -eq “NodeName”
}
if ($sClusNodeName.value -eq $sNodeName.value){
#Get the cluster object
$sClusName=$hostobj|get-monitoringobjectproperty |where {$_.name -eq “ClusterName”}
$objClus=$clusterClass|get-monitoringobject|where {$_.name -eq $sClusName.value}
#Put the cluster in mantenance mode
New-MaintenanceWindow -MonitoringObject $objClus -Comment “Planned Maintenance – scripted” -StartTime $time -EndTime $time.AddMinutes($nMinutes)
}
}
Because there is no good way (at least that I have thought of) to programmatically link remote synthetic monitors to the hosts that serve the monitored objects, these may be best dealt with by implementing manual mapping via some if…then logic based on the hostname parameter. For example, if a server named WebServer1 is hosting a website that is monitored via a synthetic monitor, we could place the synthetic monitor group in maintenance mode if the hostname passed to the maintenance mode script matches “WebServer1.”
Setting Up the Command Notifications
To create the command notification to facilitate this configuration, first create a new command notification (Administration -> Settings -> Notification). Set the file path to: C:\Windows\sytem32\windowspowershell\v1.0\powershell.exe. Set the command line parameters to be the path to the script, and then the variable for the object display name: C:\Scripts\PutNodeInPatchMaint.ps1 $Data/Context/DataItem/ManagedEntityDisplayName$
Create a new notification recipient using the command notification channel
Create an alert subscription. By setting the alert category to “Maintenance” in the alert-generating rule, it is now easy to filter the notification subscription to only that category.
About Kristopher Bash
Operating-Quadrant 
Great post. I am not able find event ID 22 on server 2003 or 2008. I wonder if this is only entered if updates are automotically applied? I am searching for some other way to start maintenance mode for updates
How are you applying updates? What about using NTServicePack events? If you’re remotely rebooting with a shutdown.exe command, I believe a USER32 event is logged.
You could also use a batch file with eventcreate.exe to log a custom event, if that works better.
I love this concept and tried it out on the OpsMgr2K7R2 envrionment and unfortunately I was unable to make this script work.
Is there any other prerequiestes that I’m missing?
WindowsPowerShell.exe PutNodeInPatchMaint.ps1 $Data/Context/DataItem/ManagedEntityDisplayName$)
is this the correct syntax?
Is there any events I’m looking for see if the command channel is working?
I also like the idea that this doesn’t have dependancy to be a singlepoint failure if setting up full high-available solution. This can work with out have to dedicate a single server to run the schedule jobs.
If only if I can make it work :^\
Do I need to specify OpsPowerShell?
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -PSConsoleFile Microsoft.EnterpriseManagement.OperationsManager.ClientShell.Console.psc1 -NoExit .\Microsoft.EnterpriseManagement.OperationsManager.ClientShell.Startup.ps1 to make this work?
I think I’d figured your syntax was looking for the computer name $Data/Context/DataItem/ManagedEntityDisplayName$ in the command channel
so its yoru script.p1 and computername
You don’t need to load the OpsMgr PowerShell add-in, as that’s in the script (the script assumes it is running on the RMS in this line: $rmsServerName=hostname)
To answer your previous question, when you set up the notification command channel, the fields should be filled out like this (reference this screenshot: http://jxkrig.bay.livefilestore.com/y1phnlyrKFPRufNlTVpYURubG_9BEIl661yLfZaGnInnEXK67CeBGVTE-7upZjmmnTaD_dDl9h_DVe5f0ZHTeWykQ)
Full path to file:
C:\windows\system32\windowspowershell\v1.0\powershell.exe
Command line parameters
C:\Scripts\PutNodeInPatchMaint.ps1 $Data/Context/DataItem/ManagedEntityDisplayName$
Note: that trailing “)” in the original post was a mistake, sorry about that.
For each of these fields, correct the path for the executable as well as the full path to the .ps1 script to reflect your configuration.
To debug the execution, you can do the following:
Change the command line parameters to:
C:\Scripts\PutNodeInPatchMaint.ps1 $Data/Context/DataItem/ManagedEntityDisplayName$ | c:\pslog.txt
That is, add a pipe character “|” and then the path to a file you want to create as a log. This should log powershell errors in calling the script to that log.
Then, you can modify the script to add some logging. So you could change the lines:
$nMinutes=20
$sHost=$args[0]
To
$nMinutes=20
$sHost=$args[0]
write-output “Running script”
write-output “The monitoring object is: ” $sHost
This would allow you to debug what’s going on as well as confirm that the script is actually being called. SCOM will log some events to the event log if there are errors in calling the script, but it’s not going to capture the error output of the executable. So once it calls powershell, I wouldn’t expect to see any events indicating problems unless the task ran for too long.
I’d like to see you get this working, so email me if you have further questions: opquad @live.com
Thanks Kristopher,
I’ll give it shot with your recommendations, I really think this is a much better soltuion then going with groups. Awesome idea to include the clustering, since that was a topic with our group for Windows Update.
I’ll definitely keep you informed.
I’ve tried all combinations including adding AV exclusions and I was unable to generate any errors to debug. Wicked wierd!
I can run the powershell when logged on to the RMS server, and it works great, even the cluster section works like a charm.
I’ve setup a rule to tigger an event ID 105 to perform a eventcreate to test and the Channel has both SMTP and command included.
I do recieve the SMTP but the command channel doesn’t generate.
R2 is abit different then your screan shots but I was able to adjust when I first started. I also have an advisory case open so I can ask to validate to test a simple script can work from a command channel first thing tomorrow.
I’ll keep you inform what I find…hitting the hay…its been a long day.
Keep me posted. I’ve been using this in my SP1 production environment for a few months now, and it works really well (except for an occasional command channel overload if too many tasks are launched at the same time). I have R2 running in a lab though, I’ll repeat the steps there and see if I encounter any problems with R2, but I won’t be able to get to that until late afternoon.
I ran a basic command notification test on my R2 test server, and it worked just fine. I used the path: C:\WINDOWS\system32\windowspowershell\v1.0\powershell.exe for the executable, and: c:\scripts\myscript.ps1 $Data/Context/DataItem/ManagedEntityDisplayName$ |out-file c:\scripts\pslog.txt as the command line parameters. I set the working directory to C:\scripts. The test script I used just wrote the $args[0] to the output, which was specified in the |out-file of the command line arguments. It output the expected ManagedEntityDisplayName. Have you made any headway with this?
The cool part of this script is that you have created a dependancy from an individual server (or service – potentially) that can look for other dependancy to place into maintenance mode.
I can see this script being very useful in an enviornment where we have managed and unmanaged systems that perform maintenance.
Just got into the office and I’m calling in on the case for this (command channel)
Got it to work…its was my bad. I had associated notepad to .ps1 and I had a syntax in your script. Its working like a champ…I’ll see if I can write-up a doc for R2, to clean up the step-by-steps.
Awesome work sir! I can see great promise with this script and how robust it can be.
I do recommend to use EventID1074 Source USER32, with description key word: restart.
It appears before Event ID 22, for patching but to ensure you have enough time the computer is in maintenance mode before a restart kills the OpsMgr Agent from sending info.
For a larger enviornment we can take chances to place something into maintenance to reduce that ‘Cry Wolf’ syndrome.
Kodos!!!!! Kris!!!!
Also, I did see the IP mmode, that is next on my list
Great! Glad to hear it. Hope it helps out in the next patch deployment.
<<<>>>
Hi David,
Have u had time to publish the R2 steps?
Thx Kris for all your work. Makes life a WHOLE lot easier.
Cheers,
John Bradshaw
I’m about to head out for a short vacation, but if David doesn’t have time to write up his R2 steps, I can work it out in my test lab. At some point in the future, I plan on revisiting this and writing a connector to do the maintenance mode scheduling based on alert subscriptions to work around the asynchronous task limits, but I’m probably two months away from starting that project.
Hey all,
I’m experiencing problems similar to David. The script works if I run it from the command shell, but the alerts don’t seem to kick of the script using the command notification.
The alerts are generated properly and I added myself as an email recipient to the subscription and I am receiving emails.
I read David’s comments and noticed that I had .ps1 files associated with notepad as well, that’s been fixed.
I tried setting it up to generate the log file, but so far no log file is being generated.
Can either of you help me figure out where my missing piece is? This will be a god-send if we can get it working!
Currently running SCOM SP1 on Win 2k8 64-bit.
I’d be glad to try to help. Can you email me the exact parameters you have defined in the path to the executable and command line parameters? opquad @live.com
Are you seeing any errors in the Operations Manager log?
Hi Khris,
when you get the alert trigger from the monitor in the ops console what do you do with the alert? i presume you have to have an alert otherwise you can’t create the subscription? it just seems to me that i’m replacing one alert with another. I’ve created a timer event to auto resolve after 30 secs but it doesn’t seem to trigger while the agent is in MM.
thanks.
The way that I approach this is to use an alert-generating rule to create an Informational alert with a low priority, which due to the class, the severity, and priority, is excluded from any notification subscriptions that generate emails (and can be excluded from alert views in the console). I have found this useful in that I can see on a Monday morning what servers were rebooted for patching over the weekend. While technically you are replacing an alert with an alert, it is more like replacing several critical alerts that would generate emails and affect availability reports with one alert that doesn’t generate an email or affect health state/availability reporting.
I’m sure there are other ways to approach this if you wanted those alerts to be closed automatically. Off the top of my head, I would say you could configure a rule that runs on a schedule and calls a powershell script configured to close any open alerts associated with the automatic maintenance mode.
Hope this helps in some way
Thanks that helps, thanks!
Another question if i may? I’m trying to run a subscription when an event is triggered to take the agent out of MM when the backup process finishes, but as the agent is already in MM no alerts are being triggered so my 2nd subscription doesn’t run. I’d rather try and keep the MM windows to the exact time the process runs as apposed to setting a time period for MM of say an hour in the initial script. Any ideas how i can get round this?
That’s a tough one. How many agents are you dealing with in this case? If it’s a small number, it may be easier to use an agent-side solution (e.g. have the agent run the scripts to end MM). I’ll try to give this some thought though.
Thanks Khris! I wanted to try and stay away from agent side responses due to the issues you highlight!
thanks again for your help!
So close! I’ve got R2 (as I guess everyone has now!)
However I’ve run into an issue…I get a permissions error even though the action account is a Domain Admin (i know!) and a Ops Admin.
I’ve run my script from the RMS logged in as the action account and everything works fine.
I set up a eventlog write so I can see what is going on – which kinda proves that the command notification works and that powershell is called – and I get
“Result = Cannot bind argument to parameter ‘MonitoringClass’ because it is null. Cannot find path ‘Microsoft.EnterpriseManagement.OperationsManager.Client\OperationsManagerMonitoring::’ because it does not exist. System error. The user does not have sufficient permission to perform the operation.”
I looked into Powershell signed scripts and have unrestricted it…any ideas?
Oh yeah thanks to the guy who posted about .ps1 files and notepad!
And thanks for this post too (am using the CISCO MP also! Brill!
Hi Kris,
This is a great post. Thank you. I haven’t tested this on our environment but planning to do it today. What happens to RMS because we do not want to put RMS or any other management server in the maintenance mode? We still use SCCM to patch our management servers. Please let me know. Thank you.
Thank you for posting this. It is exactly what I was looking for and it worked for me right away. In my situation, I am logging a custom event using the windows scripting host and needing to put the server into maintenance mode. I found other people with similar solutions but yours is the first that was easy to understand and didn’t skip steps!
Have we tested this on R2? There is no notification category within SCOM R2? Any thoughts?
Hi, thanks for your comment. This works in R2. I have used this script/method in both SP1 and R2.
Hi Kris
Thank you for your great post. How are you using this in R2 since there is no notification category? Do you have guide for R2? Thank you.
Now working a treat! Had a call in with PPS and they checked out….so I rebuilt and now works fine!
Bit of a sledge hammer to bang in a nail but glad it is working.
Cheers
Hello Kris,
you have made my day. I spent the whole day implementing this solution. What can I say. It works great! Your script does not only help when setting clusternode in MM. It also sets all virtual clients of a hyper-v server or scvmm server in mm. This makes work much much easier for me and reduces the scom alerts significantly. Thanks again.
Hi Kris. Thanks for this great script. It’s replaced our other maintenance mode scripts since it can do clusters as well. One stop shopping!
I have stumbled on an issue when run against certain agents, however.
New-MaintenanceWindow : Cannot convert ‘System.Object[]’ to the type ‘Microsoft.EnterpriseManagement.Monitoring.Monitor
ingObject’ required by parameter ‘MonitoringObject’. Specified method is not supported.
At C:\Scripts\PutNodeInMM.ps1:54 char:41
+ New-MaintenanceWindow -MonitoringObject <<<< $monObj -Comment "Planned Maintenace – scripted" -StartTime $startTime -EndTime $endTime.AddMinutes($nMinutes)
I get this on certain cluster nodes and non-cluster agents, but the script also works on other cluster nodes and non-cluster agents. I can't figure out why it errors on certain agents.
Any idea what the error means?
Okay, some of our agents have similar names, and I think that’s the issue.
$monObj=$class|get-monitoringobject |where {$_.name -like “$sHost*”}
If I put serverA into maintenance mode using this script, but I also have an agent named serverAT, it causes problems.
I will experiement replacing the $_.name -like with $_.name -eq.
Any other ideas are much appreciated by this PowerShell novice.
It sounds like you identified the cause and fix for the issue, thanks for that investigation. I had used a “like” operator in that operation to allow the script to use just a hostname in place of a fully qualified domain name. There should be no problem changing that operator to an “-eq.”
For now I’ve changed {$_.name -like “$sHost*”} to {$_.name -like “$sHost.*”}, (put a “.” between sHost and *). If I used -eq, I couldn’t get any matches using just a hostname, I’d have to use a fqdn, and we don’t want to do that.
So far this seems to work. Thanks again, Kris!
Has anybody got this to work with the Health Service Watcher? The script and the maintenance mode functions well but the health service watcher won`t go into the unmonitored state in my environment. So unfortunately if rebooting the server the health service watcher would fire an alert:-(
Any suggestions? This would be great!
I solved the problem. I added a few lines in the script getting the health service and healt service watcher nodes and putting them into maintenance mode also. If someone has the same issue I could post the lines on Monday when I am back in the office. Just let me know if its from interest. On Monday I will try to include the virtual machine health service objects of a hyper-v host when rebooting the host.
Hi Fabia!
I’m interesting to how you solved to put Health Services Watcher inot MM.
It should help me a lot if you can send me send your lines. My email address is mats.augustsson@lumagate.se
Great Post!
Was looking for something like this….
I’ve modified the script around a bit, in order to deal with our alerting that uses Health Service & Health Service Watchers group for alerting.
http://lucamihailescu.blogspot.com/2010/01/operations-manager-maintenance-mode.html
http://lucamihailescu.spaces.live.com/blog/cns!36367337A1A26F73!170.entry
Hi Luca!
I didn’t get your script to work. Can you send me your script and description to mats.augustsson@lumagate.se.
Hi Kris,
Do you have the instructions to get this working with R2?
Thanks
Mark
We are looking to have this automated. Anyone know a way to put servers that are members of an Active Directory group into maintenance mode with just a PowerShell script?
This should be a completely viable option with a custom PowerShell script that enumerates members of an A/D group and puts them in maintenance mode. If I can find some time, I’ll try to pull a script together for you.
The poweshell script is good. But SCOM notification seems to be very slow.
We have a SCOM management group with 680 agents deployed. When we have WSUS updates there are roundabout 100 servers rebooting at the same time. So we have about 100 alerts where the powershell script should run as a custom notification.
What we find is that maintenance modes start with an interval of a bit more than 1 minute between the servers. It seems as if SCOM would process one alert, generate the notification and then wait a minute till the next alert is processed. So when we have 100 servers beeing updated at the same time, the last server will start maintenance mode almost two hours later. At that time the server as already booted and the maintenance mode is not necessary any more.
Did anyone else observe this? Is there a aolution?
Michael,
We just did patching of 550+ servers and I observed our RMS being at 100% CPU and some of the maintenance mode being delayed up to 6 hours.
We have not found a solution
Mee too – I noticed that the script takes ~ 110 seconds to finish
Hello Michael
I have made a solutions that works on 500 server and quick.
———————————
A Subscription / monitor
EVENT ID 1074
————————————-
A have made a Channel command (Powershell)
D:\auto\xxxx.ps1 –rootMS: ‘xxx-rms01’ -computerPrincipalName: ‘$Data/Context/DataItem/ManagedEntityDisplayName$’ -minutes:5 -comment: ‘Server is being restarted, and will be in maintenance mode for 5 minutes from $Data/Context/DataItem/TimeRaisedLocal$ ‘ -reason: ‘PlannedOther’
————————————-
Script (xxxx.ps1)
param($rootMS,$computerPrincipalName,$minutes,$comment,$reason)
Add-PSSnapin “Microsoft.EnterpriseManagement.OperationsManager.Client” -ErrorVariable errSnapin;
Set-Location “OperationsManagerMonitoring::” -ErrorVariable errSnapin;
new-managementGroupConnection -ConnectionString:$rootMS -ErrorVariable errSnapin;
set-location $rootMS -ErrorVariable errSnapin;
$computerClass = get-monitoringclass -name:Microsoft.Windows.Computer
$computerCriteria = “PrincipalName='” + $computerPrincipalName + “‘”
$computer = get-monitoringobject -monitoringclass:$computerClass -criteria:$computerCriteria
$startTime = [System.DateTime]::Now
$endTime = $startTime.AddMinutes($minutes)
if($computer.InMaintenanceMode -eq $false)
{
“Putting ” + $computerPrincipalName + ” into maintenance mode”
New-MaintenanceWindow -startTime:$startTime -endTime:$endTime -comment:$comment -Reason:$reason -monitoringObject:$computer
}
Hi Kris.
I am a SCOM newbie and I have real problems getting this to work under R2.
The rule I created is working and alerts were sent.
The script is also working if I give it the name of the server as argument.
It seems to me as if the rule/notification is not going to feed the script with $Data/Context/DataItem/ManagedEntityDisplayName$.
The only difference I can see is that under R2 I am not able to create a notification recipient for a command channel with an address in it.
Do you have some advice?
Thanks in advance.
Nick
This article is very useful and well written, but the example assumes that 20 minutes after the initial event is logged, the server should be back in proper order.
My concern is that critical events might be skipped while SCOM is in MM during this period. This is especially important for hardware-related issues that are detected upon server startup.
What might be an automated way to select the MM end time to match the actual time that the monitored server comes back up after the reboot?
Hello!
I’m sorry but i also have Problems to run this under R2! Can someone please help me?
I’d like the idea of this script and it works absolutely fine when I trigger it manually by eventcreate (w2k8r2). I created a Windows Event Reset for target Windows Computer. the alert (on which is the powershell script is launched) is raised when event 22 (WindowsUpdateClient) has been logged, and is cleared when event 6005 (Eventlog) is logged.
SCOM never picked up the 22 event, when windowsupdate logged the event to the SCOM agent.. What can be wrong here?
Same problem as above. Script works well in R2 (thx Kris), but the server just takes way too long to place everything in to MM. The result is that Updates have long since finished before SCOM realises something has happened.
Has anyone actually got this working in Production where 100+ servers get updates at the same time?
Thx,
John Bradshaw
I have tuned the first snippit a little bit (cause it took 110 seconds for one server):
$rootMS = ”
Add-PSSnapin “Microsoft.EnterpriseManagement.OperationsManager.Client” -ErrorVariable errSnapin;
Set-Location “OperationsManagerMonitoring::” -ErrorVariable errSnapin;
new-managementGroupConnection -ConnectionString:$rootMS -ErrorVariable errSnapin;
set-location $rootMS -ErrorVariable errSnapin;
$time = [DateTime]::Now
$time >> D:\temp\log.txt
$nMinutes=150
$sHost=$args[0]
$sHost >> D:\temp\log.txt
$strServer = Get-Agent | Where {$_.DisplayName -like “$sHost”}
$strServer.HostComputer | New-MaintenanceWindow -StartTime $time -EndTime $time.AddMinutes($nMinutes) -Comment “Planned Maintenance – scripted”
$dateLimit = (Get-Date).AddMinutes( -300 )
(get-alert -criteria “Severity = ‘0’ and Name = ‘Catch Reboot'”| where { $_.resolutionstate -ne 255 -and $_.timeRaised -le $dateLimit } | resolve-alert)
… this one runs only ~15 seconds and works well, additionally it closes all the ‘old’ alerts for the Event ID 22
Hope this helps 🙂
Hello John
I have made a solutions that works on 500 server and quick.
———————————
A Subscription / monitor
EVENT ID 1074
————————————-
A have made a Channel command (Powershell)
D:\auto\xxxx.ps1 –rootMS: ‘xxx-rms01’ -computerPrincipalName: ‘$Data/Context/DataItem/ManagedEntityDisplayName$’ -minutes:5 -comment: ‘Server is being restarted, and will be in maintenance mode for 5 minutes from $Data/Context/DataItem/TimeRaisedLocal$ ‘ -reason: ‘PlannedOther’
————————————-
Script (xxxx.ps1)
param($rootMS,$computerPrincipalName,$minutes,$comment,$reason)
Add-PSSnapin “Microsoft.EnterpriseManagement.OperationsManager.Client” -ErrorVariable errSnapin;
Set-Location “OperationsManagerMonitoring::” -ErrorVariable errSnapin;
new-managementGroupConnection -ConnectionString:$rootMS -ErrorVariable errSnapin;
set-location $rootMS -ErrorVariable errSnapin;
$computerClass = get-monitoringclass -name:Microsoft.Windows.Computer
$computerCriteria = “PrincipalName=’” + $computerPrincipalName + “‘”
$computer = get-monitoringobject -monitoringclass:$computerClass -criteria:$computerCriteria
$startTime = [System.DateTime]::Now
$endTime = $startTime.AddMinutes($minutes)
if($computer.InMaintenanceMode -eq $false)
{
“Putting ” + $computerPrincipalName + ” into maintenance mode”
New-MaintenanceWindow -startTime:$startTime -endTime:$endTime -comment:$comment -Reason:$reason -monitoringObject:$computer
}
Thx Andreas and Soren.
I’m working with one of the PS guys here to get the script working from Soren, as ~500 Servers is the ball-park we need.
Cheers,
John Bradshaw
Yep…
Get an alert from the Event Create Rule……Then set up your Subscription and choose for Criteria…”created by specific rules or monitors”. Choose the rule you just created.
Works well for a sinle machine….still working on getting it going for multiple machines. See Powershell from Soren above.
mail me at jDOTbradshawATunswDOTeduDOTau if that is not clear.
Hi Soren,
Working well….thx
JB
I have made a little modification.
A Subscription / monitor System EVENT ID 22 Source: contains Update.
And now 10 minutes maintenance mode, I had some servers that come with alarms, before the servers where put in to maintenance mode.
Hope it better. 🙂
Newbie here….
Can’t get past this stage
“Create a new notification recipient using the command notification channel”, I can’t find where to start this.
Please help 😦
Hi Mike,
Under Administration/Notification/Channels, right-click/New Channel/Command
John Bradshaw
We are using SCCM Maintenance Window option to reboot servers after the patches are installed. I doubt if there is an event that gets logged in the eventlog when SCCM client is entering the MW period. If such event is logged, then we can capture that event for SCOM MM monitor that would run the above PS script.
Did anyone find a way to put servers in SCOM MM when servers are being rebooted during SCCM MW?
The event 22 contains Update is Windows Update, there make this event, not SCCM. So check the event log, if there are a event in the log before the server reboot, and try to set it.
Now I have made the best way to put server in Automatically Maintenance Mode When Servers are Rebooted for Patching. If you ask me 🙂
It set the server in Maintenance Mode just before start installing updates, and when the server has rebootet and up running, it stops Maintenance Mode.
Our helpdesk is so happy now, the only see things which are error after patchning, not windows services down, then updates ISS and SQL and so on.
Made this on you RMS server
1. Create a Subscription / monitor System EVENT ID 18 Source: contains Update.
Alert description
$Target/Property[Type=”Windows1!Microsoft.Windows.Computer”]/PrincipalName$
$Data/Context/EventDescription$
Preparing for security later to day.
2. Then a channel
D:\xxxxxx_auto\xxxxxx_WSUS_MAINT.REXX $Data/Context/DataItem/AlertDescription$
3. Then a Rexx file (use something else if you want)
/* REXX */
/* Program provides a command to put servers in maintenance mode at specific time */
/* ifm. installation of security updates. event 18 from systemlog */
arg parmstr
logfil = ‘D:\xxxxxx_auto\logs\’date(I)’-wsus-maint-input.txt’ /* what is called program */
logfil2 = ‘D:\xxxxxx_auto\wsus-timer.cmd’ /* commands to the scheduled task */
skipfil = ‘D:\xxxxxx_auto\logs\’date(I)’-wsus-maint-skip.txt’ /* errors in the input log here */
‘@echo ‘time() ‘”‘parmstr'” >> ‘logfil
/*————————————————-*/
/* valider input – exit If the input is not correct */
/*————————————————-*/
if pos(‘THESE UPDATES ON’,parmstr) = ‘0’ then ,
do
‘@echo ‘parmstr’ >> ‘ skipfil
exit
end
parse value parmstr with servernavn r2 ‘THESE UPDATES ON’ datoen ‘ AT ‘ klokken r3
parse value datoen with dd ‘.’ maaned aaaa
/*—————————————————————————————————————*/
/* Validate the content and length on the date fields */
/*—————————————————————————————————————*/
dd = strip(dd)
if pos(‘?’,dd) ‘0’ then dd = strip(dd,,’?’)
if length(dd) < '2' then dd = '0'dd
if pos('?',maaned) ‘0’ then maaned = strip(maaned,,’?’)
if pos(‘?’,aaaa) ‘0’ then aaaa = strip(aaaa,,’?’)
/*—————————————————————————————————————*/
/* When the scheduled task must be created with month as the number we just convert from name to number */
/*—————————————————————————————————————*/
if substr(maaned,1,3) = ‘JAN’ then maaned = ’01’
if substr(maaned,1,3) = ‘FEB’ then maaned = ’02’
if substr(maaned,1,3) = ‘MAR’ then maaned = ’03’
if substr(maaned,1,3) = ‘APR’ then maaned = ’04’
if substr(maaned,1,2) = ‘MA’ then maaned = ’05’
if substr(maaned,1,3) = ‘JUN’ then maaned = ’06’
if substr(maaned,1,3) = ‘JUL’ then maaned = ’07’
if substr(maaned,1,3) = ‘AUG’ then maaned = ’08’
if substr(maaned,1,3) = ‘SEP’ then maaned = ’09’
if substr(maaned,1,1) = ‘O’ then maaned = ’10’
if substr(maaned,1,3) = ‘NOV’ then maaned = ’11’
if substr(maaned,1,3) = ‘DEC’ then maaned = ’12’
klokken = substr(klokken,1,5)
dato = maaned’/’strip(dd)’/’strip(aaaa)
taskid = servernavn
if pos(‘.’,taskid) ‘0’ then parse value taskid with taskid ‘.’ rst_tsk
servernavn = strip(servernavn)
/*say taskid servernavn dato klokken*/
/*——————————————————-*/
/* write to the file later executed by the scheduled task */
/*——————————————————-*/
‘@echo cmd.exe /c d:\xxxxxx_auto\AMM_OPRET_SCHTASK.CMD’ taskid servernavn klokken dato ‘ >> ‘ logfil2
/*—————————————————————————————————————————–*/
/* If the domain controllers, there may come alarms on the running far can not see it there is to be restarted * /
/ * These alarms bother we do not see, so we put also “his partner” in maintenance mode */
/*—————————————————————————————————————————–*/
partner_server = ”
select
/*———————————————*/
/* xx.DK */
/*———————————————*/
when servernavn = ‘xx-DC4.xx.DK’ then ,
do
partner_server = ‘xx-DC5.xx.DK’
partner_taskid = ‘xx-DC5’
end
when servernavn = ‘xx-DC5.xx.DK’ then ,
do
partner_server = ‘xx-DC4.xx.DK’
partner_taskid = ‘x-DC4’
end
/*———————————————*/
/* yy.xx.DK */
/*———————————————*/
when servernavn = ‘ADMIN-DC02.yy.xx.DK’ then ,
do
partner_server = ‘ADMIN-DC3.yy.xx.DK.DK’
partner_taskid = ‘ADMIN-DC3’
end
when servernavn = ‘ADMIN-DC3..yy.xx.DK’ then ,
do
partner_server = ‘ADMIN-DC02..yy.xx.DK.DK’
partner_taskid = ‘ADMIN-DC02’
end
otherwise
do
nop
end
end
/*——————————————————-*/
/* write to the file later executed by the scheduled task */
/*——————————————————-*/
if partner_server ” then ,
do
‘@echo cmd.exe /c d:\xxxxxx_auto\AMM_OPRET_SCHTASK.CMD’ partner_taskid partner_server klokken dato ‘ >> ‘ logfil2
end
/*’@start cmd.exe /c d:\xxxxxx_auto\AMM_OPRET_SCHTASK.CMD’ taskid servernavn klokken dato */
/*’@echo ‘time() taskid servernavn klokken dato’ >> ‘ logfil */
exit
4. Then a Cmd file call AMM_OPRET_SCHTASK.cmd
@echo off
rem …………………………………………………………….
rem . This program will create a schedule task .
rem . Run the program with the parameter. .
rem . Parameter 1 is what you want the task to be named. .
rem . Parameter 2 is the full server name. .
rem . Parameter 3 is the time when the server in maintenance mode. .
rem . Parameter 4 is the date in mm/dd/åååå .
rem …………………………………………………………….
schtasks /create /RU xxxx\SCOM_scheduler /RP PASSWORD /SC ONCE /TN xxxxxx_AMM_%1 /TR “C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe D:\xxxxxx_auto\AgentMaintenanceModeV4.ps1 -rootMS: ‘xx-rms01’ -computerPrincipalName: ‘%2’ -minutes:30 -Comment: ‘Server is being patched’ -reason: ‘PlannedOther'” /ST %3 /SD %4 /V1 /F /RL HIGHEST /Z
5. Then a empty Cmd file call wsus-timer.cmd
6. Then a Cmd file call wsus-timer-admin.cmd
cmd /c d:\xxxxxx_auto\wsus-timer.cmd
del /q d:\xxxxxx_auto\wsus-timer.cmd
echo rem >> d:\xxxxxx_auto\wsus-timer.cmd
7. Then a Schedule task as call wsus-timer-admin.cmd once per hour.
Good luck
Soren – If you don’t mind, would you share this script? please send it my email ID
Hope it would be in English 🙂
Thanks for these instructions!!!
Running R2 I ran into the following issue when creating the subscriber –
1. Select a Channel Type of “Command”
2. Choose the previously created Command Channel
3. Delivery Address field is grayed out.
To get by this….
1. Briefly changed the Channel Type selection to SMTP
2. This livens up the Delivery Address field. Enter: $Data/Context/DataItem/ManagedEntityDisplayName$
3. Change the Channel type back to “Command” before saving the subscriber it will retain the correct/required Delivery Address.
I don’t see any other complaints about this but it is consistent behavior for me so I thought I’d mention it.
Hello,
Maybe I am missing something, but when I simply run the script to test it, it will put all my clusters in maintenance mode … do you have any idea on how to debug this ?
Regards
Julien
Ok I got it, in fact I did the same as I had to do to enter the first IF condition, where I had to change $blIsCluster.Value to $blIsCluster.Value.Value (or it was always false). I did add .Value after the two variables in the second IF comparaison($sClusNodeName.value and $sNodeName.value), or it was always equal.
Has anyone been able to set this up with SCOM 2012? I am trying to accomplish this but having trouble following this as the interface is slightly different. I don’t see how to select this ‘Maintenance’ category?