Using System Center 2012 for Host Cluster Remediation (Patching) and Host Cluster VM Rebalancing for Private Cloud – Part 1 of Many (Host Remediation)

Article
05/11/2012

So I’ve been working on Microsoft Private Cloud implementations for various customers. One of the common themes I run into deals with patching the fabric of the Private Cloud, meaning the actual virtual machine hosts. Using System Center 2012 Virtual Machine Manager you can easily orchestrate host remediation with built in functionality. This means that Virtual Machine Manager 2012 has an automated process to move virtual machines from each VM Host node being patched, installing the updates, and rebooting the VM Host machine. One scenario that I’m finding customers are looking for is keeping thier virtual machines on the actual VM Host it was prior to the host cluster remediation. Even thought Virtual Machine Manager will automate this sequence, often virtual machines are left on a single VM Host node in the cluster or on a different VM Host from when the host remediation started. In this first post of the series I will detail a solution that will automate host cluster remediation and prepare the virtual machines to be rebalanced to their VM Hosts they were running on prior to the System Center Virtual Machine Manager 2012 remediation job. The solution also takes in account using the local time zone of the VM host machines. This is used for environments using System Center Virtual Machine Manager 2012 that manage clusters in different time zones, but need host remediation to happen at a specific time local to the host cluster. You will see in later posts how I use System Center Orchestrator 2012 to determine the time zones of each host cluster and only remediate host clusters belonging to a particular time zone at a specified time. A few other things the script takes in account is ensuring the host clusters are assigned to an update baseline, has a compliance scan status, and that all of the guest virtual machine hosted in the cluster do not have DVD drives mounted with a local path. All of those situations mentioned can prohibit host remediation from being successful. I will provide guidance on all the Windows PowerShell scripts and System Center Orchestrator runbooks I developed to automate this process. The full automated end-to-end solution utilizes System Center products Virtual Machine Manager 2012, Operations Manager 2012, and Orchestrator 2012. Let’s begin managing your Microsoft Private Cloud.

Prerequisites

The following script assumes that you have already configured your System Center Virtual Machine Manager 2012 server with an Update Server. If you have not done this, please follow the guidance on managing fabric updates for System Center Virtual Machine Manager 2012 https://technet.microsoft.com/en-us/library/gg675084.aspx.

The Remediation Script Breakdown

Below is the remediation script that you can run against your Hyper-V host clusters in your System Center Virtual Machine Manager 2012 environment. One thing to note is that this script is intended to run as a runbook in System Center Orchestrator 2012 using the SCVMM 2012 Integration Pack. When using the run SCVMM script object you do not need to import the VirtualMachineManager module for Virtual Machine Manager 2012 because the object configuration does this for you. If you do want to utilized this script outside of Orchestrator 2012 you will have to add the Import-Module VirtualMachineManager to the script. The script itself is somewhat busy, but it will automate all the key activities needed to make the process 100% completely automated using System Center Orchestrator 2012. I will detail out the major parts in the script so that you can feel comfortable and confident using it in your Microsoft Private Cloud environments. Because the script is well documented, I will only explain the major parts. The script will also be available to download as a zip file.

Lines 12 – 97 - The script contains four globally declared functions WriteEvent, VMHostTimeZone, VMHostComplianceStatus, and UpdateRemediationBaseline. The functions are well documented and I will discuss them in more detail when the script uses them. For a quick description of the functions the WriteEvent function is used for tracing the script activity and logging to the event log. The VMHostTimeZone function will translate the VMHost time zone metric to the configured three letter time zone. The VMHostComplianceStatus function will initiate a compliancy status against VM Host Cluster. The UpdateRemediationBaseline function will add the VM Host Cluster to the approved baseline.

Lines 100 – 107 - Setting the error action setting for the PowerShell script. There are also two global variable being declared for the time zone and the baseline. These are static entries now, but when we move the script into System Center Orchestrator they will be variables using published data in the data bus.

Line 112 - Starts the Try,Catch in the scripts and does the search Where clause for the Host Clusters you are looking to target. You will have to edit this for your environment.

Line 125 – Ensures the Host Cluster is assigned to a baseline

Line 137 – Verifying the Nodes in the Host Cluster are in the required time zone.

Line 142 – Checking to see if the VM Host is not compliant. If compliant it is skipped and not added to the Job Group

Line 147 – 152 – This is a ForEach loop that will got through all the VMs on the VM Host and add the VM Host information to the Custom Field 1 property of the VM. This will be used in a later script to return the VM to the host it was running on prior to remediation. After the script runs through remediation, you will notice the Custom1 Property of the VM listed with the VM Host name that the VM is currently running on.

Line 158 – 170 – This is a sequence to determine if the VM has the DVD drive mounted and then unmounts the the DVD drive if mounted on the VMs

Line 177 – Add the VM Host node of the VM Host Cluster to the Job Group if found not compliant.

Line 196 – Initiate the VM Host Cluster Remediation Job in SCVMM

Line 197 – Writing out the SCVMM Job GUID to the event log for System Center Operations Manager to monitor and execute a System Center Orchestrator runbook that will monitor the SCVMM Job until complete and report status. You will see this detailed out in later posts on this series.

Line 213 – Script Complete Smile

Feel free to post your comments or questions. You can also email me too.

Phil Gibson | Senior Consultant | Microsoft | System Center & Private Cloud Infrastructure & Development

SCVMMClusterRemediation.ps1

<#
Purpose : Remediate Hyper-V Clusters using SCVMM 2012
Author : pgibson.online@hotmail.com [MSFT]
Disclaimer: THIS CODE AND INFORMATION IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND,
EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A PARTICULAR PURPOSE.
#>
####### Functions ###############
Function global:WriteEvent
{
<#
Purpose : Write events into the eventlog of the executing machine.
Setup : Please edit the $EventSoure variable to the EventLog source you want your script to utilize as default. Ex. $EventSource = "MyLogSource"
Example 1: WriteEvent -EventSource "TEST" -EventID 101 -EventType "Warning" -EventMessage "This is really a test."
Example 2: WriteEvent -Source "TEST" -ID 101 -Type "Warning" -Message "This is really a test."
**NOTE** : This function will default the EventLog source you supplied with a "Information" event to the "Application" log if the parameters are not used/overridden
Author : pgibson.online@live.com [MSFT]
#>
param (
[parameter(Mandatory=$false,Helpmessage="Please enter the source of the Eventlog entry. Example: -Source ""MyLogSource""")][alias("Source")][string]$EventSource = "SCVMMHostRemediation",
[parameter(Mandatory=$true,Helpmessage="Please enter the event ID of the Eventlog entry. Example: -ID 101")][alias("ID")][int]$EventID,
[parameter(Mandatory=$false)][alias("Type")][ValidateSet("Error", "Information", "Warning")][string]$EventType = "Information",
[parameter(Mandatory=$true,Helpmessage="Please enter the event description of the Eventlog entry. Example: -Message ""This is a test.""")][alias("Message")][string]$EventMessage,
[parameter(Mandatory=$false)][alias("Log")][ValidateSet("Application", "Security", "System")][string]$EventLog = "Application"
)
# Create event source if it does not exist
If (![System.Diagnostics.EventLog]::SourceExists($EventSource))
{
[System.Diagnostics.EventLog]::CreateEventSource($EventSource, $EventLog)
write-eventlog -logname $EventLog -Source $EventSource -eventid $EventID -entrytype $EventType -message $EventMessage
}
Else
{
write-eventlog -logname $EventLog -Source $EventSource -eventid $EventID -entrytype $EventType -message $EventMessage
}
}
Function global:VMHostTimeZone
{
<#
Purpose : Detects Time Zone of VMHost machine. Returns three letter time zone.
Example 1: VMHostTimeZone -TimeZone
**NOTE** : You MUST configure this function with additonal timezone if needed. Only EST,CST,MST, and PST are configured. Time Zone is represented in minutes minus GMT
Author : pgibson.online@hotmail.com [MSFT]
#>
param (
[parameter(Mandatory=$true,Helpmessage="Please enter the VMHost Time Zone Property. Example: -VMHostTimeZone -240")][int]$TimeZone
)
Switch ($TimeZone)
{
-240 { return "EST" }
-300 { return "CST" }
-360 { return "MST" }
-420 { return "PST" }
default { return $Null }
}
}
Function global:VMHostComplianceStatus
{
<#
Purpose : Detects the Compliance Status of a VMHost machine. Returns OverallComplianceState. Tested known status results are "NonCompliant", "Unknown", and "Compliant".
Example 1: VMHostComplianceStatus -VMHostName "MyVMHost"
**NOTE** :
Author : pgibson.online@hotmail.com [MSFT]
#>
param (
[parameter(Mandatory=$true,Helpmessage="Please enter the VMHost Name as string. Example: VMHostComplianceStatus -VMHostName ""MyVMHost""")][string]$VMHostName
)
$VMHost = Get-SCVMHost -ComputerName $VMHostName
$Compliance = Get-SCComplianceStatus -VMMManagedComputer $VMHost.ManagedComputer
return $Compliance.OverallComplianceState.ToString()
}
Function global:UpdateRemediationBaseline
{
<#
Purpose : Updates a Baseline with a VMHost machine.
Example 1: UpdateRemediationBaseline -VMHostName "MyVMHost" -Baseline "MyBaseline"
**NOTE** :
Author : pgibson.online@live.com [MSFT]
#>
param (
[parameter(Mandatory=$true,Helpmessage="Please enter the Cluster Host Name as string. Example: UpdateRemediationBaseline -VMHostName ""MyVMHost"" -Baseline ""MyBaseline""")][string]$ClusterName,
[parameter(Mandatory=$true,Helpmessage="Please enter the Remediation Baseline Name as string. Example: UpdateRemediationBaseline -VMHostName ""MyVMHost"" -Baseline ""MyBaseline""")][string]$Baseline
)
$Cluster = Get-SCVMHostCluster -Name $ClusterName
$RemediationBaseline = Get-SCBaseline -Name $Baseline
Set-SCBaseline -Baseline $RemediationBaseline -AddAssignmentScope $Cluster -EA 0
}
####### End Functions ###############
######## Start Main ###########
$ErrorActionPreference = "Stop"
WriteEvent -EventID 100 -EventMessage "Start Host Cluster Remediation Process."
# Set the Approved Time Zone property for the Host Cluster to be Remediated
$global:ApprovedTimeZone = "EST"
# Set Baseline for Compliance Scan
$global:ApprovedBaseline = "Pilot"
Try
{
$HostClusters = Get-SCVMHostCluster | where { ($_.ClusterName -eq "HVCLSRV01")}
ForEach ($HostCluster in $HostClusters)
{
WriteEvent -EventID 100 -EventMessage "Processing Host Cluster $HostCluster."
#Generate Remediation JobGroup SCJob GUID
$JobGroupGUID = [System.Guid]::NewGuid()
WriteEvent -EventID 100 -EventMessage "The SCVMM Host Cluster Remediation Job GUID for Host Cluster $HostCluster is $JobGroupGUID."
$Cluster = Get-SCVMHostCluster -Name $HostCluster.ToString()
WriteEvent -EventID 100 -EventMessage "Setting Baseline Assignment on Host Cluster $HostCluster to ensure Baseline is set for possible Remediation."
### NOTE: THIS WILL CREATE A JOB ERROR IS THE CLUSTER IS ALREADY ASSIGNED TO THE BASELINE. NO HARM DONE IF CLUSTER ALREADY EXISTS, BUT A FAILED JOB STATUS WILL BE PRODUCED ###
UpdateRemediationBaseline -ClusterName $HostCluster.ToString() -Baseline $ApprovedBaseline
#Get all Nodes in Cluster
$HostClusterNodes = $HostCluster.Nodes
ForEach ($Node in $HostClusterNodes)
{
$VMHost = Get-SCVMHost -ComputerName $Node -VMHostCluster $Cluster
WriteEvent -EventID 100 -EventMessage "Processing Node $VMHost of Host Cluster $HostCluster."
WriteEvent -EventID 100 -EventMessage "Verifying time zone of Node $VMHost of Host Cluster $HostCluster."
#Detect TimeZone of $VMHost
If ((VMHostTimeZone -TimeZone $VMHost.TimeZone) -eq $ApprovedTimeZone)
{
WriteEvent -EventID 100 -EventMessage "Node $VMHost of Host Cluster $HostCluster is in the correct time zone for remediation."
WriteEvent -EventID 100 -EventMessage "Detecting Compliance Status of Node $VMHost of Host Cluster $HostCluster."
If ((VMHostComplianceStatus -VMHostName $VMHost.Name.ToString()) -ne "Compliant")
{
WriteEvent -EventID 100 -EventMessage "Begin VM to VMHost pairing on Node $VMHost of Host Cluster $HostCluster."
##### - Start VM to VMHost Pairing Sequence - ###### # This will write the VMHost name in the Custom1 field property of the VM.
# This is used in a later process to rebalance the Host Cluster after remediation has taken place.
ForEach ($VM in $VMHost.VMs)
{
WriteEvent -EventID 100 -EventMessage "Setting Custom Field 1 on VM $VM on Node $VMHost of Host Cluster $HostCluster"
$CustomProperty1 = Get-SCCustomProperty -Name "Custom1"
Set-SCCustomPropertyValue -InputObject $VM -CustomProperty $CustomProperty1 -Value $VMHost.Name.ToString()
}
##### - End VM to VMHost Pairing Sequence - ######
WriteEvent -EventID 100 -EventMessage "Completed VM to VMHost pairing for Node $VMHost of Host Cluster $HostCluster"
WriteEvent -EventID 100 -EventMessage "Begin VM DVD Drive Unmount Sequence for each VM on Node $VMHost of Host Cluster $HostCluster"
##### - Start VM DVD Drive Unmount Sequence - ###### # Remediation will fail if VMs have a local path ISO mounted. ex. C:\Windows\System32\vmguest.iso
$VMsWithDVDMounted = Get-VMHost $VMHost | Get-VM | Get-VirtualDVDDrive | where {$_.ISO -ne $null}
If ($VMsWithDVDMounted.Count -gt 0)
{
ForEach ($VM in $VMsWithDVDMounted)
{
#$VM = Get-SCVirtualMachine -Name $strVMName
WriteEvent -EventID 100 -EventMessage "Unmounting DVD drive on VM $VM.Name"
$DVDDrive = Get-SCVirtualDVDDrive -VM $VM.Name.ToString() | where { $_.Bus -eq 1 -and $_.LUN -eq 0 }
Set-SCVirtualDVDDrive -VirtualDVDDrive $DVDDrive -NoMedia -EA 0
WriteEvent -EventID 100 -EventMessage "Completed Unmounting DVD drive on VM $VM.Name"
}
}
##### - End VM DVD Drive Unmount Sequence - ######
WriteEvent -EventID 100 -EventMessage "Completed VM DVD Drive Unmount Sequence for each VM on Node $VMHost of Host Cluster $HostCluster"
WriteEvent -EventID 100 -EventMessage "Adding Cluster Node Job Task for Node $VMHost of Host Cluster $HostCluster."
#Add Cluster Node Job Tasks to Job Group
Start-SCUpdateRemediation -JobGroup $JobGroupGUID.ToString() -VMHost $VMHost -VMHostCluster $Cluster
WriteEvent -EventID 100 -EventMessage "Finished Adding Cluster Node Job Task for Node $VMHost of Host Cluster $HostCluster."
}
Else
{
WriteEvent -EventID 100 -EventMessage "Node $VMHost of Host Cluster $HostCluster is currently Compliant. No Remediation will be done at this time."
}
}
Else
{
WriteEvent -EventID 100 -EventMessage "Node $VMHost of Host Cluster $HostCluster is NOT in the correct time zone for remediation. No Remediation will be done at this time."
}
}
WriteEvent -EventID 100 -EventMessage "Starting SCJob Remediation for Host Cluster $HostCluster."
Start-SCUpdateRemediation -BypassMaintenanceModeCheck -JobGroup $JobGroupGUID.ToString() -RunAsynchronously -StartNow -UseLiveMigration -VMHostCluster $Cluster
WriteEvent -EventID 101 -EventMessage $JobGroupGUID
#Clear JobGroupGUID variable
WriteEvent -EventID 100 -EventMessage "Clearing Job Group GUID variable for Host Cluster $HostCluster."
Clear-Variable -Name JobGroupGUID
WriteEvent -EventID 100 -EventMessage "Cleared Job Group GUID variable for Host Cluster $HostCluster."
}
}
Catch
{
Throw $_.Exception
WriteEvent -EventID 200 -Type "Error" -EventMessage "Host Cluster Remediation Process Failed. Error message: $_.Exception.Message"
}
Finally
{
WriteEvent -EventID 100 -EventMessage "Completed Host Cluster Remediation Process."
}

Using System Center 2012 for Host Cluster Remediation (Patching) and Host Cluster VM Rebalancing for Private Cloud – Part 1 of Many (Host Remediation)

Prerequisites

The Remediation Script Breakdown

Additional resources