Update Remediation Job Fails At Reboot Computer Step in SCVMM

Recently I was working with a customer on a update remediation issue in SCVMM 2012 R2. When remediate operation is performed on the hyper v host.  Scvmm installs the updates and waits for reboot of the host. The step reboot computer use to fail. However, the hyper v host gets rebooted but still scvmm was waiting around 35 mins and eventually it times out with below error.

Error (20502)
Virtual Machine Manager failed to restart the computer mkbhost2.mkb.com.
Recommended Action
Restart the computer somtx1.hoster.cool manually.

Error (2927)
A Hardware Management error has occurred trying to contact server mkbhost2.mkb.com.
WinRM: URL: [https://somtx1.hoster.cool:5985], Verb: [INVOKE], Method: [ScanForUpdates], Resource: [https://schemas.microsoft.com/wbem/wsman/1/wmi/root/scvmm/UpdateManagement]
Unknown error (0x80338043)

Recommended Action
Check that WinRM is installed and running on server somtx1.hoster.cool. For more information use the command "winrm helpmsg hresult" and https://support.microsoft.com/kb/2742275.

To understand why scvmm was waiting and what it does during that time. I took working scvmm etl trace. You can use the below article to enable etl tracing on scvmm.

https://support.microsoft.com/en-us/help/2913445/how-to-enable-debug-logging-in-virtual-machine-manager

In the trace i could see it waits for the host to back online and then tries to connect to wmi service on the hyper v host.

With that clue i tried to test the WMI connectivity from SCVMM to hyper v host in customer environment. I used "wbemtest" utility to test it. Go to run and type "wbemtest" on scvmm server and provide the hyper v host name as below. Click on connect, it failed with below error.

Our internal networking team helped us to find out the cause. It turns out that the below updates were missing on VMM server. However, in your case the cause might be different, it may be firewall, wmi port block or missing updates. Remediation to work the scvmm should connect to wmi service on the hyper v host and vice versa.

https://support.microsoft.com/kb/3000850 - November 2014 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2                             
https://support.microsoft.com/kb/3013769 - December 2014 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2                            
https://support.microsoft.com/kb/3172614 - July 2016 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2                                
https://support.microsoft.com/kb/3179574 - August 2016 update rollup for Windows RT 8.1, Windows 8.1, and Windows Server 2012 R2                              
https://support.microsoft.com/kb/4054519 - December 12, 2017-KB4054519 (Monthly Rollup) 

After installing the above updates on scvmm server the reboot computer job succeeded but scan job was failing. So we ended up installing the above updates on hyper v host as well.