Fighting Hyper-V Physical Host Time Drift – Utilizing NTP to Synchronize Clocks
My team at Microsoft has an engineering lab, one of which, I’ve spent a lot of time designing and building that provides our team some robustness. We run our own “mini” lab for our engineering purposes that runs very much like a large number of enterprises. By this, I mean we have our own self-hosted domain, DNS, DHCP, and application services such as ConfigMgr 2007, PKI, IIS, and SQL. This lab started off running Windows Server 2008 Hyper-V physical hosts (Dell PE2950) that were then upgraded (in-place) to Windows Server 2008 Beta and RC1. Recently, these physical hosts were cleaned and upgraded to a fresh Server 2008 R2 RTM and we used VMM to migrate our VMs off the hosts prior to doing so. The upgrade went well though there were some snags that I thought I just might share…
SCENARIO
You have n number of Hyper-V hosts running Windows Server 2008 with various VMs living on them that are “production” like. Because RTM doesn’t offer Live Migrations, you will fail to be able to migrate VMs off the hosts without some amount of downtime though SAN transfers are much quicker. When I can’t purchase new hardware but I want to take advantage of R2 features, how might I do this?
ANSWER
Over the next few days, I will spend sometime talking about this very scenario and how I did it as a way to maybe share with folks some coolness…
Kerberos & Time Drift – Like mixing Oil & Water
Issue #1: I’ve started the migration and I’m starting to get complaints that virtual machines are offline or unavailable. However, when checking the VMs on the physical hosts they are up and running and seem alive. They can ping, etc. but no clients can utilize services offered on these machines.
We ran into this issue and the root cause was due to the fact that the Hyper-V hosts time was drifting. The physical hosts are a member of a domain that is comprised of virtual machine’s that are the DCs which isn’t reality in organizations but due to this they (the DCs) didn’t have good time. This problem occurs due to the default settings for the integration services that have the virtual machines get their time from the physical hosts where they are running.
Due to this, you can disable this if you like but not one that I would recommend.
W32TIME SERVICE & AUTHORITIVE TIME SERVER
As many should already know, Kerberos authentication’s documentation for Windows says that a difference of 5 minutes by any two machines who attempt to authenticate using Kerberos will fail. The hosts were drifting utilizing the default settings for the W32time service is to use NT5DS (Active Directory). This causes issues in some cases, as we found out.
This issue can be fixed with these steps on your physical hosts:
- Open Notepad on the server
- Copy the following into Notepad -
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time]
"DisplayName"="@%SystemRoot%\\system32\\w32time.dll,-200"
"ImagePath"=hex(2):25,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,\
74,00,25,00,5c,00,73,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,73,\
00,76,00,63,00,68,00,6f,00,73,00,74,00,2e,00,65,00,78,00,65,00,20,00,2d,00,\
6b,00,20,00,4c,00,6f,00,63,00,61,00,6c,00,53,00,65,00,72,00,76,00,69,00,63,\
00,65,00,00,00
"Description"="@%SystemRoot%\\system32\\w32time.dll,-201"
"ObjectName"="NT AUTHORITY\\LocalService"
"ErrorControl"=dword:00000001
"Start"=dword:00000003
"Type"=dword:00000020
"ServiceSidType"=dword:00000001
"RequiredPrivileges"=hex(7):53,00,65,00,41,00,75,00,64,00,69,00,74,00,50,00,72,\
00,69,00,76,00,69,00,6c,00,65,00,67,00,65,00,00,00,53,00,65,00,43,00,68,00,\
61,00,6e,00,67,00,65,00,4e,00,6f,00,74,00,69,00,66,00,79,00,50,00,72,00,69,\
00,76,00,69,00,6c,00,65,00,67,00,65,00,00,00,53,00,65,00,43,00,72,00,65,00,\
61,00,74,00,65,00,47,00,6c,00,6f,00,62,00,61,00,6c,00,50,00,72,00,69,00,76,\
00,69,00,6c,00,65,00,67,00,65,00,00,00,53,00,65,00,53,00,79,00,73,00,74,00,\
65,00,6d,00,54,00,69,00,6d,00,65,00,50,00,72,00,69,00,76,00,69,00,6c,00,65,\
00,67,00,65,00,00,00,00,00
"FailureActions"=hex:80,51,01,00,00,00,00,00,00,00,00,00,03,00,00,00,14,00,00,\
00,01,00,00,00,60,ea,00,00,01,00,00,00,c0,d4,01,00,00,00,00,00,00,00,00,00[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\Config]
"FrequencyCorrectRate"=dword:00000004
"PollAdjustFactor"=dword:00000005
"LargePhaseOffset"=dword:02faf080
"SpikeWatchPeriod"=dword:00000384
"LocalClockDispersion"=dword:0000000a
"HoldPeriod"=dword:00000005
"PhaseCorrectRate"=dword:00000001
"UpdateInterval"=dword:00007530
"EventLogFlags"=dword:00000002
"AnnounceFlags"=dword:0000000a
"TimeJumpAuditOffset"=dword:00007080
"MinPollInterval"=dword:0000000a
"MaxPollInterval"=dword:0000000f
"MaxNegPhaseCorrection"=dword:ffffffff
"MaxPosPhaseCorrection"=dword:ffffffff
"MaxAllowedPhaseOffset"=dword:0000012c[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\Parameters]
"ServiceDll"=hex(2):25,00,73,00,79,00,73,00,74,00,65,00,6d,00,72,00,6f,00,6f,\
00,74,00,25,00,5c,00,73,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,\
77,00,33,00,32,00,74,00,69,00,6d,00,65,00,2e,00,64,00,6c,00,6c,00,00,00
"ServiceMain"="SvchostEntry_W32Time"
"ServiceDllUnloadOnStop"=dword:00000001
"Type"="NTP"
"NtpServer"="time.windows.com,0x9"[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\Security]
"Security"=hex:01,00,04,80,84,00,00,00,90,00,00,00,00,00,00,00,14,00,00,00,02,\
00,70,00,05,00,00,00,00,00,14,00,fd,01,02,00,01,01,00,00,00,00,00,05,12,00,\
00,00,00,00,18,00,ff,01,0f,00,01,02,00,00,00,00,00,05,20,00,00,00,20,02,00,\
00,00,00,14,00,8d,01,02,00,01,01,00,00,00,00,00,05,04,00,00,00,00,00,14,00,\
8d,01,02,00,01,01,00,00,00,00,00,05,06,00,00,00,00,00,14,00,9d,01,02,00,01,\
01,00,00,00,00,00,05,13,00,00,00,01,01,00,00,00,00,00,05,12,00,00,00,01,01,\
00,00,00,00,00,05,12,00,00,00[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TimeProviders]
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TimeProviders\NtpClient]
"DllName"=hex(2):25,00,73,00,79,00,73,00,74,00,65,00,6d,00,72,00,6f,00,6f,00,\
74,00,25,00,5c,00,73,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,77,\
00,33,00,32,00,74,00,69,00,6d,00,65,00,2e,00,64,00,6c,00,6c,00,00,00
"Enabled"=dword:00000001
"InputProvider"=dword:00000001
"AllowNonstandardModeCombinations"=dword:00000001
"CrossSiteSyncFlags"=dword:00000002
"ResolvePeerBackoffMinutes"=dword:0000000f
"ResolvePeerBackoffMaxTimes"=dword:00000007
"CompatibilityFlags"=dword:80000000
"EventLogFlags"=dword:00000001
"LargeSampleSkew"=dword:00000003
"SpecialPollInterval"=dword:00000900
"SpecialPollTimeRemaining"=hex(7):00,00[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TimeProviders\NtpServer]
"DllName"=hex(2):25,00,73,00,79,00,73,00,74,00,65,00,6d,00,72,00,6f,00,6f,00,\
74,00,25,00,5c,00,73,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,77,\
00,33,00,32,00,74,00,69,00,6d,00,65,00,2e,00,64,00,6c,00,6c,00,00,00
"Enabled"=dword:00000001
"InputProvider"=dword:00000000
"AllowNonstandardModeCombinations"=dword:00000001
"EventLogFlags"=dword:00000000
"ChainEntryTimeout"=dword:00000010
"ChainMaxEntries"=dword:00000080
"ChainMaxHostEntries"=dword:00000004
"ChainDisable"=dword:00000000
"ChainLoggingRate"=dword:0000001e[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TimeProviders\VMICTimeProvider]
"Enabled"=dword:00000001
"InputProvider"=dword:00000001
"DllName"=hex(2):25,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,\
74,00,25,00,5c,00,53,00,79,00,73,00,74,00,65,00,6d,00,33,00,32,00,5c,00,76,\
00,6d,00,69,00,63,00,74,00,69,00,6d,00,65,00,70,00,72,00,6f,00,76,00,69,00,\
64,00,65,00,72,00,2e,00,64,00,6c,00,6c,00,00,00[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TimeProviders\VMICTimeProvider\Parameters]
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TriggerInfo]
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TriggerInfo\0]
"Type"=dword:00000003
"Action"=dword:00000001
"GUID"=hex:ba,0a,e2,1c,51,98,21,44,94,30,1d,de,b7,66,e8,09[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W32Time\TriggerInfo\1]
"Type"=dword:00000003
"Action"=dword:00000002
"GUID"=hex:6e,51,af,dd,c2,58,66,48,95,74,c3,b6,15,d4,2e,a1
- Save it as useNTPTime.reg
- Double-click on the registry file to import
- Open a CMD prompt
- Type net stop w32time && net start W32time
NOTE: This step modifies the registry. Please assume all normal precautions such as backing up the registry or exporting the original values as a method to roll back if something drastically fails.
Woo.. Life is good again.
-Chris