Logon Failures Involving Virtual Machines in Windows Server 2012

Welcome back to the CORE Team blog. The General Availability date for Windows 8 and Windows Server 2012 has come and gone, and we here on the CORE Team expect more of you will be diving in and taking part in all of the excitement around these new products. To make sure you have a great experience, we endeavor, whenever possible, to make you aware of situations that may temporarily 'inconvenience' you. That is the purpose of this blog.

We have recently encountered several instances where a specific group policy configuration can affect the proper functioning of a Windows Server 2012 Hyper-V virtualization solution. This is due primarily to changes made in Windows Server 2012 Hyper-V functionality that will also be explained here.

The two scenarios we have seen thus far are:

  • Virtual machines failing to start
  • Virtual machines failing to live migrate

In both of these scenarios, the problem is the result of a logon failure. Here is an example of a pop-up error message you may see when a virtual machine fails to start.

clip_image002

The critical piece of the error message is, "Logon failure: the user has not been granted the requested logon type at this computer (0x80070569.) "

In the second scenario where a virtual machine fails to live migrate, an Event ID 21502 error message is registered in the Hyper-V-High-Availability log. The critical piece of the error message is, "Failed to create Planned Virtual Machine at migration destination. Logon failure: the user has not been granted the requested logon type at this computer (0x80070569.) "

Investigation of these events revealed that a custom Group Policy was modifying the user accounts that are allowed to Logon on as a Service on each Hyper-V server.

In Windows Server 2012, a special security group, NT VIRTUAL MACHINE\Virtual Machines is created when the Hyper-V Role is installed. Members of this group require the right to Create Symbolic Links (SeCreateSymbolicLinkPrivilege) and to Log on as a Service (SeServiceLogonRight). The SID associated with the group is S-1-5-83-0. The security group is maintained by the Hyper-V Management Service (VMMS). To ensure members of the NT VIRTUAL MACHINE\Virtual Machines security group maintain the rights they need, VMMS registers with Group Policy in order to update the local security policy whenever Group Policy is refreshed.

The NT VIRTUAL MACHINE\Virtual Machines group did not exist in previous versions of Hyper-V. As each virtual machine is started on a Hyper-V server, its account (Virtual Machine ID (VM_ID)) is added to the NT VIRTUAL MACHINE\Virtual Machines group and VMMS creates a Virtual Machine Worker Process (vmwp.exe). Examples of these processes are visible in Task Manager:

 

clip_image004

 

The VM_ID is the virtual machine account that is used to gain access to its own resources and prevent other virtual machines from gaining access to those same resources. As an example, if I run the following PowerShell command, it is easy to see the rights given to the virtual machine account to one of its resources (a virtual hard disk in this case):

 

Get-Acl -Path E:\Virtual Machines\Contoso-FS1\Virtual Hard Disks\contoso-fs1.vhdx | FL AccessToString

AccessToString : NT VIRTUAL MACHINE\E57917F3-31C3-456E-B1BA-5E45B4CC7E0C Allow Write, Read, Synchronize

BUILTIN\Administrators Allow FullControl

NT AUTHORITY\SYSTEM Allow FullControl

NT AUTHORITY\Authenticated Users Allow Modify, Synchronize

BUILTIN\Users Allow ReadAndExecute, Synchronize

 

Since the VMWP is an extension of VMMS, VMMS performs a service logon to create an access token that is used to run the VMWP. In order for this to work, the NT VIRTUAL MACHINE\Virtual Machines security group must be granted the Log on as a Service right. In previous versions of Hyper-V, the VMWP ran in the context of a different account, NETWORK SERVICE, which is an account defined by SYSTEM.

Windows Server 2008 R2 SP1 Hyper-V Server

clip_image006

To find out more information about the NETWORK SERVICE account, review this MSDN resource (https://msdn.microsoft.com/en-us/library/windows/desktop/ms684272(v=vs.85).aspx).

The error message, previously mentioned, refers to a 'user' not being granted a 'logon type'. That user, again as seen in Task Manager, is the Virtual Machine ID (VM_ID), and the logon type is 'Log on as a Service.'

clip_image008

 

Now that we understand the new changes, what needs to be done? A detailed Knowledge Base (KB) article was written in cooperation with the Directory Services team that provides additional details.

KB2779204
Starting or Live Migrating Hyper-V virtual machines may fail with error 0x80070569 on Windows Server 2012-based computers
https://support.microsoft.com/kb/2779204

Briefly, one of two things must happen:

  1. Hyper-V Administrators need to get with their Domain Administrators to review Group Policies to see if any involve specific user accounts being granted the Log on as a Service right, and, if so, have the policy modified appropriately
  2. Create an OU in Active Directory and place all hyper-V servers in that OU and block policy inheritance

Note: Option (2) is recommended by the Hyper-V Product Team

Tip: Administrators can temporarily, but quickly, recover from this error by opening an elevated command prompt and running gpupdate /force which forces a group policy refresh

Before we wrap-up, I would like to re-state one of Microsoft's long standing 'best practices' with respect to Hyper-V servers, and that is, the only Roles or Services that should ever be installed on a Hyper-V server is the Hyper-V Role and only those additional Roles or Features that directly support virtualization. The classic example is Hyper-V Failover Clusters where the Hyper-V Role and the Failover Clustering Feature complement each other by providing highly available virtualized workloads, which are the foundation of Microsoft's Cloud Strategy. If this 'best practice' is followed, no user rights modifications that could impact virtualization services should be needed.

I hope this has been helpful.

Thanks, and come back again soon.

Chuck Timon
Senior Support Escalation Engineer
Microsoft Enterprise Platforms Support
High Availability\Virtualization Team