Hyper-V authentication in Windows Server 2016 for managing remote Hyper-V servers through RSAT

Hello, my name is Ayushie Bhat and I’m a Support Engineer with the Windows Core team.

Today, I’m going to talk about the authentication in Windows Server 2016 and discuss the case of creating a VM on a CORE server from a different graphical user interface (GUI) server through Hyper-V Manager in Windows Server 2016.  In this blog, we have taken the example of a CORE server managed through a remote GUI server because that is how a variety of customers secure their environment. However, the content of this blog is applicable to all the following scenarios on OS Server 2016: GUI server managed by another GUI server, CORE server managed by remote GUI server, NANO server managed by remote GUI server or any of these servers managed by Windows 10 RSAT tools.

Now let’s discuss more about the case mentioned above of a 2016 Hyper V Core server being managed remotely through 2016 GUI server: The VM creation works fine on the CORE server locally using PowerShell but when we try to create a VM from remote GUI server, the VM creation fails with error:

‘The server encountered an error while creating <virtual_machine_name>. Failed to create a new Virtual machine.’

Below is a screenshot of the error:

1

 

The error is seen when we try to store the VM files on a local share on the GUI server or on the SOFS share in the GUI server cluster or even locally on the CORE server.

CORE server used in lab: CORE2016
GUI server used in lab: 2016GUI

Upon checking the Hyper-V event logs on the CORE server, we notice below errors:

Log Name:      Microsoft-Windows-Hyper-V-VMMS-Admin
Source:        Microsoft-Windows-Hyper-V-VMMS
Event ID:      13000
Level:         Error
Description:
User ‘XXXX\Administrator’ failed to create external configuration store at <SOFS share>\SMB-01-TEST: General access denied error. (0x80070005)

Log Name:      Microsoft-Windows-Hyper-V-VMMS-Admin
Source:        Microsoft-Windows-Hyper-V-VMMS
Event ID:      16350
Level:         Error
Description:
Cannot change the storage location of checkpoints for ‘SMB-01-TEST’ to <SOFS share>\SMB-01-TEST: General access denied error (0x80070005) (Virtual machine ID XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX). Ensure the path is valid and the directory exists. If the directory is remote, ensure that it is properly configured for sharing, and that the current user and computer accounts have read/write access. If the virtual machine is managed by a failover cluster, ensure that the file is located at a path that is accessible to other nodes of the cluster.

Log Name:      Microsoft-Windows-Hyper-V-VMMS-Admin
Source:        Microsoft-Windows-Hyper-V-VMMS
Event ID:      16352
Level:         Error
Description:
Failed to change Smart Paging store for virtual machine ‘SMB-01-TEST’ to <SOFS share>\SMB-01-TEST\:General access denied error (0x80070005). (Virtual machine ID XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX)

Notice that all the errors giving Access Denied messages. Even if Everyone is given Full Control on the SOFS share (both share and NTFS permissions), the error continues. Another curious fact is that when you use a Windows Server 2012 R2 machine as the GUI server for storing the VM files and creating the VM remotely through Hyper-V manager, the VM creation is successful.

Hence to track the root cause of the problem, we captured a network trace while reproducing the issue and found that CORE2016 was requesting the cifs/2016GUI service and it was failing with error “KDC_ERR_BADOPTION”. On the domain controller in the Active Directory Users and Computers snap-in, we added the Service Principle Name (SPN) cifs/2016GUI in the delegation properties of CORE2016.

The delegation option selected was Trust this computer for delegation to the specified services only “Kerberos Only”. After this, the VM creation still failed with the same error.

After doing some research, we found that there has been a change in Hyper-V remoting.  Between Windows Server 2012R2 and Windows Server 2016, we shifted from using the Hyper-V WMI Provider *v1* over *DCOM* to the Hyper-V WMI Provider *v2* over *WinRM*. It unifies Hyper-V remoting with other Windows remoting tools (e.g. PowerShell Remoting).  This change matters for constrained delegation because:

  • WinRM runs as NETWORK SERVICE, while the Virtual Machine Management Service (VMMS) runs as SYSTEM.
  • The way WinRM does inbound authentication stores the nice, forwardable Kerberos ticket in a location that is unavailable to NETWORK SERVICE.

The net result is that WinRM cannot access the forwardable Kerberos ticket, and VM creation fails on Windows Server 2016.  To get around this, the option is to change the delegation from “Kerberos only” to “Use any authenticated protocol”.

2

This would ensure that you do not have to depend on Kerberos only.  VM creation should work successfully after this.

To summarize, the steps to be implemented are:

  1. Enable Delegation Trust this computer for delegation to specified services only “Use any authentication protocol” on the core server object (CORE2016) in Active Directory.
  2. Add SPN cifs/2016GUI [GUI server]
  3. Restart the server for the changes to take effect or run gpupdate /force on all servers involved.

If the VMs are to be stored on a SOFS share in the GUI server cluster and the VM hosted on the CORE server cluster, we should enable the delegation on both CORE cluster nodes objects in AD to add SPN cifs/SOFS2016 <SOFS distributed name>. This will allow VM creation on the CORE servers with files stored on the SOFS share and you can create the VM from any of the GUI cluster nodes. Below are the steps for that:

  1. Enable Delegation Trust this computer for delegation to specified services only “Use any authentication protocol” on the core server object (CORE2016) in Active Directory. Do this for all CORE cluster nodes.
  2. Add SPN cifs/SOFS2016 [SOFS network name]
  3. Restart the server for the changes to take effect or run gpupdate /force on all servers involved.

3

Once these steps are followed, VM creation should be successful.

Thanks,
Ayushie Bhat
Support Engineer | Windows Core