How to recover an HNV Gateway that was deployed through Microsoft VMM 2016

Author: Manish Jha | Program Manager

This post contains information about HNV gateways that are deployed through Microsoft System Center Virtual Machine Manager 2016 (VMM 2016). Please note that with Windows Server 2016 and VMM 2016, you now have an option to use Azure style gateways that are managed by Network Controller. Steps in this guide, however, focus on the HNV style gateways that could be deployed with Windows Server 2012 R2 and VMM 2012 R2 in the past. The 2012 R2 style HNV gateways can be deployed even with Windows Server 2016 and VMM 2016 since this capability is fully compatible with our latest release. Please refer here for more information on gateways.

This post describes two ways of recovering a HNV Gateway:

  • How to recover a gateway after failure of both virtual machines that support the gateway
  • How to recover a gateway after failure of one virtual machine that supports the gateway

How to recover a gateway after failure of both virtual machines that support the gateway

We will assume that you deployed HNV gateway using a VMM Service Template. This service template helps you create a pair of virtual machines on a host cluster, and together the virtual machines and the cluster help provide high availability for the gateway that runs on them.

If both virtual machines that make up the gateway fail, you can recreate the failed gateway by deploying a service template again. When you recreate the gateway, the names of the virtual machines can differ from before, and you can deploy them to a different host cluster. Other settings, such as the subnets that are specified in the network sites, must remain the same as when you originally deployed the gateway. The following procedures provide details.

How to collect information to prepare for gateway recovery after failure of both virtual machines

1. In the Fabric workspace in VMM, select Network Service in the console tree under Networking.

2. In the Network Services pane, find the listing for the gateway that you want to recover and record the name of the gateway. In this article, this name is known as GATEWAY-NAME.

3. Right-click the listing for the gateway and then click Properties.

4. Click Connectivity, then record the information that is displayed for the frontend and the backend (network adapters and network sites).

5. Close the Properties page.

6. Make sure that you downloaded the following Windows PowerShell script:

RecoverGatewayScript.psm1

Note that this script remains unchanged from VMM 2012 R2 usage.

7. Determine the MAC address of the gateway by running the following script commands. Make sure that you substitute the correct path for PATH and the name of the gateway for GATEWAY-NAME.

Import-Module PATH\GatewayRecovery.psm1
$svcName = “GATEWAY-NAME”
$hostCredential = Get-Credential
GetMacAddress $svcName $hostCredential

How to recover after failure of both virtual machines that support the gateway

1. In VMM, delete the service listing for the two virtual machines that supported the failed gateway. To do this, click Show on the Home tab in the VMs and Services workspace, then click Services. Locate the host on which the virtual machines are deployed. In the details pane, right-click the service (not the individual virtual machines), then click Delete. When you are prompted, confirm the deletion.

2. Optionally, if you are using the same virtual machine names that you used before, and you expect that the existing DNS entries will cause problems when you redeploy, arrange to have these DNS entries removed.

3. If you are using a new host cluster instead of recovering the gateway on the existing host cluster, make sure that the new hosts are configured as dedicated network virtualization gateways. To do this, follow these steps:

  • In the Fabric workspace in VMM, make sure that Fabric Resources is selected in Show on the Home tab.
  • In the Fabric pane, click Servers, expand the host group that contains the new host cluster, and then click the host cluster.
  • In the Hosts pane, right-click one of the hosts (not the host cluster), and then click Properties.
  • Click the Host Access tab, and then click to select the This host is a dedicated network virtualization gateway, so it is not available for placement of virtual machines requiring network virtualization check box. Then click OK.
  • Repeat the process on the other host.

4. As you did for the original deployment, choose the appropriate service template (2-NIC or 3-NIC) for your environment. Review the settings in the service template to make sure that they are what you want for this deployment of the gateway.

IMPORTANT When you re-create the gateway, you must specify the same subnets in the network sites that you specified for the gateway that failed. However, the names of the virtual machines can be different, and you can deploy them to a different host cluster.

5. As you did for the original deployment, use the service template to deploy the virtual machines that will support the new gateway. Make sure that you deploy them on the intended hosts.

6. Do the following verification tasks to make sure that the service deployment was successful:

  • Confirm that the back-end virtual network adapter on the gateway is not connected. (It should not be connected yet.) To do this, follow these steps:
    1. In the VMs and Services workspace in VMM, click Services in the Show group on the Home tab.
    2. Expand All Hosts, and then click the host group that the host cluster is in.
    3. In the Services pane, expand the service until you can see the gateway virtual machines.
    4. Right-click a gateway virtual machine, click Properties, and then click the Hardware Configuration tab in the properties sheet.
    5. Under Network Adapters, confirm that there are three network adapters and that one of them is labeled Not connected. Record the name of the adapter that is not connected.
  • Start the new service, and then confirm that the virtual machines enter the Running state.
  • With the virtual machines still running, open a command prompt on the VMM server as an administrator, then type PING, followed by the name or IP address of the gateway itself. Press Enter, then confirm that a response is received from the gateway. If a response is not received, review possible causes, such as DNS settings, firewall settings and the state of the gateway cluster.

7. Choose one of the new virtual machines as “primary.” Run the following Windows PowerShell commands, substituting the name of the chosen virtual machine for VMNAME:

$vm = Get-SCVirtualMachine –Name “VMNAME”
$vm.VirtualNetworkAdapters | ft Name,VMNetwork,VirtualNetwork

8. In the resulting display, look for the adapter that is not connected. For that adapter, VMNetwork and VirtualNetwork will be blank. Identify the number of that adapter as follows:

Adapter Number
First adapter that is shown in the list: 0
Second adapter that is show in the list: 1
Third adapter that is shown in the list: 2

9. Grant the MAC address that you identified in the earlier procedure to the network adapter that is not connected. To do this, run the following Windows PowerShell command. For MACADDRESS, substitute the MAC address, and for NUMBER, substitute the number (0, 1, or 2) that you identified in the previous step:

$mac = Grant-SCMACAddress -MACAddress MACADDRESS -MACAddressPool
(Get-SCMACAddressPool -Name “Default MAC address pool”) -VirtualNetworkAdapter
$vm.VirtualNetworkAdapters[NUMBER]

10. Make sure that the virtual machine is stopped, and then apply the MAC address to the network adapter that is not connected. To do this, run the following Windows PowerShell commands. For NUMBER, substitute the same number (0, 1, or 2) that you used in the previous step:

Stop-SCVirtualMachine –vm $vm
Set-SCVirtualNetworkAdapter -VirtualNetworkAdapter
$vm.VirtualNetworkAdapters[NUMBER] –EthernetAddress $mac.Address

11. Delete the old gateway configuration information so that it will not interfere with the new gateway. To do this, you will need to know the names of the hosts in the cluster that the gateway was on before it failed. (This might be the same cluster that it is currently on.) To do this, follow these steps:

  • In any workspace in VMM, click Window on the Home tab, and then click PowerShell.
  • Run the following commands. For GATEWAY-NAME, substitute the gateway name. For HOST1 and HOST2, substitute the computer names of the physical hosts that were in the host cluster when the gateway failed. Run these commands even if you are using the same cluster you were using before the gateway failed.

$svcName = “GATEWAY-NAME”
$hostCredential = Get-Credential
$gwHosts = @(“HOST1”, “HOST2”)
CleanupGatewaysBeforeMigration $svcName $gwHosts $hostCredential

12. Restart the virtual machine by running the following command:

Start-SCVirtualMachine –vm $vm

13. Update the connection string for the gateway as follows. Do this even if you are using the same host cluster and the same computer names for the virtual machines.

  • Look up the existing connection string by running the following commands:

$ns = Get-SCNetworkService -Name $svcName
$ns.ConnectionString

For example, the connection string might resemble the following:

VMHost=GW-HV-CL01.contoso.com;GatewayVM=GW-VM-CL01.contoso.com;BackendSwitch=DatacenterSwitch

  • Construct an updated version of the connection string by pasting the old string into a text editor such as Notepad and then updating it as follows (make sure that you leave the semicolons as-is).
    1. If the cluster name of the host cluster differs from what it was before, change the VMHost= setting.
    2. If the computer name of the “primary” virtual machine differs from what it was before, change the GatewayVM= setting to the new name.
  • Update the connection string by running the following command. When you run the command, substitute the correct connection string for CONNECTIONSTRING.

Make sure that you include ;Migrate=true at the end of the string inside the quotation marks, and -Force at the end after the last quotation mark.

Set-SCNetworkService -NetworkService $ns -ConnectionString
“CONNECTIONSTRING;Migrate=true” -Force

  • If the Set-SCNetworkService command reports any errors, fix them and run the command again.

14. From a tenant virtual machine that uses the gateway, test the gateway. For example, use a network command such as PING.

How to recover a gateway after failure of one virtual machine that supports the gateway

If one of the virtual machines that support the gateway is functioning correctly but the other is not, you can use capabilities that are built into VMM services to “scale out” the gateway so that it again provides redundancy. To do this, follow these steps:

1. In VMM, remove the virtual machine that no longer functions. To do this, follow these steps:

  • In the VMs and Services workspace, select the host group where you deployed the service that runs the gateway.
  • In the Show group on the Home tab, click Services.
  • In the Services pane, expand the service.
  • Right-click the virtual machine that has failed, and then click Delete.
  • When you are prompted, confirm the deletion.

2. In Failover Cluster Manager, evict the failed node from the guest cluster that supports the gateway. To do this, follow these steps:

  • Open Failover Cluster Manager, expand the cluster in the console tree, expand Nodes, and then look at the status for each node. One node will be Up, and one node will be Down.
  • Right-click the node that has a status of Down, click More Actions, and then click Evict.

3. In the Services pane, right-click the gateway service itself (not a virtual machine), then click Scale Out.

4. On the Select Tier page in the Scale Out Tier Wizard, click Next.

5. On the Identity page, enter a name for the new virtual machine and click Next.

6. On the Select Host page, make sure that you select the host that held the virtual machine that failed, then click Next.

7. If the Configure Settings page appears, enter the computer name for the new virtual machine under Operating System Settings. Make sure that this computer name is not already being used by a computer in your environment.

8. Click Next.

9. On the Add Properties page, click Next.

10. On the Summary page, review your settings, and then click Scale Out.

NOTE You can track the progress of the scale out operation in the Jobs window. The operation can take 15 minutes or longer. You can perform other tasks in the VMM console while you monitor the job.

11. After the Create virtual machine job is completed successfully, verify that the new virtual machine was added and that it is started in the VMs and Services workspace. The new node will probably start to function within about five minutes.

12. To verify that the gateway is functioning, connect to the new virtual machine, and then run Get-NetCompartment at a Windows PowerShell prompt. If multiple compartments are listed, the new virtual machine is functioning and will help provide high availability for the gateway.

Manish Jha | Program Manager | Microsoft