Automated Disaster Recovery Testing and Failover with Hyper-V Replica and PowerShell 3.0 for FREE!

Hyper-V Replica is a new built-in feature of both Windows Server 2012 and our FREE Hyper-V Server 2012 products.  Hyper-V Replica enables Hyper-V hosts or clusters to enable distance replication of running VMs to remote Hyper-V hosts over a standard IP WAN connection.  It provides a very cost-effective disaster recovery solution in the event of a primary data center outage. To get this feature in the VMware world, we'd pay gobs of extra money to license "Site Recovery Manager", but in Hyper-V it's just included as part of the core feature set - no extra cost.

To learn more about Hyper-V Replica, Kevin Remde, my friend and colleague, has a great new post over on his "Full of I.T." blog.  Be sure to check it out!

How Can I Automate Disaster Recovery and DR Testing with Hyper-V Replica?

There's two ways to manage the Hyper-V Replica feature:

  • GUI - We can use the built-in Hyper-V Manager tool to manage Hyper-V Replica settings, enable VMs for replication, and execute planned failovers, unplanned failovers and test failovers ... but, being a GUI tool, it's intended for interacting with Hyper-V Replica on an individual VM-by-VM basis.
     
  • Script - PowerShell 3.0 (of course - what else?!) includes 22 Cmdlets for configuring, enabling, monitoring, and managing Hyper-V Replica on an automated basis. 

Since most organizations have many VM's and often want an automated solution for managing Disaster Recovery, the basic components of a scripted solution is what we'll focus on this article.

Enabling Hyper-V Replica on each Hyper-V Host

The first step to leveraging Hyper-V Replica is enabling it on each Hyper-V Host.  With PowerShell 3.0, this can easily be done using the "Set-VMReplicationServer " Cmdlet as follows:

Set-VMReplicationServer -ComputerName HYPERV_HOST -ReplicationEnabled $true -AllowedAuthenticationType Kerberos -ReplicationAllowedFromAnyServer $true -DefaultStorageLocation DRIVE:\FOLDER

If you've got Windows Firewall enabled on each of your Hyper-V hosts, you'll also need to make sure that the Hyper-V Replica inbound port is open on each Hyper-V host. You can do this using the "Enable-NetFirewallRule" PowerShell 3.0 Cmdlet, which replaces the now deprecated "netsh advfirewall" command line tool.

Enable-NetFirewallRule -displayname "Hyper-V Replica HTTP Listener (TCP-In)”

Enabling Replication for each Virtual Machine

After Hyper-V Replica is enabled on each Hyper-V host, the next step is to enable replication for each Virtual Machine currently running on your primary Hyper-V hosts.  To do this, you can use the "Enable-VMReplication" PowerShell 3.0 Cmdlet as follows:

Enable-VMReplication -VMName VM_NAME -ReplicaServerName DEST_HYPERV_HOST -ReplicaServerPort 80 -AuthenticationType Kerberos -CompressionEnabled $true -RecoveryHistory 0

Note that once VM replication is enabled, you can then use the "Set-VMReplication" PowerShell Cmdlet should you need to modify any of the initial replication settings.

Checking in on Replication

Once a VM is enabled for replication, you can check on the replication status by using the "Measure-VMReplication" PowerShell cmdlet as follows:

Measure-VMReplication -VMName VM_NAME

Name State Health LReplTime PReplSize(M) AvgLatency AvgReplSize(M) SuccReplCount
---- ----- ------ --------- ------------ ---------- -------------- -------------
Win8Ent01 Replicating Normal 10/5/2012 9:57:31 AM 0.0039 00:08:41 3,691.15 3 of 3 

In the Cmdlet output above, we see the state (Replicating), the overall Heatlh and Last Replication Time, as well as some useful insights for WAN capacity planning to support replication: AvgLatency and AvgReplSize (in MB). To use these statistics for planning your Recovery Point Objective ( RPO ) and bandwidth requirements between hosts, check out this article posted by one of my peer IT Pro Technical Evangelists, Tommy Patterson.

Testing VM Failover

Testing Disaster Recovery Failover with traditional solutions is usually a painful process - to test failover procedures, normally you have to completely failover your production workloads after-hours to your DR site and then, often times, spend the entire remainder of your maintenance window trying to fail everything back to production before users come back to work ... Not so with Hyper-V Replica! Hyper-V provides the ability to test failover at any time by using the "Start-VMFailover" cmdlet with the -AsTest parameter.  Here's an example:

Start-VMFailover -ComputerName DEST_HYPERV_HOST -VMName VM_NAME -AsTest

When testing failover using the Cmdlet above, Hyper-V will create a copy of the replicated VM that is disconnected from all virtual networks.  This allows you to start that VM and verify that all services are running properly, without risk of exposing the replicated VM to your production network and causing potential network conflicts.  Cool stuff!

Checking for Data Center Connectivity

Hyper-V Replica is intended as a data center, or site-wide, recovery solution. As such, we'd normally want to execute a failover in situations where the primary data center or Hyper-V replica server is no longer reachable.  To test for Hyper-V Replica connectivity between the primary and secondary sites, we can create a PowerShell function that leverages the "Test-VMReplicationConnection" Cmdlet as follows:

Function PrimarySiteAvailable {

Param ([string]$HyperVHost)

$Test = Test-VMReplicationConnection -AuthenticationType Kerberos -ReplicaServerName $HyperVHost -ReplicaServerPort 80 -ErrorAction SilentlyContinue

If ( $Test -match "was successful") {

Return $True

}

Else {

Return $False 

}

}

Now, we have a function we can call from the remote Hyper-V Replica host periodically to test connectivity to the primary Hyper-V Replica host and site as follows:

$IsPrimarySiteUp = PrimarySiteAvailable -HyperVHost PRIMARY_HYPERV_HOST

If ($IsPrimarySiteUp -eq $False) { ... Insert Failover or Notification Commands Here ... }

Of course, we could get a lot more sophisticated with our code to test connectivity by including additional support for multiple retries, timing logic, running as a scheduled task, etc ... but this example is sufficient to demonstrate the basic capabilities.

CAUTION! True automated datacenter failover can be tricky to implement due to potential "Split Brain" scenarios with some application configurations.  Be sure that you test your logic to account for these conditions with the applications that you use in your environment.  Many organizations stop just short of fully automating the failover process and instead have a notification sent to the Admin team when a primary site outage is detected.  The Admin team can then triage the situation and determine if a failover is warranted.  To speed the failover process, the Admin team can then choose to use a script containing the Cmdlets below for each VM requiring failover.

At last! Performing Scripted Failover ...

We almost there .... we just need to add the actual Cmdlets that perform VM failover into a PowerShell script to automate the failover process when we invoke it.  We've already seen the needed Cmdlet above when we were testing failover - our old friend "Start-VMFailover".  We'll just remove the -AsTest parameter to perform a real unplanned failover.  Here's an example:

$VM = Start-VMFailover -ComputerName DEST_HYPERV_HOST -VMName VM_NAME -PassThru -Confirm:$false

Start-VM -VM $VM

What's Next?

In this article, we walked through examples of the PowerShell 3.0 Cmdlets that you can leverage to easily implement an enterprise disaster recovery solution using Windows Server 2012 and Hyper-V Replica.  To continue your learning, check out these resources next ...

  • Hyper-V Replica: Learn more about Hyper-V Replica in this post on Kevin's blog!
  • PowerShell: Get up to speed on PowerShell 3.0 in this post on Matt's blog!
  • Server Management: Interested in learning more about server management in Windows Server 2012?  Check out this post on Brian's blog!
  • Want to Get Certified on Windows Server 2012? Become an "Early Expert" for FREE at https://EarlyExperts.net!

How Are You Using Hyper-V Replica?

Have you found a unique and interesting use case for Hyper-V Replica and/or PowerShell 3.0?  Be sure to share your story in the comments below!

HTH,

Keith