Hi, I’m Kerim Hanif and and I work as a Program Manager on the SCVMM team. Today I am going to talk about the High Available VMM server (HA VMM) feature in SCVMM 2012 Beta.
During the evolution of the VMM server product from VMM 2007 to VMM 2008 R2 we saw many customers tying their workflows to VMM and even more using VMM to carve out and give compute resources to their internal departments. One consistent evolution fact of the product is that we see more and more VMM servers used as mission critical servers in customer environments. This fact makes VMM service’s highly availability feature very important for many customers, with this new feature VMM is able to sustain a failure on the VMM Service Level.
In previous versions VMM was unable to provide this feature and the workaround was to install VMM in a highly Available Virtual Machine, so that if something was to happen to the host, the HA VM could failover to another host in the cluster. The problem with this approach is that it doesn’t take in account the problems that could go wrong with the VMM service itself or problems with the operating system that the VMM is running on, if something goes wrong in these places, VMM service would go down.
So a huge ask from our customers was to make VMM service itself cluster aware, and in VMM 2012 version did exactly that.
Here are the pre-requisites for a HA VMM server installation:
1. Failover clustering feature added, cluster created and configured (Windows Server 2008 R2 is the minimum OS version supported as node servers)
2. Windows Automated Installation Kit (AIK) for Windows 7 installed on all nodes that will be used as VMM servers http://www.microsoft.com/downloads/en/details.aspx?FamilyID=696dd665-9f76-4177-a811-39c26d3b3b34&displaylang=en
3. Server and instance name of a SQL Server 2008 or SQL Server 2008 R2 cluster or remote SQL server (best practice to use clustered SQL Server with HA VMM servers)
4. For our DKM (Distributed Key Management) requirement, either logged on, installing VMM with an account that has “edit” permission on the Active Directory container (can be a lower level container doesn’t have to be the root) or DKM group pre-created on Active Directory and its name available to provide at setup (more on DKM requirement later)
HA VMM Installation Steps
When we were designing this feature we wanted it to be very easy and simple. Installation of VMM in an HA or standalone fashion is very similar and it is integrated into the usual standalone installation.
1. To install VMM in an HA fashion you just need to start installation of VM on one of the clustered nodes, and select install from our splash screen.
2. After accepting our EULA you will get our feature selection screen, as you can see one of the setup improvements that we did for this version of VMM is to chain the various VMM installations together.
3. Once you select VMM Server feature, we will detect that you are running this server on a failover clustering node and will offer you to start HA VMM setup instead, you will need to select YES at this dialog to start HA VMM setup. Note that it is supported to install VMM in a standalone fashion on a cluster node; all you have to do is select NO at this dialog box.
4. Once you select YES to the HA VMM opt-in question, setup will select the features that you need for this installation. In this version of VMM, regardless of its high availability aspect, it is a requirement to install VMM Console on all machines that VMM Server is installed; therefore in this dialog we will select VMM Server and VMM Console.
Another important thing to note here is that we actually do not recommend selecting Self-Service Portal during HA VMM installation, but it is allowed at Beta code, this will be fixed at RTM timeframe and we will gray out Self-Service Portal selection in this view.
5. After this page we will ask you standard questions about registration information, product key (another improvement; you can pass this product key section empty in VMM 2012), Microsoft Update configuration (if not configured previously), installation location and we finally we will come to the database configuration page.
In the database configuration page you will need to provide the server name, instance name and database name that setup will use.
There are many options here:
1. You can ask setup to create a new database (logged on user needs to have permission to create a database on the server name provided)
2. Use an existing database (if logged on user don’t have permission to create a database, database admins can pre-create an empty database and VMM can add its tables to that database during installation)
3. Provide different credentials other than logged on users credentials
Please note that as mentioned before the best practice is to use a clustered SQL server for HA VMM installations.
If you leave a port or instance name boxes empty in this page we will use the defaults for that box (e.g. if you leave port number empty we will use 1433, or if you leave the instance name empty we will use default instance on the SQL server that you provided.)
We will use the provided SQL server instance’s defaults for log and database file locations, if you like to provide different locations you can;
1. Pre-create an empty database with its log and database file locations pointing to where ever you want them to, and then provide this empty database to VMM as existing database during install.
6. After database configuration page you will come to an HA VMM specific cluster configuration page. This page will be different for different configurations, for example for IPv6 and DHCP configured servers you will not see the second portion of the page and will only provide the cluster service name.
Cluster service name here basically is the name in active directory the users and admins will use to identify this HA VMM service. When choosing this name, make sure that it is a unique name that is easy to identify the HA VMM service.
7. Another important setup page in HA VMM installation is the account configuration page.
There are two things that are mandatory in this page for HA VMM installations:
a. HA VMM server installation requires a domain account as a startup account for the VMM service. You won’t have the choice to use a local system here. It is best practice to use a dedicated domain account created just for VMM as a service account here.
b. The other mandatory place in this page is for VMM to store its encryption keys in AD.
As mentioned at the beginning of the blog we use Distributed Key Management (DKM) to let users and processes running on different machines securely share data. Once an HA VMM node fails over to another node, the VMM service on that failed over node starts accessing the VMM database and uses the encryption keys conveniently stored under a container in AD to decrypt the data that is being held securely encrypted in the VMM database.
· The AD container distinguished name that will contain DKM data needs to be written in the LDAP Data Interchange Format (LDIF) at this screen.
· If the logged on user has permission to create a container in AD then the group won’t have to be pre-created. The group name can be anything VMM admin chooses, and the container in AD doesn’t have to be a root container.
Example#1: If domain name is contoso.com and the DKM group name was decided to be “VMMDKM”, user can writer CN=VMMDKM,DC=contoso,DC=com under the DKM and since the logged on user has permission to create this container, VMM setup would create this container in contoso.com domain.
Example #2: If user has no permission to create a container in AD then he/she needs to coordinate with the AD admins to create this group and get the container’s “distinguished name” from AD admins prior to HA VMM installation. Make sure to ask AD admin to provide the following rights to the setup user;
1. Generic Write
2. Generic Read
3. Create Child
One convenient way to get this information from AD admin could be to give him/her a ready script to run in his/her environment. This way you don’t have to explain much, they would just run the script and let you know when it successfully executes.
One easy way to create an ADSI script is to use “ADSI Scriptomatic” tool to create a script; you can get this tool from http://www.microsoft.com/downloads/en/details.aspx?FamilyID=39044e17-2490-487d-9a92-ce5dcd311228&DisplayLang=en
8. Once you are done with these pages you will see VMM port selection page and after that you will see the library creation page. For HA VMM installation this page is just there for warning purposes because setup does not create a default library share after HA VMM installations.
The reason behind this is that when creating high availability for VMM servers, it is important that not only the VMM server feature but all components that constitutes VMM service are also highly available (hence the best practice recommendation for clustered SQL Server in the previous step)
After HA VMM installation, a new library server and share needs to be added to VMM . It is best practice to use a HA file server for HA VMM library server.
9. After going through the installation summary page your installation will start and in couple of minutes end with a successful installation of the first node of your HA VMM server.
10. After the first node installation you can easily add another node to this HA VMM cluster that you just created, to do that simply start the VMM setup on the second node where you want to install HA VMM.
After going through the EULA page and selecting the VMM server feature checkbox you will see a similar popup as the first node installation, but this time we will detect the HA VMM and ask “if you want to add this server as a node”. If you say YES, there will be minimum amount of pages of setup and your second node will be added. You will need to repeat this on all of the nodes that you want to add to this HA VMM installation.
Important SCVMM 2012 HA VMM Facts
a. It is a fault tolerant service feature, but this not does increase scale/performance
b. There can be as many as 16 nodes in an HA VMM installation but there can be only one node active at any time.
c. When VMM console connects it asks for a VMM server name and port number. Make sure to provide the cluster name of the HA VMM service instead of a node name here. Connecting to a node name will not be allowed.
d. You can do a planned failover (i.e. to install a patch, do maintenance to a node etc..) using failover clustering UI; there is no way to failover HA VMM service using the VMM console in this version of VMM.
e. You can only see the active node of the HA VMM service from the failover clustering UI or using Get-SCVMMServer PowerShell commandlet at beta timeframe.
There are three new parameters under Get-SCVMMServer
1. IsHighlyAvailable – True/False
2. FailoverVMMNodes – FQDN of all nodes that this HA VMM installation contains
3. ActiveVMMNode – FQDN of the active node
When you do a planned failover make sure to do following:
- Always perform inside a maintenance window that is communicated to SCVMM users. All running tasks and all connections to VMM consoles and Self-Service Portals will be stopped at failover time.
- Running jobs that failed due to the failover will not start automatically after failover. If the particular job supports restarting it will be possible to restart this job but this process will not be automatic.
- Ensure that when connecting VMM console the VMM cluster service name is used to enable reconnecting to the VMM service after planned failover.
Uninstall HA VMM Service
- To uninstall an HA VMM server, simply go to any node and manually uninstall VMM server on that node, repeat this until you come to the last node, during last node un-installation, setup will warn you that this is the last node of the HA VMM Installation and removing this node will remove the clustering resources.
- Note: You cannot uninstall HA VMM from an active node of a multi-node cluster; you will need to start from the inactive node first.