With the release of Microsoft Windows Server 2008 with Hyper-V and Microsoft Hyper-V Server 2008, a virtualized Exchange 2007 SP1 server is no longer restricted to the realm of the lab; it can be deployed in a production environment and receive full support from Microsoft. This past August, we published our support policies and recommendations for virtualizing Exchange, but many people have asked us to go beyond that guidance and weigh-in on the more philosophical question: is virtualization is a good idea when it comes to Exchange?
Due to the performance and business requirements of Exchange, most deployments would benefit from deployment on physical servers. However, there are some scenarios in which a virtualized Exchange 2007 infrastructure may allow you to realize real benefits in terms of space, power, and deployment flexibility. Presented here are sample scenarios in which virtualization may make sense, as well as checklists to help you evaluate whether the current load on your infrastructure makes it a good candidate for virtualization.
Small Office with High Availability
Some organizations are small but they still require enterprise-class availability. For example, consider Contoso Ltd., a fictitious company that regards email as a critical service and has several small branch office site(s) consisting of 250 users. Contoso wants to keep their e-mail environment on-premises for legal reasons and they want to have a fully redundant email system. Contoso's users have average user profiles and the mailboxes are sized at 2 GB.
Before Hyper-V was introduced, to get full redundancy for all Exchange server roles, Contoso would have to deploy 7 physical servers: 2 servers for AD and DNS, 1 server for file and print, 2 servers running CAS and Hub, and 2 servers running the Mailbox role in a CCR environment. The servers are assumed to have 2 quad core processors and the amount of RAM is based on the installed roles. Each CCR node would have 4 GB of RAM, and each of the other servers would have the minimum 2 GB of RAM to support the users and traffic pattern. With user profiles this size, the average load would be 35-45% of CPU on a server that has eight cores.
Flip the page to today and Contoso can give the same level of redundancy and availability with only 3 servers by using virtualization. Each physical server would run each of the roles as a Hyper-V guest. In this scenario, 3 physical servers with 2 quad core processors and 16 GB of RAM would have sufficient capacity to serve Contoso's users. Along with RAM and processors, the servers need to have multiple NICs and redundant paths for storage. Since Contoso still has the same number of Exchange servers to manage, they have not benefitted much from the O&M perspective, but think about the space, power, and HVAC impact. Each of the virtual servers would be configured with 2 virtual CPUs.
The following diagram illustrates the scenario. Note that the Hub Transport Exchange 2007 Server serving as the File Share Witness (FSW) is on a separate Hyper-V host from the CCR Mailbox Nodes to eliminate any single point of failure in the clustering solution and to provide true clustering capability.
Figure 1 - Possible Small Branch Office Design for Exchange 2007 SP1 on Hyper-V
In this scenario, you could save a possible 25,754 kWh and $22,516 per year. This information was based on having 7 physical servers and then changing to 3 physical and 7 virtual servers. The Microsoft HyperGreen Tool was used to gather these numbers:
The factors for sizing the virtualized server are not any different from sizing physical servers. No matter if you are using physical or virtualized servers, the CCR nodes will need 4GB of RAM and they would need to support 48 IOPS for the databases and 19 IOPS for the transaction logs. As you can see for this scenario, the IOPs requirements are very low and should be able to be maintained in a virtualized environment. For the small amount of users that would be hosted on the virtualized Exchange servers, you would be fine to use VHDs for the drives. It is recommended that you move to external storage if your user count is much higher than what is depicted here.
Remote or Branch Office with High Availability
In the early days of Exchange server, organizations needed to place local Exchange servers in remote and branch offices to provide sufficient performance. With improvements such as Cached Exchange Mode and Outlook Anywhere (RPC over HTTPS), consolidating those servers to a central datacenter became the recommended approach. However, in some situations, poor network connectivity to remote offices still requires some organizations to have a local Exchange server. Often the user populations at these locations are so small that it doesn't make sense to dedicate a whole physical server to the Exchange environment. The technical considerations in this scenario are the same as described in the "Small Office with High Availability" scenario above. For an example of how a company used Hyper-V in this scenario, refer to the case study on Caspian Pipeline Consortium.
In order to provide redundancy for a remote site, some organizations may require a Warm Site that contains a duplicate of the primary production Exchange 2007 infrastructure. The intent of this standby site is to provide as near to the same level of functionality as possible in the event of the loss of the primary site. However, keeping a duplicate infrastructure for standby purposes, although useful for high SLA requirements, can be prohibitively expensive for some organizations. In that event, it is possible to provide a virtual duplicate of the entire primary site using Hyper-V. A typical Warm Site configuration utilizing physical Exchange 2007 servers would include one or more servers configured together as a standby cluster and one or more other servers configured as a CAS/Hub server. To achieve redundancy of just the messaging services within the Warm Site, a total of four physical servers would be needed. By contrast, a Hyper-V-based solution with only three physical servers can provide an organization with a Warm Site that includes two Mailbox servers in a CCR environment, as well as and redundant CAS, and Hub servers. Thus, by virtualizing Exchange in this scenario, you can provide a higher level of services to your users while also saving on hardware, power and cooling costs as well as space requirements when compared to a similarly configured physical solution. The following diagram illustrates one such configuration.
Figure 2 - Possible Warm Site Disaster Recovery Configuration using Hyper-V
The Primary Site uses physical hardware due to the demanding size and profile of the user population. In this scenario, the Warm Site is designed to support the entire population of users from the Primary Site. Careful consideration must be given therefore to the configuration of a virtual environment that will support the user population, even if it will be for a temporary period and at a certain level of reduced service performance.
The diagram illustrates that the Warm Site would rely on a standby cluster, with one of the nodes configured as an SCR target, as the primary recipient of regular log copies from the primary site. A pair virtual Domain Controllers would also be in the Hyper-V environment for AD integration. The SCR target is a two-node failover cluster. In the event of a site failure, the standby cluster would be activated by using Restore-StorageGroupCopy and the CMS would be recovered by using the /recoverCMS switch. The same procedures for recovering from a disaster using a standby cluster still apply despite the fact that the standby cluster is virtualized. Once the standby cluster is online and hosting the CMS from the failed site, client access to messaging services and data will be restored once DNS and Active Directory replication has occurred.
The virtual Warm Site must be able to provide an adequate level of service to users in the event of the loss of the Primary Site with the understanding that there will probably be a reduced level of service due to the WAN/Internet link(s) to the Warm Site. However, since the site is designed to provide emergency functionality, and only for a brief period, this reduced level of service should be a reasonable expectation. It would be understood however that, while the Primary Site is down, there is site no resilience for the Warm Site.
In this scenario, you could save a possible 33,005 kWh and $28,225 per year. This information was based on having 8 physical servers and then changing to 3 physical and 8 virtual servers. The Microsoft HyperGreen Tool was used to gather the numbers above.
There are situations in which a company, agency, or governmental department may need a complete network infrastructure that can be deployed to specific locations at a moment's notice. This infrastructure is then connected to the organization's network via satellite or similar remote WAN technology. For example, a non-governmental organization (NGO) may need to react to a disaster and set up local servers to serve an affected community. This subset of servers would need to be completely self-contained and able to provide all necessary server services to the personnel located in the target location.
In such situations, the mobile Local Area Network must be easy to transport. It must have a small and highly efficient form factor, providing all of the local users' necessary services while taking up the smallest amount of space possible. Given the probable remoteness of the locations in which the Mobile LAN would be deployed, it must also provide fault tolerant capabilities to ensure that there is no single point of failure in a location where spare parts may be hard to come by.
In this scenario, a Hyper-V server can be used to host Exchange as well as file server services and domain infrastructure services in a compact form factor. Virtualization of Exchange 2007 with Service Pack 1 requires that the Hyper-V host server does not host any other IO-intensive applications installed on the host server. You can have Exchange 2007 SP1 and other applications running as VMs on the same host.
The following diagram illustrates a possible configuration for a Hyper-V server hosting the Exchange 2007 and Domain infrastructure systems. Due to the nature of this scenario, you will see that none of the Exchange server roles have been combined.
Figure 3 - Possible Mobile LAN Configuration with Exchange 2007 SP1
The diagram illustrates a comprehensive network solution for an organization that requires all necessary server services to be provided locally regardless of location. The solution is as small as possible while also allowing for a high degree of fault tolerance and system availability.
In the rack, you would also need enough network infrastructure equipment to support the Hyper-V servers and the workstations. The Hyper-V systems would use either an iSCSI or Fiber Channel Storage Area Network (SAN). The SAN should give enough spindles to provide the necessary performance for the guest systems. With this scenario, you would have everything you needed in a 42U rack.
In this scenario, you could save a possible 91,012 kWh and $73,891 per year. This information was based on having 14 physical servers and then changing to 3 physical and 14 virtual servers. The Microsoft HyperGreen Tool was used to gather the numbers above.
You'll notice that in each of the scenarios described above, if Exchange were deployed on physical infrastructure without Hyper-V, hardware resources would have been underutilized. To help you determine if your Exchange environment is a candidate for server consolidation, we've prepared the following checklists. If these checklists reveal that your hardware is not being fully utilized, you should consider the following possible actions:
- If you are a small organization, you may be able to reduce your server footprint using virtualization, down to as few as three physical servers with full redundancy of Exchange roles
- If the underutilized hardware is at a branch or remote office that cannot be consolidated to a central datacenter, you may be able to reduce your server footprint there using virtualization
- In other situations, you may want to make your Exchange environment a bit leaner by revisiting how well your server capacity matches up with your user load. You can reduce the number of physical servers in order to boost utilization to the desired levels--implementing virtualization is not required
Keep in mind that underutilized hardware is simply a signal that your Exchange environment has excess capacity. This may be by design (to accommodate usage spikes or expected growth) or by accident. Some "breathing room" is desirable, and we have factored this into our checklists. Also, keep in mind that hardware utilization is not the only factor to consider when deciding whether to use virtualization with Exchange. Adding virtualization to an Exchange environment introduces additional complexity in a number of areas, including backup, monitoring, and storage configurations (see the TechNet article for details).
Checklist #1 - Performance Counters
The following checklist outlines performance metrics to monitor that may indicate that your Exchange environment may lack the resource intensive performance that requires a purely physical Exchange 2007 infrastructure. These counters need to be gathered from physical servers, not virtualized servers. As the main planning for Exchange 2007 SP1 infrastructure revolves in great part around planning for the Mailbox Server, the checklist deals mostly with Mailbox Server-based metrics.
The idea with these counters is to collect data for a minimum of 1 week. Once the data has been gathered you can compare the results to the expected values. If the observed values are less than the expected values below, then your server hardware is being underutilized.
The counters in the table below are from the TechNet article Exchange 2007 Monitoring without System Center Operations Manager. You will need to make sure that you monitor your Hyper-V servers after they are put into production. You can monitor "Processor Performance" on Windows 2008 so you know the operating system is not slowing down your processors frequency. This could lead you to believe you have high CPU utilization, when in fact you have low CPU utilization and the processors are just throttled down to save power.
|Common Performance Counters (All Exchange Servers)||Processor\% Total||Should be less than 40% average|
|System\Processor Queue Length (all instances)||Should be less than 5 (per processor)|
|Network Interface(*)\Bytes Total/sec|
For a 1000-Mbps network adapter, should be below 30-35 Mbps
|Mailbox Server-Specific Performance Counters|
MSExchangeIS Client (*)\RPC Average Latency
Should be less than 30 ms on average
|Process(Microsoft.Exchange.Search.ExSearch)\% Processor time|
Should be less than 1% of overall CPU typically and not sustained above 3%
MSExchange Store Interface(_Total)\RPC Latency average (msec)
Should be less than 100 ms at all times
MSExchange Store Interface(_Total)\RPC Requests outstanding
Should be 0 at all times
CCR, LCR and SCR Mailbox Server -Specific Performance Counters
Should be less than 10 at all times for CCR and SCR. Should be less than 1 at all times for local continuous replication (LCR).
Table 1 - Performance Counters Checklist
Checklist #2 - User Profile Factors
A less time-consuming (and less precise) way of judging whether your whether the load on your servers is fully utilizing your hardware, is to compare processor cores to user profiles, using the general sizing guidelines provided in the topic, Planning Processor Configurations.
To determine the user profile of your organization, refer to the chart in the article (also reprinted below).
User type (usage profile)
Send/receive per day approximately 50-kilobyte (KB) message size
5 sent/20 received
10 sent/40 received
20 sent/80 received
30 sent/120 received
Table 2 - Knowledge Worker Profiles for Outlook Users
As noted in the article, a rule of thumb for sizing is that 1,000 active average profile mailboxes will require a 1 x processor core (for example, a 4,000-mailbox server with an average usage profile requires 4 x processor cores). A heavy usage profile requires more processor cycles than an average profile, thus for planning purposes, use 750 active, heavy-profile mailboxes per processor core. Using this logic, we can summarize how many users are needed to fully utilize a processor core:
Light User Profile
Recommended Number \ Processor Core = 2000
Average User Profile
Recommended Number \ Processor Core = 1000
Heavy User Profile
Recommended Number \ Processor Core = 750
Very Heavy User Profile
Recommended Number \ Processor Core = 500
Table 3 - User Profiles Factors Checklist
Since the recommended number of Average Users is 1,000 per processor core, having an active user population of 500 or less average profile mailboxes would indicate that there are not enough users to fully utilize one mailbox server processor core. Keep in mind that for physical Exchange servers, the maximum number of processor cores efficiently used by the Mailbox server role is eight, http://technet.microsoft.com/en-us/library/aa998874.aspx. Deploying mailboxes on servers with more than eight cores will not provide significant scalability improvements.
The use of this table to gauge the number of Mailbox server processors core is a good starting point for reviewing other Exchange server roles, since the Exchange infrastructure planning methodology for Hub Transport and Client Access Servers is in part based on the processor core ratio (e.g., Mailbox:Hub and Mailbox:CAS). For example, the Mailbox:Hub Server ratio is 5:1 if the Hub servers have antivirus software installed on them and 7:1 if the Hub servers do not have antivirus software installed. Therefore, if a user population is not likely to fully utilize one Mailbox server processor core, then it logically is not likely to fully utilize a Hub Transport processor core either. This can indicate that using virtualization for the Hub Transport and/or the Client Access Servers may be warranted.
With the release of Windows Server 2008 with Hyper-V and more recently, Microsoft Hyper-V Server 2008, you have new options for deploying Exchange Server 2007 SP1. In many situations, keeping Exchange on physical hardware provides better manageability and a lower TCO than using virtualization. However, there are some scenarios in which a virtualized Exchange 2007 SP1 infrastructure may allow you to realize real benefits in terms of space, power, and deployment flexibility.
The degree to which your current hardware is underutilized is a key factor in determining whether the benefits of virtualization outweigh the complexities that are introduced by adding a virtual layer beneath Exchange. The benefits of virtualization are usually reserved for environments in which the Exchange deployment is too small to fully tax the underlying servers. Small Exchange deployments, remote sites with poor connectivity, certain disaster recovery sites, and "Mobile LAN" deployments are examples of good candidates for virtualization.
Exchange in most organizations is a mission critical application. If you remember this as you design your virtualized environment, and follow the Microsoft support policies and recommendations for virtualized Exchange servers, then you will set yourself up for success.
- Microsoft Support Policies and Recommendations for Exchange Servers in Hardware Virtualization Environments
- Windows Server Virtualization Validation Program
- Exchange 2007 System Requirements
- Exchange Mailbox Server Storage Calculator
- Exchange 2007 - Monitoring Without System Center Operations Manager
- Server Virtualization with Advanced Management (SVAM) Service Offering