When selecting hardware for your Exchange servers, there are many things that you must consider. Two of the most critical resources are processor and memory.
This article, an update to an article originally posted on 3/13/2006, provides processor & memory guidelines for minimum, recommended, and recommended maximum configurations. There are several factors that go in to choosing the right hardware for an Exchange 2007 deployment. This article provides rule of thumb guidance on the primary factors which determine to proper memory and processor configuration for a given Exchange 2007 role. There are other factors to consider as well which we do not directly cover in this article. Storage and network hardware are other critical pieces to Exchange 2007 sizing but these areas will be covered in separate articles. We will provide additional guidance as available throughout the Exchange Server 2007 development process.
Exchange 2007 server hardware requirements are different than previous versions of Exchange (2003)
The primary hardware difference between Exchange 2003 and Exchange 2007 is the move from a 32-bit platform (Exchange 2003) to a 64-bit platform (Exchange 2007). Exchange 2007 will only be supported in production environments when it is running on an x64 edition of Windows Server 2003. The change from a 32-bit platform to a 64-bit platform requires a new approach to choosing server hardware for Exchange; especially processor and memory:
Processor types supported by Exchange 2007
Exchange 2007 is only supported in production when the following are true:
- Running the x64 version of Exchange 2007
- Running on Windows 2003 x64 Editions
- Running on x64 compatible processors
The following processors support x64 versions of Windows 2003, thereby supporting Exchange 2007 deployments:
Each of these vendors also ship x64-capable desktop processors which can also run x64 versions of Windows 2003 (e.g. AMD Athlon64 and Intel Pentium D with EM64T) but for the sake of simplicity, this article will concentrate on processors designed for server deployments.
It's important to note that the Intel Itanium (IA64) processor will not work with Windows 2003 x64 Editions, and thus it will not work for Exchange 2007 deployments. Exchange 2007 is designed to run only on x64 capable processors such as those listed above; Exchange 2007 will not run on Itanium based systems.
Regardless of which server processor you select, it is necessary to have the server product pass the Designed for Windows test suite to ensure Microsoft support. Servers listed on the Windows Server Catalog meet these criteria. If your server is not listed, check with your vendor to see if either the "Designed for Windows" logo testing is in progress or the server has passed the testing and is pending a website update.
Multi-core processor considerations for Exchange 2007
Exchange 2007 benefits significantly when running on multi-core processors. The performance benefit for Exchange from dual-core technology depends upon the specific processor utilized. The findings from Exchange 2003 dual-core testing have been summarized in Microsoft Knowledge Base article 827281, CPU and memory scalability for Exchange Server 2003 and for Exchange 2000 Server. Also, the performance benefit of specific dual-core implementations can be seen by comparing the MMB3 results of a 4-processor, single-core based server to a 2-processor, dual-core based server. These results have been published at the Performance Benchmarks for Computers Running Exchange Server 2003 Web site. The benchmark is in the process of being updated for Exchange 2007. Exchange 2007 benchmarks will be published when available.
Multi-core processors are an attractive option for Exchange 2003 and Exchange 2007 servers based on price and performance. Please ask your server vendor about dual-core benefits for Exchange, specific to a given hardware architecture.
Hardware specific memory considerations for Exchange 2007
Exchange 2007 enables much better memory utilization than Exchange 2003 due to its 64-bit architecture. Because of the virtual address space limitations of a 32-bit platform, Exchange 2003 is limited to using 4 GB or less of physical memory. In contrast, customers running Exchange 2007 on Windows 2003 x64 Editions can efficiently utilize upwards of 32 GB of memory (Mailbox role). Note; 32 GB is not a physical limitation, rather the most cost-efficient recommended max. This change needs to be factored in when choosing server hardware. The following factors should be considered:
- Server Maximum Memory Configuration - Different server architectures have different memory limits. We recommend that you check the following technical specifications of the server to determine the criteria that affect the maximum memory configuration to ensure that the appropriate amount of RAM can be installed in to an Exchange 2007 server economically:
- Memory Speed - Some server architectures require slower memory to scale up the memory to ten's of gigabytes in a given server (e.g., maximum server memory is limited to 16GB with PC3200 or 32GB using PC2700). You should check with the manufacturer to make sure that the memory configuration target for Exchange 2007 is compatible in terms of speed.
- Memory Module Size - What is the largest memory module size the server will support? Generally, the larger the memory module, the more expensive it is; 2x1GB DDR SDRAM memory modules generally cost much less than 1x2GB DDR SDRAM memory modules. When planning for an Exchange 2007 server, make sure the maximum memory module size allows you to meet your target memory requirements.
- Total Number of Memory Slots - How many memory modules will a given server support? The total number of slots multiplied by the maximum memory module size will provide the maximum memory configuration for the server. Also, keep in mind that memory modules must sometimes be installed in pairs.
Applying the processor and memory configuration factors to specific Exchange 2007 server roles
The following chart can be used to assist in purchasing server hardware destined to be used for Exchange 2007 server roles. The goal of this chart is to provide minimum, recommended and recommended maximum configurations.
Minimum: The minimum processor/memory configuration suitable for specific Exchange 2007 server roles (also defined in System Requirements). Minimum hardware requirements must be met to receive Microsoft Product Support Services.
Recommended: This is the recommended processor/memory configuration for specific Exchange 2007 server roles. Recommended can be defined as “the best configuration based on price and performance.” The recommended configuration also provides a balance between processor and memory capacity. The goal is to match the memory configuration to the processor configuration so the system will effectively utilize the processors without becoming bottlenecked on memory and vice versa.
Recommended Maximum: This is the maximum recommended processor/memory configuration for specific Exchange 2007 server roles. Maximum is defined as “the upper bound of viable processor/memory configurations for Exchange 2007 based on price and performance.” Maximum is a guideline and not a “support criteria” and does not take in to account the resource requirements of 3rd party applications. The recommended maximum may change over time based on price changes and technology advancements.
Processor Configuration Table for Exchange 2007
Edge Transport Role
Edge transport is extremely efficient in design and thus requires moderate processing power. Also, fault tolerant Edge Transport deployments will utilize multiple servers to provide redundancy. The Recommended configuration on 2 x Processor Cores takes this fault tolerant deployment consideration in to account. Large enterprises, with a large volume of inbound/outbound SMTP traffic will be able to utilize 4 x Processor Core servers to reduce aggregate Edge Transport server count. Processor utilization is based on several factors such as: message rate, average message size, number of agents enabled, anti-virus configuration, 3rd party applications etc.
Hub Transport Role
The recommended configuration for Hub Transport is 4 x Processor Cores in deployments where Hub Transport is deployed along with several Mailbox servers and thousands of mailboxes. 1-2 x Processor Core configurations can be considered for deployments where there are not enough mailboxes (message traffic) to utilize a 4 x Processor Core configuration. Processor utilization is based on several factors such as: message rate, average message size, number of agents enabled, anti-virus configuration, 3rd party applications etc.
Client Access Server Role (CAS)
Exchange 2007 architecture has moved significant client specific functions from the Microsoft Exchange Store to the Client Access Server. Messages are now converted on the CAS server when accessed with non-MAPI clients (e.g. POP3/IMAP4). Also, Outlook Web Access rendering is now performed on the CAS server as opposed to the Mailbox Store in previous versions of Exchange. These architectural changes allow CAS to offload significant processing from the Mailbox role and to allow it to effectively utilize 4 x Processor Cores. Similar to Hub Transport, servers with 1-2 x Processor Cores can be utilized for CAS in deployments where there are not enough mailboxes/non-MAPI client traffic to utilize 4 x Processor Core servers.
Unified Messaging Role (UM)
The recommended configuration for the Unified Messaging role is 4 x Processor Cores. Multiple Cores are well utilized on the UM server for several architectural functions such as Voicemail WAV to WMA conversions. Similar to previous roles, 1-2 x Processor Core configurations may be utilized when the mailbox count/UM utilization does not necessitate 4 x Processor Core configurations.
The recommended configuration for the Mailbox role is based predominantly on mailbox count and user profile. A 4 x Processor Core server provides a good balance between price and performance and should be able to host several thousand mailboxes. Rule of thumb sizing for the Mailbox server role requires an understanding of the average client user profile. This profile can be collected using the Microsoft Exchange Server Profile Analyzer or with 3rd Party tools/applications. The following table lists generic/common Outlook knowledge worker profiles:
There are several factors that go in to server sizing for Exchange 2007 which are above and beyond the user type listed above. These include Exchange 2007 features such as Local Continuous Replication (LCR), Forefront Mailbox Virus Scanning, 3rd Party applications, 3rd Party mobile devices, and Microsoft Outlook client version/mode (Online vs Cached Exchange Mode etc.). Rule of thumb sizing used primarily for budgeting purposes can be accomplished by calculating that 1000 Average profile mailboxes will require 1 x Processor Cores. E.g. A 4000 Mailbox server with an Average usage profile can be estimated to require 4 x Processor Cores. A Heavy usage profile will effectively double this calculation (500 Mailboxes/Processor Core). The maximum number of processor cores the Exchange 2007 Mailbox role will efficiently utilize is eight. Deploying Exchange 2007 Mailbox on servers with greater than eight cores will not provide significant scalability improvements.
***Concurrency: Concurrency is defined as the percentage of the total number of users on a server that are connected and using the server at a given peak period of time. Concurrency is difficult to measure since clients may use multiple clients and different versions of Outlook have various connection counts to the server. For a fully utilized server, concurrency is generally in the 75-80% range. The guidance in this article assumes this average concurrency profile.
Multi-Role (Mailbox, Hub, CAS)
The Multi-Role configuration has similar guidance and limitations as the Mailbox role. To accommodate CAS and Hub Transport utilization on a single server with the mailbox, reduce the 1000 Mailboxes/core calculation based on the Average client profile by 20% (800 Mailboxes/core) when performing rule of thumb sizing for Multi-Role servers. The maximum recommended processor core configuration is listed at 4 for the Multi-Role configuration to indirectly provide guidance on the maximum number of users recommended to be hosted in this scenario. Neither Clustered Continuous Replication (CCR) nor Single Copy Cluster (SCC) support hosting the Hub nor CAS server so a Multi-Role server has to be non-clustered by definition. It is a good idea to cluster Mailbox servers that host thousands of user to ensure patch management and server failures do not have a significant impact on up time. The more users hosted on a Mailbox server the more user “pain” is caused by server downtime. For this reason, the recommended maximum processor core configuration for the Multi-Role server is listed at 4. While the this configuration will perform well up to 8 processor cores, it is not recommended due to the availability concerns outlined above.
Memory Configuration Table for Exchange 2007
Once the number of processor cores have been estimated to be required per server role are understood; baseline memory recommendations can be applied. The following table illustrates the Minimum, Recommended and Maximum memory configurations for Exchange 2007 server roles.
Edge/Hub Transport Roles
The Edge and Hub Transport roles do not require substantial quantities of memory to perform well in optimal conditions. Generally, 1GB of RAM per processor core is sufficient to handle all but the most demanding loads/scenarios. There are two significant memory factors that must be taken in to account for large deployments:
Large Queue Scenarios: Exchange 2007 Edge/Hub Transport servers are designed to handle scenarios where extremely large queues build up (e.g. 1 million messages in a single server queue). Edge/Hub Transport servers hold the queued message recipient information in memory to optimize the SMTP send and message retry operations. When sizing Edge/Hub Transport servers for large queue scenarios, use the following table to estimate the memory requirements:
The recommended maximum memory configuration of 16GB is based on the Edge/Hub Transport servers handling 1 million messages with an average number of recipients each. Most deployments will be optimally configured with the “Recommended” memory configuration of 1GB per processor core.
Client Access Server Role (CAS)
Memory utilization on the Client Access Server has a linear relationship with the number of client connections and the transaction rate. For this reason, there are no other substantial memory factors to take in to account.
The memory configuration process for the Mailbox role is more involved than the other roles since the optimal memory configuration depends upon the mailbox count and the client profile (similar to estimating processor core requirements). Memory sizing for the Mailbox role is critical to reducing the I/O requirements of the server. The more memory you add to the Mailbox server, the less I/O will be generated by the Exchange databases. There is a point of diminishing returns; in which adding additional memory to the server may not be justified based on price/performance. Memory guidance outlined here takes this point of diminishing returns in to account based on current memory prices and performance metrics. Also, defining the memory configuration of the Exchange 2007 Mailbox server is required prior to defining the storage requirements/configuration. The following table can be used to assist in estimating the memory requirements of a given mailbox server with a given number of hosted mailboxes with a given profile type (taken from previous profile table).
Recommended Maximum Memory Configuration for Mailbox
Recent x64 based servers have the ability to scale up their memory configuration to 64GB and beyond. There are several reasons why Microsoft does not recommend maximum memory configurations for Mailbox that go beyond 32GB.
Cost: Based on current memory prices (specifically 4GB DIMMs), it is cost prohibitive to expand the physical memory capacity of a server beyond 32GB. Generally, the cost of physical RAM is linear up to 32GB; beyond this the cost trend is exponential. Beyond 32GB, for many configurations, it is less expensive to add disk drives as opposed to memory.
Non-Transactional I/O: The Mailbox server utilizes additional physical RAM by caching more data and thus reducing the database I/O footprint for transactional I/O (I/O that is generated by send/receive/client processing of email). There are several sources of non-transactional I/O generated on the Mailbox server. These include Online Maintenance (e.g. Online Defrag), Offline Maintenance (e.g. offline defrag, database repair operations), Backup/Restore operations, Mailbox Management Processes (e.g. Email Lifecycle Management (ELC)), All of these operations require database I/O to properly maintain and recover the server. Although Exchange 2007 has reduced transactional I/O significantly; adequate storage performance is still required for proper maintenance of the Mailbox server. For this reason, there is a point of diminishing returns when adding memory to the server. In general, the purpose of adding memory to the Exchange Mailbox server is to reduce the storage requirements (specifically performance) and thus storage cost. Due to non-transactional I/O requirements; the storage requirements of the system may not be significantly reduced by adding server memory beyond 32GB.
Cold State Operation: Cold state is defined as the state of the Mailbox server immediately following a server reboot or store.exe process restart. The Database Cache, which is used to cache database read/write operations, is small in size (or “cold”) during this period so it has a significantly diminished ability to reduce read I/O operations. As the Mailbox server processes messages, the Database Cache Size grows which increases the effectiveness of the cache and subsequently reduces the I/O footprint of the server. The larger the physical memory size of the server the longer it takes the Database Cache size to reach its optimal size. If the storage is designed/sized for a server with a large amount of physical RAM (>32GB), and the I/O profile of the users assumes an optimal Database cache state (large/warm cache); then the client experience may be compromised due to insufficient disk performance during these “cold state” periods. Similar to the Non-Transactional I/O case; the storage subsystem requirement may be the same for a server that has been populated with 32GB of RAM as a server that has been populated with greater than 32GB of RAM.
While the Exchange 2007 Mailbox role will utilize memory greater than 32GB, for the reasons outlined above; 32GB is the maximum recommended memory config and is considered the point of diminishing returns in terms of both cost and performance.
Recommended Minimum Memory Configuration for Mailbox based on the number of Storage Groups
The maximum number of Storage Groups configurable in Exchange 2007 has been increased to 50 (up from 4 with Exchange 2003). This increase provides much greater flexibility in server/storage architecture, but the increase has a significant effect on the memory utilization of the Exchange 2007 Mailbox server. Increasing the number of Storage Groups primarily effects the Database Cache utilization of ESE (Extensible Storage Engine). The ESE Database Cache is used for both read and write activity. Due to the way Checkpointing works, adding a Storage Group effectively increases the amount of the Database Cache used for write activity. This has a positive impact of reducing database write I/O; but if too many Storage Groups are configured on a server with insufficient physical memory, the effectiveness of the database read cache may be reduced which may have an overall negative effect on the performance of the server. For this reason, it is important to maintain a ratio between the number of Storage Groups configured and the physical RAM of the server.
Multi-Role (Mailbox, Hub, CAS)
The Multi-Role configuration has similar guidance and limitations as the Mailbox role. To accommodate CAS and Hub Transport role on a single server with the Mailbox role, the recommended base memory configuration is 4GB as opposed to 2GB. Memory guidance based on mailbox count and profile is the same as the Mailbox role. The maximum recommended memory configuration is listed at 8GB for the Multi-Role configuration to indirectly provide guidance on the maximum number of users recommended to be hosted in this scenario. Neither Clustered Continuous Replication (CCR) nor Single Copy Cluster (SCC) support hosting the Hub nor CAS server so a Multi-Role server has to be non-clustered by definition. It is a good idea to cluster Mailbox servers that host thousands of users to ensure patch management and server failures do not have a significant impact on up time. The more users hosted on a Mailbox server the more user “pain” is caused by server downtime. For this reason, the recommended maximum memory configuration for the Multi-Role server is listed at 8GB. While more memory is supported in this configuration, it is not recommended due to the reason outlined above.
With effective planning and an understanding of the basic processor and memory requirements for specific Exchange 2007 server roles, a balanced/cost effective topology can be easily attained.