Lately, we’ve been getting a lot of questions related to Exchange 2007 memory consumption and why Exchange uses so much of the paging file. Well, there are some valid reasons why and some misconceptions that need to be corrected.
Comparing memory usage between Exchange 2007 and Exchange 2003
First, let’s start with why Store.exe uses so much RAM. If we take a step back in time to the Exchange 2003 era, this blog was quite active on how to tune memory on an Exchange 2003 server. I am not going to go in to specifics or great detail, but one thing to call out is that on the 32-bit architecture, we were limited to addressing 4GB of virtual memory on any given server. So essentially, any 32bit program could address up to 4GB of virtual memory. This address space is typically split so that applications could address 2GB of memory and 2GB would be for the Windows Executive. By adding the /3GB switch in the boot.ini, applications could now access up to a 3GB virtual address space and lower the Executive down to addressing only 1GB, essentially halving the memory that is can be addressed or is available for kernel drivers, paged/non-paged pool memory, PTE’s etc. The larger the load that you put on a server has the potential to exhaust important resources on the server which eventually causes a failure or server outages. Any type of memory leak in these areas could be detrimental to the stability of the server. With the numerous posts on this blog regarding the /3GB, /USERVA and /PAE switches, what drivers to update, how to tweak memory to allow Exchange to run optimally, etc., it was always a juggling act to try to keep Exchange servers online. Cluster servers were especially touchy as http.sys will start rejecting HTTP connections when non-paged pool hit a certain level, thus causing cluster failovers.
Due to these constraints of the 32-bit architecture, we needed a way to balance I/O based applications on the server. Essentially, there are two types of I/O for the Windows operating system, Buffered and Unbuffered I/O. For Buffered I/O, only portions of the working set can be contained in memory; those portions not in use are retained in the page file. For Unbuffered I/O, the entire working set can be contained in memory. On any given system, we needed to maintain a balance so that each would get the right resources that they need. This is one of the main reasons why we needed to put a maximum on the ESE database cache to leave us enough room for the system cache.
In most implementations, the default of 900MB of database cache was used which was sufficient for most deployments, but some implementations had the need for more cache increasing it to a maximum of 1.2GB of memory on systems of 2GB or more of physical RAM. This maintained the balance that was necessary so that the system cache could perform efficiently.
So you may ask, why did I go through all of the above to explain how Exchange 2003 worked? Well, one of the aspects of Exchange 2007 that people ask about is the difference between Exchange 2007 memory usage to Exchange 2003. Indeed, the memory usage is much higher,and there are very good reasons for this. Let me explain further.
Exchange 2007 Design Considerations
During the design phase of Exchange 2007, we needed to find a way to scale out servers further than Exchange 2003 did due to the rising demands of email usage in companies. For many companies, email is their life line and if an exchange server should go down, they could be losing money at an astronomical rate. One of the performance bottlenecks in Exchange 2003 was disk latencies which was mostly due to the memory constraints of the 32-bit platform. Ultimately, -, we had to read more from disk than what was available in memory. This required a well performing disk subsystem to handle the sustained I/O necessary to respond to client requests in a timely manner. In many cases where disks were the underlying problem, a re-design of the disk subsystem was necessary to allow the current amount of IOPS that were being generated on the server to be handled by the disks. This could range from adding more spindles to the disk arrays, rebuilding different RAID solutions to provide enough sustained throughput based on current user load.
Why we learned to love the 64-bit Platform
With the 64-bit architecture being available, we made a conscious decision that Exchange 2007 would only be supported in production on 64-bit hardware. This was mainly done to overcome the limitations that the 32bit platform had. The design goal here was to reduce cost of Exchange deployments in which storage turned out to be the number one driver of cost and support overhead. By moving to the 64-bit platform, we are able to reduce our I/O requirements thus bringing us back to easier balancing of I/O and capacity requirements.
Now, one of the results of being able to address more memory is the capability to cache more memory for each application. To allow this to happen for Exchange, the ESE Database Cache Size limit was removed to allow Exchange the ability cache more pages in memory. Accessing pages in memory is extremely fast, so the more data we cache in memory, the less we actually have to read andwrite from the disk. When following our best practice guidance around storage group deployment and memory sizing, Exchange 2007 reduces the amount of I/O required overall. This gave us rather huge performance gains. For more information regarding I/O improvements on the 64-bit platform for Exchange, see the following blog post http://msexchangeteam.com/archive/2006/09/08/428860.aspx.
With no limitation in Database Cache for Exchange 2007, the memory usage for the store process will naturally be much higher compared to what Exchange 2003 used due to the many benefits discussed earlier. Memory allocation for the ESE cache is dynamic, so Jet will use all available memory on a system and return it as needed. For example, if a server had, let’s say, 16GB of physical RAM installed, the database cache could consume approximately 14GB of memory, roughly 2GB less than the total amount of physical RAM in the server, leaving enough memory for the system cache or other applications running on the server.
The algorithm is relatively simple for the ESE Database Cache. The DBA (Dynamic Buffer Allocation) will grow the cache by comparing the amount of I/O the databases are doing with the amount of I/O the system is doing in terms of hard page faults (Memory\Transition pages repurposed/sec). So if the database is doing more I/O, and Exchange is hitting hard page faults, we will increase the cache. If the database is doing less I/O than the hard page faults, then we will decrease the cache. The goal is to keep the ESE cache in balance with the disk cache so we don’t end up paging ourselves to death. In some cases, on larger servers, Online maintenance during the night could cause up to a 4GB swing in memory for the Database cache alone due to the aggressive nature of how we touch pages during this process. (It may also be prudent to stagger your OLM schedules to assist with this.) Each server configuration is different, so it is important to have a monitoring solution in place or a Performance monitor log to track memory usage over time if memory pressure is occurring. If memory usage gets to a point where some driver or application attempt to allocate a large block of contiguous memory, and there is not one available, then the OS may decide to start trimming working sets which is a really bad thing performance wise. I’ve discussed this on my own blog http://blogs.technet.com/mikelag/archive/2007/12/19/working-set-trimming.aspx and possible remediation’s to work around those type scenarios if you fall in to this bucket.
Other Server Memory Considerations
So we’ve talked a lot about memory usage for ESE database cache, but this cache alone is not the only significant consumer of memory on an Exchange server, it is just the majority. From an Exchange perspective, we have other processes going on such as Content Indexing, log shipping in continuous replication and underlying .NET applications that Exchange makes use of now.
Most of Exchange has been rewritten in managed code using the .NET framework except for Store, DAV, and DSAccess. Managed code offers some significant advantages over non-managed code such as the ability to compile applications in real-time so that you don’t have to worry about running your application on different architectures or platforms and the ability to manage memory efficiently. .NET applications use the Common Language Runtime (CLR) to allow easier coding in different languages because they share the same runtime. Any code that you develop with a language compiler that targets the runtime is called managed code as mentioned here: http://msdn2.microsoft.com/en-us/library/ddk909ch.aspx.
Reasons for Memory Paging
One of the primary reasons for memory paging in Exchange 2007 is Content Searching, which uses the system cache since this is buffered. Performing more searches increases the file cache you have which in turn reduces the amount of IOPS that are generated. Content Indexing itself does not make use of the file cache as this is designed to work on a memory structure that is not affected by the file cache. Master merges of index data does use a small amount of the file cache, but it very short lived. Database I/O is actually unbuffered, so this not affected by the system cache. This is why the ESE cache doesn’t continuously grow (ESE cache I/O is less than the system cache I/O). It’s being balanced out by the content searching I/O (and potentially other applications that use buffered memory I/O) and thus the system cache. CCR log shipping is also unbuffered while log files are being copied over SMB, so this is also not affected by the system cache.
Now couple all of this memory management with other 3rd party applications and drivers that are installed on the server which are all competing for the same resources, you can see that this is vastly different than what Exchange 2003 used to be. There is actually no comparison at all.
How Exchange makes use of the Paging File
So this brings up the second part of this document where I will now talk about the paging file. So you may ask, if I have so much memory in the server, why is a paging file needed at all? Our system requirements talk about needing one for getting a memory dump on the server, but what does Exchange really use it for? I get this very question asked very frequently. In certain cases, it is true that Windows 2003 does not require a paging file. It is however completely incorrect that Exchange does not need a paging file for normal runtime operations when running on top of that OS. There is always going to be a need to page out unused or rarely used memory to the page file so that we can make maximum use of RAM. For example, there could be a large amount of VM pages used during system startup that are not useful under main load. Unexpected transient conditions could cause large upward swings in memory that could cause us to spill pages of memory in to the paging file. There are enough processes outside of the cache manager that consume a significant amount of memory in Exchange 2007 (e.g. ESE and all processes that utilize the CLR) that there is a need for a substantially sized paging file as the system balances the memory footprint of the competing processes. Without a paging file (or with one that is too small) there is a very high chance that applications will encounter OOM (out of memory) conditions and both the performance and the functionality of the server will suffer and may lead to a possible blue screen.
The cost of having a 32GB paging file on a server with 32GB of physical RAM on disk is pretty minimal, so having a properly configured paging file (RAM+10MB) will be able to sustain any unexpected growths in memory where we need more commit charge to handle current load. The other benefit is that if we should get ourselves in to a working set trimming issue, we would at least have the ability to flush these pages out to the paging file and then page them back in when a particular page needs to be accessed again. Excessive working set trimming on an Exchange 2007 server is not normal and should be investigated. Again, check out my blog post on working set trimming problems and why this is bad for performance.
Why does Exchange allocate so much memory in the paging file?
Another question that generally comes up is why does Exchange allocate so much memory in the paging file? Depending on the amount of memory that is consumed for the store process, we will need to at least allocate enough memory in the paging file to match the current working set that is in RAM on the server. If a system wide working set trim should occur, we would be able to page the current pages in the store working set out to the backing store or paging file. It is important to monitor Memory\Pages/sec on the server to detect if this working set trimming problem is causing any performance related problems on the server. You should be able to see by adding Process\Working Sets a sharp decline in this value along with a sharp increase in Memory\Pages/sec. With the other processes on the server possibly using parts of the system or file cache, there is always a balancing act trying to compete for memory on any given server. The good news is that Windows 2008 has made some significant changes in memory management to help prevent these system wide working set trims and to manage memory more efficiently. With server applications having the need for using more RAM as technology advances, it is imperative that memory management works effectively to prevent any negative performance problems on any given server. For more information on some of these memory management changes in Windows 2008, see http://www.microsoft.com/whdc/system/sysinternals/memmgt.mspx.
I hope this helps clarify some of the reasons why Exchange uses more memory overall and provide clarity for these most common questions.
A special thanks goes to Andrew Goodsell, Matt Gossage, and Ross Smith IV for their invaluable information in this area and Nick Basile for tech reviewing this document.
Support Escalation Engineer
|Share this post :|