Exchange 2013: Store, FAST, and ESE Cache Demystified…Hopefully!

After getting great community response on my Exchange 2013 Mail flow Demystified blog I decided to do a similar one on Store, FAST, and ESE Cache. Much like mail flow a lot has changed with Exchange 2013 in terms of Store. So hopefully I can help clarify exactly what has changed in 2013 with these components. First let’s start with Store since this is the core to your Mailbox Server.

With Exchange 2013 Store.exe got a much needed overhaul (hasn’t changed much since Exchange 4.0) and was rewritten entirely in managed C# code. The  store.exe process is now broken out into 2 processes: Microsoft.Exchange.Store.Service.exe and Microsoft.Exchange.Store.Worker.exe. The Store.Service process is responsible for managing and terminating all of the Store.Worker processes. Each database on the mailbox server will get its own Store.Worker process regardless if it an active or passive database copy. You can see in the image below both processes and how the database WorkerProcessID (PID) correlates to Store.Worker process. By splitting Store into multiple process this minimizes the chance a single database brings down the entire store.exe and crashing\failing over all databases on that server.

Anyone who has worked with Exchange 2010 or older knows if you look at processes, Store.exe took the majority of the server’s memory (75-90%). This by design and is the default behavior. ESE (store.exe) will release memory back to the system if it is needed by another process. You can also set the msExchESEParamCacheSizeMax registry key to specify the max memory ESE can consume on that server. This design worked great for Exchange 2010 and previous versions but due to significant changes in 2013 this process had to get architected.

Due to the fact that Exchange 2013 needs more memory for other processes like Search (FAST) and Transport we had to limit the amount of RAM that ESE used. Now with 2013 ESE (Store) will use 25% of the total RAM in the system. So if your server has 48GB of memory ESE will utilized 12GB of that for cache. Now that 12GB will need to get broken up to support each of the databases hosted on that server. Before I get into the exact formula ESE will use approx. 80% of the 12GB for Active copies and 20% for passive copies. Easy enough right? Not so Fast. There is a problem that can arise if you don’t properly set the MaximumActiveDatabases on each mailbox server (Set-Mailbox –MaximumActiveDatabases). So let’s look at two examples below and you will see exactly what I mean.

Example 1 – Our MBX server has 48GB of memory and hosts 8 databases (4 Active and 4 Passive). In this scenario we didn’t set the MaximumActiveDatabases parameter on the MBX server. You can see that out of 12GB allocated for ESE we are only utilizing 7.2GB in a typical scenario.

Total Server RAM = 48GB
48GB*25%=12GB allocated for ESE cache
There are 4 active copies on the server
There are 4 passive copies on the server (these can potentially become active)
12GB/(4+4) = 1.5GB
Active database cache = 1.5GB
Passive database cache=1.5GB*0.20 = 0.3GB
Max Cache Target for active and passives = 12GB (25% RAM)
Typical (Used) Cache Target = (4*1.5GB + 4* 0.3GB = 7.2GB)

Example 2 – Our MBX server has the same 48GB of memory and still hosts 8 databases (4 Active and 4 Passive). In this scenario we set the MaximumActiveDatabase parameter on the MBX server to 4. Now you can see that in the typical scenario we utilize 12GB (100%) of the allocated ESE Cache.

Total Server RAM = 48GB
MaximumActiveDatabases = 4
48GB*25%=12GB allocated for ESE cache
There are 4 active copies on the server
There are 4 passive copies on the server (these can’t become active because our max is 4)
12GB/(4+4*.20) = 2.5GB
Active database cache = 2.5GB
Passive database cache=2.5GB*0.20 = 0.5GB
Max Cache Target for active and passives = 12GB (25% RAM)
Typical (Used) Cache Target = (4*2.5GB + 4* 0.5GB = 12GB)

You can see how making sure you set the MaximumActiveDatabase in your environment (if required) can make a huge difference. If we are talking about a standard user who utilizes 5MB of ESE Cache this could be the difference of 204 mailboxes more\less per active database. Proper configuration = Optimal performance!!

Lastly I want to talk about the new search platform used by Exchange called FAST. Many will be familiar with the FAST search technology because it is used by SharePoint. I am not going to go into the new Search Infrastructure internals but focus on its processes and memory requirements on the MBX Server. The Microsoft Exchange Search service starts a controller processes named hostcontrollerservice.exe. In turn the Host Controller process starts 4 worker processes named noderunner.exe. You can see both of these processes below. Each of these nodes has a different responsibility in the Search Infrastructure including Content, Query, Index, and Admin. You will notice that these worker process take up a significant amount of memory. As Best Practice plan to allocate approx. 15% of the total system memory to the entire Search Infrastructure\Processes. This means in the examples above we would use 7.2GB for Search.