Why is Exchange Store.exe so RAM hungry?


This seems to be one of most favorite questions ever: “How come that Exchange Store.exe keeps on grabbing more and more memory, even on the server that is not very busy? Is there a memory leak?”. This is many times followed by “I am rebooting my server on the weekly basis to keep store memory consumption in check”.


Well, let’s see what we have here.


It is absolutely normal for Store.exe to grab as much RAM on the server as it can possibly get – as long as Store thinks it needs it to optimize performance. Store was written to do so. It does this as it wants to do as much stuff in memory as possible, without having to go to the slow page file. It is a common misconception – that Store.exe’s increasing memory consumption is a “memory leak”. This behavior is expected since Exchange 5.5 days:


182505 XADM: Memory Usage of Store.exe Is Higher in Exchange 5.5
http://support.microsoft.com/?id=182505


That being said:


1. Exchange Store will grab as much RAM as it can if it thinks it needs it, yes. But – we constantly monitor the performance of the system in regards to memory usage and we can use this data to infer when we need more memory and when other applications or the OS needs more memory.  We then use this data to act accordingly. This scheme allows the system to act as if there is explicit control when in fact it is actually a few autonomous applications cooperating in a disconnected manner. That means that we should NEVER see a “out of memory” message by any application on the server because of the Store – unless there is a leak on the server, of course… or the page file is too small. If there was a malfunction in this Store mechanism it would cause a lot of paging.  That is a big performance problem, but shouldn’t cause actual errors.


2. Store memory utilization can go up to 1.2 GB or sometimes even more when viewing through Task Manager. I have not seen it go over 1.5 GB, but it would typically not take more, even if there is 8 GB of RAM in the machine. So – Store taking 1.2 GB is not an indication of the problem all by itself.


3. Exchange Store is not the only product behaving like this… SQL does something very similar, for example. That is one of reasons why we do not necessarily encourage putting SQL and Exchange on the same server, as they will be fighting over whatever RAM is in the server.


All of the above being said, let’s go into what can be manifestations of real problems:


– Are there actual performance problems that might be related to store taking up RAM? For example, other services on the server slow down noticeably when store gets “large”, events are logged in the application / system log mentioning lack of memory?


– Does client access to the server slow down when Store.exe grabs a lot of RAM?


– Are there any “out of memory” errors on the server at all? Popups when trying to start applications/services on the server?


If not – there is most likely no problem. Again – Store.exe taking up a lot of RAM is NOT a problem on it’s own, as the memory can be returned to the OS when needed by other processes. Store is simply taking advantage of the RAM – as it is in the machine :)


One more note – if Exchange 2003 SP1 is applied to the server and you used to monitor Store memory utilization, you could be seeing an increase in memory consumption unless you change the monitoring settings. This is normal and is covered in more detail here:


867628 Monitoring programs report that the Store.exe process consumes
http://support.microsoft.com/?id=867628


Nino Bilic

Comments (13)
  1. drebin says:

    Wow.

    The was the worst thing I’ve ever read.

    I can’t believe you just tried to justify a program eating 1.5GB (thats 1,500,000,000 bytes, you know)..

    If you’re car gastank is on empty, do you tell yourself "Is this affecting my driving at all?" or "Is the car still running fine?"

    Just because it isn’t affecting anything right now, that doesn’t mean you don’t have to do something about it!! How is one to do capacity planning – when your application will eat whatever memory it finds???

    Wow. That’s just horrible.

  2. Nino Bilic says:

    Thanks for your comment :)

    The truth of the matter is – the 1.5 GB figure is quite extreme. I did see it on very busy servers with 1000s of mailboxes though, and it was not a cause of the problem at all.

    The whole point of this post was to shed some light on this as I have heard people rebooting their servers just because "Store was utilizing XYZ MBs" – so not because there was a performance problem on the server, but because of perception that Store is taking up more RAM than some specific number.

    That in itself is not a problem – so it does not have to be fixed by a reboot. Especially not when there are 4 GBs of RAM in the machine for example (was true in the case where Store took 1.5 GB), and other applications that run on the server can get all the RAM they need.

  3. Nino Bilic says:

    Sorry, just saw teh "capacity planning" part of the questuion – the behavior of Store (taking up more and more RAM if it needs it and if it is available) is not something that you necessarily need to try "compensate" for by adding more RAM to the machine. The process that Store goes through should prevent it from "starving" the OS and other applications on the server – so they have no RAM. So – no matter how much RAM you have on the server (unless it is very small for server application – where you will probably see problems with Exchange or no Exchange) – Store will act accordingly and stop growing after certain point.

  4. Nick Wade says:

    When I was responsible for building and running in Exchange 2000 Clusters; if I put 2GB or 3GB of RAM in a box and Store *didn’t* use 1.2GB on average (well, obviously user load is a factor here) then I KNEW I had a problem with that box. 1.2GB of RAM and idling on 3 to 4% CPU is a happy cluster node that’ll run for months in that state – reboot – are you mad?

    When you’re supporting gobs of users on the one Exchange server, shouldn’t it be a dedicated server, and shouldn’t it use as much physical RAM as possible? In memory ops are so much more performant than those that must refer out to the page…

    Next stop – fast fast FAST transaction log volumes :-) – then 2000 / 3000 corp users *concurrent* per box should be no problem at all.

    Cheers

  5. Nick Wade says:

    P.S. Oops, nearly forgot

    Nino – good, reassuring article, good tips, thanks! ;-)

  6. Jeff Centimano says:

    The first comment (while being way too harsh and a little unprofessional) is typical of new Exchange admins. I run into this quite often as a consultant. Your article will be added to my "Knowledge Gems" folder to distribute to other newbie admins when this question comes up in the future. Thanks for taking the time to write this article. Keep up the good work.

  7. drebin says:

    Jeff, for the record, I am no longer an Exchange admin – but have done it since the olden days. Secondly, sorry you cried over my post. The concept of having a poorly behaving product is one thing, but to justify it’s behaviour was a little over the top for me.

    To assume that Exchange is the only thing on that box is crazy. There could be other connectors – or what about backup software and a local tape drive. Imagine the backup taking 10 hours because it can’t get enough RAM to do it more efficiently?

    At least something like SQL Server let’s you have some control over how much RAM you want to give it.. but this main post was a little out of control in my book.. Sorry I offended.

  8. Nino Bilic says:

    drebin,

    I am not sure – I guess it is possible I was not clear enough in my post – obviously it would be bad if Store does not care if anything else is running on the machine and just takes up RAM as it wants. That’s why it said:

    1. Exchange Store will grab as much RAM as it can if it thinks it needs it, yes. But – we constantly monitor the performance of the system in regards to memory usage and we can use this data to infer when we need more memory and when other applications or the OS needs more memory. We then use this data to act accordingly. This scheme allows the system to act as if there is explicit control when in fact it is actually a few autonomous applications cooperating in a disconnected manner.

    In other words – we will decrease our (store) memory utilization as needed on the server too. Hope that clears it up?

  9. drebin says:

    Understood – and sorry to make a stir.. :-)

  10. Joey Winn says:

    Thank you all for your comments / information. I’m one of those ‘new’ Exchange Admins. I’ve only been doing it for about three years. (That’s ‘new’, right?)

    Well I actually DID encounter some problems with store.exe causing user-side memory errors (Exchange 2003). Two company users received a "90% memory used" – type of error message on their workstation. Sure enough, Store.exe had used up a huge amount of RAM (about 650Megs, out of our 1Gig available) which resulted in our server utilizing about 90% of the RAM available. No, there were no performance problems (unless you count the error and inability to send e-mails with any decent-sized attachment). Ok yeah, so there WERE performance problems. haha. Here comes my question….

    Can I limit store.exe to a specific amount of alloted RAM (perhaps 500 Megs for example)?

    Thanks. – joeywinn@yahoo.com

    Joey Winn

    Network Administrator

    Bellomy Research, Inc.

  11. Nino Bilic says:

    Joey,

    There is no way to actually limit the amount of RAM that Exchange Store will use. So – you can not set a cap (limit) on it. That was always the case for Exchange – even in 5.5, even though the UI in Performance Optimizer suggested that you can do this – it would actually limit the buffers for the store rather than putting the cap on the memory use.

    That being said – I would really suggest to open up a case with our support on this – if you are seeing memory errors.

  12. Colin Lee says:

    One check I would do is to ensure that you have the /3GB and /Userva=3030 switch in the boot.ini as you have 1Gb or more of RAM.

    http://support.microsoft.com/default.aspx?scid=kb;en-us;316739

  13. blast from the past says:

    What Nino tried to explain above (memory backoff alogrithm) is called affectionately DBA which stands for Dynamic Buffer Allocation. Basically depending on the memory pressure being put on the system (and being sampled at a fairly aggressive rate), the store will release memory back to the system. For example, go to your Exchange server and start up IE or MMC. Both spike memory usage at thier load time. If you are watching the memory allocated to each process you will see Exchange store give up memory to the system (so it can then turn around and give it to IE / MMC). You will have to trust me.. this is how it works.

    Now it wont release ALL of it at once, but will release it over (a relatively short) time. On the flip side, we don’t aggressively eat it as it becomes available. Again based on memory pressure we will consume free memory in chunks. Think of it as Exchange’s private memory manager that takes input from the entire system and is a benevolent participant.

    The reason for doing it gradually is that you don’t end up in a seesaw effect on the server where you are reacting to every absolute request for memory etc. Realize that the "memory" you are evicting is essentially the database cache and if you evict too aggressively and then re-allocate you end up with unpredictable and in-efficient IO patterns and your hit ratio on the cache drops (plus all the added IO’s required to flush the cache pages to disk and the subsequent reads to fill the cache).

    This now explains why on an Exchange server unused memory is essentially a wasted resource and if you can effectively approprirate and evict the resource, you net make your Exchange server as efficient as possible.

Comments are closed.