Optimizing Outlook 2007 Cache Mode Performance for a Very Large Mailbox


My mailbox size is 7.4GB. I send/receive around 200 emails a day; so that 7.4GB of email goes back several years. I don’t use too many rules so most of my email ends up in my Inbox (30k items).

If you have read Exchange mailbox sizing guidance, you will notice that my mailbox size is higher than the recommended max OST size (2GB), and I store more items in my Inbox than the recommended (5k). Since I am one of the contributors to this guidance, I felt I should come clean (by writing this blog); and explain how I use my email that meets the following criteria:

  • Large Mailbox (several years worth of email stored in the Exchange Store)
  • No PST usage
  • Good laptop cached mode user experience
  • Good workstation/home computer cached mode user experience
  • The ability to search my entire mailbox
  • My email processing does not over burden the server

Before I get started, I should probably tell you what hardware/software I am using

Software

Office Workstation

Home Computer

Laptop

Windows Vista SP1 RC1 (x86)

2 core 2.66ghz

2 core 3.2ghz

2 core 2Ghz

Office 2007 SP1

4GB Memory

2GB Memory

2GB Memory

 

10K SCSI disk

7.2k SATA disk

5.4K SATA disk

*The software is the same across all machines.

**The disks sited are those which host my Outlook 2007 OST (I am the Exchange Storage Program Manager; I have lots of disks in all of my computers).

Not surprisingly, I have a different user experience processing my mailbox depending upon which machine I am on. Also not surprisingly, my experience has changed as my mailbox size has grown. As my mailbox began to exceed 2GB, I noticed that my laptop experience began to degrade. It took longer to sync down new messages, longer to do sorts etc. As my mailbox exceeded 4GB, I noticed the same thing on my home computer. I now experience the same slowness on my office workstation with my 7.4GB mailbox (don’t ask what my laptop experience is like). This is expected since each machine has a different performing hard disk (which my Outlook 2007 OST is stored on) and the fact that Outlook storage performance will slowly degrade as the mailbox size increases. Outlook development folks have the same max mailbox size as me (~10GB), so they are also very much aware of the performance issues associated with a large OST; and are working toward addressing the issue in future versions of Outlook. They made great progress in improving OST scalability with Outlook 2007 Sp1, but I hear there is still work to do.

My background is in software performance (especially storage performance); so I couldn’t live with the status quo. I was stuck, I needed my large mailbox; but I also needed a great/fast user experience so I could effectively process my mailbox. All of my machines had sufficient memory (2GB or more) so I couldn’t solve the problem by adding memory (to provide the Windows System Cache with more memory to buffer the OST IO’s). I basically had the following options:

1. Increase the IO performance for the OST. This would involve doing one of the following:

a. Installing the fastest hard disk available (15k 2.5" SAS). This would require a new disk controller.

b. Building a RAID10 array of several disks. Since I am a storage guy, the RAID10 approach (combined with a nice battery backed write caching array controller) was rather appealing.

c. Installing a flash disk. Flash is still a bit too expensive and the write performance does not match the read performance (not even close) on today’s devices. I wanted a good user experience both when Outlook 2007 was syncing down lots of new messages, as well as when the OST was up to date.

2. Run Outlook 2007 in Online Mode. This moves the IO burden from the client to the server and doesn’t address the disconnected laptop scenario.

3. Limit the amount of email synced to my OST. Outlook 2007 has the flexibility to limit what folders get synced to your OST. You can reduce the size of your OST by configuring sync policies on a per folder basis. By configuring this, I could have my 7.4GB mailbox but only sync down a subset (say 2GB) to my OST.

Option 1 was what I was originally targeting but it didn’t make much sense to solve the IO problem only on my workstation and home computer (2.5" SAS disk or RAID array); but not solving the problem on my laptop. A flash disk would work for all my computers, but I wasn’t willing to invest that much cash to solve this problem; nor was I happy with the random write performance of these devices. Option 2 was a non starter since it breaks for my disconnected scenarios (Wi-Fi is not quite ubiquitous). I could use Online Mode for my workstation and home computer; but I would be adding additional burden to the server in the process; and still wouldn’t have a laptop solution. This left me with Option 3; limit the amount of data I sync to my OST. This is what I did.

Step 1: Designate a small set of "archive folders" and move old items in to them (or us MRM to automate this)

We utilize MRM (Messaging Records Management) policies in our Exchange 2007 deployment. The policy is configured like this:

  • Any email older than 1 year gets moved to "Managed Folders\Will Expire" folder. I have 90 days to move any email moved to this folder to another "Managed Folder" with a longer expiry policy before it gets deleted.
  • "Managed Folders\Research and Development" is a Managed Folder that has no expiry policy. I move messages in to this folder that are related to Research and Development that I want to keep longer than a year. The majority of my email relates to R&D so the majority of my email makes it in to this folder eventually. To help organize the folder, I have created subfolders based on year beneath this folder:

Managed Folders 
      Research and Development (No Expiry) 
            2004 
            2005 
            2006 
            2007

Personally, I manually manage moving old R&D messages in to the correct R&D managed folder (messages from the year 2005 go in to the 2005 folder etc…) MRM could also be configured to do this for me (and for everyone else in the Organization for that matter), but would not provide the flexibility of picking and choosing which messages get moved to the Managed Folders. If you wanted to move all messages of a given age for all users to a given Managed Folder, an MRM policy could easily be configured to do this. This is ideal for organizations who have unlimited expiry policies and want to provide the ability to easily mass move old items out of the primary folders and in to a Managed Folder hierarchy. More info on MRM can be found here: http://technet.microsoft.com/en-us/library/bb123507.aspx

My mailbox is now segmented so all of my email from the last year is in my personal mailbox folders (Inbox, Sent Items etc..); and all of my email that is older than one year is stored in the year based folder hierarchy underneath "Managed Folders\Research & Development".

Step 2: Configure Outlook 2007 to not sync the "archive folders"

Now that my mailbox is segmented, I want to configure Outlook 2007 (Cached mode) to only sync the email for the past year. I rarely access email that is older than 1 year (a couple times a month at most), so it is a worthwhile trade off to sync only recent email to my OST and thus improve the client experience. If I need to access email that is older than a year, I either bring up Outlook 2007 in Online Mode or use Outlook Web Access (OWA). Since accessing +1year old email is so rare, I have yet to hit a scenario where I was on the road and disconnected from a network and needed to access an old email. If it was a critical situation and I needed to access that email immediately, I could always access it via my Smart Phone and Exchange Active Sync. Configuring Outlook 2007 to not sync the "archive folders" is straightforward:

1. In Outlook 2007 (Exchange Cached Mode), right click the folder you want to set the sync policy on and choose Properties (e.g. Managed Folders\Research and Development\2006)

2. Click the Synchronization tab, and then click Filter.

3. Click the More Choices tab.

4. Set the Size (kilobytes) to "less than" 0. This is a simple way of configuring Outlook to not sync anything for this folder.

5. Click OK

6. If you already synced items to this folder and want to clean them out of the OST, then you need to go back to the Folder Properties General tab and click Clear Offline Items. This will expunge already synced items from the folder based on your new policy. I usually perform these steps on a newly created OST (to prevent the "archive folder" syncs occurring in the first place).

7. Follow steps 1-6 for every "archive folder" you do not want to sync to your OST. I recommend using a just a few archive folder for this reason.

8. Follow steps 1-7 every time you create a new Outlook Cached Mode profile on any computer. I followed these steps for my office workstation, my home computer and my laptop.

Step 3: Enjoy the results

I have been very happy with Outlook 2007 performance after segmenting my mailbox and only sync’ing a year’s worth of kept email (~2GB). This approach has allowed me to meet all of my criteria:

  • Large Mailbox (several years worth of email stored in the Exchange Store)
    • I have access to several years worth of email from any device
  • No PST usage
    • Since my mailbox is so large, I do not need PST’s. PST’s are a security/reliability risk as well as being a pain to manage and use. Life is much simpler with all of my email in one place, my mailbox (accessible via all of the standard clients). Keeping all of my email in my Exchange mailbox is the best/safest place to store it. It is stored on RAID10 storage, replicated via CCR, and is backed up on a regular basis. Before my large mailbox, my PST’s were stored on non-raided storage, not replicated and backed up maybe once or twice a year.
  • Good laptop Outlook 2007 cached mode user experience
    • By segmenting my mailbox in to two buckets (less than a year old and over a year old) and configuring Outlook 2007 folder sync policies, I have dramatically improved the client performance; even on my 5.4k laptop disk.
  • Good workstation/home computer cached mode user experience
    • Outlook 2007 cache mode performance screams on my workstation and home computers (again, thanks to mailbox segmentation and Outlook 2007 folder sync policies).
  • The ability to search my entire mailbox
    • For those few times a month I need to access email that is older than a year, I just do a quick search in OWA or open up Outlook in online mode.
  • My email processing does not over burden the server
    • By running all of my clients in cached mode, I reduce my personal impact on the server (IO and CPU processing), by offloading some of the mail processing work to my client. The server costs do increase as the mailbox size increases when running in Online Mode. Running in cached mode solves this.

I have found that a large mailbox, combined with a few Outlook configuration tweaks can make for a great user experience. I have the best of both worlds; my Outlook experience is snappy, and I have access to all of my email from any standard Microsoft client. Till Outlook improves the OST scalability further, this is the configuration I live with on a daily basis. I look forward to the day where I can keep full copies of my mailbox synced to my client; but for now, I am satisfied using this "creative approach".

To most of you, a 7.4GB mailbox is pretty big (huge even). I have worked with several customers recently who still have 25MB max mailbox sizes (I would fill it up with my profile in 2.5 days!). I have also worked with customers who have unlimited mailbox sizes and have users with mailboxes in the 20GB -> 60GB range! I have not tested this selective sync folder approach to 60GB mailboxes, but in principle it should work. It has got to be better than the 60GB OST experience!

Matt Gossage


Share this post :


Comments (17)
  1. Matthew Loraditch says:

    Thanks for the info. I’ve got a few users with mailboxes between 5 and 10GBs and this will be useful for me getting them to be manageable. I really hope that OST performance gets improved in Office 14. More and more people have gigantic mailboxes and whether they actually need the old emails or not they keep them and the Outlook team has gotta make it so these mailboxes are usable.

  2. Kevin G says:

    I wish PST would be removed.  I have users that believe storing PST on a network share is great….

  3. Ian_m says:

    Any idea how to propagate these setting into all subfolders below the unsynched one ?

    A lot of our users already move e-mail to yearly folders but each project/task/customer etc is in a seperate sub folder, meaning the properties of a lot of subfolders has to be set, which means this handy tip will not get done !!!

  4. Rickp says:

    You will have an even faster OS experience with Windows XP vs. Vista SP1 RC1.  I’m not trying to flame.  Kevin, there is a GPO for Office that prevents PST creation

    http://support.microsoft.com/kb/896515

    There are also 3rd party tools which will find mapped PSTs and suck them off to SAN automatically for users.  Lookup Quest Archive Manager for Exchange.

  5. Brian Hoyt says:

    I have to follow-up Rickp here and further ask why.  Using two identical hardware machines with one running XP SP2 and the other running Vista (SP1) Outlook 2007 performance is horrible on the Vista machine in comparison.  I had to upgrade the HDD on some of our large mailbox users from 5.4K to 7.2K laptop drives to make the machine more useable.  I don’t know what is happening in the disk usage on Vista to cause this but it would be great if it could get fixed.  There is a limit to disk i/o to throw at this issue on laptops.

  6. Robert says:

    Or… one can adopt the tenets of Inbox Zero. Life is nice when your Inbox is empty. :)    inboxzero.com.

  7. OSTPST says:

    Nice article, but I fail to see the point. How are you not impacting the server? You have almost 8GB of email that needs to be backed up and virus scanned daily. No PST?….what happens if your boss comes to you one day and tells you that it is your last day. You’ll have to find time (if allowed) to pull all that email down so you have it for future reference.

    Here is what I do. Every year gets a .PST file and once a week I move the Inbox and Sent items into the current .PST file. Then I use X1 search engine to locate any email that I am looking for. Then I use Remote Desktop/Terminal Services and/or port 80 remote control software like logmein.com to share amongst computers.

  8. Dustin Lema says:

    Ummm…so the newest version of Outlook (2007) isn’t quite ready for the 2+GB ost size?  I’ve got clients and users that are routinely over 5GB and some to 20GB….and there’s no telling them they can’t have that much email when they just footed the $50K-$150K to upgrade their mail system to the latest and greatest.  Integrate the UM aspect of E2K7 and watch the MB sizes grow.

    I’ve got a similar scenario to Matt’s post whereby the laptop ost load is slow….and I’ve got a 7200rpm drive.  The OST is 2.5GB.  There should be a method where the OST only loads the inbox and (maybe) the last 5 folders used….that way it blasts open….then in the background, it can more slowly load the remaining folders/items unbeknownst to the user.  This might be the interim workaround instead of waiting to load a whole OST.

  9. OscarSotoCL says:

    Mmm, i don´t know about internal structure of ost files, but, why don´t use a second or third OST file. Could outlook use more than One OST file?. Are there any benefits using two or tree shorter OST Files.  I Think outlook could hanlde automatic creation of OST and limit each OST to a maximun size. 2GB?, 3GB?

    Greetings

  10. Allards says:

    I think people with such large mailboxes should learn to cleanup there old mail. Removing useless attachments, deleting nonsense e-mail and emptying the trash folder cleans the mailbox quit well. I can’t believe that there is good reason to store everything. How much time it will cost only to scan the headers, 3 months?

  11. Mike Crowley says:

    -do you need to use mrm for this?  Can you use the regular mailbox folder structure instead?

    -do you need e2k7 for this or just o2k7

    -you mentioned outlook 2k7 sp1, the kb article is very long, and full of hotfixes, whats changed with sp1?

  12. Frank says:

    I am migrating to 2007 (from 2003), and at the same time, I setting up a new 2007 server in a different geographic location. I have 70 users that I would like to move to the new server, and many of them have 4GB+ mailboxes.

    The only way that I could find to move huge mailboxes internationally was to disconnect a message store, and use an external HD with sneakernet (okay a 10hr international flight) and then use the "move-mailbox –configurationonly" cmdlet.

    Can you recommend any way to move *huge* mailboxes without downtime, without affecting users, and without a lot of international bandwidth?

  13. mattgos says:

    Ian_m

    Nope, sorry, there is no way to propogate your sync settings to sub folders.  What I describe in this article requires a search based mailbox structure (simple organization combined with search to find what you need).  It does not work well with the “create  sub folder for every category” type of mailbox processing.

    Rickp

    I am not going to argue the performance aspects of XP vs Vista.  Regardless of which OS you use, you will hit performance issues with large OST’s at some point.  The methodology I described above works well on both OS’s.  

    OSTPST

    The server resources to maintain my 8GB mailbox are minimal.  It gets fully backed up once per week (incrementals daily) and does not get virus scanned daily (we scan at the transport level).  Since I am on CCR there is a fast recovery mechanism so the chance of going to the backups is very, very small (we have yet to do it in over a year in production).

    I am not sure about where you work, but where I work my email is not mine, it is the corporations.  If I am terminated, I will not be taking my work email with me; it stays with my job.  This issue highlights an aspect of the security risks with PST’s.  Not using them is more secure for exactly the scenario you describe.

    Dustin Lema

    Yes, +2GB OST on Outlook 2007 present a performance problem.  The point where it becomes painful depends on several factors; most importantly the hardware (disk perf).  The whole point of this article is to provide a method for folks to have those 20GB mailboxes and still have a good cached mode user experience.

    Mike Crowley

    No, you don’t need to use MRM for this, but it reduces the mailbox management burden on the user (for this scenario).

    E2k7 is required for the MRM functionality (and better performance due to 64bit and lower IOPS on the server).  The manual method will work with E2k3 just fine.

    A series of performance optimizations were made to Outlook 2007 SP1 to improve the OST performance.

    Frank

    Sorry, there is currently no ingenious answer to your question.  We understand this is a serious issue and is something we hope to address down the road.

  14. DC_Ranger says:

    mattgos, ref Frank and geographic locations

    You have all the technology at hand, the process of moving a mailbox from one geographic location to another shouldnt be more difficult that synchronizing a handheld; allowing Exchange to recieve synchronization from Exchange 03/07 to a temporary store, then migrate when BITS completes the actualy data transfer..how hard can it be….

    A manual workaround has so far been to set up a cached mode client on the site remote to the source data store, let it synch, then extract a PST, then import into 07….

  15. behzad says:

    I think an efficient way to deal with this is to have an option (definitely server-based, possibly client-based as well) to limit the overall size of the OST based on its total size as well as percent free space on local drive. Then simply do first in first out. There should also be an option to limit OST usage based on age of items (e.g. only cache items that are 180 days old or newer). Then you can effectively manage OSTs, for example:

    – Cache items based on age (e.g. 180 days)

    – Limit size to 4GB (if that means you can keep only 37 days worth of emails, that’s what you get)

    – Limit size to no more than 50 percent free space (if you only have 3GB available, your OST is limited to 1.5GB)

    I believe Symantec and EMC provide similar capabilities in their archive products with their client-side offline storage and Outlook Cached Mode should definitely do the same. Obviously improving overall performance within Outlook beyone 2GB is also a must as others have noted (2GB is way too small with UM and general corporate mail trends).

  16. TechieBird says:

    Great info – we deal with performance degradation due to huge OST size and/or large item count all the time where I work.  The biggest problem for us is that we use cached mode as a buffer to save the mailbox servers getting hammered by client-side operations that poll the mailbox – the most obvious example is desktop search tools, but there are other scenarios.

    Don’t get me wrong, I’m not knocking your approach because it’s absolutely valid… but sadly for us I think it would break some of the protection cached mode allows us to give our servers :o(

  17. Neil says:

    Well, I do not know if this is related, but I cannot delete, or move ANYTHING in my Outlook email.  Including tasks.  

Comments are closed.