How high item counts can influence 623 events
In my earlier blog about 623 events I mentioned there are a great many possible causes. Twice in the last 1.5 months I have seen a situation where high item counts were the major contributing factor for 623 events.
In both cases the dumps were collected and they revealed that the store was busy working on an "empty" search. What this means is that the store was working with the internal MAPI Search Folders that had been constructed for a folder that had too many items in it.
With Exchange 2010 the architecture has changed, but in Exchange 2007 and lower the store creates and maintains numerous MAPI Search Folders on each folder (also known as Restricted Search Views as outlined in this article https://technet.microsoft.com/en-us/library/cc535025(EXCHG.80).aspx). There is a cap on the number, but until that cap is reached every unique Search or Sort that is done on the folder is tracked. The resulting MAPI Search Folder is then updated with every change to the folder. Typically the store maintains 3-5 MAPI Search Folders per folder, but if you are in Outlook (online mode) and change the Sort order to one Exchange did not already have built then Exchange will build a new MAPI Search Folder. The database engine will then maintain that MAPI Search Folder until it expires (typically 40 days).
So you're probably thinking - Why is this bad? Aren’t these just indexes? That is what databases do.
Even if these were just indexes you would only be partially correct. The issue comes in the number of MAPI Search Folders the database engine can end up maintaining and the resources it consumes to constantly update them. Most databases do not dynamically create and destroy indexes. If your IT staff or developers make you a SQL database they will optimize the indexes to give the greatest possible speed to the forms and queries that will be commonly used. This allows them to account for performance considerations at design time. Exchange and Outlook are creating most of these MAPI Search Folders on the fly. This makes it impossible to foresee all performance scenarios. In addition to this the Search Folder is more than an index. It is actually a separate table that is linked back to the original. Thinking of the MAPI Search Folders as an index is too much of a simplification.
When a new item is added to a folder and the Store begins updating the MAPI Search Folders those updates all go into a single transaction. If there are 30 MAPI Search Folders to update then all of the pages that will be dirtied get loaded into the Exchange Version Store as part of a single transaction. That causes the consumption of Version Buckets to rise (as seen in the counter "\Database(Information store)\Version buckets allocated" ). The number of items influences how long it takes to traverse each MAPI Search Folder and carry out all the proper page splits, writes, etc. for the update. If there are a lot of MAPI Search Folders that are very large they will each take longer to load, examine, and update. Therefore pages in the Version Store will be held longer. The longer those pages are held the greater the chance we will have a 623 event occur.
Now if this is just one mailbox that is over by a little bit this probably won't hurt and won't cause a problem. However, if you have numerous mailboxes with high item counts and the users are active in the folders with the higher counts, then the cumulative effect can snowball into a 623 event.
By default Exchange only maintains a maximum of 11 Restricted Search Views per folder and updates each for 40 days (after which, if it hasn't been used in 40 days, it is removed). However if an online mode client has multiple regional settings this can turn into one set of 11 per language. Also Administrators can raise the number from 11 to whatever they like (a max of 50 is recommended in https://technet.microsoft.com/en-us/library/cc535025(EXCHG.80).aspx). Raising the number from 11 can be desirable because you can prevent dynamically created Views from being bumped out of the list and replaced by more recently requested views (if users are constantly causing turnover between views it can cause other performance problems).
You can get short term relief from running Outlook /cleanviews. However, because the views are created dynamically you can very quickly return to the problem. In the relatively rare case where this problem is occurring it would be better to reduce item counts to avoid the problem. You can check item counts with the steps outlined in this blog: https://blogs.technet.com/b/exchange/archive/2009/12/07/3408973.aspx