Recently we've seen some cases in Exchange Support where the error event 623 gets generated immediately at the start of the Information Store service. So far we've only witnessed this on some Exchange 2003 Servers. Please note that this is specifically about this event being generated after the IS start. There were other causes of even 623 that we had fixes for already.
When this behavior occurs, you may see the Information Store appear to take upwards of 45 minutes to fully respond at service startup. Monitoring "Version Buckets Allocated" (viewable with Show Advanced Counters - see Nageshs' excellent post here: http://msexchangeteam.com/archive/2006/04/19/425722.aspx) will show the Store is immediately running high (over 70%) and until the number falls the Information Store will be unresponsive to clients and ESM. After Version Buckets Allocated falls, the server responds fine and no other issues are observed. 623 errors go away. Restarting a 3rd party server that ties into users' mailboxes (if present) or restarting the Information Store service may cause the issue to occur again.
This problem occurs because of a large amount of hidden search folders that have been created by applications (other than Microsoft Exchange) that have access to users' mailboxes. When the Information Store starts, it becomes available to the host of 3rd party applications which might reconnect and want to sync the contents of the search folders at the same time. These search folder updates can then result in the search folders for a user's mailbox to all be updated at the same time. When a mailbox has a large item count in the Inbox folder (more than 5,000 items) you can experience higher than normal store CPU % utilization and Version Buckets Allocated spikes which can lead to version store out of memory problems. Depending on the type of search performed, the impact can be greater or smaller. Once version store cache has been depleted, the offending transaction gets canceled or it times out and is rolled back and everything moves along as if nothing happened. That's why the event 623 eventually corrects itself.
To avoid this scenario, there are a few things you can do to monitor this:
- Keep your Inbox item count down to 5,000 or less. In some cases with this problem we've seen 60,000 to 80,000 items per user Inbox. To find out if you have a problem like this, we suggest you use the Exchange Server Profile Analyzer tool which we blogged about here.
- Keep an eye on the number of search folders querying against the Inbox folder. This will require you to run ISINTEG on your server (please see below for what exactly to look for). Most people don't realize is that some third party applications that plug into Exchange (for example Fax servers, Mobile device synch servers, Unified Messaging clients, desktop search clients) create hidden search folders and restricted views. Each time a change happens in the folder that is being monitored (an modification, deletion, addition) - backlinks to the search folders are looked at and we will evaluate each search folder to see if this new items meet that set view.
- It is possible that when 3rd Party products are upgraded older versions of search folders are not cleaned up as well. In some cases we've seen users with well over 150 hidden search folders. Just a few users with high item counts in their Inbox and this many hidden search folders can cause some serious trouble for your environment.
So - how do you do #2 above?
You'll have to run: isinteg -s servername -dump -l logfilename
Then open up the "logfilename" file and look for the following:
 Folder FID=0028-00000002451E
Scope FIDs(search folder only)=
What we're looking at here is a high number of Search folders (Search FIDs above) and Search Backlinks that - when they have to generate or update - have to scan over 29000 items each (MsgCount above). This is the crux of the 623 version store problem at startup that you might be seeing.
At this time, we do not have a simple solution for this problem... If you have this problem and have identified it using the above step #2 (as situation described in step #1 can be solved by reorganizing the folders/number of items), please contact our Exchange support line. Once we have a better way of resolving this, we'll post about it here.
For more information on Search Folders, review:
KB260322 - How To Search Folders with the SetSearchCriteria Method http://support.microsoft.com/kb/260322/en-us
Best Practices for Exchange 2003 Search Folders (there are several subsections here to look at as well)
Creating Search Folders:
Exchange Store Search Folders: