Case of the corrupted calendar items

First off, I’d like to apologize for the large gap in blog posts. Even 6 1/2 months in I am still trying to get a complete handle on all things PFE, thus the gap.

In all the areas our customers present issues with, we in premier field engineering are only second to our PSS (Premier support services) & CTS (Consumer technical support) in the level of detail and volume of odd issues. The following blog post outlines one such issue.

Background..

mixed Exchange 2003 & 2007 server environment. Large number of users 50k +. Clients using a mix of Outlook 2003 and 2007 in accessing the mailboxes. This particular group of users that experienced the calendaring item “corruption” was on exchange 2003 and have recently moved to Outlook 2007 Sp2. The reported issues were as follows..

Calendar reminders either not triggering or unable to be cleared

Calendar items mysteriously being removed from the calendar, even off of the meeting organizers

Opening a reminder and the calendar item itself is completely blank including subject / attendees / etc..

After some initial triage we found this to be an issue only with Cache mode clients. Once we moved the users to Online Mode, the issues immediately disappeared. We tried then creating a new local OST and even a new profile with immediate issues reappearing. This initially led us to think this was an issue with the way the OST file was being created with Outlook 2007 sp2. We had them do some initial testing with some SP3 clients to rule out a patch or bug that could have been fixed. An interim work around was to leave any users reporting the issue in online mode until the root cause could be located. The only issue with this is there was a portion of the affected users that were mobile and although an interim “manual” syncing of certain folders worked, it introduced additional steps the users had to perform which was deemed unacceptable.

During this investigative process one way the client was trying to resolve this issue was using the MFCMAPI tool. They were able to manually deleted the reminders which couldn’t be dismissed by the end users, but couldn’t address missing items in the local OST that were there when viewed using OWA. This meant they had a partial fix, but even this fix was a long manual process which wasn’t feasible in the long run.

Then we tried to look at some of the commonalities of the affected users. Did they all work for one department? did they all use the same Outlook client? Did they all use a similar type of mobile device and tried modifying mail (more specifically calendar) items from mobile devices? Now I have to give this customer a lot of credit here. They published some internal best practices documents to try and minimize possible causes of mailbox corruption (reduce delegates, don’t use more than one version of outlook for mailbox management, don’t use mobile devices for calendar item management, etc..) After some more analysis we found ALL the affected users were in one data store on one server. They then moved the mailboxes to another store on another server and the users no longer had the issues.

NOTE: Some of them did have to recreate their OST local cache, but once complete all was well

After the mailstore was empty, they noted they couldn’t delete the database as it appeared to have some orphaned items in it. they forced the maintenance cycle on the database which then allowed them to finally and permanently delete the EDB and STM files.

Case solved.