Holy COW! Changes to Recoverable Items versioning in Exchange 2010 SP2 RU3

Over the course of the last several months, we’ve received reports of excessive growth of the Recoverable Items folder (what many term Dumpster, although we no longer refer to it as such). There are two main scenarios that can contribute to this excessive growth.

  1. Single Item Recovery and Litigation Hold versioning
  2. Calendar Version Logging

Single Item Recovery and Litigation Hold

Single Item Recovery and Litigation Hold enable preservation of data within the mailbox. If you are unfamiliar with these features, see my previous post Single Item Recovery in Exchange 2010 for more information.

In addition to retaining data within the mailbox, both Single Item Recovery and Litigation Hold also enable versioning. Essentially, when an item is changed, a copy-on-write (COW) is performed to preserve the original version of the item. The original item is placed in the Recoverable Items\Versions folder. This folder is not exposed to users.

Given that Exchange 2010 includes a change in how clients connect and access mailbox related data, the copy-on-write activity occurs in the Exchange System Objects (XSO) layer on the Client Access server.

What triggers a copy-on-write?

  • For messages and posts (IPM.Note* and IPM.Post*), copy-on-write captures changes in the subject, body, attachments, senders/recipients, and sent/received dates.
  • For other types of items, copy-on-write captures any change to the item except for moves between folders and read/unread status changes.
  • Drafts are exempt from copy-on-write to prevent excessive copies when they’re auto-saved.

What can cause excessive growth in the Recoverable Items folder?

Overflowing-Trash-Can_thumb[1]Copy-on-write behavior can lead to excessive growth. We discovered that the following scenario using Microsoft Outlook as a leading cause of excessive copy-on-write:

  1. You create a calendar appointment.
  2. You attach an Office document to the appointment and save it
  3. Later, you decide to open the appointment to reference something in the Office document. You open the document.
  4. Behind the scenes, Outlook begins auto-saving this open appointment and its corresponding open document (by default Outlook auto-saves every 3 minutes).
  5. Each auto-save event triggers a copy-on-write. But because the auto-save is saving both the Office document and the appointment, there are two copy-on-write events. For each additional attachment, a subsequent copy-on-write version would also be created.

Ouch, is right.

Since attachments are part of the message, it doesn’t make sense to create a copy for every attachment in the item. Essentially, when you save an item that has an attachment, Outlook performs the following:

  1. CreateAttachment
  2. SaveAttachment
  3. SaveMessage

The copy-on-write was taking place at both SaveAttachment and SaveMessage. Digging into the code, we found that the call to SaveAttachment ultimately calls the Flush method (a means by which the client syncs the state with the server) on the message it’s associated with after the attachment is saved. It was this Flush call that was signaling the copy-on-write code to act.

After further analysis, we realized that the copy-on-write logic is triggered on any Flush call. This was a breakthrough finding because Flush can be initiated in many scenarios, potentially leading to the cause of the thousands of copy-on-write events that several customers saw in their environments.

With Exchange 2010 SP2 RU3 and later, copy-on-write now understands the difference between Flush and Save operations and will only be triggered when a Save operation occurs.

Calendar Version Logging

Calendar Version Logging is the process by which calendar changes that occur within a mailbox are saved via copy-on-write. Calendar Version Logging was introduced in Exchange 2010 to help troubleshoot and repair calendar reliability issues.

The design of Calendar Version Logging is to create a log every time a calendar item has been changed. These logs provide a history of the meeting. You can use the Get-CalendarDiagnosticLog cmdlet to review the history and determine which clients performed a destructive action. The second use for Calendar Version Logging is by the Calendar Repair Assistant, which uses the logs when trying to determine the history of a given calendar item for inconsistency detection.

Calendar Version Logging is enabled by default on mailboxes in Exchange 2010. You can disable or enable this on a mailbox via the CalendarVersionStoreDisabled property. Note, the property name is CalendarVersionStoreDisabled, so the default value of $false means that Calendar Version Logging is enabled by default.  Depending on the mailbox configuration, a different process is followed for storing copies of the calendar items:

  1. If the mailbox is not enabled for Single Item Recovery or Litigation Hold, then stripped versions of calendar items are maintained in the root of the Recoverable Items folder for 120 days. The stripped versions (body and non-first level or non-embedded message type attachments are removed) are created using copy-on-write.
  2. If the mailbox is enabled for Single Item Recovery or Litigation Hold, full copies of the calendar items are maintained in the Recoverable Items\Deletions or Recoverable Items\Versions folders. A stripped version is created via the copy-on-write infrastructure whenever a hard delete operation is performed against the calendar item in the Recoverable Items\Deletions or Recoverable Items\Versions folders. This stripped version is placed in the root of the Recoverable Items folder and kept for 120 days. The only time a stripped version is not created is when the item’s age in the Recoverable Items\Deletions or Recoverable Items\Versions folder is more than 134 days (120 + 14). This can happen if you change the retention period, if the Mailbox Folder Assistant hasn’t been executing, if Litigation Hold was disabled, etc.).

Due to the aforementioned issue with how the copy-on-write logic did not distinguish between Flush and Save operations, Calendar Version Logging, in some cases, could consume a large percentage of the Recoverable Items folder quota (and if you recall, the warning threshold is 20GB and the hard quota is 30GB).

Although SP2 RU3 addresses the copy-on-write issues, to address concerns that Calendar Version Logging could consume all of the Recoverable Items folder’s quota, in SP2 RU2 we introduced a change to the architecture whereby Calendar Version Logging now takes into account the Recoverable Items folder size prior to initiating a copy-on-write.

If the folder size is greater than the RecoverableItemsWarningQuota, Calendar Version Logging is disabled for the mailbox. The RecoverableItemsWarningQuota value that is used depends on the mailbox’s settings:

  1. If the mailbox has the UseDatabaseQuotaDefaults set to $true, then the mailbox database’s RecoverableItemsWarningQuota will be used.
  2. If the mailbox has the UseDatabaseQuotaDefaults set to $false, then the mailbox’s RecoverableItemsWarningQuota will be used.

When Calendar Version Logging is disabled, the following event is generated in the Application event log on the Client Access server:

Event ID: 5003
Source: MSExchange Mid-Tier Storage
Task Category: CopyOnWrite
Level: Information
Description: User mailbox <legacyExchangeDN> has exceeded the dumpster warning quota. Calendar logging has been disabled for the mailbox.

And that’s not all we’re doing. I won’t go into specifics at this early stage of development, but we’re working to further improve Calendar Version Logging to minimize the impact this feature has in your deployment. When I can share more, I will do so via this blog.

Ross Smith IV
Principal Program Manager
Exchange Customer Experience

Comments (9)
  1. Gulab says:

    Thank you for explain it, It's very help in troubleshooting. Thank you so much Smith!

  2. Dean Uemura (MVP - Exchange) says:

    Thanks for the heads-up Ross!  I just know I'm going to run into this soon.

  3. Frank (www.frankysweb.de) says:

    Ross, thank you! very great explanation!

  4. turbomcp says:

    Thank you Mr. Smith:)

    but seriously thank you

    i hope it also resolved the issues with third party archiving software which created multiple copies when they runs job(to create shortcuts for items with attachments for example…)

  5. Dan Sheehan says:

    @Ross – speaking of improving the recoverable items (dumpster) – any chance you all are planning on reintroducing the ability for users to recover items deleted from a specific folder and or the folder heirarchy?

    In Exchange 2007 you could restore everything deleted in just a specific folder such the Inbox or the Calendar individually. While this was tedious if you wanted to recovery everything deleted in a mailbox across multiple folders, you could restore a folder hierarchy and put things back the way they were in a very controlled manner.

    In Exchange 2010 deleted items all seem to go into a single folder/bucket, and when you restore items they get restored to the folder you are currently focus on, not where they came from nor the hierarchy they were in.

    While this might not sound like that big of an issue, it makes troubleshooting things like a POP client downloading everything from the Inbox a lot harder after the fact. Also if a user deletes a number of folders with a specific hierarchy and items in them, and they want their mailbox put back the way it was, we have to resort to a backup file to restore their mailbox since the recover deleted items doesn't seem to maintain this level of detail like the previous version of Exchange did. I.E. the lack of folder heirarchy retention makes certain troubleshooting harder, and makes more work for us in that some recoveries require a mailbox restore versus a few minutes of combing the mailbox and recovery items from where they were deleted.

    Sorry if this is a little off topic.

  6. @Dan – Unfortunately this was one takeaway when we moved to the Recoverable Items architecture (moving away from a special flag/view to delineate deletion (blogs.technet.com/…/single-item-recovery-in-exchange-server-2010.aspx) to an actual folder structure).  Currently there is no MAPI property that defines source folder or folder hierarchy, so that sort of information isn't maintained and thus cannot be recovered.  I've let the component team know about your feedback as they are open to hearing feedback on this subject.


  7. pesospesos says:

    great post Ross.  I'd like to add my hat to Dan's ring – if there's some way to work this info back into the recovery items it would be most helpful.  thanks!

  8. pete says:

    Hey Ross,

     Thank you for the Great Post, I would also like to add to Dans request. We would like to see a mixture of both, we liked the ability to recover the items back into the original folders they were deleted from (Exch 2007)but also hated that you had to search each folder for deleted items, now with Exch 2010 you get the ability to see all deleted items but the ability to put back were they cam from is very limiting.

    I have always thought it would be nice to click on the top level of the mailbox to see all items that can be recovered and then drilling down to the individual folders to see the items that could be restored from that particular folder if need be, all with restoring back to the original location.

  9. GabooN says:

    How can we clean up and or recover this space?  I have a user that is at 20GB already for the RecoverableItemsRoot

Comments are closed.