Mailbox Quarantine

One corrupted mailbox can have the potential to disrupt service by taking down the entire information store, thereby affecting all users on that server. Mailbox quarantine has been introduced in Exchange Server 2010 to help prevent this situation.

What is Mailbox Quarantine?

Mailbox quarantine is a feature in the Exchange Server 2010 information store. Based on values in the registry, the store detects a mailbox or mailboxes that have the potential to or have caused the store to crash and quarantines them for specific period. The mailboxes that have the potential to crash the store are called Poisoned mailboxes.

When does quarantining happen?

Quarantining of mailboxes can occur in two situations:

  • A thread that is doing work for a mailbox has crashed.
  • More than 5 threads allocated to process a mailbox, have not progressed for long time.

How does it work?

The store will tag a mailbox that has the potential to crash the store. The tag includes the number of times that mailbox has caused the store to crash and a time stamp. If the store is crashed by a mailbox, a registry key is created. The path to the registry key is:

HKLM\SYSTEM\CCS\Services\MSexchangeIS\Servername\Private-dbguid\Quarantined Mailboxes\ {Mailbox GUID}

It will have the following two values:

  • CrashCount The number of times the mailbox has crashed the store.
  • LastCrashTime The last time the mailbox crashed the store.

The key is not created until the store has been crashed at least one time by a mailbox.

Each time a database is mounted, the store checks the registry to see if any mailboxes hosted on this particular database is tagged. If the registry indicates that a mailbox has caused the store to crash the mailbox will be quarantined.

By default, if a mailbox has been identified as a threat 3 times in 2 hours then that mailbox will be quarantined for 6 hours.

These default settings can be modified by creating the following key:

HKLM\SYSTEM\CCS\Services\MSexchangeIS\ParameterSystem\Servername\Private-dbguid\Quarantined Mailboxes

Using the following values:

  • MailboxQuarantineCrashThreshold – The number of times a mailbox can be identified before the store quarantines it.
  • MailboxQuarantineDurationInseconds – The number of seconds a mailbox remains in quarantine before the store releases it.

These two values do not exist by default. They should be created only if there is a need to change the default behaviour.

A background process in the store runs every 2 hours (this default can’t be changed) to check the registry values for each mounted database. The store checks the CrashCount and LastCrashTime values and performs any of the following four actions:

  • If all tagged mailboxes have a CrashCount value less than the MailboxQuarantineThreshhold (default value of 3) in the last 2 hours, then the dbguid registry value for the mailbox located at HKLM\SYSTEM\CCS\Services\MSexchangeIS\Servername\Private-dbguid\Quarantined Mailboxes is deleted.
  • If a tagged mailbox has a CrashCount is greater than the value MailboxQuarantineThreshhold (default value of 3) and a mailbox is not quarantined then the mailbox will be quarantined immediately.
  • If a mailbox has been quarantined longer than the default 6 hours or the time specified in the value MailboxQuarantineDurationInSeconds then it will be released immediately.
  • If a mailbox is quarantined for less than the default six hours or time specified in the value MailboxQuarantineDurationInSeconds then it will remain quarantined.

What happens when clients try to access a quarantined mailbox?

When a client attempts to access a mailbox the following occurs:

1. The store will return an error code ecMailboxQuarantinedand basedon this, XSO throws the transient exception MapiExceptionMailboxQuarantined to signal transport and other XSO clients

2. Every 5 minutes transport tries to deliver message sent to a quarantined mailbox

3. Outlook clients see the following pop up when they try to access a quarantined mailbox

clip_image002[1]

4. OWA displays the following pop up error message when trying to access a quarantined mailbox

clip_image004[1]

Only clients such as MFCMAPI that can pass Open-As-Admin flag can access a mailbox while it is in quarantined state. Even Exchange processes such as content indexing and mailbox assistants cannot access the mailbox.

For example, a move mailbox request will fail with the following pop up error:

clip_image006[1]

Resetting a quarantined mailbox

It is possible to reset a quarantined mailbox by deleting the quarantine registry key for that mailbox located at:

HKLM\SYSTEM\CCS\Services\MSexchangeIS\Servername\Private-dbguid\Quarantined Mailboxes\ {Mailbox guid}.

The database then has to be dismounted and remounted or the IS service restarted for the reset to take effect immediately. Unless the underlying issue is not resolved, the mailbox could crash the store and become quarantined again.

Troubleshooting

Application log

The following event will be logged when a mailbox is automatically quarantined:

Log Name: Application

Source: MSExchangeIS

Event ID: 10018

Task Category: General

Level: Error

Description: The mailbox for user /o=AMERICAS/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=test1 has been quarantined. Access to this mailbox will be restricted to administrative logons for the next 6 hours.

The following event will be logged when a mailbox is automatically removed from the quarantine:

Log Name: Application

Source: MSExchangeIS

Event ID: 10019

Task Category: General

Level: Error

Description: The quarantine of the mailbox for user /o=AMERICAS/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=test1 has expired. Access to the mailbox has been restored.

Shell Command

We can also use the Get- MailboxStatistics cmdlet to see if a mailbox has been quarantined.

Get-MailboxStatistics –identity test1 | FL Isquarantined

Isquarantined : True

Performance Monitor

The store also provides the performance monitor counter: MSExchangeIS Mailbox\Quarantined Mailbox Count. This counter shows the number of quarantined mailboxes on a specific server.

EXTRA

Finally we can used EXTRA to trace data. Select Function from Trace Types and use the tag tagQuarantineMailboxunder component Store .

clip_image007

Thanks to Hamza Hassen and Jonathan Runyon for putting all this information together which will help so many of us certainly…