Why Exchange Databases Might Remain Dirty After ESEUTIL /R Recovery


You’re trying to recover your Exchange database from the “dirty shutdown” state to “clean shutdown.” You use ESEUTIL /R and after several ill-fated attempts the databases remain dirty. What’s happening? Danijel Klaric. a Microsoft Premier Field Engineer based in Germany, sheds some light on why the Exchange 2007/2010 databases might remain dirty after a seemingly “successful” ESEUTIL /R recovery, and provides pointers on how to solve this. Enjoy his article!


Normally I wouldn't think that this following topic would ever be a problem, but as I faced the issue two times in the last three weeks, I thought it would be good to shine some light on the subject.

Both customers are using Exchange 2010 with Native Data Protection, and have enabled a seven day lag on one of their database copies to enable their admins to recover items past the “single item recovery” period, or to recover deleted folder structures. Remember that single item recovery doesn't preserve the folder structure as mails moved to the dumpster are stored in a flat hierarchy.

The Symptom

They thought they were following appropriate Microsoft guidelines to use logs needed to recover the database from the “dirty shutdown” state to “clean shutdown”, so that the database would be usable as a recovery database for extracting the needed content.

1) Collect the database and the required logs

First grab a copy of the edb file and needed logs (i.e. logs needed from database perspective and logs they would like to roll forward to). This can be accomplished by suspending the database copy, and then copying the needed files to a separate location or create a snapshot of the volume which can be reverted to at a later time.

2) Check database state

Run the following: eseutil /mh "c:\DBRecovery\Mailbox Database 0436312751.edb"

eseutil /mh "c:\DBRecovery\Mailbox Database 0436312751.edb"

Check the database header for log files generation needed for the recovery, in this case: Log Required 124-124 (0x7c-0x7c) which means file E000000007C . These log files are needed at a minimum to recover the database to a clean shutdown without data loss.

Side note: If you try to recover the database with only “Log Required” logs and NOT “Log Commited” logs it will throw the following error:

Side note: If you try to recover the database with only “Log Required logs” and NOT “Log Commited” logs it will throw the error:

You can continue with the recovery using /a. And the last committed transaction will be removed from the database and the database will have a clean and consistent state. Otherwise just add the log files mentioned in the “log committed” field to your log directory and this error will disappear.

3) Check consistency of needed logs

Before starting to apply the logs you would like to, it´s important to check if all logs are consistent and available. If you need to handle a large number of logs and have used the Windows Explorer to copy the files, occasionally I’ve seen that some files are missing when sorting them in the Windows Explorer view.

To ensure that you will not fail afterwards when performing the recovery, take a moment and check these. Use eseutil /ml together with your log directory path and log prefix, in my example E00.

Log file check within directory: eseutil /ml c:\DBRecovery\E00

Log file check within directory: eseutil /ml c:\DBRecovery\E00

4) Recover database state to “clean shutdown”

After they finished both checks successfully, they started the recovery using:

eseutil /R E00 /l "c:\DBRecovery" /d "c:\DBRecovery\Mailbox Database 0436312751.edb"
eseutil /R E00 /l "c:\DBRecovery" /d "c:\DBRecovery\Mailbox Database 0436312751.edb"

It failed with an Error “-1216”. They did some Internet research and found that this could be fixed by using the “/i"

So their next try was:

eseutil /R E00 /l c:\DBRecovery /d "c:\DBRecovery\Mailbox Database 0436312751.edb" /i

eseutil /R E00 /l c:\DBRecovery /d "c:\DBRecovery\Mailbox Database 0436312751.edb" /i

And YES it results in “Operation completed successfully in 0.140 seconds.” Then they checked the Application event log and YES, there is no Error.

Application event log

Their last check before mounting the database was eseutil /mh "c:\DBRecovery\Mailbox Database 0436312751.edb."   After all that work - STILL a “dirty shutdown” Sad smile.

Note:

The get quick access to their data, they executed the eseutil command using /P for a repair to get the database into a clean shutdown. Microsoft strongly discourages using eseutil /P because it may lead to data loss. The /P should be a measure of extreme desperation when there’s no way at all to recover your database via /R.

Troubleshooting Steps – How did we solve it?

They showed me their eseutil command:

eseutil /R E00 /l "c:\DBRecovery" /d "c:\DBRecovery\Mailbox Database 0436312751.edb” /i

I saw three things I wouldn't expect to see:

  1. “/i” is a parameter for skipping missing log stream attached databases. In older versions of Exchange one log stream was used per storage group. As a storage group could have contained multiple databases, the “/i” option was provided when only one out of multiple databases has to be recovered.   As Exchange 2010 doesn’t use storage groups any more, this parameter shouldn’t be used any more.
  2. “/d” parameter with the fully qualified path to the EDB file.  You HAVE to use the directory path and NOT the full path to the edb file.
  3. They also didn’t specify a system path, so the actual path of the cmd prompt is used. This directory is used to check for, or to create, an E0x.chk file. I always recommend specifying the same path as the edb and log directory to keep the files together, and ensure that the files you expect to be used are the ones which are always used.

First, I asked them to remove the “/i” parameter and to try it again, they told me that this resulted in the previously mentioned error:  “-1216 (JET_errAttachedDatabaseMismatch)”.

Checking the EventLog after removing the “/i” parameter, it stated that the database file has not been found at path “C:\DBRecovery\Mailbox Database 0436312751.edb\Mailbox Database 0436312751.edb" which makes sense.

Checking the EventLog

After removing the file name and ending “\”, and optionally adding the system directory parameter to the command, we ran eseutil /R E00 /l “c:\DBRecovery” /s “c:\DBRecovery” /d “c:\DBRecovery”.   Running this modified command, everything worked as expected and the database ended up in a “clean shutdown” without any error message.

eseutil /R E00 /l “c:\DBRecovery” /s “c:\DBRecovery” /d “c:\DBRecovery”.

Note: If the database still stays in “dirty shutdown”, check the specified system directory for an e0x.chk file from your previous attempts. This checkpoint file can be safely removed for this recovery process if only logs that are needed for your recovery exist in the log directory. Following that, the command should complete successfully.

So why was this happening?

I was wondering why two different customers would make the “same” mistake using eseutil /r, so I started some Internet research and found sites providing examples using “eseutil /r /d” with the full path to the edb. This will simply not work.  Most of these mention that sometimes this will not work – and recommend using /p to perform a repair of the database. As mentioned, this shouldn’t be the recommended way as long as the edb file and the required logs exist and are available to be used.

Another possible misunderstanding would be that eseutil also incorporates another “MODES OF OPERATION” which use “/d” as a parameter –> Database defrag. So if you use “eseutil /d” to defrag a database you have to specify a complete file path.

Hope this helps. And after reading through this, just sit back and use the ESEUtil parameter which you should try out with your Exchange 2007/2010 server (without any risk):  “eseutil /ESE.”   And give them their credits providing such a solid database Smile.

Thanks goes to Alexandre Costa and James Cameron reviewing this content.


Posted by Frank Battiston , MSPFE Editor