Hierarchy Replication Fails Due To Zero GUID

Exchange 2010 Sp1 Rollup Update 4 released a couple of days ago, and I want to briefly mention one of the fixes it includes. The fix I’m talking about is KB 2506049, “The hierarchy of a new public folder database on an Exchange Server 2010 SP1 server is not replicated”. You’ll know you’ve hit this problem if you get the very specific events mentioned in the article with error code 80070057. Specifically, we should see this in the 3079:

Unexpected replication thread error 80070057 on database "First Storage Group\Public Folder Store (<Server_Name>)".

EcReplFolderMessagesUnpack
EcReplMessageUnpack

If you’re not seeing that, then you’re not hitting this problem.

If you are hitting this problem, it means that your current PF databases which have hierarchy and content have a GUID in them somewhere of 00000000-0000-0000-0000-000000000000. It’s a GUID of all zeroes, and it’s not valid. Starting with Exchange 2010 SP1, we block these GUIDs from entering a new database. So, if you have this sitting around in your old PF databases, and you bring up a new clean database on 2010 SP1, the hierarchy fails to replicate because we refuse to let this GUID replicate in. You’ll never notice this problem until you try to create a new database on 2010 SP1 or later.

What I want to focus on, however, is the fix. The article states the following:

After you apply this update on all the computers that have public folder databases, follow these steps:

  1. Unmount all public folder databases in the organization.
  2. Run the following command on all the computers that have public folder databases in the organization:
    isinteg -s <server> -test replidguid -fix
  3. Mount all public folder databases.

Notes

  • The command must be run simultaneously on all the computers that have public folder databases in the organization.
  • Do not follow these steps if there are any public folder servers that are not running Update Rollup 4 for Exchange Server 2010 SP1, such as Exchange Server 2003, Exchange Server 2007, or earlier versions of Exchange Server 2010.
  • If you try to run the command on only a subset of computers that have public folder databases in the organization, public folder replication issues occur. Additionally, the issues cannot be resolved.

This command will strip the zero GUID out of the databases. Yes, the command must be run simultaneously on all public folder databases in the org. This means that all public folder servers must be on at least Exchange 2010 SP1 RU4. If you have any Exchange 2007 or 2003 public folder servers, you cannot use this fix.

Even if all your public folder databases are on Exchange 2010 SP1 RU4 or newer, you must dismount all of them at the same time and run this command against all of them. They must all be dismounted at the same time and must not be mounted until the command has been run.

To put it another way, a fixed database must not replicate with an unfixed database. In order to achieve this, all databases must be dismounted before you start running the isinteg command. Once the command completes, you can mount that fixed database if you want. But be very careful. It is probably safer to just leave them all down until you are sure they have all been fixed.

If you fix one or more databases and then accidentally mount one that has not been fixed, you run a high risk of introducing a form of logical corruption that cannot be fixed, ever, at all. When the article says that the issues cannot be resolved, we really mean it. The only option at that point is to pull the public folder data out to PSTs and start over with clean public folder databases everywhere.

Please use caution when using this fix, and make sure that when this isinteg command is run, all public folder databases have been dismounted and are not mounted again until they are fixed.

The good news is that this fix is extremely fast. This isinteg command should complete in less than a minute.