Windows 2008 / Exchange 2007 SP1 - ESE 522 errors on CCR passive or SCR target machine.

There are some users that are experiencing errors on CCR (cluster continuous replication) passive nodes or SCR (standby continuous replication) target machines. 

Here is an example error:

Event ID     : 522
Raw Event ID : 522
Record Nr.   : 3965
Category     : General
Source       : ESE
Type         : Error
Generated    : 6/4/2008 4:48:58 PM
Written      : 6/4/2008 4:48:58 PM
Machine      : server.domain.com
Message      : Microsoft.Exchange.Cluster.ReplayService (7012) Log Verifier e0a 31573001: An attempt to open the device name "\\source\share$" containing "\\source\share$\" failed with system error 5 (0x00000005): "Access is denied. ".  The operation will fail with error -1032 (0xfffffbf8).

For more information, click https://www.microsoft.com/contentredirect.asp.

The error -1032 (0xfffffbf8) translates to Jet_errFileAccessDenied.

The errors most commonly occur when:

1)  Replication is paused and resumed automatically to accomidate a backup operation.

2)  Replication is paused and resumed using suspend-storagegroupcopy and resume-storagegroupcopy (or through the GUI equivalents in the Exchange Management Console).

3)  The replication service is restarted or the entire node / target is restarted.

4)  Databases are mounted and dismounted using dismount-mailboxdatabase and mount-mailboxdatabase (or though the GUI equivalents in Exchange Management Console).

5)  The CMS (clustered mailbox server) is stopped and restarted using stop-clusteredmailboxserver and start-clusteredmailboxserver (or though the GUI equivalents in Exchange Management Console).

If you manually review the log file directories on the passive or target machines they are relatively in sync with the source machine. 

Likewise, when using any replication health check including get-storagegroupcopystatus, get-storagegroupcopystatus -standbymachine, and test-replicationhealth no errors are displayed.

Here is an example get-storagegroupcopystatus on a two node Exchange 2007 SP1 / Windows 2008 CCR cluster.

 

Name                  SummaryCopySt atus CopyQueueLeng th ReplayQueueL ength LastInspecte dLogTime
---- -------------     -------------   ------------      ------------
2008-MBX3-SG1 Healthy           0               0               12/21/200...
2008-MBX3-SG2  Healthy           0               0               12/21/200...

 

Here is an example of test-replicationHealth

                         

Server                   Check                          Result Error                   
------          -----                          ------ ----                   
2008-NODE1 PassiveNodeUp                  Passed
2008-NODE1 ClusterNetwork                 Passed 
2008-NODE1 QuorumGroup                    Passed
2008-NODE1 FileShareQuorum                Passed
2008-NODE1 CmsGroup                       Passed 
2008-NODE1 NodePaused                     Passed 
2008-NODE1 DnsRegistrationStatus          Passed 
2008-NODE1 ReplayService                  Passed
2008-NODE1 DBMountedFailover              Passed 
2008-NODE1 SGCopySuspended                Passed
2008-NODE1 SGCopyFailed                   Passed
2008-NODE1 SGInitializing                 Passed
2008-NODE1 SGCopyQueueLength              Passed 
2008-NODE1 SGReplayQueueLength            Passed                  

 

The error is caused by an invalid function call from the replication service to the operating system.  This causes the operating system to respond with access denied, and the replication service to respond by logging the ESE 522 event.

This issue is scheduled to be corrected in Exchange 2007 SP1 RU7.  There is currently no incremental update for the issue.

The issue can be considered benign if:

1)  Visual inspection of the log file directories show that replication is occurring.

2)  Get-storagegroupcopystatus or get-storagegroupcopystatus -standbymachine shows all storage groups in healthy state (some storage groups may be in an initializing state, in which case logs will need to be generated to determine status).

3)  Test-replicationhealth when run on the passive node or target server shows all tests passed.