Part 5: Datacenter Activation Coordination: How do I force automount consensus?

In order for any database availability group (DAG) that has Datacenter Activation Coordination (DAC) mode to be able to mount any databases, the DAG itself much reach automount consensus. In the following blog posts, I discuss and outline situations where automount consensus may not be achieved, resulting in dismounted databases.  A DAG in DAC mode which cannot achieve automount consensus will not automatically mount databases, and administrators are blocked from manually mounting them, as well.

 

Let’s look at an example to illustrate this scenario.  In this example, I have a 4-member DAG comprised of servers DAG-1, DAG-2, DAG-3, and DAG-4

 

image

 

DAG-4 experiences a hardware failure, goes down, and is scheduled for replacement.  Although the server is offline, it is still a member of the DAG and it’s underlying cluster.

 

image

 

Next, a power failure occurs resulting in DAG-1, DAG-2, and DAG-3 going offline.

 

image

 

After power is restored, DAG-1, DAG-2, and DAG-3 are booted up.

 

image

 

The cluster service is successfully initialized and a cluster is formed.  You can verify the nodes are operational by running

 

Windows 2008:

PS C:\> cluster node
Listing status for all available nodes:

Node Node ID Status
-------------- ------- ---------------------
DAG-4 1 Down
DAG-3 2 Up
DAG-2 3 Up
DAG-1 4 Up

 

 

Windows 2008 R2:

PS C:\> Import-Module FailoverClusters

PS C:\> Get-ClusterNode | fl name,state

Name : DAG-1
State : Up

Name : DAG-2
State : Up

Name : DAG-3
State : Up

Name : DAG-4
State : Down

 

Using the Get-MailboxDatabase cmdlet, we can see that, even though the servers are up and the cluster is running, no databases are mounted:

 

[PS] C:\>Get-MailboxDatabase -Status | fl name,mounted

Name : Mailbox Database 1252068500
Mounted : False

Name : Mailbox Database 1757981393
Mounted : False

Name : Mailbox Database 1370762657
Mounted : False

Name : Mailbox Database 1511135053
Mounted : False

 

Attempting to mount a database (with or without the –Force parameter) results in the following error:

 

[PS] C:\>Mount-Database "Mailbox Database 1252068500"
Couldn't mount the database that you specified. Specified database: Mailbox Database 1252068500; Error code: An Active Manager operation failed. Error An Active Manager operation encountered an error. To perform this operation, the server must be a member of a database availability group, and the database availability group must have quorum. Error: Automount consensus not reached.. [Server: DAG-1.exchange.msft].
+ CategoryInfo : InvalidOperation: (Mailbox Database 1252068500:ADObjectId) [Mount-Database], InvalidOperationException
+ FullyQualifiedErrorId : FD5FDFE8,Microsoft.Exchange.Management.SystemConfigurationTasks.MountDatabase

So what needs to be done to achieve automount consensus and mount the databases?  It’s simple: the administrator can run Start-DatabaseAvailabilityGroup.  The cmdlet can be run using the –MailboxServer parameter to specify one of the mailbox servers that is functional.  After the cmdlet is run, the specified server will have its DACP bit set to 1.  The other servers in the DAG will see that the DACP bit on the specified server is set to 1 and this will cause them to set their DACP bits to 1.  And once all of the DAG members have DACP set to 1, automount consensus is reached, and the databases can be mounted. Here is an example:

 

[PS] C:\>Start-DatabaseAvailabilityGroup -Identity DAG -MailboxServer MBX-1

[PS] C:\>Get-MailboxDatabase -Status | fl name,mounted

Name : Mailbox Database 1252068500
Mounted : True

Name : Mailbox Database 1757981393
Mounted : True

Name : Mailbox Database 1370762657
Mounted : True

Name : Mailbox Database 1511135053
Mounted : True

 

Start-DatabaseAvailabilityGroup can also be used in scenarios where the remaining survivor of a DAG failure is a single member that is using the witness server boot time to achieve automount consensus.  In this case the cmdlet forces the member’s DACP bit to 1 and updates the boot times in the registry for future restarts.

 

As a side note of caution, administrators should be careful when using Start-DatabaseAvailabilityGroup.  It is possible that a split brain condition could be created by using this cmdlet to start the wrong DAG members, or it by using it at the wrong stage of a datacenter switchover. To ensure your datacenter switchovers go smoothly, I recommend using the recently released Datacenter Switchover Tool.

 

========================================================

Datacenter Activation Coordination Series:

 

Part 1:  My databases do not mount automatically after I enabled Datacenter Activation Coordination (https://aka.ms/F6k65e)
Part 2:  Datacenter Activation Coordination and the File Share Witness (https://aka.ms/Wsesft)
Part 3:  Datacenter Activation Coordination and the Single Node Cluster (https://aka.ms/N3ktdy)
Part 4:  Datacenter Activation Coordination and the Prevention of Split Brain (https://aka.ms/C13ptq)
Part 5:  Datacenter Activation Coordination:  How do I Force Automount Concensus? (https://aka.ms/T5sgqa)
Part 6:  Datacenter Activation Coordination:  Who has a say?  (https://aka.ms/W51h6n)
Part 7:  Datacenter Activation Coordination:  When to run start-databaseavailabilitygroup to bring members back into the DAG after a datacenter switchover.  (https://aka.ms/Oieqqp)
Part 8:  Datacenter Activation Coordination:  Stop!  In the Name of DAG... (https://aka.ms/Uzogbq)
Part 9:  Datacenter Activation Coordination:  An error cause a change in the current set of domain controllers (https://aka.ms/Qlt035)

========================================================