Keeping up to date with what’s happening with Set-MailboxDatabase II
Tony Redmond is absolutely correct when he says “No matter how long you work with software, it’s impossible to master every possible detail, especially when you’re talking about a complex product like Exchange 2013.” With the pace of innovation faster today than ever before, and even faster tomorrow, it can be difficult to stay on top of the endless waves of changes that are occurring. Even with rigorous testing, substantive validation and other confirmatory processes, a small change can elude even those closest to the process.
I call this post Keeping up to date with what’s happening with Set-MailboxDatabase II because it builds upon Tony’s post about the changes to the Set-MailboxDatabase cmdlet in Exchange 2013.
The first change Tony mentions is the description for the AutoDagExcludeFromMonitoring parameter. The purpose of this parameter is to exclude a database from Managed Availability’s ServerOneCopyMonitor. We consider the ServerOneCopyMonitor, which executes only on DAG members and not standalone Mailbox servers, to be our most important redundancy protection mechanism from an HA perspective. As Tony also mentions, it is the latest incarnation of the database redundancy checks we introduced and I blogged about in May 2010. We think it is very important that all customers monitor for situations in which database redundancy is compromised and immediately take action to restore database redundancy to avoid catastrophic data loss. What started as an external script was later shipping in the box and configured automatically on all Mailbox servers (which is why the Task Scheduler service is a requirement for installing the Mailbox role in Exchange 2010). And in Exchange 2013, the checks are now done in code as part of Managed Availability. As a result, there are no scheduled tasks configured by Exchange, and the Task Scheduler service is not required to install the Mailbox role in Exchange 2013.
The ServerOneCopyMonitor checks databases once a minute on all DAG members. It specifically is looking for databases that where:
- The database copy is (Healthy || Mounted) &&
- ServerComponentState is NOT Offline &&
- The database copy is NOT Activation Blocked &&
- The server is NOT exceeding MaxActive &&
- The copy queue length < AutoDatabaseMountDial &&
- The server is NOT Activation Disabled
30 consecutive failures are considered an escalating condition by Managed Availability. In this event, the ServerOneCopyMonitor is notified and transitions into an Unhealthy state.
When databases are being monitored for redundancy, events will be logged in the Application event log when redundancy is compromise and/or restored (event 4113 when you are down to a single copy of a database, and event 4114 when redundancy has been restored). The AutoDagExcludeFromMonitoring parameter is a mechanism to tell the ServerOneCopyMonitor not to alert on a specific database (for example, a non-replicated database that is hosted on a DAG member). The default value for this parameter is False. Set the value of AutoDagExcludeFromMonitoring to True, and ServerOneCopyMonitor does not report on its redundancy. Or at least that is the way it is supposed to work, and that is what I wrote on TechNet about how it should work, too.
And that is the way it will work in Exchange Server 2013 Cumulative Update 6 (CU6). But in CU5 and earlier, that’s not quite how it works. As Tony reported, about 15 minutes after setting AutoDagExcludeFromMonitoring to True, it mysteriously reverts back to False, and the ServerOneCopyMonitor generates 4113 alerts about databases that should have been excluded. And that brings us back to those small changes that can elude even those closest to the process. As it turns out, the resetting of AutoDagExcludeFromMonitoring from True wasn’t a huge mystery or major code bug. Rather, it was a function of a datacenter (Exchange Online) process that made its way into the enterprise (on-premises) product. Can you think of a feature in Exchange 2013 that is mysterious and makes changes to your servers behind the scenes and under the covers? That’s right…it was Managed Availability again. As it turns out, Exchange 2013 on-premises ships with an Exchange Online responder called EnableDatabaseMonitoringResponder that is part of the DataProtection health set used in Exchange Online. I bet you can guess from the name of the responder and the context of this blog as to what that responder does.
As you probably guessed, that responder runs every 15 minutes, and if it finds any databases excluded from monitoring, it re-includes them by changing the value of AutoDagExcludeFromMonitoring from True to False. The primary purpose of the AutoDagExcludeFromMonitoring parameter is to allow administrators to exclude a non-replicated database in a DAG from single copy alerting. But in Exchange Online, we don’t have any non-replicated databases in our DAGs. All databases in all Exchange Online DAGs are replicated. As such, databases should not be excluded from monitoring, and AutoDagExcludeFromMonitoring should always be set to False. To ensure that we catch cases in Exchange Online where AutoDagExcludeFromMonitoring was somehow set to True, we created the EnableDatabaseMonitoringResponder to check the value of AutoDagExcludeFromMonitoring and change it back to False if it finds it set to True. And because the EnableDatabaseMonitoringResponder was included in Exchange Server, this behavior is happening in on-premises environments, too.
As I mentioned before, we have fixed this in a later build, and it will be included in CU6. In the meantime, if you have set the value of AutoDagExcludeFromMonitoring to True because you want to exclude a database, you should be able to override the EnableDatabaseMonitoringResponder by using a server or global override; for example:
Add-GlobalMonitoringOverride -Identity DataProtection\EnableDatabaseMonitoringResponder -ItemType Responder -PropertyName “Enabled” -PropertyValue “0”
For the above override, also add either –Duration or –Build to specify a time length or have the override apply to a specific build of Exchange 2013.
After this override takes effect, AutoDagExcludeFromMonitoring should remain set to True.
The second change Tony mentions is the MaintenanceSchedule parameter. I actually opened a bug to remove this parameter from Exchange 2013 way back in September 2012. No, that is not a typo. That was one of three mailbox database parameters that we deprecated and discussed removing from Exchange 2013:
- 1. MaintenanceSchedule – because we no longer have scheduled maintenance
2. PublicFolderDatabase - because we no longer have public folder databases
3. QuotaNotificationSchedule – because we no longer schedule quota notifications; we check at logon time now
We went through several rounds of discussions about this, and decided that instead of removing the parameters, they should be deprecated. The primary reason for this was to avoid any interoperability issues in mixed environments that contained Exchange 2013 and legacy (e.g., Exchange 2010 and/or Exchange 2007) servers and databases. For example, the PublicFolderDatabase parameter is used by Exchange admins to point Exchange 2013 users to legacy public folder databases in single-forest coexistence deployments.
We also wanted to be particularly sensitive about any custom scripts that might be out there in use by admins that were leveraging these parameters. If we left the parameters in, but returned a Not_Supported error, that would break these scripts. If we took the parameters out altogether, that would break these scripts. Of course, with functionality being removed, these scripts need to be fixed anyway, but we wanted the experience to be as frictionless as possible. So we opted for a third path – deprecate the parameters in content and in the product.
So for the MaintenanceSchedule parameter, we documented in TechNet that “This parameter has been deprecated in Exchange 2013 and it no longer does anything.” However, as Tony and fellow MVP Jaap Wesselius discovered, that last part is not true. Sort of. It depends on how you look at it. If you look at it from a pure parameter perspective, then TechNet is right. The parameter no longer does anything. But if you look at it from cmdlet perspective, Set-MailboxDatabase -MaintenanceSchedule does do something. It will change the value of the database’s MaintenanceSchedule property; but like the parameter of the same name, the property does nothing on Exchange 2013 databases because scheduled maintenance no longer exists in Exchange 2013. But to avoid any further customer confusion, I changed the description of this parameter in TechNet to read as follows:
“This parameter has been deprecated in Exchange 2013. While it can be used to change the MaintenanceSchedule property of a database, that property is ignored in Exchange 2013 because scheduled maintenance no longer exists.”
That change will go live with the next content refresh to TechNet.
As Tony says at the end of his post, “All of this goes to prove that you have to keep your wits about you to keep up to date with changes that are made to a major product like Exchange 2013. It makes the job interesting – or demanding.” I choose to think of it as interesting. 