Exchange 2013 Cumulative Update 2 – Where are my databases?

  • Article

In Exchange Server 2013 Cumulative Update 2 (CU2) we introduced support for a maximum of 100 mounted mailbox databases per server (active or passive) when an Enterprise Edition license is applied.  This was an increase from Exchange Server 2013 RTM and Exchange Server 2013 Cumulative Update 1 (CU1).

 

It has been recently discovered that when updating to CU2 from Exchange Server 2013 RTM or Exchange Server 2013 CU1, the database limit is not increased to 100. This results in the inability to mount more than 50 databases on a server running the Enterprise Edition of Exchange Server 2013. When installing Exchange Server 2013 CU2 directly, the database limit is applied correctly.

 

The Core Problem

 

In this example, I deployed a new install of Exchange Server 2013 CU1 on Windows Server 2012 (although, the operating system doesn’t matter here). After the Setup completed, I performing an LDP dump of the information store object, and noted a database limit of 5.

 

Expanding base 'CN=InformationStore,CN=TEST-MBX-0,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=EXCHANGE,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft'...
Getting 1 entries:
Dn: CN=InformationStore,CN=TEST-MBX-0,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=EXCHANGE,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft
adminDisplayName: InformationStore;
cn: InformationStore;
distinguishedName: CN=InformationStore,CN=TEST-MBX-0,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=EXCHANGE,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft;
dSCorePropagationData: 0x0 = ( );
instanceType: 0x4 = ( WRITE );
msExchESEParamCircularLog: 0;
msExchESEParamCommitDefault: 0;
msExchESEParamDbExtensionSize: 256;
msExchESEParamEnableIndexChecking: TRUE;
msExchESEParamEnableOnlineDefrag: TRUE;
msExchESEParamLogFileSize: 5120;
msExchESEParamPageFragment: 8;
msExchESEParamPageTempDBMin: 0;
msExchESEParamZeroDatabaseDuringBackup: 0;
msExchMaxRestoreStorageGroups: 1;
msExchMaxStorageGroups: 5;
msExchMaxStoresPerGroup: 5;
msExchMaxStoresTotal: 5; name: InformationStore;
objectCategory: CN=ms-Exch-Information-Store,CN=Schema,CN=Configuration,DC=exchange,DC=msft;
objectClass (3): top; container; msExchInformationStore;
objectGUID: 52c2fc98-b8b4-4d33-b35b-ca00cb3fcfff;
showInAdvancedViewOnly: TRUE;
uSNChanged: 24300;
uSNCreated: 24300;
whenChanged: 10/27/2013 3:58:35 PM Pacific Daylight Time;
whenCreated: 10/27/2013 3:58:35 PM Pacific Daylight Time;

-----------

 

This is normal and expected because all unlicensed installations (e.g., Trial) are treated as Standard Edition, and Standard Edition can have a maximum of 5 mounted databases.

 

Next, I entered an Enterprise Edition license key. This updated the value msExchMaxStoresTotal to a value of 50, which is the expected value for the Enterprise Edition of Exchange Server 2013 RTM and Exchange Server 2013 CU1.

 

Expanding base 'CN=InformationStore,CN=TEST-MBX-0,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=EXCHANGE,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft'...
Getting 1 entries:
Dn: CN=InformationStore,CN=TEST-MBX-0,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=EXCHANGE,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft
adminDisplayName: InformationStore;
cn: InformationStore;
distinguishedName: CN=InformationStore,CN=TEST-MBX-0,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=EXCHANGE,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft;
dSCorePropagationData: 0x0 = ( );
instanceType: 0x4 = ( WRITE );
msExchESEParamCircularLog: 0;
msExchESEParamCommitDefault: 0;
msExchESEParamDbExtensionSize: 256;
msExchESEParamEnableIndexChecking: TRUE;
msExchESEParamEnableOnlineDefrag: TRUE;
msExchESEParamLogFileSize: 5120;
msExchESEParamPageFragment: 8;
msExchESEParamPageTempDBMin: 0;
msExchESEParamZeroDatabaseDuringBackup: 0;
msExchMaxRestoreStorageGroups: 1;
msExchMaxStorageGroups: 100;
msExchMaxStoresPerGroup: 5;
msExchMaxStoresTotal: 50;
msExchMinAdminVersion: -2147453113;
msExchVersion: 4535486012416;
name: InformationStore;
objectCategory: CN=ms-Exch-Information-Store,CN=Schema,CN=Configuration,DC=exchange,DC=msft;
objectClass (3): top; container; msExchInformationStore;
objectGUID: 52c2fc98-b8b4-4d33-b35b-ca00cb3fcfff;
showInAdvancedViewOnly: TRUE;
uSNChanged: 27284;
uSNCreated: 24300;
whenChanged: 10/28/2013 4:44:30 AM Pacific Daylight Time;
whenCreated: 10/27/2013 3:58:35 PM Pacific Daylight Time;

-----------

 

Next, I upgraded the server to Exchange Server 2013 CU2. Once Setup completed, I reviewed the value of msExchMaxStoresTotal, and found that it did not get updated as expected.

 

Expanding base 'CN=InformationStore,CN=TEST-MBX-0,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=EXCHANGE,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft'...
Getting 1 entries:
Dn: CN=InformationStore,CN=TEST-MBX-0,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=EXCHANGE,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft
adminDisplayName: InformationStore;
cn: InformationStore;
distinguishedName: CN=InformationStore,CN=TEST-MBX-0,CN=Servers,CN=Exchange Administrative Group (FYDIBOHF23SPDLT),CN=Administrative Groups,CN=EXCHANGE,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=exchange,DC=msft;
dSCorePropagationData: 0x0 = ( );
instanceType: 0x4 = ( WRITE );
msExchESEParamCircularLog: 0;
msExchESEParamCommitDefault: 0;
msExchESEParamDbExtensionSize: 256;
msExchESEParamEnableIndexChecking: TRUE;
msExchESEParamEnableOnlineDefrag: TRUE;
msExchESEParamLogFileSize: 5120;
msExchESEParamPageFragment: 8;
msExchESEParamPageTempDBMin: 0;
msExchESEParamZeroDatabaseDuringBackup: 0;
msExchMaxRestoreStorageGroups: 1;
msExchMaxStorageGroups: 100;
msExchMaxStoresPerGroup: 5;
msExchMaxStoresTotal: 50;
msExchMinAdminVersion: -2147453113;
msExchVersion: 4535486012416;
name: InformationStore;
objectCategory: CN=ms-Exch-Information-Store,CN=Schema,CN=Configuration,DC=exchange,DC=msft;
objectClass (3): top; container; msExchInformationStore;
objectGUID: 52c2fc98-b8b4-4d33-b35b-ca00cb3fcfff;
showInAdvancedViewOnly: TRUE;
uSNChanged: 27284;
uSNCreated: 24300;
whenChanged: 10/28/2013 4:44:30 AM Pacific Daylight Time;
whenCreated: 10/27/2013 3:58:35 PM Pacific Daylight Time;

-----------

 

In this state, when the administrator tries to mount more than 50 databases (any combination of active or passive) on this server, they will receive an error message:

 

[PS] C:\>Mount-Database DB51
Couldn't mount the database that you specified. Specified database: DB51; Error code: An Active Manager operation failed. Error: The database action failed. Error: Operation failed with message: MapiExceptionTooManyMountedDatabases: Unable to mount database. (hr=0x8004060e, ec=-2147219954)

Diagnostic context:
Lid: 65256
Lid: 10722 StoreEc: 0x8004060E
Lid: 1494 ---- Remote Context Beg ----
Lid: 37952 dwParam: 0x144489A
Lid: 39576 StoreEc: 0x977
Lid: 35200 dwParam: 0x39F8
Lid: 58864 StoreEc: 0x8004060E
Lid: 43248 StoreEc: 0x8004060E
Lid: 48432 StoreEc: 0x8004060E
Lid: 54336 dwParam: 0x14448C8
Lid: 1750 ---- Remote Context End ----
Lid: 1047 StoreEc: 0x8004060E [Database: DB51, Server: TEST-MBX-0.exchange.msft].
+ CategoryInfo : InvalidOperation: (DB51:ADObjectId) [Mount-Database], InvalidOperationException
+ FullyQualifiedErrorId : [Server=TEST-MBX-0,RequestId=797928f4-fe83-4e1f-8545-e01a72aaf79d,TimeStamp=10/28/2013 7
:53:32 PM] 5FE3BF1B,Microsoft.Exchange.Management.SystemConfigurationTasks.MountDatabase
+ PSComputerName : test-mbx-0.exchange.msft

 

In addition, three events will be logged to the Application event log:

 

Log Name: Application
Source: MSExchangeIS
Date: 10/28/2013 1:00:27 PM
Event ID: 40003
Task Category: High Availability
Level: Error
Keywords: Classic
User: N/A
Computer: Test-MBX-0.exchange.msft
Description:
Exceeded the max number of 50 databases on this server.

Log Name: Application
Source: MSExchangeRepl
Date: 10/28/2013 1:00:27 PM
Event ID: 3154
Task Category: Service
Level: Error
Keywords: Classic
User: N/A
Computer: Test-MBX-0.exchange.msft
Description:
Active Manager failed to mount database DB51 on server Test-MBX-0.exchange.msft. Error: An Active Manager operation failed. Error: The database action failed. Error: Operation failed with message: MapiExceptionTooManyMountedDatabases: Unable to mount database. (hr=0x8004060e, ec=-2147219954)
Diagnostic context:
Lid: 65256
Lid: 10722 StoreEc: 0x8004060E
Lid: 1494 ---- Remote Context Beg ----
Lid: 37952 dwParam: 0x14A9D25
Lid: 39576 StoreEc: 0x977
Lid: 58864 StoreEc: 0x8004060E
Lid: 43248 StoreEc: 0x8004060E
Lid: 48432 StoreEc: 0x8004060E
Lid: 54336 dwParam: 0x14A9D54
Lid: 1750 ---- Remote Context End ----
Lid: 1047 StoreEc: 0x8004060E

Log Name: Application
Source: ExchangeStoreDB
Date: 10/28/2013 1:00:27 PM
Event ID: 226
Task Category: Database recovery
Level: Error
Keywords: Classic
User: N/A
Computer: Test-MBX-0.exchange.msft
Description:
At '10/28/2013 1:00:27 PM' the copy of database 'DB51' on this server didn't mount because the number of mailbox database copies on this server exceeds the supported limit. The error returned by failover was "There is only one copy of this mailbox database (DB51). Automatic recovery is not available.".

 

Similar errors and events will occur when attempting to activate a passive database copy on a server with 50 mounted databases or when adding a passive database copy to a server that already has 50 database copies assigned.

 

How to Correct this Condition

 

Fortunately, fixing this problem is pretty easy. You simply re-enter your Enterprise Edition product key on the server, and restart the Microsoft Exchange Information Store service.

 

Moving Forward

 

For the immediate future, we expect this problem to exist when upgrading from Exchange Server 2013 CU1 to any future CU. However, once the server is re-licensed on CU2 or newer, the condition will remain resolved.