In this case, the problem was initially manifesting itself as the following error when trying to create a new mailbox in the first database created with installation of the server:
Active Directory operation failed on adserver.domain.com. This error is not retriable. Additional information: Insufficient access rights to perform the operation.
Active directory response: 00002098: SecErr: DSID-03150E8A, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0
The user has insufficient access rights.
Further it was found that the customer got a similar error when trying to create a new database:
Active Directory operation failed on adserver.domain.com. This error is not retriable. Additional information: Access is denied.
Active directory response: 00000005: SecErr: DSID-03152492, problem 4003 (INSUFF_ACCESS_RIGHTS), data 0
+ CategoryInfo : NotSpecified: (0:Int32) [New-MailboxDatabase], ADOperationException
+ FullyQualifiedErrorId : 36F6B416,Microsoft.Exchange.Management.SystemConfigurationTasks.NewMailboxDatabase
This ended up coming down to a case of the Exchange Trusted Subsystem not having the correct permissions on the Databases container in AD to create the new database. The questions here might be, “What is the Exchange Trusted Subsystem? And why does its permissions matter? Shouldn’t the permissions of the administrator on the container be what’s important?”
Well, to answer these questions, all commands in Exchange 2010 are now actually executed by the Exchange Trusted Subsystem, after going through user access checks in Role Based Access Control (RBAC).
More about RBAC and permissions can be looked at here:
The second article notes:
If RBAC allows an action to proceed, the action is performed in the context of the Exchange Trusted Subsystem and not the user's context. The Exchange Trusted Subsystem is a highly privileged universal security group (USG) that has read/write access to every Exchange-related object in the Exchange organization. It's also a member of the Administrators local security group and the Exchange Windows Permissions USG, which enables Exchange to create and manage Active Directory objects.
So, how did we figure out that the Exchange Trusted Subsystem permissions were the problem?
A DSACLS of the Databases container showed the Exchange Trusted Subsystem group missing from the Permissions on the container, so I could only wonder where else it is missing. It should have an inherited Full Control there, inherited from the CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domain,DC=com container.
The following document references the Exchange 2010 Deployment permissions:
This gets set during setup /prepareAD.
In this case, setup /prepareAD had been run again to try and solve the issue, but did not work. So why not?
When we checked, we found that we did have the Exchange Trusted Subsystem group with full control on the Microsoft Exchange container. That’s why the prepareAD didn’t fix it.
So that means that somewhere between the Microsoft Exchange container and the Databases container, the permissions inheritance was disabled. You can see the Inheritance setting in the Advanced area on the Security for each container. You can use ADSIEdit to open the Configuration container, and then drill down to these containers:
First Organization – (check here)
Administrative Groups – (check here)
Exchange Administrative Group (FYDIBOHF23SPDLT) – (check here)
Here is an example of what this looks like:
If the permissions inheritance is disabled on one of the ones I have indicated to check, it needs to be re-enabled, and the permissions should then be fixed.
In our case, it was missing at the Administrative Groups level. Once that was fixed, new mailboxes and new databases could be created without error.