Canonical Ordering Of Mailbox Permissions In Exchange

An Access Control List, or ACL, is basically just a list of Access Control Entries, or ACEs. Each ACE contains a flag indicating whether it’s an Allow or a Deny ACE, a mask that shows which rights are being allowed or denied, and a SID to which those allow or deny rights are applied.

Evaluating an ACL to determine if a user has certain rights is pretty simple. We take the user’s token, which is a list of all SIDs that apply to this user (all groups he’s a member of), and then we iterate through the ACL looking at each ACE in the order in which it appears. We keep evaluating ACEs until we hit one of two conditions: either all rights requested have been granted, or any one right requested has been denied. At that point, evaluation stops, and we don’t even bother looking at the rest of the ACEs.

Because of this process, in order to achieve the desired behavior where denies take precedence over allows, the ACL must be in a specific order. If the ACL is ordered correctly, we say it is canonical. If the order is incorrect, we say it is non-canonical, because it does not follow the established rules for the ordering of ACEs in an ACL.

For example, let’s say we have the following very simple ACL, with only two ACEs:

The S-1-1-0 SID is the SID for the Everyone group. In this ACL, we have two ACEs. The first ACE denies all rights to the Everyone group. The second one grants all rights to the Everyone group. When we go through the process of checking a user’s rights, the first ACE we hit says that all rights are denied to Everyone. Because of that, we stop right there, and the evaluation result will be that all rights are denied. In this case, canonical order was followed, so we have the expected behavior – the user is denied access to the object.

Now imagine the ACEs are not in canonical order:

In this case, when we evaluate the ACL, the first ACE we hit says that all rights are allowed. The evaluation stops right there, and the result is that the user is allowed to access the object, despite the fact that we have also denied all rights to Everyone! So in this case, the ACL is non-canonical, and the result is that the permissions don’t function as expected – the user is allowed access.

Note that the above is a much simplified example just to illustrate why canonical order exists. For full information on canonical ordering of ACEs, see MSDN.

Canonical ordering is not enforced by Windows itself – it is enforced by the tools that are usually used to manage those ACLs. If you set permissions on an Active Directory object using Active Directory Users And Computers, those permissions will be canonical. If you set rights on a file using Explorer, the ACL on the file will be canonical.

However, it’s entirely possible for a programmer to write a tool that saves ACLs to these objects in non-canonical order. In fact, in some cases this has been done intentionally. Back in Exchange 2003, you could choose to hide distribution group membership. Doing so caused us to purposely put the ACL in non-canonical order. After hiding the membership, attempting to go to the Security tab in ADUC resulted in the message “Windows can not edit the permissions on ‘whatever’ because they have been written in a nonstandard format by another application. To enable editing, you must use the application to restore the permissions to a standard format.” This message was because the ACL was non-canonical.

Mailbox permissions require canonical ordering just like AD permissions or file permissions (public folder permissions have their own canonical order that is different from Windows canonical order – that’s a topic for another post). If you run Add-MailboxPermissions to modify the permissions and the ACL is not canonical, the cmdlet reports an error:

The ACL for the object "whatever" is not in canonical order (Deny/Allow/Inherited) and will be ignored.

This makes it impossible to edit the mailbox rights using the cmdlet.

I recently ran across a case where some mailboxes migrated from Exchange 2003 to Exchange 2007 were in this state. We were not able to determine what application made the ACLs non-canonical. To fix the problem, I wrote a simple application called FixMailboxSD and made the tool and the source available on Codeplex: https://fixmailboxsd.codeplex.com/. This uses the old CDOEXM interfaces from Exchange 2003 to put the mailbox rights back in canonical order.

So, if you need an example of how to play with mailbox rights from C# using CDOEXM, you can take a look at this simple tool. Or, if you run into this same problem in your environment, feel free to use the tool to correct the mailbox rights on any affected mailboxes.