Exchange Management Shell and Mailbox Anchoring

Update: behavior described in this blog post has changed in later Cumulative Update; please see Remote PowerShell Proxying Behavior in Exchange 2013 CU12 and Exchange 2016.

Coming to the next CUs for Exchange 2013 and Exchange 2016 there are some changes to how the Exchange Management Shell (EMS) connects to Exchange. In previous versions, we have seen that customers who were reliant on a load balanced solutions for third party apps and scripts may get routed to a non-Exchange 2016 server. This would lead customers to some broken administrative experiences based on the reliance of Exchange 2016 cmdlets and features. Today we will dive deeper into these changes to help the Exchange Administrator understand how these changes will affect their Exchange Organization. In Exchange 2010 we used session affinity to provide a long-standing association between a particular client and a particular Client Access Server. With Exchange 2013 and continuing on into Exchange 2016 we have removed the requirement for session affinity. Also with the release of Exchange 2016 we added an ability to proxy between Exchange 2013 and Exchange 2016. This allowed us to have an Exchange 2013 server public facing and proxy traffic back to an Exchange 2016 server. So in order to ensure that when a connection to the Exchange Management Shell (EMS) always goes to an Exchange 2016 server we have introduced mailbox anchoring into Exchange 2013 CU11 and Exchange 2016 CU1. In mailbox anchoring, CAS locates where the mailbox resides by querying Active Directory for the user's mailbox GUID. As soon as we obtain the GUID, HTTP Proxy refers to Active Manager to locate the database containing the user's mailbox. Once CAS knows where the user's mailbox is located, the protocol request is then proxied to the appropriate mailbox server. If that server goes down and the database is a part of a DAG, the Client Access Proxy will proxy the session to the new active mailbox server for the corresponding database. This process is utilized currently for most client protocols, such as OWA, ECP, and EWS with the exception of Remote PowerShell. This is great for the client but what about the Exchange Management Shell (EMS)? In Exchange 2013 RTM through CU10 and Exchange 2016 (RTM), we do not use the mailbox anchoring logic to proxy the PowerShell session we just connect to the local server or proxy to another available server in the Exchange Organization. This means if I log into an Exchange 2013 Server I cannot manage any new cmdlets that became available in Exchange 2016. For that I would have to logon directly to an Exchange 2016 Server. Starting with Exchange Server 2013 CU11 and Exchange Server 2016 CU1, the Exchange Management Shell (EMS) session will also be using mailbox anchoring. If the aforementioned environment has all servers upgraded to Exchange 2013 CU11 and Exchange 2016 CU1 the behavior of Exchange Management Shell changes. Now when a user logs on to the Exchange Server and open up EMS, the session will be proxied to the server where the user’s mailbox is located. If the user does not have a mailbox, we will utilize the arbitration mailboxes for the mailbox anchoring logic. Wherever the arbitration mailbox is mounted is where the EMS session will be proxied. It is important to understand when you upgrade your existing Exchange Server 2010/Exchange Server 2013 organization to Exchange Server 2013/Exchange 2016, you must move the arbitration mailboxes to a database on highest version of Exchange Mailbox server. Same applies to administrators that have mailbox associated. You should move the mailbox after installing and verifying Exchange was working properly. If you don’t move the arbitration mailbox to higher version of Exchange, you may run into issues like not getting version appropriate tasks or tasks not being saved to the administrator audit log when you run the Search-AdminAuditLog cmdlet. We primarily use the Arbitration mailbox “SystemMailbox{bb558c35-97f1-4cb9-8ff7-d53741dc928c}” however if that mailbox is unavailable we can use the other arbitration mailboxes to make our connection. So let’s cover how to handle a couple of scenarios that you may run into.

Installing Exchange 2016 into an Exchange 2013 Organization

After you complete the installation of your first Exchange 2016 server and verify that everything is working properly you need to move all arbitration mailboxes to Exchange 2016. To do so you can use the Exchange admin center or Exchange Management Shell and move all of the Microsoft Exchange Migration, Federation, and System Mailboxes. To move all the arbitration mailboxes via Exchange Management Shell we would run:

[PS] C:\PowerShell> Get-Mailbox -Arbitration | New-MoveRequest -TargetDatabase <Exchange_2016_DataBase>

To move all the arbitration mailboxes via Exchange Admin Center we would run:

1. In the EAC, go to Recipients > Migration. 2. Click New Add Icon, and then click Move to a different database. 3. On the New local mailbox move page, click Select the users that you want to move, and then click Add Icon. 4. On the Select Mailbox page, search for Microsoft Exchange add all the mailboxes that are in the list below:

5. Click OK, and then click Next. 6. On the Move configuration page, type the name of the migration batch, and then click Browse next to the Target database box. 7. On the Select Mailbox Database page, add the mailbox database indicating where you want to move the system mailbox. Verify that the version of the mailbox database that you select is Version 15.1, which indicates that the database is located on an Exchange 2016 server. 8. Click OK, and then click Next. 9. On the Start the batch page, select the options to automatically start and complete the migration request, and then click New.

How do you know which server you are connected to?

The Exchange Management Shell console will always display the name of CAS server that received the initial connection. However, the connection may be proxied to a different mailbox server in the background. You can use following trick to identify the mailbox server to which the connection is being proxied to. Open up Exchange Management Shell, and run a simple query to determine which Exchange Server you are connected to: As you can see from the above EMS Session even though I was on the host E1501 when I ran the cmdlet Get-ExchangeCertificate it returned the certificate for server E1503. If you want to dig deeper you can go into the logging directory for the HTTPProxy PowerShell and see what server, you proxied to. (Before you can search in this path you will need to open the path in File Explorer to take ownership) As you see above we proxied the connection to e1503.contoso.com. Since your PowerShell connection is actually running on the server e1503, therefore all cmdlets run will be against that server. If the cmdlet you are using supports the -server switch, this can be utilized to ensure which server the cmdlet is run against. For example, if I run Get-PowerShellVirtualDirectory I can specify which server I want to communicate with.

Fail Overs

If you have an admin mailbox and the database that hosts the mailbox or DAG that the mailbox is in is down, we will then proxy to the database that has the arbitration mailbox (which is why we recommend moving this mailbox as noted before). This way cmdlets can always be run against the highest version Exchange server in the organization.

AD Site Proxying

In a multisite or geographically dispersed datacenters we will proxy the EMS session back to where the admin mailbox is mounted. In most scenarios this is fine, however there may be some tasks that would require us to have an administrator in the local site. In order to accomplish this, you will need to have either the arbitration mailboxes in the site and logon to the server with an administrator account that has no mailbox or you can create an admin account with a mailbox in the corresponding site.

Critical Failures

If you open EMS and are not able to connect to any Exchange Servers due to loss of communication, then you will need to add the PSSnapin for Exchange to your local PowerShell Session. Rob Whaley Beta Engineer for Exchange and Office 365