Exchange Server 2007 used the Move-Mailbox cmdlet to move mailboxes between mailbox stores. Move-Mailbox makes a RPC connection to both the source and target mailbox databases and then starts the move process. Exchange Server 2010 uses the Mailbox Replication Service (MRS), a service that runs on all Exchange 2010 Client Access Servers, for mailbox move operations. MRS handles all mailbox moves including offline and online moves.
Exchange 2007 vs. 2010
This is a high level comparison of Exchange Server 2007 vs. Exchange Server 2010. There is a radical change in mailbox moves from Exchange 2007 to Exchange 2010 environment. In Exchange 2010, we implemented online mailbox moves.
When you move a mailbox in previous versions of Exchange Server, the source mailbox gets locked and then the content is copied to the new mailbox on the target mailbox database. After the content is copied, the new mailbox is unlocked and the old one is deleted. This results in downtime for the user for the duration of the mailbox move operation. As long as you had smaller mailboxes, the downtime is not a big deal since this happens fairly quickly. With larger mailboxes (check out the Large Mailbox Vision whitepaper as well as Astrid’s blog on the Top 10 Exchange Storage Myths), the downtime can be unacceptable.
In Exchange 2010, we implemented online mailbox moves and we also implemented changes to the Store in Exchange 2007 SP2 so that when you move and upgrade from Exchange 2007 to 2010 you will benefit from online mailbox moves.
The following terms are used in Exchange Server 2010 move operation:
- Online mailbox move – Move mailbox operation wherein users are able to access their mailbox almost for the entire time of the operation except for the last part. Exchange Server 2010 uses online mailbox move.
- Offline mailbox move is a move operation wherein users cannot access their mailboxes during the move.
- Local Move is a move operation wherein the source and target mailboxes exist in the same forest and organization.
- Remote Move is a move operation wherein the source and target are in different forests and organization.
- Push is a move operation where the target is either an Exchange Server 2007 or Exchange Server 2003 and the source is an Exchange 2010 server in a local move. A push can also occur between Exchange 2010 where the source and the target are in different forests.
- Pull is a move operation where the target is an Exchange 2010 server.
Which Exchange Versions supports what?
The following table outlines the type of mailbox moves supported by different Exchange Server versions.
Mailbox Moves and Personal Archives
Exchange Server 2010 introduced Personal Archives. An archive mailbox appears as an additional mailbox in Outlook 2010 or OWA. In Exchange 2010 RTM, an archive mailbox is moved along with the mailbox if one exists. Since archive mailboxes exist only in Exchange Server 2010, mailbox moves to legacy Exchange servers will fail if the mailbox being moved has an archive mailbox.
Exchange 2010 CMDlets for move mailbox
Mailbox moves can be performed using the EMC or the Shell (EMS. You can use the following CMDlets from the Shell to manage mailbox move requests. It’s worth noting that suspending and resuming a move request can only be done through the Shell. Just like any other operation in Exchange 2010, you need to have certain permissions to perform the commands. The following table shows the RBAC permissions required for each cmdlet.
Note: RBAC permission can be granted to administrators either by assignment of a management role or membership in a built-in role group.
The Microsoft Exchange Mailbox Replication Service
The Microsoft Exchange Mailbox Replication Service (MRS) is a Windows service and is dependent only on the Microsoft Exchange Active Directory Topology service and Net.TCP Port Sharing service. There is a pre-requisite in Exchange Server 2010 setup that checks if the Net.TCP Post Sharing service is set to automatic. If it’s not, setup fails on CAS Servers. MRS is built on the Windows Communication Foundation (WCF), a part of .Net Framework 3.0 stack. MRS uses a configuration file MSExhangeMailBoxReplication.Exe.Config on every Client Access Server for its configuration information. By default it’s located in C:\Program Files\Microsoft Exchange Server\V14\Bin folder. The process associated with the service is MSExchangeMailBoxReplication.exe
How the Mailbox Replication Service works
When a move mailbox request is issued, the command creates a message in the System mailbox of the target database. From there MRS picks up the request and makes a MAPI.net connection to both the source and target databases. After a successful MAPI connection is made, MRS creates a mailbox in the target database and starts incremental synchronization of data. When it reaches to a point where it is about to complete the move, it locks the mailbox, updates Active Directory attributes, unlocks the mailbox and deletes the source mailbox.
Here is the flow of the move operation in detail for online move.
Local Online Mailbox Move:
- Administrator creates the move request using the New-MoveRequest command.
- The New-MoveRequest makes the following checks for mailbox being moved.
- Gets the target and source mailbox server version
- Checks the database versions to verify they are supported Exchange versions.
- Determines the push or pull operation by the Exchange version information
- Checks for an archive mailbox, if one is found, then adds it to the move request. Also checks to make sure you are not moving a mailbox with an archive to legacy Exchange system.
- Checks the rule limit for legacy exchange servers
- Checks mailbox quotas
- The New-MoveRequest command creates a request message in the target database’s System Mailbox as special message.
- The following attributes are added to a user account for the mailbox in Active Directory. These attributes are used to store information about moving the mailbox and some are updated throughout the move.
These attributes will not be removed after the move request is completed unless a Remove-MoveRequest is run. If these attributes are not removed then another New-MoveRequest cannot be issued for the same mailbox.
- The New-MoveRequest command then “tickles” an MRS. A tickle is an operation where the command contacts an MRS directly to alert it to a new move request that is ready for pick up and processing. Which MRS is contacted is chosen at random from the CAS servers in the same AD site as the mailbox server where the target mailbox database is located.
- Mailbox Replication Service scans the mailbox databases for new interesting events. When it discovers the new interesting event, it then logs into the System Mailbox and gets information from the Move request messages.
- It then updates the message in the System Mailbox on the Mailbox Server’s MRS that owns the moving of the mailbox.
- The Mailbox Replication Service will then update the msExchMailboxMoveStatus attribute on the mailbox object in Active Directory.
- It will then log into the source and target mailboxes using MAPI.Net and start the synchronization of the user data. This type of synchronization is also referred to as a heavy pipe operation.
- Once the initial synchronization is complete, most all of the mailbox data will be synchronized to the target mailbox. The Mailbox Replication Service will then lock the mailbox.
- Mailbox Replication Service will then complete the synchronization of the data including any new or changed items. This last synchronization data is typically not a full synchronization; instead it is a moving of changed and new items.
- The Mailbox Replication Service will then update the following attributes in Active Directory on the mailbox account to point to the new mailbox.
- MSExchangeVersion (Set the appropriate Exchange Version)
- Proxy Address (Typically changed in Cross forest moves)
- The move history is then written to the user’s mailbox.
In the following screenshot, we can see the location of the MailboxMoveHistory using MFCMAPI.
Contents of the MailboxMoveHistory folder:
- The Mailbox Replication Service does not remove a move-mailbox request message from the System Mailbox. The message is removed when the move request is removed by the Remove-MoveRequest cmdlet.
- The Mailbox Replication Service then removes the source mailbox from the source database. It then changes the move status on msExchMailboxMoveStatus and msExchMailboxMoveFlags attributes to indicate that the move completed on the mailbox in Active Directory.
- Once the status has been changed to “Completed” the mailbox can be accessed again.
An Offline move works similar to the steps listed above with the exception that it will lock the mailbox database so no one can access the mailbox.
Mailbox Replication Service Queue
Each MRS keeps track of all move requests in its Active Directory site. It does this by scanning all System Mailboxes in the site. As mentioned earlier, each move request command creates a message in the System Mailbox of the target database. These messages are saved in the following folders:
- MailboxReplicationService Move Jobs
- MailboxReplicationService Move Reports
- MailboxReplicationService SyncStates
The messages contain queue information about the move request. This queue information can be accessed using the Get-MoveRequestStatistics command with the MoveRequestQueue parameter.
Below is a screenshot of a System Mailbox opened using MFCMAPI that has move request messages. Shown are the 3 different folders that the MRS uses to store information about move requests.
System Mailbox in MFCMAPI
The MailboxReplicationService Move Jobs and MailboxReplicationService Move Reports folders contain information about the move request stored as messages within the folders. Each message in the folder represents a single mailbox move represented by the msExchMailboxGUID attribute of the mailbox enabled account. The below screenshot shows the contents of the MailboxReplicationService Move Reports folder.
Mailbox moves and database failures
By default Mailbox Replication Service waits 30 seconds before attempting to reconnect to a database if it encounters transient problems during a move operation. It will try to reconnect every 30 seconds until a successful connection or 60 retries. If it cannot connect after 60 retries then it puts the move request into a failed state. The default retry interval and maximum number of retries can be changed by editing the MAXRetries and RetryDelay values in the MsexchangeMailboxReplication.exe.config file.
Mailbox moves and High Availability
Move mailbox operations that involve databases in a DAG environment are different than move mailbox operations on standalone databases.
The active database, the passive database and log shipping are factors that affect move mailbox in a DAG environment. MRS checks with the Active manager component of the Exchange replication service before, during and right before completing a move request to see if the active copy is up, if log shipping is not lagging behind and if the passive copies are keeping up. The action taken depends on a property of the database called DataMoveReplicationConstraint. If this value is not set, then the move operation assumes SecondCopy option if the database has a copy. That is the move operation does not take into consideration log shipping and the passive copies. If this value is set the action depends on what the actual value is.
Possible values for the DataMoveReplicationConstraint property are:
- None – The move operation treats the move just as it treats move mailbox operation on a standalone database. This is the default if the database is not replicated.
- SecondCopy – If the database is replicated then at least one passive mailbox database copy must have the changes synchronized. This is the default value.
- SecondDatacenter – If the database is replicated to two AD sites then at least one passive mailbox database copy in another AD site must have the changes replicated.
- AllDatacenters – If the database is replicated to multiple AD sites then at least one passive mailbox database copy in each AD site must have the changes replicated.
- AllCopies – If the database is replicated then all passive mailbox database copies must have the changes replicated.
The DataMoveReplicationConstraint property can be set by running the Set-MailboxDatabase with DataMoveReplicationConstraint parameter.
Mailbox Replication Service and High Availability Configuration
In addition to the DataMoveReplicationConstraint property of a database, the following two settings in msExchMailboxReplication.exe.config file also control the behavior of move mailbox that involves a DAG.
- DataGuaranteeCheckPeriod – Controls how often the MRS checks with the active manager. The default value is 5 minutes with a minimum value of 30 seconds and maximum of 2 hours.
- EnableDataGuaranteeCheck – When enabled the MRS checks with the active manager on the status of the mailbox databases. The default value is True.
Note: In order for the Mailbox Replication Service to check with active manager to see if the mailbox database is healthy and not behind processing log files both the EnableDataGuaranteeCheck in the MSExchangeMailboxReplication.exe.config and the DataMoveReplicationConstraint on the mailbox database need to be enabled. If they are enabled then the MRS will throttle back on processing the data transfer when the active manager reports database replication is not in a healthy state.
Database Availability Group Failover
During a move operation if the active database becomes unavailable then MRS contacts the active manager to see which copy will take over. MRS then will logon to the mailbox on the new database and will continue the move from where it left off. This is provided that the DataMoveReplicationConstraint property is set to other than none and also the database was not down for longer than 30 minutes. (Or there is another copy satisfying the constraint. If say the database has 3 copies, it’s entirely possible that MRS will just continue working after a failover even if the original server is down). If DataMoveReplicationConstraint is set to none then MRS will try to connect to the same database every 30 seconds for the next 30 minutes. The 30 minute is from the maximum retry of 60 times every 30 seconds. Off course this value can be changed in the in msExchMailboxReplication.exe.config file.
Cross Forest Moves
Exchange 2010 has the ability to move mailboxes between Active Directory forests. The MRS is responsible for moving mailboxes to an Exchange 2010 Mailbox server. When it comes to moving mailboxes from one Exchange forest to an Exchange 2010 forest, there are two move types:
- Remote – An Exchange 2010 Client Access (CAS) server is present in the source forest
- Remote Legacy – There is no Exchange 2010 CAS server in the source forest
When there is no Exchange 2010 CAS server in an Exchange forest that is the source of a mailbox move, the MRS in the Exchange 2010 target forest is designed to process the move request in a manner similar to previous versions of Exchange. In this case the MRS communicates directly to the Active Directory (AD) Directory Service in the source forest, as well as the mailbox server where the source mailbox is located.
When the source forest is also an Exchange 2010 forest, the MRS is designed to move mailboxes between the forests using a new feature that simplifies and improves the process.
In previous versions of Exchange in order to move mailboxes between different Active Directory forests, the administrator would have to allow direct MAPI access to servers and configure trusts and give other administrators full access to each other’s Exchange Organization. This way of moving mailboxes was not going to be effective with moving mailboxes to Exchange Online and between forests in Exchange 2010.
To overcome these issues Exchange 2010 introduces the Mailbox Replication Proxy Service (MRSProxy). The Mailbox Replication Proxy service works in conjunction with the MRS to facilitate the required communication between the source and target servers in each Exchange 2010 forest. Each CAS server that has an instance of the MRS also has an instance of the MRSProxy service as part of the implementation. Essentially, Mailbox Replication Proxy Service is a web service for the Mailbox Replication Service. It is part of the Exchange Web Services (EWS). The Mailbox Replication Proxy service will proxy MAPI, ExRPCAdmin and LDAP requests between local and remote forests when moving mailboxes. These requests will be HTTP requests that the Mailbox Replication Proxy Service will proxy these requests to the Mailbox Replication Service. The Mailbox Replication Service then communicates with the mailbox servers and sends the data back through the Mailbox Replication Proxy Service. The Mailbox Replication Proxy Service then communicates back to the Mailbox Replication Proxy server that initiated the request.
Many thanks to Stephen Gilbert and Jonathan Runyon for providing detailed information on Local Online Mailbox Move and their contributions to this blog. Thanks to Otavio Pereira, Matt Richoux and Nasir Ali for their review.
– Hamza Hassen, Nagesh Mahadev