Troubleshooting issues where the hybrid migration endpoint cannot be created

This article is meant to help you troubleshoot scenarios where the connection from Office365 to your hybrid migration server cannot be established: it can  be about the first time when you attempt to create the migration endpoint, but it also applies to cases where the endpoint was created a while ago and now you cannot connect anymore to the migration server and none of the mailboxes can be migrated.

It is also good to know that the steps mentioned here will not necessarily deliver you the solution, but for sure they will help you in isolating the issue.


Introduction to remote move requests

I will take this opportunity to discuss about the process of remote moves and about the factors that matter in a hybrid scenario, where you would like to move your mailboxes to the cloud.

After a hybrid setup is deployed, most of the customers will create the migration endpoint in Office365 Exchange Admin Center, as a first step in the migration process.

The interesting part is that if you managed to successfully run the hybrid wizard, you would expect  to have the same success when creating the migration endpoint. This is a wrong expectation, because in some cases, for example due to firewall configuration (URL filtering or IP restriction), your attempt to create a migration endpoint on Office365 side can fail.

It is worth mentioning that with the new Hybrid Configuration Wizard you will get the benefit of having the “remote move” migration endpoint automatically created into Office365 EAC. But like I said above, due to some configuration issues, even the new HCW might be unable to create the endpoint on your behalf.

Soon we will simplify this process for our customers, especially when they just need to perform a remote move migration to the cloud, by giving them the option to deploy a simplified hybrid configuration. For example, in the case where you do not need free-busy or centralized mailflow features , you can choose only the options that you are interested in.

Before starting the migration, you must ensure that your on-premises Exchange servers meet the prerequisites. Here I want to mention that you need to first enable the MRSproxy on each of your CAS servers, as this endpoint is responsible for the accepting the requests for remote moves and to proxy them to the servers that are running Mailbox Replication Service.

The ability of a Client Access server to accept these MRS requests is disabled by default. To allow the Client Access server to accept incoming move requests, you must enable the MRS Proxy endpoint. You can refer to below article for more details:

https://technet.microsoft.com/en-us/library/dn155787(v=exchg.150).aspx

 

The Mailbox Replication Service  is the service responsible for cross-forest mailbox moves and remote move migrations between your on-premises Exchange organization and Exchange Online:

  • in Exchange 2010 the MRS service is running on the CAS server.
  • in Exchange 2013, MRS is running on the Mailbox server role; during cross-forest and remote move migrations, a Client Access server acts as a proxy for incoming move requests for the Mailbox server.
  • in Exchange 2016 we have a new architecture, where the Mailbox server integrates both CAS and Mailbox roles, hence the MRS service is running on the Mailbox server and the CAS role acts first like a proxy, as in Exchange 2013 case.

Creating migration endpoint in Office365

In Office365 EAC, you are basically provided with 2 options  to create a “Remove Move” migration endpoint:

a) First you can use Autodiscover and mention below parameters:

- the email address of an on-premises mailbox: this will be used by Autodiscover service to detect the server running the MRSproxy endpoint.

-the local Exchange administrator credentials: these will be used to connect remotely to your hybrid server and to authorize the incoming migration requests.

You can see an example in the next picture :

 

autodiscover_endpoint1

b) If Autodiscover can’t detect your migration server, you can input manually the external hostname of your server, as in below example:

 

failed_endpoint1

 

Below diagram explains how the cloud MRSproxy is connecting  to a server from your organization, in order to migrate a mailbox:

 

In above example we have an environment with multiple CAS servers and a Load Balancer in front of them:

  •  Exchange 2013 CAS server, that has MRSproxy enabled and acts like a proxy for the MRS requests.
  •  Exchange 2013 Mailbox Server, that runs the Mailbox Replication Service.
  • Exchange 2013 multi-role server, where we have both MRSproxy and Mailbox Replication Service on a single place.

When we try to establish a connection from the cloud with the on-premises MRSproxy endpoint, the request is passing through the firewall, from here it is forwarded to the load balancer and this can choose one of the 2 CAS servers as a destination for the incoming MRS request.


Configuration requirements for remote move requests

Firewall considerations:

Let's  imagine an environment configured like in the mentioned example. Here we can have a lot of factors that can prevent the incoming MRS calls reaching the destination server.

If you are not aware about the way your firewall is configured or you are having a third-party company that is managing it, it is strongly recommended to double check with the responsible team that the firewall meets below pre-requisites:

  • Your firewall must allow the incoming connections on URL containing /ews/mrsproxy.svc, in case it is using URL filtering, or if possible, you can allow all the incoming traffic on port 443.
  • If you are using an IP filtering configuration in the firewall, you should make sure that you have the updated Exhange Online IP list in the firewall configuration; you can find IP ranges list for Exchange Online in below article:

https://support.office.com/en-us/article/Office-365-URLs-and-IP-address-ranges-8548a211-3fe7-47cb-abb1-355ea5aa88a2?ui=en-US&rs=en-US&ad=US#BKMK_EXO

  • In case you want to be informed every time when we update this list, you can choose the option “ Subscribe via RSS " from above url.
  • If your firewall requires pre-authentication or you are using a solution that is performing SSL offloading in front of CAS servers, please note that these configurations are not supported in remote moves scenarios, due to the fact the on-premises MRS expects to receive the incoming traffic over port 443 with SSL encryption and it refuses the connection if these conditions are not met.
  • If you are using a TMG server, make sure that you created a rule to publish EWS service, in order for this service to be accessible from outside the network.

Load Balancer considerations:

  • When you have multiple Exchange 2010 Client Access Servers in the organization with a Load Balancers in front of them, it is strongly recommended that the load balancer to maintain the session affinity(or IP persistence), more precisely all the migration requests for specific mailbox should be processed by the same CAS server.
  • If you have Exchange 2010 with a such miss-configured load balancer, you will most likely experience slowness during the remote moves, due to a lot of transient errors like this ones: The remote endpoint no longer recognizes this sequence. This is most likely due to an abort on the remote endpoint. The value of wsrm:Identifier is not a known Sequence identifier. The reliable session was faulted" or "Couldn't switch the mailbox into Sync Source mode/SourceMailboxAlreadyBeingMovedTransientException".
  •  Exchange Server 2013 and Exchange Server 2016 do not require session affinity for the load balancing layer. This means that regardless of where a connection originates, it always connects, either directly or by proxy, to the Mailbox server that has an active copy of the database where that user’s mailbox is located.
  • Some of the existing Load Balancers might not support this configuration and are providing only the option to forward the incoming requests based on the CAS servers availability/loading. In this scenario, the suggested option would be to bypass the Load Balancer during the migration, to better isolate the issue.

Troubleshooting steps

If you managed to understand the process behind the migration endpoint creation and how MRS works with remote moves, the troubleshooting process will become much easier and you will have the answers to these very common questions:

Why is Office365 unable to detect my migration server?

What are the firewall requirements for remote move migrations?

How can I make sure that the firewall is not blocking the migration requests?

What is the server from my organization that should process the incoming MRSproxy requests?

Where do I see the incoming requests for my MRSproxy endpoint?

In these scenarios we should agree that the cause of the issue lies on the on-premises environment, where the communication between cloud MRSproxy  and local MRSproxy service is failing somewhere locally.

 

These are the most relevant steps that we should perform on-premises during the troubleshooting phase:

  • check the EWS configuration on each CAS server
  • test MRS health on the on-premises Exchange Servers
  • browse to the MRSproxy external URL
  • use Test-MigrationServerAvailability from Exchange Online
  • check the IIS logs on CAS servers
  • check HttpProxy and HTTPERR  logs from CAS servers

The order of the steps might vary from one case to another, so it depends on you to choose the order in which you want to perform them.

Once you managed to complete all these steps , you will most likely be able to identify the cause of the issue.

1. Check the EWS configuration on each CAS server

Ensure that the MRS proxy endpoint is enabled on every CAS server that will be used in the migration process.

Collect below output from local Exchange Management Shell:

Get-WebServicesVirtualDirectory|fl ExternalAuthenticationMethods,Externalurl,MRSproxyEnabled,Server

The output can be similar to below example:

ExternalAuthenticationMethods : {Basic, Ntlm, WindowsIntegrated, WSSecurity, OAuth}

ExternalUrl                   : https://mail.pronicsi.com/EWS/Exchange.asmx

MRSProxyEnabled               : True

Server                        : EX2015

Here is important to check that Windows authentication and Basic authentication are enabled and that you have the correct ExternalUrl populated on each CAS server.

 

2. Confirm that the local MRSproxy service works fine

  • Check first that the MSExchangeMailboxReplication.exe service is running on the migration servers.
  • Perform some local move requests to confirm that MRS service works fine.
  • You can check if Microsoft Exchange Mailbox Replication service is consistent on your Mailbox/ CAS servers by using the command Test-MRSHealth . Please refer to below article for more details:

https://technet.microsoft.com/en-us/library/ee332325%28v=exchg.160%29.aspx?f=255&MSPPError=-2147217396

3. Browse to the MRSproxy external URL

  • This can help you to confirm that your MRS proxy URL is reachable from outside the network, by accessing it from an internet browser.
  • Since MRSproxy endpoint is running under Exhange Web Services, it is expected that all the MRS request will be encapsulated in EWS calls, so the URL should have this format: https://mail.contoso.com/ews/mrsproxy.svc
  • The middle portion “mail.contoso.com” is the external URL for your EWS service.
  • If MRSproxy service is accessible from outside the network, you should receive a credential prompt.
  • If no credential prompt occurs, this indicates that your EWS service is not accessible externally and the firewall could block the MRS incoming requests, so you need to double check the firewall configuration and see if you can find any relevant logs in the firewall.

 

4. Test-MigrationServerAvailability

This test is very useful because you can see more details about why the connection with your on-prem server cannot be created and you can use the received error message or the HTTP response code to investigate the issue further.

You need to connect to Exchange Online from PowerShell and test the connection with the migration server( for remote moves), using below commands:

$Cred=Get-Credential (input the local Exchange Admin credentials)

Test-MigrationServerAvailability -ExchangeRemoteMove -Autodiscover -EmailAddress user@contoso.com -Credentials $Cred

If you want to test the connection without Autodiscover, you can use below command:

Test-MigrationServerAvailability -ExchangeRemoteMove -RemoteServer mail.contoso.com -Credentials(Get-Credential)

If both of the tests failed, you might have to check the firewall logs for any incoming MRS request, to see if these are not blocked or if they are allowed, to see what is the destination CAS server.

 

 5.Check the IIS logs from CAS servers

If you are sure that the firewall is not the problem, you can see further if the incoming MRS calls have reached out to correct destination, by checking the IIS logs on the CAS servers.

Depending on Exchange Server versions and the roles installed, the CAS server can process the request directly or it may forward it to a Mailbox server, that hosts the mailboxes that you want to migrate.

If you have multiple servers in the organization, some of them might have important services disabled or not enough resources assigned; these are usually called "passive" servers. If the migration request will somehow reach out to one of these servers, we can expect to have a failure.

Therefore, to avoid these doubts, you can check the IIS logs on the destination CAS servers and you will clarify if the migration calls are present there or not.

The location of the IIS logs is by default C:\inetpub\logs\LogFiles and in these logs we should search for entries that contain EWS/mrsproxy.svc as destination URL.

Note that by default the IIS logs are stamped in UTC time format, regardless the time that you set on your server.

Every call to MRS service should be generating some IIS logs. So you should try either to browse to the MRSproxy external URL or to use the command Test-MigrationServerAvailability to reproduce the issue and in a few minutes you can check the IIS logs, to find any entries for EWS/mrsproxy.svc.

If no IIS entry can be found on the CAS servers that are meant to be the destination servers or if the migration requests can be seen on the passive/wrong CAS server logs , this indicates clearly that you should reconfigure the firewall rules or the load balancers, if you are using any.

Please be aware that each incoming request for the on-premises MRSproxy will generate several entries/calls on the IIS logs, so if you will see multiple lines with mrsproxy.svc, consider it normal.

In the next picture we ran Test-MigrationServerAvailability from Exchange Online PowerShell, so we can see an example with the IIS log entry generated by a failed connection to MRSproxy:

test-migrationserver

When checking the IIS logs in the hybrid server, we see below entry:

2016-09-15 08:20:18 10.2.0.5 POST /EWS/mrsproxy.svc&CorrelationID=<empty>;&ClientId=NF0C4KC30EGFQUUPAZ7QW&cafeReqId=13bae44a-b1c7-402f-aa2f-763aee410ffa; 443 - 132.245.35.172 - - 401 1 2148074254 156

From this log entry, we can all agree that this is not a firewall problem, since our request reached to the correct destination, but rather an authentication problem on the CAS server, where we got a 401 HTTP response. In my simulation, the issue occurred due to invalid credentials of the local Exchange admin account used for testing the MRS connection,

Below is an example of an IIS log generated by a successful connection:

2016-09-15 08:23:57 10.2.0.5 POST /EWS/mrsproxy.svc &CorrelationID=<empty>;&ClientId=SR0WSDR7B0IXI4TCOLJG&cafeReqId=d26343b6-8790-4ccc-954d-936aed081d88; 443 pronicsi\nicsi 132.245.35.172 - - 200 0 0 343

In this log you can also see the admin account used to establish the remote connection with MRS, along with the HTTP response 200, meaning that our request was accepted by the server.

If you receive a different HTTP response error during this test, you might have to deal with a wrong configuration on IIS side. Like I said before, the MRS proxy endpoint is configured under EWS, which is actually a virtual directory in IIS.

You might find useful below article, for identifying possible causes for various HTTP status codes in IIS:

https://support.microsoft.com/en-us/kb/943891

 

6. Check HttpProxy and HTTPERR logs from CAS servers

If you are running  Exchange Server 2013 or/and Exchange Server 2016 and you saw the MRS requests on the IIS logs from CAS servers, but you are not sure about where the issue might be, you can extract more logs from other places on the CAS server.

HTTP proxy logs for EWS can be found at the following path :

C:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Ews

In this log we can see where the CAS server is going to proxy the MRS request, it can be either another CAS server or to a stand-alone Mailbox server. You need to find the entries for /EWS/mrsproxy.svc, like in below example:

HTTPproxylog

 HTTP error logs that are located to below path:

C:\Windows\System32\LogFiles\HTTPERR

This is an example of some MRS related calls from HTTP error logs :

httpERR


 

I hope you found this article useful in troubleshooting issues related to hybrid migration endpoints and I am looking forward to  receiving any feedback from your side.