O365 – Error when trying to move a mailbox to the cloud “CreatingFolderHierarchy CommunicationErrorTransientException”

By: Caio Ribeiro César

 

When dealing with moving mailboxes to the cloud, we should remember that there are several steps involved in order to make it work. Troubleshooting online mailbox moves can simple as increasing your upload rate or somewhat complex as the example in this article.

At first, we need to understand what technology is being used for the mailbox move. In this example, we are dealing with a hybrid mailbox move (or rich mailbox move), but there are other migration methods that can be used – for a complete list, please refer to this link.

Since we are dealing with a hybrid mailbox migration, let`s list some of the components that might bring issues while doing the move:

- Exchange On-Premises Performance;

- Exchange On-Premises Components (Mailbox Replication Service, MRSProxy);

- Mailbox Permission;

- Organization Relationship issues;

- Specific steps of the migration and other.

 

In summary, the rich mailbox move procedure consists of 9 steps:

 

1.       Starting the process (command New-RemoteMoverequest is run by the admin, received by O365 MRS);

2.       Validation – Mailbox Replication Service will confirm that the MSExchMailboxGuid from the mailbox is valid;

3.       Creation of the mailbox in the store - that will be migrated in the cloud environment;

4.       MRS (O365) connects to the local environment in order to find the source mailbox (again, using the property of MsExchMailboxGuid);

5.       On-premises MRS finds the source mailbox and sends the information to MRSProxy;

6.       MRS service from the cloud opens an WebServices connection with the on-premises CAS server;

7.       Data migration starts;

8.       When the data migration ends, the source mailbox (on-premises) is converted to a remote mailbox (or mail enabled user “MEU”) and a TargetAddress is added with the onmicrosoft address;

9.       The MEU from the cloud is converted to mailbox.

 

What is a “MEU”? When dealing with hybrid environments, we can`t have two mailboxes for a single user. The object has a single attribute of MSExchMailboxGuid so it can be matched in both environments, it is a mailbox in one of the environments and consequently a MEU in the other.

 

MEU is an object without attribute values for HomeMDB/homeMTA/msExchHomeServerName, but with an SMTP address configured in TargetAddress pointing to the destination mailbox.

 

Now that we are aware of some of the steps of the rich mailbox migration, let`s focus on the issue.

 

When trying to rich-move a specific mailbox to the cloud, the move request would stop in 10% with the error message:

 

3/4/2014 2:56:14 PM [BLUPR05MB258] The Microsoft Exchange Mailbox Replication service 'BLUPR05MB258.namprd05.prod.outlook.com' (15.0.888.6 caps:0FFF) is examining the request.

3/4/2014 2:56:20 PM [BLUPR05MB258] Connected to target mailbox caioc.onmicrosoft.com\9eb4312d-de3d-4184-a94-db6a2e54120b (Primary)', database 'CA-MDB', Mailbox server 'CASCAIOC.COM' Version 14.3 (Build 123.0), proxy server 'CASCAIOC.COM' 14.3.151.0 caps:05FFFF.

3/4/2014 2:56:21 PM [BLUPR05MB258] Connected to source mailbox 'caiocsource.onmicrosoft.com\9eb4312d-de3d-4184-a924-db6a2e54120b (Primary)', database 'NAMPR05DG017-db022', Mailbox server 'BLUPR05MB258.namprd05.prod.outlook.com' Version 15.0 (Build 888.0).

3/4/2014 2:56:22 PM [BLUPR05MB258] Request processing continued, stage CreatingFolderHierarchy.

3/4/2014 2:56:22 PM [BLUPR05MB258] Initializing folder hierarchy from mailbox 'caioc.onmicrosoft.com\9eb4312d-de3d-4184-a924-db6a2e54120b (Primary)': 207 folders total.

3/4/2014 2:56:22 PM [BLUPR05MB258] Folder creation progress: 0 folders created in mailbox 'caioc.onmicrosoft.com\9eb4312d-de3d-4184-a924-db6a2e54120b (Primary)'.

3/4/2014 2:59:19 PM [BLUPR05MB258] Transient error CommunicationErrorTransientException has occurred. The system will retry (1/60).

 

In this error message we can understand that:

 

- The request stops in the creation of the folder hierarchy “CreatingFolderHierarchy”;

- The mailbox being migrated contains 207 folders, but the process fails before the data migration starts and nothing is migrated to the cloud;

- The error message is “CommunicationErrorTransientException”.

 

We need more information about what exactly is causing this failure so then we can create a solid Action Plan.

 

We need a verbose report of the move request and then filter it with error messages only. The command below will generate an xml report with debugging information of the move request:

 

Get-MoveRequestStatistics caioc -IncludeReport | Export-Clixml caioc.xml

This xml needs to be filtered with error messages only. We can create a filter of report failures:

 

$stats = Import-CliXml caioc.xml

$stats.report.failures[-1]

 

Result:

Timestamp         : 3/4/2014 7:04:16 PM

FailureType       : CommunicationErrorTransientException

FailureCode       : -2146233088

MapiLowLevelError : 0

FailureSide       : Target

FailureSideInt    : 2

ExceptionTypes    : {Exchange, Transient, MRS, MRSTransient}

ExceptionTypesInt : {1, 2, 10, 11}

Message: The call to 'https://cascaioc.domain.com/EWS/mrsproxy.svc CASCAIOC.com (14.3.151.0 caps:05FFFF)' failed. Error details: The server was unable to process the request due to an internal error.  For more information about the error, either turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the <serviceDebug> configuration behavior) on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework 3.0 SDK documentation  and inspect the server trace logs.. --> The server was unable to process the request due to an internal error.  For more information about the error, either turn on IncludeExceptionDetailInFaults (either from ServiceBehaviorAttribute or from the <serviceDebug> configuration behavior) on the server in order to send the exception information back to the client, or turn on tracing as per the Microsoft .NET Framework 3.0 SDK documentation and inspect the server trace logs.

MessageData       :

DataContext       : --------

                    Operation: IDestinationFolder.SetSearchCriteria

                    Operation: IDestinationFolder.SetSearchCriteria

OperationSide: Target

                    caioc.onmicrosoft.com\9eb4312d-de3d-4184-a924-db6a2e54120b (Primary)

Restriction: Restriction: AND[count:2, AND[count:7, NOT[CONTENT[ptag:0x1a001f(MessageClass), Prefix, IgnoreCase, val:[Tag:0x1a001f(MessageClass),Value:"IPM.Appointment"(string)]]]; NOT[CONTENT[ptag:0x1a001f(MessageClass), Prefix, IgnoreCase, val:[Tag:0x1a001f(MessageClass), Value:"IPM.Contact"(string)]]];                 NOT[CONTENT[ptag:0x1a001f(MessageClass), Prefix, IgnoreCase, val:[Tag:0x1a001f(MessageClass),Value:"IPM.DistList"(string)]]];NOT[CONTENT[ptag:0x1a001f(MessageClass), Prefix,                  IgnoreCase, val:[Tag:0x1a001f(MessageClass), Value:"IPM.Activity"(string)]]]; NOT[CONTENT[ptag:0x1a001f(MessageClass), Prefix, IgnoreCase, val:[Tag:0x1a001f(MessageClass),               Value:"IPM.StickyNote"(string)]]]; NOT[CONTENT[ptag:0x1a001f(MessageClass), IgnoreCase, val:[Tag:0x1a001f(MessageClass), Value:"IPM.Task"(string)]]];                   NOT[CONTENT[ptag:0x1a001f(MessageClass), Prefix, IgnoreCase, val:[Tag:0x1a001f(MessageClass), Value:"IPM.Task."(string)]]]]; AND[count:2, AND[count:4,                    PROPERTY[ptag:0xe090102(ParentEntryId), NotEqual, val:[Tag:0xe090102(ParentEntryId), Value:cb = 46, lpb =                    00000000297ED05D0A0CD211921C006097D538A20100BC42FDA7843F4947929EED9833A2252700000A6BE4BC0000]]; PROPERTY[ptag:0xe090102(ParentEntryId), NotEqual,

                  val:[Tag:0xe090102(ParentEntryId), Value:cb = 46, lpb = 00000000297ED05D0A0CD211921C006097D538A20100BC42FDA7843F4947929EED9833A2252700000A6BE4BD0000]];                   PROPERTY[ptag:0xe090102(ParentEntryId), NotEqual, val:[Tag:0xe090102(ParentEntryId), Value:cb = 46, lpb                    00000000297ED05D0A0CD211921C006097D538A20100BC42FDA7843F4947929EED9833A2252700000A6BE4BE0000]]; PROPERTY[ptag:0xe090102(ParentEntryId), NotEqual,                    val:[Tag:0xe090102(ParentEntryId), Value:cb = 46, lpb = 00000000297ED05D0A0CD211921C006097D538A20100BC42FDA7843F4947929EED9833A2252700000A6BE4BB0000]]]; AND[count:2, AND[count:2,                    NOT[OR[count:2, AND[count:3, OR[count:2, NOT[EXIST[ptag:0x10950003]]; PROPERTY[ptag:0x10950003, Equal, val:[Tag:0x10950003, Value:0(int)]]]; EXIST[ptag:0x10900003]                   PROPERTY[ptag:0x10900003, Equal, val:[Tag:0x10900003, Value:1(int)]]]; AND[count:2, PROPERTY[ptag:0x80920003(NamedProp), Equal, val:[Tag:0x80dd0003(NamedProp), Value:2(int)]];                   EXIST[ptag:0x80920003(NamedProp)]]]]; OR[count:3, AND[count:2, EXIST[ptag:0xe2b0003]; BITMASK[ptag:0xe2b0003, NotEqualToZero, mask:0x1]]; AND[count:2, EXIST[ptag:0x10950003]                   PROPERTY[ptag:0x10950003, GreaterThan, val:[Tag:0x10950003, Value:0(int)]]]; AND[count:2, NOT[AND[count:2, PROPERTY[ptag:0x8093000b(NamedProp), NotEqual,                   val:[Tag:0x80e4000b(NamedProp), Value:True(bool)]]; PROPERTY[ptag:0x80940003(NamedProp), Equal, val:[Tag:0x80e50003(NamedProp), Value:2(int)]]]]; OR[count:2,                   CONTENT[ptag:0x1a001f(MessageClass), Prefix, IgnoreCase, val:[Tag:0x1a001f(MessageClass), Value:"IPM.Task."(string)]]; CONTENT[ptag:0x1a001f(MessageClass), IgnoreCase,                   val:[Tag:0x1a001f(MessageClass), Value:"IPM.Task"(string)]]]]]]; CONTENT[ptag:0x8003101f(NamedProp), IgnoreCase (mv), val:[Tag:0x8002101f(NamedProp), Value:"FII"(string)]]]]]                   EntryIDs: [count:1, [len=46, data=00000000297ED05D0A0CD211921C006097D538A20100007CA93C08CCD1119219006097D538A20000002882E50000]]

                 Flags: Recursive, NonContentIndexed, FailOnForeignEID

                    --------

                  Search folder: '/Finder/Sent directly to me, entryId [len=46, data=00000000297ED05D0A0CD211921C006097D538A20100ACCB14784BE62B4283BA485D3DB115F1000008147CF50000], parentId                   [len=46, data=00000000297ED05D0A0CD211921C006097D538A20100007CA93C08CCD1119219006097D538A20000002882EC0000]

DataContextData   :

StackTrace       :    at Microsoft.Exchange.MailboxReplicationService.CommonUtils.CallWCFService[ExceptionT](Action serviceCall, String epAddress, Action`1 faultHandler, VersionInformation

                    serverVersion)

                       at Microsoft.Exchange.MailboxReplicationService.CommonUtils.CallService(Action serviceCall, String epAddress, VersionInformation serverVersion)

                       at Microsoft.Exchange.MailboxReplicationService.MailboxReplicationProxyClient.CallServiceWithTimeout(TimeSpan timeout, Action serviceCall)                      at Microsoft.Exchange.MailboxReplicationService.MailboxReplicationProxyClient.Microsoft.Exchange.MailboxReplicationService.IMailboxReplicationProxyService.IDestinationFolder_Se                   tSearchCriteria(Int64 folderHandle, RestrictionData restriction, Byte[][] entryIDs, Int32 searchFlags)

                       at Microsoft.Exchange.MailboxReplicationService.RemoteDestinationFolder.Microsoft.Exchange.MailboxReplicationService.IDestinationFolder.SetSearchCriteria(RestrictionData                   restriction, Byte[][] entryIds, SearchCriteriaFlags flags)

                       at Microsoft.Exchange.MailboxReplicationService.DestinationFolderWrapper.<>c__DisplayClass28.<Microsoft.Exchange.MailboxReplicationService.IDestinationFolder.SetSearchCriteria>                    b__27()                      at Microsoft.Exchange.MailboxReplicationService.ExecutionContext.Execute(Action operation)

                       at Microsoft.Exchange.MailboxReplicationService.ExecutionContextWrapper.Execute(Action operation, Boolean measure)

                       at Microsoft.Exchange.MailboxReplicationService.DestinationFolderWrapper.Microsoft.Exchange.MailboxReplicationService.IDestinationFolder.SetSearchCriteria(RestrictionData                   restriction, Byte[][] entryIds, SearchCriteriaFlags flags)

                       at Microsoft.Exchange.MailboxReplicationService.CommonUtils.ProcessKnownExceptions(ActionactionDelegate, FailureDelegate failureDelegate)                       at Microsoft.Exchange.MailboxReplicationService.MailboxCopierBase.CopyFolderProperties(FolderRecWrapper folderRec, ISourceFolder sourceFolder, IDestinationFolder destFolder,                  FolderRecDataFlags dataToCopy, Boolean& isContentAvailable)

                       at Microsoft.Exchange.MailboxReplicationService.MoveBaseJob.<>c__DisplayClass3a.<>c__DisplayClass3e.<CreateFolderHierarchy>b__37()                     at Microsoft.Exchange.MailboxReplicationService.ExecutionContext.Execute(Action operation)

                       at Microsoft.Exchange.MailboxReplicationService.MoveBaseJob.<>c__DisplayClass3a.<CreateFolderHierarchy>b__36(FolderRecWrapper folderRec, EnumFolderContext context)

                       at Microsoft.Exchange.MailboxReplicationService.FolderMap.EnumSingleFolder(FolderRecWrapper folderRec, EnumFolderContext ctx, EnumFolderCallback callback, EnumHierarchyFlags                    flags)

                       at Microsoft.Exchange.MailboxReplicationService.FolderMap.EnumSingleFolder(FolderRecWrapper folderRec, EnumFolderContext ctx, EnumFolderCallback callback, EnumHierarchyFlags                    flags)

                       at Microsoft.Exchange.MailboxReplicationService.FolderMap.EnumSingleFolder(FolderRecWrapper folderRec, EnumFolderContext ctx, EnumFolderCallback callback, EnumHierarchyFlags                    flags)

                       at Microsoft.Exchange.MailboxReplicationService.FolderMap.EnumerateSubtree(EnumHierarchyFlags flags, FolderRecWrapper root, EnumFolderCallback callback)                      at Microsoft.Exchange.MailboxReplicationService.MoveBaseJob.CreateFolderHierarchy()

                       at Microsoft.Exchange.MailboxReplicationService.CommonUtils.ProcessKnownExceptions(Action actionDelegate, FailureDelegate failureDelegate) InnerException    : FaultException: The server was unable to process the request due to an internal error.  For more information about the error, either turn on IncludeExceptionDetailInFaults

                    (either from ServiceBehaviorAttribute or from the <serviceDebug> configuration behavior) on the server in order to send the exception information back to the client, or turn on

                    tracing as per the Microsoft .NET Framework 3.0 SDK documentation and inspect the server trace logs

 

In this error message, we can understand that the MRS service can`t copy the Search Folder “Sent directly to me”, when trying to create the content hierarchy.

 

A Search Folder is a virtual folder that provides a view of all email items that match specific search criteria. For example, the Unread Mail Search Folder enables you to view all unread messages in one folder, even though the messages might be saved in different folders.

 

When asking the end-user to open his Outlook, we could see the “Sent Directly to Me” search folder listed:

 

clip_image001

 

The end user removed the search folder (no email is lost, only the search folder is removed and he can recreate it after the migration). And we could re-run the move request with success.

 

Another option is to remove all search folders by running the command “outlook.exe /cleanfinders”:

 

clip_image002

 

If the folder is not available in Outlook and the cleanfinders option doesn’t` fix the issue, we can use MFCMapi (online mode) to validate search folders permissions.

 

You can download MFCMAPI and configure the end-user Outlook profile to online. Thereafter, open the tool and select the option “Session” and “Logon and Display Store Table”.

 

Select then the user profile, the mailbox will appear in the tool. Click on it twice so a new window will open:

 

clip_image003

 

On the left side, we can find the specific folder:

 

clip_image004

 

When right-clicking the folder, there are two options: remove folder or reset permissions. Both options would resolve the incident.