SharePoint 2010 – User Profile Replication Engine (UPRE)–Troubleshooting

  • Article

 

Hello @all,

it’s a long time ago as a wrote my last blog post… since November 2010 I have changed my job role inside of Microsoft (from Support Escalation Engineer to a Premier Field Engineer). With the new role I had the chance to join into cool projects around the topic SharePoint and here come my next post.

In a global MySite deployment project for 70.000 users we have 3 farms: 1 in Europe, 1 in America and 1 in ASIA. All 3 Farms should host SharePoint MySite for each local user. This means: America users have their MySite in America farm, ASIA users have their MySite in ASIA farm and Europe users have their MySite in Europe. Based on performance, I believe in this design we reduce the IO traffic from one continent to another.

Caused of this design we have 3 User Profile Stores, one in each farm and for local search we need to replicate the user profiles with using the user profile replication engine (UPRE).
Comment: we could create a remote UPA connection and only one Farm host the global User Profile Store, but this create a lot of traffic for each MySite page rendering request. In case of Social networking functionality with outlook the amount of additional MySite request increase and we need much more hardware for one farm. Additionally we create a single point of failure.

So what we did: create 3 farms, use the replication:

  1. install SharePoint Administration Toolkit to get the User Profile Replication Engine installed
  2. as next step: we need to start the replication with using powershell cmdlets, first we need to collect all user profile properties which we want to replication and then we need to start the Full or incremental replication.

So we did this:

 get-spprofilepropertycollection –source https://Europe.demo.com | 
start-spprofileservicefullreplication -source https://Europe.demo.com 
-destination https://America.demo.com –EnableInstrumentation 
-MaxNumberOfThreads 15 -DoSocialReplication

An user profile has a size of around 1 MB and with a amount of 70.000 users a full replication mean we copy 70.000 MB = 70 GB data from Europe to America. Later we can easily start the increment replication.

 

With these action the whole magic of the replication is finished, but the learning of this project is not finished:

1. the replication was very slow and our calculation show us: a Full Replication of 70.000 user profiles took around: 20 days (measured over IIS w3c logs and UPRE logs). From replication engine we have 4 log file to troubleshoot the behavior:

image

2.  Inside the Error logs we see many time the following Event Entries and Exception:

 

Event Log Error:

Microsoft.ResourceManagement.Service: System.ArgumentException: Unable to open database connection

Parameter name: connectionString ---> System.Transactions.TransactionException: The partner transaction manager has disabled its support for remote/network transactions. (Exception from HRESULT: 0x8004D025) ---> System.Runtime.InteropServices.COMException (0x8004D025): The partner transaction manager has disabled its support for remote/network transactions. (Exception from HRESULT: 0x8004D025)

 

Detailed Error:

Log Name: Application

Source: Forefront Identity Manager

Date: 2/23/2011 11:04:04 AM

Event ID: 3

Task Category: None

Level: Error

Keywords: Classic

User: N/A

Computer: XXXXXXXXX

Description:

Microsoft.ResourceManagement.Service: System.ArgumentException: Unable to open database connection

Parameter name: connectionString ---> System.Transactions.TransactionException: The partner transaction manager has disabled its support for remote/network transactions. (Exception from HRESULT: 0x8004D025) ---> System.Runtime.InteropServices.COMException (0x8004D025): The partner transaction manager has disabled its support for remote/network transactions. (Exception from HRESULT: 0x8004D025)

   at System.Transactions.Oletx.IDtcProxyShimFactory.ReceiveTransaction(UInt32 propgationTokenSize, Byte[] propgationToken, IntPtr managedIdentifier, Guid& transactionIdentifier, OletxTransactionIsolationLevel& isolationLevel, ITransactionShim& transactionShim)

   at System.Transactions.TransactionInterop.GetOletxTransactionFromTransmitterPropigationToken(Byte[] propagationToken)

   --- End of inner exception stack trace ---

   at System.Transactions.Oletx.OletxTransactionManager.ProxyException(COMException comException)

   at System.Transactions.TransactionInterop.GetOletxTransactionFromTransmitterPropigationToken(Byte[] propagationToken)

   at System.Transactions.TransactionStatePSPEOperation.PSPEPromote(InternalTransaction tx)

   at System.Transactions.TransactionStateDelegatedBase.EnterState(InternalTransaction tx)

   at System.Transactions.EnlistableStates.Promote(InternalTransaction tx)

   at System.Transactions.Transaction.Promote()

   at System.Transactions.TransactionInterop.ConvertToOletxTransaction(Transaction transaction)

   at System.Transactions.TransactionInterop.GetExportCookie(Transaction transaction, Byte[] whereabouts)

   at System.Data.SqlClient.SqlInternalConnection.GetTransactionCookie(Transaction transaction, Byte[] whereAbouts)

   at System.Data.SqlClient.SqlInternalConnection.EnlistNonNull(Transaction tx)

   at System.Data.ProviderBase.DbConnectionInternal.ActivateConnection(Transaction transaction)

   at System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject)

   at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)

   at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)

   at System.Data.SqlClient.SqlConnection.Open()

   at System.Workflow.Runtime.Hosting.DbResourceAllocator.OpenNewConnection(Boolean disallowEnlist)

   at System.Workflow.Runtime.Hosting.DbResourceAllocator.Init(String connectionStr)

   --- End of inner exception stack trace ---

   at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.ActivateHost(ResourceManagementWorkflowDefinition workflowDefinition)

   at Microsoft.ResourceManagement.Workflow.Hosting.WorkflowManager.StartWorkflowInstance(Guid workflowInstanceIdentifier, KeyValuePair`2[] additionalParameters)

Event Xml:

<Event xmlns="https://schemas.microsoft.com/win/2004/08/events/event">

  <System>

    <Provider Name="Forefront Identity Manager" />

    <EventID Qualifiers="0">3</EventID>

    <Level>2</Level>

    <Task>0</Task>

    <Keywords>0x80000000000000</Keywords>

    <TimeCreated SystemTime="2011-02-23T10:04:04.000Z" />

    <EventRecordID>699449</EventRecordID>

    <Channel>Application</Channel>

    <Computer>XXXXXXXXXXXXXXX</Computer>

    <Security /> 
</System>

<EventData>

    <Data>Microsoft.ResourceManagement.Service: System.ArgumentException: Unable to open database connection

Parameter name: connectionString ---&gt; System.Transactions.TransactionException: The partner transaction manager has disabled its support for remote/network transactions. (Exception from HRESULT: 0x8004D025) ---&gt; System.Runtime.InteropServices.COMException (0x8004D025): The partner transaction manager has disabled its support for remote/network transactions. (Exception from HRESULT: 0x8004D025)

   at System.Transactions.Oletx.IDtcProxyShimFactory.ReceiveTransaction(UInt32 propgationTokenSize, Byte[] propgationToken, IntPtr managedIdentifier, Guid&amp; transactionIdentifier, OletxTransactionIsolationLevel&amp; isolationLevel, ITransactionShim&amp; transactionShim)

   at System.Transactions.TransactionInterop.GetOletxTransactionFromTransmitterPropigationToken(Byte[] propagationToken)

   --- End of inner exception stack trace ---

   at System.Transactions.Oletx.OletxTransactionManager.ProxyException(COMException comException)

   at System.Transactions.TransactionInterop.GetOletxTransactionFromTransmitterPropigationToken(Byte[] propagationToken)

   at System.Transactions.TransactionStatePSPEOperation.PSPEPromote(InternalTransaction tx)

   at System.Transactions.TransactionStateDelegatedBase.EnterState(InternalTransaction tx)

   at System.Transactions.EnlistableStates.Promote(InternalTransaction tx)

   at System.Transactions.Transaction.Promote()

   at System.Transactions.TransactionInterop.ConvertToOletxTransaction(Transaction transaction)

   at System.Transactions.TransactionInterop.GetExportCookie(Transaction transaction, Byte[] whereabouts)

   at System.Data.SqlClient.SqlInternalConnection.GetTransactionCookie(Transaction transaction, Byte[] whereAbouts)

   at System.Data.SqlClient.SqlInternalConnection.EnlistNonNull(Transaction tx)

   at System.Data.ProviderBase.DbConnectionInternal.ActivateConnection(Transaction transaction)

   at System.Data.ProviderBase.DbConnectionPool.GetConnection(DbConnection owningObject)

   at System.Data.ProviderBase.DbConnectionFactory.GetConnection(DbConnection owningConnection)

   at System.Data.ProviderBase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory)

   at System.Data.SqlClient.SqlConnection.Open()

   at System.Workflow.Runtime.Hosting.DbResourceAllocator.OpenNewConnection(Boolean disallowEnlist)

   at System.Workflow.Runtime.Hosting.DbResourceAllocator.Init(String connectionStr)

   --- End of inner exception stack trace ---

   at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.ActivateHost(ResourceManagementWorkflowDefinition workflowDefinition)

   at Microsoft.ResourceManagement.Workflow.Hosting.WorkflowManager.StartWorkflowInstance(Guid workflowInstanceIdentifier, KeyValuePair`2[] additionalParameters)</Data>

  </EventData>

</Event>

 

Resolution:

As we can see from the callstack and from Event log the issue hit the component: Forefront Identity Manager. SharePoint 2010 User Profiles will be handled with FIM and the replication of user profile uses FIM too. The issue itself happens in the .net layer and below: System.Transactions.TransactionException: The partner transaction manager has disabled its support for remote/network transactions.

I’ve found an solution object in the Microsoft KB: https://support.microsoft.com/kb/817064/en-us. The KB was created for Windows Server 2003 but also works for our environment Windows Server 2008.

Action Plan: Enable Network DTC Access

https://technet.microsoft.com/en-us/library/cc753510(WS.10).aspx

Steps to enable network DTC access

1. Click Start, point to Control Panel, and then click Add or Remove Programs.

2. Click Add/Remove Windows Components.

3. Select Application Server, and then click Details.

4. Select Enable network DTC access, and then click OK.

5. Click Next.

6. Click Finish.

If you are running Windows Server 2003 Service Pack 1 (SP1), you must follow these additional steps:

1. Click Start, click Run, type comexp.msc, and then click OK to open Component Services.

2. Expand Component Services, expand Computers, right-click My Computer, and then click Properties.

3. On the MSDTC tab, click Secuity Configuration under Transaction Configuration, click to select the Network DTC Access check box under Secuity Settings, and then click to select the following check boxes under Transaction Manager Communication:

o Allow Inbound

o Allow Outbound

4. On Microsoft Cluster Server (MSCS) clusters, you cannot select Mutual Authentication Required. Therefore, click to select one of the following check boxes:

o Incoming Caller Authentication Required

o No Authentication Required

Note For more information about these options, click the following article number to view the article in the Microsoft Knowledge Base:

899191 (https://support.microsoft.com/kb/899191/ ) New functionality in the Distributed Transaction Coordinator service in Windows Server 2003 Service Pack 1 and in Windows XP Service Pack 2

5. Make sure that the Logon Account is set to NTAUTHORITY\NetworkService.

6. Click OK. A message box explains that the MS DTC Service will be stopped and restarted, and that all dependent services will also be stopped and restarted. Click Yes.

Note If this is a Majority Node Set (MNS) cluster, do not use the MNS resource as the storage device for MS DTC. MS DTC requires a storage resource such as a physical disk.

 

At the end of the story, enable DTC access improve the replication a lot, over 100 Profiles per minute. The Full replication finish successful in less than 12 h, so it’s even better than 20 days.

Now the replication runs much faster… Smile

 

Now the second issue: By default SharePoint user profiles have the following setting configured:

- User Profile notification is enabled

image

Question: What happens during the first Full User Profile replication?

Answer: The replication operation is an update operation on each user profile from source user profile store to target user profile store. In case the source adds new colleagues into the target, then the related user receive an email from target environment, because a new colleague was added… In case of many users, we see a lot of emails…. So it’s very important to disable the 2nd check box: “Notify me when someone adds me as a colleague”, otherwise users will receive an email of colleague was added.

Based on many emails, I would ever suggest to disable all email notification before announcement.

How to get an overview of enabled/disabled User Profile Notifications? (using powershell)

 #Load the SharePoint snap-in
Add-PsSnapin Microsoft.SharePoint.PowerShell;

#Load the SharePoint assemblies
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.Office.Server");
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.Office.Server.UserProfiles");

#Specify the MySite URL
$MySiteUrl = "https://mysite.demo.local";

#Get the server context for the profile manager
$site = Get-SPSite $MySiteUrl;
$ServerContext = Get-SPServiceContext $site;
$UPManager = new-object Microsoft.Office.Server.UserProfiles.UserProfileManager($ServerContext);

#Count variables
$ucount = 0;

write-host ""
write-host "Dump User Email Notification"
write-host "----------------------------"
write-host ""


$enumProfiles = $UPManager.GetEnumerator();
"Total User Profiles available:" + $UPManager.Count
$count=0;

# Email notification flag is handled via Property: "SPS-EmailOptin"
# SharePoint Notification emails is a integeter with 3 bit flags - SPS-EmailOptin (int)
# This field has 3 values one for each email type

# Bit flag:   0 =  enabled –> 0 (integer value)
# Bit flag: 111 = disabled –> 7 (integer value)

write-host ""
write-host ""

$disabled  = 0;
$enabled   = 0;
$undefined = 0;

foreach ($oUser in $enumProfiles)
{
    $count = $count + 1;
    $u = $oUser.Item("Accountname").ToString();
    $notification = $oUser["SPS-EmailOptin"].Value;

    Switch ($notification) 
    {
       "0"  { Write-host "enabled: "  $u; $enabled++;}
       "7"  { Write-host "disabled: " $u; $disabled++;}
    default { Write-host "undefined: "$u; $undefined++}
    }
}
Write-host "-----------------------------------------------"
Write-host "Total"
Write-host "-----"
Write-host "disabled User: "  $disabled
Write-host "enabled User: "   $enabled 
Write-host "undefined User: " $undefined 


#Dispose of site object
$site.Dispose();

I hope it helps.

Regards

Patrick