SharePoint 2010: Unable to start UPA sync service ,stuck at “Starting”

As discussed in my previous blog, its been observed that starting User profile sync job is not an easy task at times. This post describes errors and the solution that I recently worked on. The issue was on a SharePoint 2010 farm which is migrated from 2007 farm.

Issue & Observations
Unable to start User Profile sync service from Central administration site ->system settings->Services on server. The service freezes at "starting" and never start. Below are the information from event viewer and verbose enabled ULS logs.

From ULS

Exception occured while connecting to WCF endpoint: Microsoft.SharePoint.SPEndpointAddressNotFoundException: There are no addresses available for this application. at Microsoft.SharePoint.SPRoundRobinServiceLoadBalancer.BeginOperation() at Microsoft.Office.Server.UserProfiles.MossClientBase`1.get_Channel() at Microsoft.Office.Server.UserProfiles.MossClientBase`1.ExecuteOnChannel(String operationName, CodeBlock codeBlock) at Microsoft.Office.Server.UserProfiles.ProfilePropertyServiceClient.ExecuteOnChannel(String operationName, CodeBlock codeBlock)

UserProfileApplicationProxy.InitializePropertyCache: Microsoft.Office.Server.UserProfiles.UserProfileException: There are no addresses available for this application. ---> Microsoft.SharePoint.SPEndpointAddressNotFoundException: There are no addresses available for this application. at Microsoft.SharePoint.SPRoundRobinServiceLoadBalancer.BeginOperation() at Microsoft.Office.Server.UserProfiles.MossClientBase`1.get_Channel() at Microsoft.Office.Server.UserProfiles.MossClientBase`1.ExecuteOnChannel(String operationName, CodeBlock codeBlock) at

Microsoft.Office.Server.UserProfiles.ProfilePropertyServiceClient.ExecuteOnChannel(String operationName, CodeBlock codeBlock) --- End of inner exception stack trace --- at Microsoft.Office.Server.UserProfiles.ProfilePropertyServiceClient.ExecuteOnChannel(String operationName, CodeBlock codeBlock) at Microsoft.Office.Server.UserProfiles.ProfilePropertyServiceClient.GetProfileProperties() at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.RefreshProperties(Guid applicationID) at Microsoft.Office.Server.Utilities.SPAsyncCache`2.GetValueNow(K key) at Microsoft.Office.Server.Utilities.SPAsyncCache`2.GetValue(K key, Boolean asynchronous) at Microsoft.Office.Server.Administration.UserProfileApplicationProxy.InitializePropertyCache()

From Event viewer

Log Name: Application

Source: Microsoft-SharePoint Products-SharePoint Foundation

Date: 3/4/2011 12:30:23 PM

Event ID: 8077

Task Category: Topology

Level: Error

Keywords:      

User: Domain\sp_farm

Computer: Computer FQDN

Description:

"There are no instances of the User Profile Service started on any server in this farm. Ensure that at least one instance is started on an application server in the farm using the Services on Server page in Central Administration."

Event Xml:

<Event xmlns=" https://schemas.microsoft.com/win/2004/08/events/event" >

  <System>

    <Provider Name="Microsoft-SharePoint Products-SharePoint Foundation" Guid="{6FB7E0CD-52E7-47DD-997A-241563931FC2}" />

    <EventID>8077</EventID>

    <Version>14</Version>

    <Level>2</Level>

    <Task>13</Task>

    <Opcode>0</Opcode>

    <Keywords>0x4000000000000000</Keywords>

    <TimeCreated SystemTime="2011-03-04T18:30:23.718750000Z" />

    <EventRecordID>18688</EventRecordID>

    <Correlation />

    <Execution ProcessID="3356" ThreadID="5948" />

    <Channel>Application</Channel>

    <Computer>Computer name</Computer>

    <Security UserID="S-1-5-21-1834648779-3785927647-1620804944-5169" />

  </System>

  <EventData>

    <Data Name="string0">User Profile Service</Data>

  </EventData>

</Event>

-----------------------------------------------------------------------------------------------------------------------------------

Log Name: Application

Source: Microsoft Resource Management Service

Date: 3/4/2011 1:00:07 PM

Event ID: 0

Task Category: None

Level: Error

Keywords: Classic

User: N/A

Computer: Computer name

Description:

Service cannot be started. System.Data.SqlClient.SqlException: Cannot open database "Sync DB" requested by the login. The login failed.

Login failed for user domain\sp_farm'.

   at Microsoft.ResourceManagement.WindowsHostService.OnStart(String[] args)

   at System.ServiceProcess.ServiceBase.ServiceQueuedMainCallback(Object state)

Event Xml:

<Event xmlns=" https://schemas.microsoft.com/win/2004/08/events/event" >

  <System>

    <Provider Name="Microsoft Resource Management Service" />

    <EventID Qualifiers="0">0</EventID>

    <Level>2</Level>

    <Task>0</Task>

    <Keywords>0x80000000000000</Keywords>

    <TimeCreated SystemTime="2011-03-04T19:00:07.000000000Z" />

    <EventRecordID>18718</EventRecordID>

    <Channel>Application</Channel>

    <Computer>sharepoint</Computer>

    <Security />

  </System>

  <EventData>

    <Data>Service cannot be started. System.Data.SqlClient.SqlException: Cannot open database "Sync DB" requested by the login. The login failed.

Login failed for user 'Domain\sp_farm'.

   at Microsoft.ResourceManagement.WindowsHostService.OnStart(String[] args)

   at System.ServiceProcess.ServiceBase.ServiceQueuedMainCallback(Object state)</Data>

  </EventData>

</Event>

-----------------------------------------------------------------------------------------------------------------------------------

Log Name: Application

Source: Forefront Identity Manager

Date: 3/4/2011 1:00:07 PM

Event ID: 3

Task Category: None

Level: Error

Keywords: Classic

User: N/A

Computer: sharepoint

Description:

.Net SqlClient Data Provider: System.Data.SqlClient.SqlException: Cannot open database "Sync DB" requested by the login. The login failed.

Login failed for user 'domain\sp_farm'.

   at Microsoft.ResourceManagement.Data.Exception.DataAccessExceptionManager.ThrowException(SqlException innerException)

   at Microsoft.ResourceManagement.Data.DatabaseConnection.Open(SqlConnection connection)

   at Microsoft.ResourceManagement.Data.DatabaseConnection.Open(DataStore store)

   at Microsoft.ResourceManagement.Data.TransactionAndConnectionScope..ctor(Boolean createTransaction, IsolationLevel isolationLevel, DataStore dataStore)

   at Microsoft.ResourceManagement.Data.TransactionAndConnectionScope..ctor(Boolean createTransaction)

   at Microsoft.ResourceManagement.Data.DataAccess.RegisterService(String hostName)

   at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.RegisterService(String hostName)

   at Microsoft.ResourceManagement.Workflow.Hosting.HostActivator.Initialize()

   at Microsoft.ResourceManagement.WebServices.ResourceManagementServiceHostFactory.CreateServiceHost(String constructorString, Uri[] baseAddresses)

   at Microsoft.ResourceManagement.WindowsHostService.OnStart(String[] args)

Event Xml:

<Event xmlns=" https://schemas.microsoft.com/win/2004/08/events/event" >

  <System>

    <Provider Name="Forefront Identity Manager" />

    <EventID Qualifiers="0">3</EventID>

    <Level>2</Level>

    <Task>0</Task>

    <Keywords>0x80000000000000</Keywords>

    <TimeCreated SystemTime="2011-03-04T19:00:07.000000000Z" />

    <EventRecordID>18717</EventRecordID>

    <Channel>Application</Channel>

    <Computer>sharepoint</Computer>

    <Security />

  </System>

  <EventData>

    <Data>.Net SqlClient Data Provider: Sys

-----------------------------------------------------------------------------------------------------------------------------------

Log Name: Application

Source: Microsoft.ResourceManagement.ServiceHealthSource

Date: 3/4/2011 1:00:07 PM

Event ID: 2

Task Category: None

Level: Error

Keywords: Classic

User: N/A

Computer: sharepoint

Description:

The Forefront Identity Manager Service could not bind to its endpoints. This failure prevents clients from communicating with the Web services.

A most likely cause for the failure is another service, possibly another instance of Forefront Identity Manager Service, has already bound to the endpoint. Another, less likely cause, is that the account under which the service runs does not have permission to bind to endpoints.

Ensure that no other processes have bound to that endpoint and that the service account has permission to bind endpoints. Further, check the application configuration file to ensure the Forefront Identity Manager Service is binding to the correct endpoints.

Event Xml:

<Event xmlns=" https://schemas.microsoft.com/win/2004/08/events/event" >

  <System>

    <Provider Name="Microsoft.ResourceManagement.ServiceHealthSource" />

    <EventID Qualifiers="0">2</EventID>

    <Level>2</Level>

    <Task>0</Task>

    <Keywords>0x80000000000000</Keywords>

    <TimeCreated SystemTime="2011-03-04T19:00:07.000000000Z" />

    <EventRecordID>18716</EventRecordID>

    <Channel>Application</Channel>

    <Computer>sharepoint</Computer>

    <Security />

  </System>

  <EventData>

    <Data>The Forefront Identity Manager Service could not bind to its endpoints. This failure prevents clients from communicating with the Web services.

A most likely cause for the failure is another service, possibly another instance of Forefront Identity Manager Service, has already bound to the endpoint. Another, less likely cause, is that the account under which the service runs does not have permission to bind to endpoints.

Ensure that no other processes have bound to that endpoint and that the service account has pe

-----------------------------------------------------------------------------------------------------------------------------------

Log Name: Application

Source: Microsoft.ResourceManagement.ServiceHealthSource

Date: 3/4/2011 1:00:07 PM

Event ID: 26

Task Category: None

Level: Error

Keywords: Classic

User: N/A

Computer: sharepoint

Description:

The Forefront Identity Manager Service was not able to initialize a timer necessary for supporting the execution of workflows.

Upon startup, the Forefront Identity Manager Service must initialize and set a timer to support workflow execution. If this timer fails to get created, workflows will not run successfully and there is no recovery other than to stop and start the Forefront Identity Manager Service.

Restart the Forefront Identity Manager Service.

Event Xml:

<Event xmlns=" https://schemas.microsoft.com/win/2004/08/events/event" >

  <System>

    <Provider Name="Microsoft.ResourceManagement.ServiceHealthSource" />

    <EventID Qualifiers="0">26</EventID>

    <Level>2</Level>

    <Task>0</Task>

    <Keywords>0x80000000000000</Keywords>

    <TimeCreated SystemTime="2011-03-04T19:00:07.000000000Z" />

    <EventRecordID>18715</EventRecordID>

    <Channel>Application</Channel>

    <Computer>sharepoint</Computer>

    <Security />

  </System>

  <EventData>

    <Data>The Forefront Identity Manager Service was not able to initialize a timer necessary for supporting the execution of workflows.

Upon startup, the Forefront Identity Manager Service must initialize and set a timer to support workflow execution. If this timer fails to get created, workflows will not run successfully and there is no recovery other than to stop and start the Forefront Identity Manager Service.

Restart the Forefront Identity Manager Service.</Data>

  </EventData>

</Event>

-----------------------------------------------------------------------------------------------------------------------------------

Log Name: Application

Source: Microsoft.ResourceManagement.ServiceHealthSource

Date: 3/4/2011 1:00:06 PM

Event ID: 22

Task Category: None

Level: Error

Keywords: Classic

User: N/A

Computer: sharepoint

Description:

The Forefront Identity Manager Service cannot connect to the SQL Database Server.

The SQL Server could not be contacted. The connection failure may be due to a network failure, firewall configuration error, or other connection issue. Additionally, the SQL Server connection information could be configured incorrectly.

Verify that the SQL Server is reachable from the Forefront Identity Manager Service computer. Ensure that SQL Server is running, that the network connection is active, and that the firewall is configured properly. Last, verify the connection information has been configured properly. This configuration is stored in the Windows Registry.

Event Xml:

<Event xmlns=" https://schemas.microsoft.com/win/2004/08/events/event" >

  <System>

    <Provider Name="Microsoft.ResourceManagement.ServiceHealthSource" />

    <EventID Qualifiers="0">22</EventID>

    <Level>2</Level>

    <Task>0</Task>

    <Keywords>0x80000000000000</Keywords>

    <TimeCreated SystemTime="2011-03-04T19:00:06.000000000Z" />

    <EventRecordID>18712</EventRecordID>

    <Channel>Application</Channel>

    <Computer>sharepoint</Computer>

    <Security />

  </System>

  <EventData>

    <Data>The Forefront Identity Manager Service cannot connect to the SQL Database Server.

The SQL Server could not be contacted. The connection failure may be due to a network failure, firewall configuration error, or other connection issue. Additionally, the SQL Server connection information could be configured incorrectly.

Verify that the SQL Server is reachable from the Forefront Identity Manager Service computer. Ensure that SQL Server is running, that the network connection is active, and that the firewall is configured properly. Last, ver

Cause

In this case, the SSP was not migrated properly. Some of the database information was not properly registered in the SharePoint server registry.

Resolution

- Stop User profile service and User profile sync service on ALL SharePoint servers in the farm

- Clear timer config cache by following the "resolution" documented at https://support.microsoft.com/kb/939308

- Deleted the Pending timer job related to the UPA sync service provisioning using Central administration->Monitoring->check job status

- Confirmed that the security token service has only windows and anonymous enabled ( from IIS manager->SharePoint web services->Security token service->Authentication (under IIS))

- On the SharePoint server access the registry editor and confirm that the information related to FIM ( database servername , sync DB name etc ) are properly populated, if found wrong , back up the registry key and edit the values to reflect the right information.

HKLM\system\currentcontrolset001\services\FIMservice

HKLM\system\currentcontrolset002\services\FIMservice

HKLM\system\currentcontrolset\services\FIMservice

- Delete all FIM related certificates from certificate store ( for account and system) . Make sure that no FIM related certificate is listed in any folders when looked at certificate manager.

- From Central administration->Application management->Manage service application, Make sure the User profile service application and its proxy are started, If found stopped, its always recommended to recreate them taking necessary backups (mysites)

- Start User profile service on the machine where you would like to start the User profile sync service (use central administration console)

- Once the service is started, try starting the User profile sync service from the central administration site