2. Fast Search Foundation for Exchange Server 2013, Mailbox database Content index State is 'Failed'. AdminNodes are Terminating.

  • Article

Here was the situation: Search function stopped working on Exchange server 2013. Users are not getting any results.  
No Issues noticed with Microsoft Exchange Search service and HostCotrollerService. 

However, Get-MailboxdatabaseCopyStatus reported "Failed" state on all Mailbox databases.

- Restart "HostControllerService".
  Service restarted, but AdminNodes failed to start  with HostCotrollerService.  

- Resetting the Content index on every database do not help, because Admin nodes are not running.
  Index won't be generated if AdminNodes cannot talk to each other.

Following Exception event is logged in event logs,

 An operation attempted against a FAST endpoint Experienced an exception. This Operation may be
retried. Error details:Microsoft.Exchange.Search.Fast.PerformingFastOperationException: An
Exception was received during a FAST operation.--> System.ServiceModel.EndpointNotFoundException: Could
not connect to net.tcp://localhost:3803/Management/ContentTransformation/FlowService.The connect
connection attempt lasted for a time span of 00:00:02.0468713. TCP error code 10061: No Connection could be
made because the Target actively refused it 127.0.0.1:3803.  --->System.Net.Sockets.SocketException:
No connection could be made because the target machine actively refused it 127.0.0.1:3803 at
System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)   at
System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)

   --- End of inner exception stack trace ------------------------------------------------------------------------------

There are many methods to identify the cause, ULS Logging, 

 hostcontrollerservice.exe (0x3B94) - 0x5F0C -HostController  - HostController: Restarting node AdminNode1 on system
Fsis marked for automatic startup - threadId: 3 

hostcontrollerservice.exe (0x3B94) - 0x5F0C - HostController - HostController: Restarting node ContentEngineNode1 on
System Fsis marked for automatic startup - threadId: 3 

hostcontrollerservice.exe (0x3B94)- 0x5F0C - HostController - HostController: Restarting node IndexNode1 on System
Fsis marked for automatic startup - threadId: 3

hostcontrollerservice.exe (0x3B94) - 0x5F0C- HostController - HostController: Restarting node InteractionEngineNode1
on system Fsis marked for automatic startup - threadId:3 

NodeRunner.exe (0x5D94) - 0x58C4 - ULS Init -Completed NodeRunner.exe,Microsoft.Ceres.Diagnostics.Native.dll) 

Each node runner terminates with following error :

NodeRunnerIndexNode1-Fsis (0x1688) - Search Platform Services   - Unexpected exception in node
activator:  System.ArgumentException: An item with the same key has already been added. at
System.Collections.Generic.Dictionary`2.Insert(TKeykey, TValue value, Boolean add) at
Microsoft.Ceres.CoreServices.Node.BootstrapPropertyLoader.AddEnvironmentProperties(IDictionary`2
result) at Microsoft.Ceres.CoreServices.Node.NodeActivator.InitializeBootstrapProperties(IDictionary`2
overlay) at Microsoft.Ceres.CoreServices.Node.NodeActivator.ActivateNode(IDictionary`2 configuration) 

 In general Error would indicate that ‘AdminNodes’ are trying to add a Directory list and it is
duplicated. The duplicated entry is not same as the existing list.
To list Environment variable Properties via PS cmdlet

Get-ChildItem Env:

Look for multiple PSModulepath entries listed in Environment Variable  or PSModulepath entry with trailing space

    PSModulePath=%Unders%\Administrator.Domain\WindowsPowerShell\Modules;c:\Windows\System32\WindowsPowerShell\v1.0\Modules

   PSModulePath = c:\Program Files\MicrosoftSecurity Client\MpProvider

                       [* Note a Trailing Space in " PSmodulepath = " ]

In Our case Second PSModulePath entry was added by installing the Endpoint Protection Agent that
created an additional PSModulePath environment variable that contains a trailing space.

To fix the issue:

1. Install https://support.microsoft.com/kb/2865173
 if one of the following applications are installed on Exchange 2013 / Windows 2012 server 

  • Microsoft System Center 2012 Configuration
    Manager Service Pack 1
  • Microsoft Forefront Endpoint Protection 2010
  • Microsoft System Center Configuration Manager 2007

2. Remove any trailing space from Environment variable “ PSModulePath = "  and restart HostControllerService. 

    PSModulePath= c:\ProgramFiles\MicrosoftSecurity Client\MpProvider  [ <---- Correct Entry ]

3. Fix the first entry by adding MPdirectory and Delete the Second one

PSModulePath=%Unders%\Administrator.Domain\WindowsPowerShell\Modules;c:\Windows\System32\
WindowsPowerShell\v1.0\Modules;c:\ProgramFiles\MicrosoftSecurity Client\MpProvider   

 - Restart HostConterllerService and Microsoft Exchange Search Service.

4. If one Entry for PSModulePath Remove the Trailing space in [ PSModulePath= "  ]  Restart HostControllerService.

 Manju