Exchange 2013 Search not Working (3) :NodeRunner.exe process Terminated with an "OutOfMemoryException"

  • Article

Background:

Microsoft Exchange Host Controller Service starts 4 worker processes, each named NodeRunner (Admin Nodes) . The Individual functionality of Each NodeRunner process is conferred by  configuration. The NodeRunner.exe  which activates single Admin node in a process of its own. NodeRunner.exe is a standalone process, it derives some of the operating properties from  NodeRunner.exe.config the application configuration file during the start of Admin Nodes.

* if you are familiar with SPS, you probably know how to Limit the Memory allocated to NodeRunner.exe Process using the application configuration file. ( Do Not implement SPS resolutions/ workaround in Exchange  server 2013).

The default setting for minimum memory requirement for NodeRunner.exe  is 0. NodeRunner process can consume and change its memory requirements dynamically based on the requirement and available memory. However, you can set the Upper limit and restrict the amount of memory that node runner can access using the parameter "memoryLimitMegabytes" in the NodeRunner.exe.config application Configuration file. ( **** Lessons from SPS...  ). If you limit the memory usage by NodeRunner.exe and if we can't allocate the memory for NodeRunner.exe  operation, we fail with an OutOfMemoryException. Check for the following events if you are experiencing this issue and for a resolution.

[ * limiting memory allocation for NodeRunner.exe in Exchange server 2013 using this method is not supported ].

Symptoms :

A - Content Index stats on all mailbox databases are "FailedAndSuspended"

B - ULS Logs :    

 NodeRunnerIndexNode1-Fsis (0x59D4) 0x6390 Search Search Component  
 IndexRouter[AA51A323-AFF5-490F-BEC8-40300AB578BC12]: Caught exception when preparing generation GID[38]: (IndexNode1-00000000-0000-0000-0000-000000000000-AA51A323-AFF5-490F-BEC8-40300AB578BC12.Single: bad allocation): Microsoft.Ceres.SearchCore.FastServer.FastServerException:
IndexNode1-00000000-0000-0000-0000-000000000000-AA51A323-AFF5-490F-BEC8-40300AB578BC12.Single: bad allocation   
at Microsoft.Ceres.SearchCore.FastServer.Indexer.PrepareGeneration()    
at Microsoft.Ceres.SearchCore.Indexes.FastServerIndex.GenerationBuilder.FlushGeneration()    
at Microsoft.Ceres.SearchCore.Indexes.AbstractIndex.AbstractProtectedGenerationBuilder.FlushGenerationInternal()    
at Microsoft.Ceres.SearchCore.Indexes.AbstractIndex.AbstractBuilder.PrepareCloseGeneration()    
at Microsoft.Ceres.SearchCor... 

c - Event Logs:

The NodeRunner.exe process throws an exception when trying to allocate more memory than the limited memory.                                                                                                                    

ID:1026
Level:Error
Source:.NETRuntime
Message: Application: NodeRunner.exe
Framework Version:v4.0.30319
Description: The process was terminated due to an unhandled exception
Exception Info: System. OutOfMemoryException
Stack:
atMicrosoft.Ceres.CoreServices.Framework.Container.ComponentManager.FailingInvoker()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object,Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object,
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback,System.Object)
at System.Threading.ThreadHelper.ThreadStart() 

ID:1000
Source: Application
Error Message:
Faulting application name: NodeRunner.exe,
version: 16.0.847.30,
time stamp: 0x52fef8d5
Faulting module name: ntdll.dll,
version: 6.2.9200.16579,
time stamp: 0x51637f77
Exception code: 0xc00000fd
Fault offset:
0x000000000001b45c
Faulting process id:0x12b0
Faulting application start time:0x01cf4e43dce6bead
Faulting application path: C:\Program Files\Microsoft\ExchangeServer\V15\Bin\Search\Ceres\Runtime\1.0\NodeRunner.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll Report Id:29ffb9d1-ba37-11e3-9427-00155d0a730c
Faulting package full name: Faulting package-relative application ID: 

ID:  1026
Level:Error
Source: .NET Runtime
Application: NodeRunner.exe
Framework Version:v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info:
System.AccessViolationException
Stack:  
atMicrosoft.Ceres.SearchCore.FastServer.Indexer.PrepareGeneration()
atMicrosoft.Ceres.SearchCore.Indexes.FastServerIndex.GenerationBuilder.FlushGeneration()
atMicrosoft.Ceres.SearchCore.Indexes.FastServerIndex.GenerationBuilder.PublishGeneration()
atMicrosoft.Ceres.SearchCore.Indexes.AbstractIndex.AbstractBuilder.CloseGeneration() atMicrosoft.Ceres.SearchCore.Journal.RegistrationHandler.CloseGenerationToUsers(Microsoft.Ceres.SearchCore.Journal.RecoveryState)
atMicrosoft.Ceres.SearchCore.Journal.Journal.RecoveryEndGeneration(Microsoft.Ceres.SearchCore.Services.GenerationId)
at Microsoft.Ceres.SearchCore.Journal.RegistrationHandler.RedoJournalEntry(Microsoft.Ceres.SearchCore.Services.Journal.IJournalEntry, UInt16,Microsoft.Ceres.SearchCore.Journal.RecoveryState)
at Microsoft.Ceres.SearchCore.Journal.Journal.RecoverEntry(Microsoft.Ceres.SearchCore.Services.Journal.IJournalEntry, UInt16)
at Microsoft.Ceres.SearchCore.Journal.DoubleBuffered.DoubleBufferJournal.RecoveryScan(Microsoft.Ceres.SearchCore.Services.Journal.CheckpointId, Microsoft.Ceres.SearchCore.Journal.JournalEntryRecovered, Microsoft.Ceres.SearchCore.Journal.ResumeCheckpointing, System.Collections.Generic.IDictionary`2<UInt16,Microsoft.Ceres.Common.Utils.IFactory`1<Microsoft.Ceres.SearchCore.Services.Journal.IJournalEntry>>) atMicrosoft.Ceres.SearchCore.Journal.Journal.DoLogicalRecovery()
atMicrosoft.Ceres.SearchCore.Journal.Journal.EnsureFullRecovery()
atMicrosoft.Ceres.SearchCore.Journal.JournalComponent.DoReadying()
atSystem.Threading.Tasks.Task.Execute()
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object,Boolean)
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef) atSystem.Threading.Tasks.Task.ExecuteEntry(Boolean)
at System.Threading.ThreadPoolWorkQueue.Dispatch()

 

ID: 1000
Level: Error
Message:
Faulting application name: NodeRunner.exe, version: 16.0.847.30, time stamp: 0x52fef8d5
Faulting module name: MSVCR110.dll, version: 11.0.51106.1, time stamp: 0x5098826e
Exception code: 0xc0000005 Fault offset: 0x000000000003c920
Faulting process id: 0x6670
Faulting application start time: 0x01cf4e437e35974d
Faulting application path: C:\Program Files\Microsoft\Exchange Server\V15\Bin\Search\Ceres\Runtime\1.0\NodeRunner.exe
Faulting module path: C:\Windows\SYSTEM32\MSVCR110.dll Report Id: cc6806ff-ba36-11e3-9427-00155d0a730c Faulting package full name: Faulting package-relative application ID: 

Fix NodeRunner Exception issue :

Events in ULS logs and event logs indicate NodeRunners.exe are terminating with exception, because of  OutOfMemory. Either the server is outofmemory or .Net CLR is forcing memory allocation limitation to the NodeRunners.exe process. NodeRunner tried to allocate more memory and terminates with an exception .
Navigate to following location , edit the file  

C:\Program Files\Microsoft Office Servers\15.0\Search\Runtime\1.0\noderunner.exe.config

Check for Key and the value   <nodeRunnerSettings memoryLimitMegabytes="0" />.

if it is set to any value than 0, change it to 0 and restart  "Microsoft Exchange Host Controller Service"

 

 - Manju