Troubleshooting Exchange 2010 Search


Since Exchange servers are now storing an ever increasing amount of data, it is imperative that people can quickly search and retrieve the content from their mailbox.  This may have been possible without search capabilities in the past with a 20 MB mailbox quota (remember those days ?!?), but as quotas move to double gigabyte figures effective search is a hard requirement.

Exchange Search is a different animal compared to the full-text indexing that was available in Exchange 2003 and has numerous performance, feature and scalability changes.  The search methodology is one of always up to date rather than Exchange 2003’s crawl model.  Unlike Exchange 2003, content indexing is enabled by default on all Exchange 2010 mailbox servers so there is minimal configuration or setup apart from installation of the Office 2010 Filter Pack.  Throttling also ensures that Exchange 2007/2010 search does not impact server performance.

Users running Outlook in online mode will leverage the Exchange search service, as will those on OWA.  Administrators performing discovery searches will also be using Exchange Search.  Thus it is clearly an important aspect of the Exchange environment, and it is also part of the Best Copy Selection process which is considered when Active Manager is deciding which copy of a mailbox database to activate in a Database Availability Group.

 

What are some of the issues that can arise, and what can be done about them?  Issues can include the following:

  • Slow search performance

  • Unable to index certain documents

  • Unable to active database copies in a DAG

 

Slow Search Performance

Exchange 2010 SP1 RU4 addressed some important Exchange Search issues.  For example:  It takes a long time to return results when you perform an Advanced Find search on a mailbox by using Outlook in online mode in an Exchange Server 2010 SP1 environment.  In addition, there have been multiple other fixes for Exchange Search.  At this time only Exchange 20103 SP3 is supported, and when this post was updated Exchange 2010 SP3 RU9 was the latest available update. 

If the Exchange search service is not able to index documents quickly, check how the server is performing and pay particular attention to disk performance.

 

Exchange Unable To Index Certain Documents

iFilters are required so that Exchange Search can index content in various file formats.  In Exchange 2007 and Exchange 2010 RTM you must manually register the IFilters by editing the registry or running the provided script.  Exchange 2010 SP1 helps us out as Exchange setup will automatically register the IFilters from the Office Filter Pack with Exchange Search.   

If you want other file formats to be indexed, e.g. PDF then an IFilter for that type needs to be installed and registered.  Note that additional work items are required, which are discussed in detail in this post

 

Unable to Activate Database Copies

You will not be able to use the Exchange Management Console to activate a mailbox database copy when its Content Index is failed.  Exchange Management Shell (EMS) can be used in conjunction with the SkipClientExperienceChecks switch.

Note that the various database copy states can be found here in the Get-MailboxDatabaseCopyStatus cmdlet section.

 

Troubleshoot The Exchange Search Catalog – Troubleshoot.ps1

What else can we do to troubleshoot the Content Index for issues where searches are stalling or it is deadlocked?  Exchange provides a range of scripts that are present in the \Scripts directory.  By Default this is C:\Program Files\Microsoft\Exchange Server\V14\Scripts for Exchange 2010.  In the Exchange Management Shell there is a variable called $EXScripts which abstracts the physical location.  It also makes it very easy to run scripts!

Exchange 2010 SP1 introduced the Content Index Troubleshooter Script – Troubleshoot-CI.ps1.  This script can monitor and perform troubleshooting on a server’s CI catalogs and can detect and resolve symptoms such as

  • Deadlock: Exchange Search deadlocks waiting on threads from MSSearch
  • Corruption: One or more search indices are corrupted
  • Stall: Similar to deadlock in that the indices are not getting updated
  • Backlog: The Search catalog is backlogged resulting in missing index searches

This Troubleshoot-CS.ps1 script is designed to run on  a single server. 

 

Reset The Exchange Search Catalog – ResetSearchIndex.PS1

If the above trouble shooter is unable to correct the issue, it may be necessary to completely reset on one machine or potentially on all servers that have a copy of the mailbox database in question. There are two ways to do this, either:

  1. Running the script ResetSearchIndex.ps1 script
  2. Manually stopping the search services, removing the catalog folder and restarting the search services

If using the script to reset the index for one database

ResetSearchIndex.ps1 [-force] <dbname>

Or if all instances are to be reset:

ResetSearchIndex.ps1 [-force] –all

This will stop the Exchange Search service, delete the catalog files for the given database and then start the Exchange Search service  after which the catalog is rebuilt.

As noted above in the second option, it is possible to manually perform the same actions.  If manually stopping the  Exchange Search service (MsExchangeSearch), removing the catalog folder and starting the Exchange Search service it is critical that all of these steps are performed.  A common mistake is to delete the catalog folder with Exchange Search service running.  This means that the search service believes that it has indexed the content and that the indexes exist on disk.  However they do not as the  admin has just removed them.  For this reason the Exchange Search service must be stopped and then started backup so that Exchange knows that there is no catalog and it must be rebuilt. 

 

Resetting All Instances Of a Database’s Search Catalog

Normally the above process works well, but from time to time I encounter catalogs that have persistent issues where a little more invasive action is required.  In these situations the CI index is set to be disabled for the database, old copies of the catalog files are removed and then the generation process can start afresh.  This is shown in the following steps, but is time consuming and impacts users who run Outlook in online mode, OWA users and administrators performing E-Discovery tasks. 

  • Disable the CI for the database.   

Set-MailboxDatabase -IndexEnabled $False
  • Stop Exchange Search service on all copies

  • Remove the catalog folder on all copies

  • Re-enable CI for the database

Set-MailboxDatabase -IndexEnabled $True
  • Start the Search service on the active server to rebuild the CI on the active copy.  Look for the EventID in the app log to confirm the that the rebuild has started and then completed. 

  • Then start search service on passive copies

  • Then update the catalog on passive

Update-Mailboxdatabasecopy –CatalogOnly
  

Cheers,

Rhoderick

This post was originally located on MSPFE. Moved here for editing. 

Comments (4)

  1. David Schryer says:

    Hi ... the manual process above for "Resetting All Instances Of a Database’s Search Catalog" . One of the steps, "Stop Exchange Search service on all copies", I did this and the MSEXCHANGESEARCH service has restarted on all the servers that have copies of that database ... and the catalog folders have been re-created. I presume this is because in Services it is set to Automatic ???
    Also, the active copy is in the process of rebuilding the indexes right now. Can I 1) stop the services again on those servers right now, 2) set the service to manual, 3) and delete the catalog folders ?????
    PS - I was on your blog this week. We had an MS Exchange consultant here at the time. He noticed your name and mentioned he's a buddy of your's ... Jon Ninaber.

  2. Hi David,

    Yes Jon and I were discussing this last night 🙂

    I suspect that you have SCOM or the ilk, and that is running the troubleshoot-CI script. Is that the case, and did you put the boxes into maintenance mode?

    I also assume you are doing this on Exchange 2010?

    Cheers,
    Rhoderick

    1. David Schryer says:

      - Yes, using Exchange 2010 SP3 ... UR12 I think.
      - Huh, didn't know about this CI Troubleshooter script ! Just read about it ... neat ... will try using it. As you gathered, no, we are not running it.
      - No SCOM either (Jon can tell you that story). No, I didn't put the boxes in mtce mode; I presume you're talking about using the StartDagServerMaintenance.ps1 script ?
      - So, were you trying to say the CI script caused the Exchange Search Service to start up again ?

      1. The maintenance mode comment was related to SCOM, but not applicable if you don't have that.

        Jon will call you later today 🙂 On phone with him right now.

        Cheers
        Rhoderick

Skip to main content