Exchange Search Fails After Installing Exchange 2007 SP3


 

Symptoms

1. After installing Exchange Server 2007 Service Pack 3, in some cases, you may find that searches in Outlook online mode or OWA may not return any results for any new email that has arrived after installing Exchange 2007 SP3 for any users with mailboxes on the affected database(s).

2.  If you run ResetSearchIndex.ps1 on the indexing for the affected database(s), searches in online mode or OWA may not return any results for any users with mailboxes on the affected database(s).

3. If you run ResetSearchIndex.ps1 on the indexing for the affected database(s), if you check the CatalogData folder(s) that correspond to the affected database(s), you may find that either no .CI files are generated at all or only only a small number .CI files are generated and these files are only a few KB each.

Cause

The ultimate reason why this does not happen with Exchange 2007 Service Pack 2 or earlier, but does occur in some cases after installing Exchange 2007 Service Pack 3 is unknown.  However, the immediate cause for the content indexing files failure to generate is explained in detail below.

1. When the Microsoft .NET Framework 2.0 loads a managed assembly, the managed assembly calls the CryptoAPI function to verify the Authenticode signature on the assembly files to generate publisher evidence for the managed assembly.

2. The CryptoAPI function checks a Certificate Revocation List (CRL) that is available at http://crl.microsoft.com . This action requires an Internet connection.

3. If the Internet connection is blocked, the outgoing HTTP requests may be dropped. Therefore, an error message is not returned. This problem may also occur if DNS cannot resolve http://crl.microsoft.com . This long delay causes the CRL check to time out.

4.  The Micrsoft Search Indexer service (Microsoft.Exchange.Search.ExSearch.exe) is a managed assembly that is loaded by Microsoft .NET Framework 2.0. After installing Exchange 2007 SP3, if the Exchange 2007 SP3 server or DNS cannot resolve http://crl.microsoft.com for any reason, this affects the Microsoft Search Indexer service in such a way that any new email fails to be indexed. If DNS or the Exchange 2007 SP3 server cannot resolve http://crl.microsoft.com for a long enough time, the cached CRLs will expire.

5. MSSearch 3.0  and MSSearch 3.1 is hardcoded to call the “WINTRUST!WinVerifyTrust” to verify modules when they load into process space.

6. When the service starts up, VerifySignature is called when the configuration blob is created via SetProperty().

7. Also, when a search thread is created from the thread pool via WorkItemThread(), VerifySignature is called each time.

8. VerifySignature will then call WinVerifyTrust(). This is the API that actual performs the CRL check.

9. WINTRUST will ultimately use the WINHTTP API to make the internet connection.

Resolution

NOTE: The following is the official resolution from Exchange Development:

1. Add the following entry in the HOSTS file on the Exchange 2007 SP3 server:

127.0.0.1              crl.microsoft.com

2. Save the HOSTS file.

3. Stop the Microsoft Search Indexer Service.

4. Stop the Microsoft Search (Exchange) service.

5. Rename the CatalogData folder(s) for the database having the issue – it should be in the same folder as the database(s) with the issue.  If there is not enough available room, then either save the CatalogData folder to another location or delete the CatalogData folder.

6. Start the Microsoft Search (Exchange) service.

7. Start the Microsoft Search Indexer Service.

8. Wait for Event ID 110 in the application log to signify that creating the new index has finished.

9. Check the CatalogData folder(s) to see if .CI files are now being generated.

10. Check searches in Outlook online mode and OWA to see if search now works.

More Information

1. When MSSearch performs the CRL check, it tries to access crl.microsoft.com, and will wait 15 seconds (tbc) for a response. Furthermore, if there are proxy servers configured (especially bad proxy entries), MSSearch attempts to access the CRL URL via each proxy address. Adding the 127.0.0.1 crl.microsoft.com entry to the HOSTS file forces the server to check against itself, which immediately responds saying that the CRL is not available. This bypasses the timeout period.

2. Click on the following links for more information on WinVerifyTrust and WinHTTP:

WinVerifyTrust Function
http://msdn.microsoft.com/en-us/library/aa388208(VS.85).aspx

About WinHTTP
http://msdn2.microsoft.com/en-us/library/aa382925.aspx

Using WinHTTP tools
http://msdn2.microsoft.com/en-us/library/aa384084.aspx

Bob Want, Senior Support Escalation Engineer, Enterprise Communications Services, Microsoft


Comments (1)

  1. Frank Tanner says:

    This did not work for us, despite getting the 110 entries in our Event Logs.  Any other ideas?

Skip to main content