Connectivity verifier memory issues caused by optional update KB971737

Recently Microsoft Customer Service and Support have seen cases with the firewall process (wspsrv.exe) in ISA Server 2006 “leaking” memory until it reaches the 32-bit process limits or face too much memory fragmentation to continue working properly.

On an ISA Server with high loads, memory fragmentation can always cause issues and that’s why we recommend setting the HeapDeCommitFreeBlockThreshold registrykey on high load systems. This is to improve how the heap manager decommits the memory and to minimize the heap fragmentation. This is described in detail in http://support.microsoft.com/kb/315407.

However even though setting this key, can help to improve the overall stability and performance of your server, it will only be a temporary workaround if you’re facing a leak inside the process. In a nutshell, a memory leak is memory allocated in a process that is not freed at a later point in time. If enough memory allocations like this happen, the process may run out of memory.

For more details about memory leaks I suggest that you have a look at some of Mark Russinovich’s blog posts like “Pushing the Limits of Windows: Virtual Memory” or recorded sessions around “Pushing the Limits of Windows” which can be found here: http://technet.microsoft.com/en-us/sysinternals/bb963887

A leak inside the ISA firewall process (wspsrv.exe) can be caused by many things. E.g. we’ve seen issues with 3rd party Add-Ins, drivers, and unexpected behavior in our application. Identifying the cause of a leak can be a very complex troubleshooting task and in most cases you’ll need to collect Performance Monitor logs to see how the allocated memory is behaving. E.g. how much does it increase by per day, does it ever decrease etc. If the memory allocations keep increasing over time, then this may be a sign of a leak and it may be necessary to collect LeakTrack memory dumps using DebugDiag and analyze the dumps for leaks.

In order to avoid you having to do this, we’re doing our best to identify and fix the leaks we find and to include these fixes in regular software updates.

As mentioned at the beginning, I want to let you know about one issue we identified recently. The issue we identified is unfortunately introduced in an optional update of the winhttp.dll for Windows 2003 x86 (http://support.microsoft.com/kb/971737). This update implements EAP Authentication in Windows HTTP services. There’s no need to install this update on ISA Server 2006, because we don’t use this feature of the winhttp.dll.

The leak occurs when you configure HTTP connectivity verifiers in ISA that verify HTTP connections to web servers that use Windows Integrated (NTLM/Kerberos) authentication.

To prevent running into this issue, please either uninstall the optional update from KB971737 or review your connectivity verifiers, and check whether any of them verifies against an authenticating HTTP server. If so, disable or move to an anonymous server. Please note that HTTP connectivity verifiers may also be used in Web Farms.

If you’re not sure if the update is installed on your ISA server, then please open your Control Panel -> Add Remove Programs -> Show Updates and review the list of installed updates

clip_image002

Or open Accessories -> System Tools -> System Information -> Loaded Modules

Verify the version number of winhttp. The version number of the leaking dll is 5.2.3790.4584.

clip_image004

Summary

Please don’t install KB971737 on computers running ISA Server. If it’s installed, please uninstall the fix if you’re using Connectivity verifier on Webserver with Integrated Authentication.

Author

Philipp Sand
Microsoft CSS Forefront Security Edge Team

Technical Reviewer

Lars Bentzen

Microsoft CSS Forefront Security Edge Team