High CPU Utilization by Wspsrv.exe Process


1. Introduction


 


The Firewall service is ISA Server's   main user mode service; it runs with the Network Account credentials and under the context of the process wspsrv.exe. The CPU utilization of this process might vary according to the server load and other external elements, for example: antivirus, third party plug ins, etc.


 



 


The goal of this post is to help you to understand that in some scenarios where the wspsrv.exe process is consuming too much CPU the issue might be an external component that is not necessarily related to ISA Server itself.  To exemplify that this post describes a real scenario where the wspsrv.exe process was using almost 100% CPU for approximately 3 to 4 minutes and then backs it up to normal utilization. During this time, since ISA Server was too busy it couldn’t answer in a timely manner to the requests coming from the internal network. As result of that clients were receiving "time- out" errors when trying to browsing Internet.  This 3 to 4 minutes interval was enough to cause help desk calls and chaos to the end users.


 


2. Data Gathering


 


For this type of scenario you can user Performance Monitor to capture the main set of objects. Use the guidelines created by Jim Harrison at: http://blogs.isaserver.org/shinder/2008/10/21/tips-for-isa-firewall-performance-assessment. Also, install DebugDiag on ISA and get a full user mode dump for the process wspsrv.exe when the issue is happening.


 



 


 


3. Analyzing the Data


 


One of the most valuable elements for this type of scenario is the dump file of the wspsrv.exe process and even without having the Microsoft Internal symbols you will still be able to review some key data using the public symbols. Download the debugging tools and configure the symbols according to the Debugging Tools and Symbols: Getting Started Page in Windows Hardware Developer Central.


After you get the dump you can use windbg to analyze this dump file. Here are some conclusions from this dump file:


 


1. By using the command !runaway 7 we were able to find out the top threads that were using resources:


 


0:000> !runaway 7


User Mode Time


  Thread       Time


 253:98c       0 days 0:00:59.296 << Thread using more CPU


 263:1104      0 days 0:00:56.828


  70:1ad8      0 days 0:00:51.156



 Kernel Mode Time


  Thread       Time


 253:98c       0 days 0:00:35.093


 263:1104      0 days 0:00:33.265


  13:13dc      0 days 0:00:32.218



 Elapsed Time


  Thread       Time


   0:243c      0 days 20:41:59.956


   1:23e4      0 days 20:41:59.643


   2:1ec0      0 days 20:41:59.456


 


...removed the complete list since it was very long.


 



Note:  The parameter 7 comes from 1 (user time) + 2 (kernel time) + 4 (time elapsed since thread start)


 


2. We switched to the thread 253 by using the command below:


 


0:000> ~253s


eax=00002000 ebx=46d66810 ecx=00000193 edx=00003800 esi=7fffffff edi=ffffffff


eip=7c82860c esp=4b90cea0 ebp=4b90cedc iopl=0         nv up ei ng nz ac pe cy


cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000297


ntdll!KiFastSystemCallRet:


 


3. From there we got the call stack for this thread (253), which was the one that was consuming more user mode space within wspsrv.exe process, causing high CPU consumption:


 


0:253> kbnL


 # ChildEBP RetAddr  Args to Child             


00 4b90ce9c 7c827d29 71b21af5 0001f0ac 00000001 ntdll!KiFastSystemCallRet


01 4b90cea0 71b21af5 0001f0ac 00000001 4b90cec8 ntdll!NtWaitForSingleObject+0xc


02 4b90cedc 71b2c507 0001f0ac 000268f4 00000000 mswsock!SockWaitForSingleObject+0x19d


03 4b90cf54 71c02fee 000268f4 4b90cf8c 00000001 mswsock!WSPRecv+0x203


04 4b90cf9c 37cb47da 000268f4 4b90d018 00000001 ws2_32!recv+0x83


WARNING: Stack unwind information not available. Following frames may be wrong.


05 4b90cfc0 37cb4880 4b90d018 00000001 00000001 IcapClientApi!TmDbgger::write+0xaba


06 00000000 00000000 00000000 00000000 00000000 IcapClientApi!TmDbgger::write+0xb60


 


3. We start to analyze patterns and the first 6 threads had a similar stack where this third party component was showing up there. We got more information about this component using the command lmvm:


 


0:253> lmvm IcapClientApi


start    end        module name


37cb0000 37cca000   IcapClientApi C (export symbols)       IcapClientApi.dll


    Loaded symbol image file: IcapClientApi.dll


    Image path: D:\Program Files\Microsoft ISA Server\IcapClientApi.dll


    Image name: IcapClientApi.dll


    Timestamp:        Fri May 26 17:58:55 2006 (4476D15F)


    CheckSum:         00000000


    ImageSize:        0001A000


    File version:     5.0.0.1164


    Product version:  5.0.0.1164


    File flags:       0 (Mask 3F)


    File OS:          40004 NT Win32


    File type:        1.0 App


    File date:        00000000.00000000


    Translations:     0409.04b0


    CompanyName:      XXXXXXXXXX


    ProductName:      XXXXXXXXXX


    InternalName:     XXXXXXXXXX


    OriginalFilename: XXXXXXXXXX


    ProductVersion:   5.0.0.1164


    FileVersion:      5.0.0.1164


    PrivateBuild:     Build 1164


    SpecialBuild:     1164


    FileDescription:  XXXXXXXXXX


    LegalCopyright:   XXXXXXXXXX.


    LegalTrademarks:  XXXXXXXXXX


    Comments:         XXXXXXXXXX


 



Note:  As you can see this is a very old version of this component.


 


4. Conclusion


 


In this case wspsrv.exe was externalizing the problem since it was the process that was consuming more CPU, however within the process we have many threads running and not always those threads are exclusively using calls from the ISA Server components itself. This is a classic example of external components that can make wspsrv.exe use more CPU than what it should.  In this particular scenario the problem was fixed by the vendor of the third party application by updating the software to the latest version at that time.


 


5. References


 


Here are some good articles for troubleshooting high CPU utilization scenario (regardless of having ISA Server):


 


Troubleshooting High CPU Utilization


Troubleshooting High CPU Hangs with a Debugger


Isolating the Threads Consuming High CPU


How to track down High CPU in User Mode Applications - A live debug!


 


 


Authors


Yuri Diogenes


Sr Security Support Escalation Engineer


Microsoft CSS Forefront Edge Team


 


Congyong Su


Sr Support Engineer


Security Support Team,
Microsoft Global Technical Support Center, Asia Pacific & Greater China


 


Technical Reviewer


Vic Singh Shahid


Security Escalation Engineer


Microsoft CSS Forefront Edge Team


 

Comments (0)

Skip to main content