How to troubleshoot high cpu in the System process
I noticed that we didn’t have a general troubleshooter for the System Process.
Since I work mostly on the Server space, the samples are more server centric but it will work the same way on the clients.
APPLIES TO
• Windows 2000
• Windows 2000 Server
• Windows XP
• Windows Server 2003
• Windows Vista
• Windows Server 2008
• Windows 7
• Windows Server 2008 R2
Problem description:
-----------------------
The System process consumes a significant amount of CPU time (30% to 100%) time on a Windows based system.
Note: The PID for System process is always 4.
During this time, your system might feel very sluggish.
What is the System process?
The System process is a kernel mode process which runs system threads (the kernel and loaded device drivers) taking care of network i/o and/or disk i/o, et al.
Cause(s):
-------------------
1. 3rd party applications such as:
A. Antivirus on the remote machines keeps on scanning the server.
B. Firewall programs.
2. Issues with network interface card (NIC) drivers/firmware.
3. Scripts and batch files that place excessive load on the serer, possibly run from the remote machines.
4. Windows Security Auditing configured aggressively as to Audit, in particular audits on success.
Tools:
------
1. Task Manager (built-in to the O.S.)
2. Process Explorer https://technet.microsoft.com/en-us/sysinternals/bb896653.aspx
Make sure to enable the Microsoft public symbol to get stack information.
3. Perfmon (built-in to the O.S.)
For those that are new to perfmon, the Performance Analysis of Logs (PAL) Tool might make your life a little easier.
4. KernRate (Windows 2000 or Windows Server 2003, no longer created for Windows Server 2008 and Windows Server 2008 R2)
Kernrate Viewer https://www.microsoft.com/downloads/details.aspx?familyid=d6e95259-8d9d-4c22-89c4-fad382eddcd1&displaylang=en
5. Windows Performance Toolkit (XPerf is what replaced KernRate for Windows Server 2008 and Windows Server 2008 R2) https://msdn.microsoft.com/en-us/performance/default.aspx
6. Network trace
Microsoft Network Monitor 3.3 https://www.microsoft.com/downloads/details.aspx?FamilyID=983b941d-06cb-4658-b7f6-3088333d062f&displaylang=en
7. Complete memory dump.
972110 How to generate a kernel dump file or a complete memory dump file in Windows Server 2003
https://support.microsoft.com/?id=972110
969028 How to generate a kernel or a complete memory dump file in Windows Server 2008
https://support.microsoft.com/?id=969028
Troubleshooting:
-------------------
Proactive action plan (to try preventing the problem):
-------------------------------------------------------------
1. Update the kernel filter drivers for the antivirus and/or firewall program(s).
If you are not able to, go thru:
816071 How to temporarily deactivate the kernel mode filter driver in Windows
https://support.microsoft.com/?id=816071
2. Update the NIC driver/firmware
3. Update the NIC teaming software/driver/firmware, if there are no updates, break the NIC teaming since we at Microsoft do not support it. It’s supported by the OEM vendor.
Reactive action plan (while the problem is occurring):
-------------------------------------------------------------
1. Unplug the network cable
If the problem goes away, the issue is coming over the network.
Restart in Safe Mode with Networking, if the problem still occurs:
Get a network trace and see where the traffic is coming from. Use the experts for top chatter.
Focus on one of the clients and see what that client is doing.
2. MSConfig.exe (built-in, except for on Windows 2000)
Note: This will not prevent the antivirus/firewall drivers from loading in memory.
Disable all 3rd party services and all startup group items.
Restart
If the problem doesn’t reproduce, then enable the startup group items
Restart
If the problem doesn’t reproduce, then enable the half the 3rd party services
Restart
And go thru the process of elimination.
3. Otherwise let’s grab the Process Explorer, Perfmon, KernRate or XPerf, Network Trace and complete memory dump.
MORE INFORMATION
The third-party products that this blog discusses are manufactured by companies that are independent of Microsoft. Microsoft makes no warranty, implied or otherwise, about the performance or reliability of these products.
I highly recommend reading thru Mark Russinovich’s blog:
The Case of the System Process CPU Spikes
https://blogs.technet.com/markrussinovich/archive/2008/04/07/3031251.aspx
Additionally the two books that he and David Solomon co-wrote have good information regarding the System Process:
Windows Internals 4th Edition https://www.microsoft.com/learning/en/us/Book.aspx?ID=6710&locale=en-us
Windows Internals 5th Edition https://www.microsoft.com/learning/en/us/book.aspx?ID=12069&locale=en-us