Case in the Corner Series: Spiking CPU

[Today’s post comes to us courtesy of Damian Leibaschoff from Commercial Technical Support]

This was an interesting situation that developed during April of 2009; this is what happened behind the scenes.

Around April 20 (as early as April 17) we started getting reports of 2008 servers with Exchange 2007 where an SVCHOST was spiking one of the CPU cores, our main objective was to find a server that was reproducing this behavior. Lucky for me, my own server was experiencing the same behavior. By looking at the PID (process identifier) of the SVCHOST and getting a list of services being hosted there by using tasklist /svc (SVCHost is a shared service process for hosting multiple services) I was quickly able to determine that the actual service causing the spike was WUAUCLT (Windows Update Client service). Many readers may remember similar issues with this service especially on XP clients, they may also remember that the AU client was partially at fault and that MSI had a lot to do with this issue, you will see how this is relevant (again) in the next paragraph…

I decided to take a process monitor capture while the issue was happening to see if I could identify any obvious activity causing the spike in the CPU. (if you never used the Windows Sysinternals tool set, you need go there now and have a look at the great tools available <link>). Since the issue was with CPU utilization, I used the stack trace view to see what process/function call was actually triggering the activity, and we finally got down to MSI.DLL being the trigger as shown below. Next, I jumped to the registry key that was being repeatedly accessed (also shown in the Procmon capture), and found it had to do with Exchange 2007 anti-spam updates. My SBS 2008 server had the automatic Anti-Spam updates enabled, product of having selected the option to install Forefront Security for Exchange during the setup.

We now had enough information to start digging into what was triggering the issue. We knew it was related to updates, most specifically to Exchange Anti-Spam updates (block list). We took a peek at Windows Server Update Services only to find a very large number of non-expired, superseding, updates (over 40). Once we manually removed them the issue subsided .

I contacted the Exchange Sustained Engineering team and they brought in their update people. Turns out that they had been working on an issue with their updates in WSUS, and by accident released over 40 updates at the same time; this was enough to cause the AU client to go into this state on a low powered server like mine. The Exchange team immediately expired the extra updates that had been published to WSUS and the issue was resolved on the backend within the hour, with most folks never realizing they had an issue (it would take WSUS to be synchronized again for the change to be effective for the affected server).

A quick disclaimer here in case you see this scenario presented in a different venue: We have shared this scenario with Mark Russinovich as he collects these types of successful experiences with the Windows Sysinternals toolset for use in presentations and training.

Hope you enjoyed this, look for our next post pretty soon.

Damian