Troubleshooting KMS with Process Monitor (ProcMon)

Dougga here with a short and simple post.  I wanted to share an issue that was resolved using tools from SysInternals.

Recently I helped discover an odd issue with some KMS servers losing their KMS Server Key and becoming a KMS Client.

So where to start troubleshooting… Do you have the right key? Try slmgr.exe (this is the command line tool to manage Site Licenses) to output KMS key information. Collect logs. Collect diagnostic information. Dig through previous cases at Microsoft. Review great blogs such as this one on how to ramp up on KMS. Search on www.bing.com. Nothing looks wrong.

We had one piece of information that turned out to be very helpful. This issue appeared to happen sometime after reboot of the KMS server and once fixed never reverted until it was rebooted.

With that piece of information we decided to leverage a monitoring tool (ProcMon) from SysInternals to trace what is happening on reboot. Specifically we relied on two features in ProcMon (boot logging and process tree view) that I want to show you that helped pin point this issue.  Here is a closer look at the two features I would like to bring out.

Feature 1: Enable Boot logging

You can collect boot traces with ProcMon to collect data during a boot of a machine. From the menu click OPTIONS then ENABLE BOOT LOGGING and ProcMon will monitor on reboot. Since this issue was occurring at reboot that was the path we followed.

clip_image002

Word of caution. ProcMon collects a lot of information. If you use this feature be sure to stop trace as soon as the repro happens to minimize the size of the collected data.

In our case the ProcMon wrote a little over 1GB of data from that short trace.

Feature 2: Process Tree (off of the Tools menu)

Using Process Tree, all the processes that ran during data collection from ProcMon are visible in a format similar to process explorer (another tool). If you have a ProcMon trace, click on TOOLS then PROCESS TREE and you will be happy with what you find. The image below is not from this case, but is here to show where this option is. You can then use this to select a process to filter the ProcMon trace.

clip_image004

Figure 1 Sample image from lab

Data was collected and we opened the Process Monitor trace and using TOOLS / PROCESS TREE (with no filtering) we were able to see a script that ran during startup installing a product key.  It isn’t this easy every time, but this time, the answer was in the Process Tree for us and not much effort beyond that was needed. This image is from the real data collected, but only a small snip of the image is here to maintain privacy.

clip_image006

You can see the VB script running slmgr.vbs with a switch of ipk in the image above. 

Software License Manager, sometimes referred to as SL Manager (Slmgr.vbs), is a script used to configure and retrieve Volume Activation information and IPK is the switch to Install Product Key.  So each time the server was rebooted the product key in the script (which was a client product key) was installed and disabled the ability for the KMS server to issue product keys. 

Once this script was removed the issue was resolved.  It is also of value to note here that it is not necessary to update the product key or re-install it on each reboot.  This turned out to be an oversight in the build process and was corrected.

It can be that simple. You may have to browse a little (or a lot) through the data in ProcMon and it is helpful to know what you need to look for.  In this case we looked for anything related to licensing and product keys.

Summary

I was convinced there must be something misconfigured or a problem with the license – though I could not find anything. It is okay to be on that road, but when you have exhausted all the configuration possibilities and all looks good and there are no similar cases that say “this is it,”  then it is time to change your thinking.  Change it  from something is wrong to it is doing what is supposed to be doing (or what it is being told to do) and start following the processes.

Could we have found this without ProcMon? Sure. However, with ProcMon the script was exposed and easy to find. ProcMon can be used for many issues, but consider using ProcMon when things are not broken so you will learn how to use the tool before you need it.

You can find some AWESOME examples and videos on how to use this tool and others from SysInternals. Just go to http://technet.microsoft.com/en-US/sysinternals and you will find all the Sysinternal Tools, a link to Mark Russinovich’s blog, a link to videos, and much more.

Doug Gabbard “Just keeping it simple”