Identifying 32-bit Kernel Memory Issues

I'm a Premier Field Engineer (PFE) and I go onsite with customers on a regular basis to conduct Health Checks. This is my first blog posting on TechNet, but I figured it would be on something important versus "Hello World". ;-) 

More and more I am seeing customers who are not aware of kernel memory issues on the 32-bit Windows architecture. If you are running 32-bit Windows 2000 or 32-bit Windows 2003, then check the kernel memory. Lack of kernel memory can lead to system-wide hangs which seem unexplainable, so this is a serious issue. Always generally use and/or recommend 64-bit or Windows Server 2008 server to avoid these issues (Windows Server 2008 and Windows Vista have automatically adjusting kernel memory pool sizes). This issue and other performance issues are addressed in the Vital Signs workshop (written by Shane Creamer), which can be delivered by my team, Premier Field Engineering (PFE) - just contact your Technical Account Manager (TAM) if you are interested in this course.

Here is a kernel memory chart for Windows 2003 Server:

Check your 32-bit servers to see if they are within 80% of the memory pool sizes for Pool Paged Memory and Pool non-Paged memory according to the chart above. The Free Page Table Entries (PTEs) listed in the chart above is the starting amount based on the specification of the hardware and boot.ini switches. Here are the performance counters to monitor:

Memory\Free System Page Table Entries

Memory\Pool Nonpaged Bytes

Memory\Pool Paged Bytes

For example, if I have a server that is a Windows 2003 SP1 (SP1 is needed to accurately see the Free PTE data in performance monitor) with 4GB of memory and /3GB switch turned on, then Free System PTE’s start off at 15,000 (according to the chart). If the value gets below 5,000, then the system could hang temporarily. This system has a Paged Pool memory maximum of 258MB and Non-Paged Pool maximum memory of 154MBs. If the respective “Memory\Pool Paged Bytes” and “Memory\Pool Nonpaged Bytes” counter values come within 80% of these maximum pool sizes, then the system could hang temporarily – in this case if the values go over 206MB’s and 123MB’s respectively, then it’s a critical issue.

Fix for Win2003 SP1 systems with /3GB and low on PTE’s: If the system is low on PTE’s, running Windows 2003, and using /3GB switch, then consider using the /USERVA switch to give back some of the memory to the kernel. Note, this only works for Free System PTE issues. For more information on the USERVA switch, go to:

How to use the /userva switch with the /3GB switch to tune the User-mode space to a value between 2 GB and 3 GB

https://support.microsoft.com/kb/316739

Lack of Paged Pool or non-Paged Pool Memory: If the system is low on Paged Pool or non-Paged pool memory, then first consider opening a support case with Microsoft to address this. Alternatively, you can use a free and public tool called Poolmon.exe to see what DLL’s are using kernel memory (see the article below). Most kernel memory leaks can be tracked back to a usermode process. To identify which user mode process is responsible, reboot the system (so you start off with a clean system), start a performance monitor log intending to run for a week or more capturing the Memory and Process objects, then analyze the perfmon log looking for memory leaks and/or handle leaks in one or more of the processes.

How to Use Memory Pool Monitor (Poolmon.exe) to Troubleshoot Kernel Mode Memory Leaks

https://support.microsoft.com/kb/177415

In summary, always consider 64-bit and always keep an eye on kernel memory.

In addition, I and a few of my colleages wrote a tool that can help identify these and many other performance issues by analyzing performance counter logs. The tool is located at https://www.codeplex.com/PAL. Codeplex.com is Microsoft's open source web site.