Windows Server 2003 Rapidly Approaches End-Of-Life, Watch Out For Performance Bottlenecks
Summary: Still running Windows Server 2003? Besides making the well-advised recommendation of moving to Windows Server 2012, Frank Plawetzki , a Senior Microsoft Premier Field Engineer based in Germany, runs us through how to monitor servers remotely on a large scale, what to look for from a performance standpoint, and digs into some details on how to properly deal with page and non-paged pool memory to insure these older systems don’t get dinged by memory-hungry applications. Enjoy.
Most IT administrators should be aware that Windows Server 2003 will reach its end-of-life by July 2015. Since Windows Server 2003 has run for so long now and 64-bit/x64-systems are so common nowadays, I did not expect to have to deal with paged-pool and non-paged-pool issues again, but I was wrong.
There are still many customers, who run Windows Server 2003 in production today and many will continue to do so until end-of life of Windows Server 2003 or close to that date for various reasons. While this date is still two years away, current systems should at least be checked to insure things are running properly. In my example here-in, we’ll check that we don’t have 3rd-party applications depleting paged-pool or non-paged-pool memory since many of those servers are running as 32-bit (x86) installations.
Although Windows Server 2003 is running stably, some 3rd-party applications today consume large amounts of this paged-pool or non-paged-pool memory which is very limited on x86 systems, Problems start to arise when a x86-System has the /3GB switch set in boot.ini, in order to increase available memory for the application on that server which in fact decreases the amount of available paged-pool or non-paged-pool memory.
Problems may start to arise when 3rd-party applications have not been developed for the x86 architecture or weren’t optimized for the limited paged- and non-paged-pool memory available on x86-systems.
In my case, one 3rd-party application was consuming more than 50MB of the total available paged-pool memory of ~150MB. The result was that the affected server stopped working as soon as paged-pool-memory consumption was increased only a little bit more during normal work cycles. As soon as paged-pool (or non-paged-pool) memory is depleted, only a reboot can make the server work again.
This article gives a rough overview, what the /3GB switch does: Exchange memory use and the /3GB switch.
The general architecture of those resources is described here: Pushing the Limits of Windows: Paged and Nonpaged Pool. As you can see from the tables in the latter article, an x86-System in theory should have enough paged-pool or non-paged-pool memory available, i.e. something around 650MB for Windows Server 2003.
Practical operation often shows different values: When you take a x86-system with 4GB RAM as an example, you will usually see a maximum of available paged-pool memory of around 360 MB and a maximum of non-paged-pool memory of around 260 MB. Remember, those limits depend on the used hardware, certain BIOS and software settings, and are calculated during boot time and are fixed until the server gets rebooted.
Sometimes the available paged-pool maximum will even be much less, like in this example:
How to check what amount of paged- or non-paged pool memory is available on a system
The amount of paged- and non-paged-pool memory available on a system (a.k.a. paged- and non-paged MAX) is determined from the hardware, hardware settings like hot-add memory usage, and software settings in the boot.ini and in the registry for memory manager.
You should definitely check your systems for the amount of available paged- and non-paged-pool memory, in order to be sure they are running healthy and can bear a load peak. The link above explains how you can check the practical limits as in the screenshot above, without attaching a debugger for live debugging to the kernel of the running system.
If your servers are equally configured and use the same hardware, those paged- and non-paged-pool memory limits should be identical on your servers. You can then compare the paged- and non-paged-pool memory max with the amount of paged- and non-paged pool currently consumed on your server.
The performance counters for the current consumption that you want to look at are:
-
- “\Memory\Pool Paged Bytes”
- “\Memory\Pool Nonpaged Bytes”
You can also quick-check the current consumption via task manager, even on systems where no symbols are deployed.
As soon as the paged- or non-paged-pool memory consumption on your server exceeds 80% of the available memory of this pool, the usage is critically high. At this point, you need to identify which driver or component is consuming what amount of memory. The inbox tool POOLMON helps with that and shows the real-time consumption. For more information on POOLMON see this article: How to use Memory Pool Monitor (Poolmon.exe) to troubleshoot kernel mode memory leaks.
Also, with those values, you can create a baseline to get an idea what a healthy memory pattern of a server in your organization looks like during normal production.
How to monitor performance counters on your servers large scale and remotely
You most likely will quickly come to a point where you want to set up a baseline and create a set of performance counters that you want to monitor on all your servers. Windows ships an inbox tool, which makes deploying performance counters on remote systems very easy. The tool is called LOGMAN and can also help query the performance counters which are available on a remote system or extract certain values from a .blg file which is used to save the monitored performance counters to.
As an example, the following command will create a performance counter set on <server> and create a new file as soon as the size limit of 1024 is hit:
Logman create -s <Server> counter MS_Baseline -f bin -max 1 -si 1 -v mmddhhmm -o c:\test\MS_Baseline -cf C:\test\MS_Baseline_Template.txt -cnf 1024
The MS_Baseline_Template.txt file in our example would contain those two entries.
“\Memory\Pool Paged Bytes”
“\Memory\Pool Nonpaged Bytes”
For simplicity, we are only using two performance counters in this example. There are certainly more performance counters you need to look at in order to check whether your server is healthy or not.
After you created the performance counter set on the remote server, you need to start it:
Logman start -s <Server> MS_Baseline.
LOGMAN is a bit tricky regarding the various switches and combination of switches that you can use, so you have to check the syntax carefully and do some tests until the tool creates the pool of performance counters you want, in order to monitor in the exact way you want it to happen, e.g. for the sampling intervals or size of the .blg files. The LOGMAN tool uses performance monitor on the remote system which creates a .blg file that you can open in performance monitor on you client workstation and check the various performance counters.
Summary
If you really still need to run Windows Server 2003, especially the x86-version for some more months, you should at least ensure that the servers are running healthy. You can only achieve that by creating a performance baseline and checking various performance counters.
Microsoft Support or Premier Field engineering is happy to help with questions that you may have around which performance values are important and at what values they should be, according to your workload.
This article wants to point your attention to the fact, that even in times when everybody is running x64-systems, you still have to watch your 32-bit systems carefully for performance. Another (better) way to solve those kind of issues is to migrate to a modern 64-bit operating system like Windows Server 2012.
Written by Frank Plawetzki, Senior Microsoft Premier Field Engineer; Posted by Frank Battiston, MSPFE Editor