This blog provides some guidance on how to optimize Exchange 2000/2003 on Windows OS for paged and non-paged pool issues.
Event log might show
Event ID 2020 Event Type: Error Event Source: Srv Event ID: 2020 Description: The server was unable to allocate from the system paged pool because the pool was empty. Event ID 2019 Event Type: Error Event Source: Srv Event ID: 2019 Description: The server was unable to allocate from the system NonPaged pool because the pool was empty.
Other symptoms of pool exhaustion on the system include application or process hangs, out of resource errors reported by drivers or applications, the server becomes slow or refuses additional requests and connections, or all of the above!
When a machine boots up, the Memory Manager creates two dynamically sized memory pools that kernel-mode components use to allocate system memory. These two pools are known as the Paged Pool and NonPaged Pool. Pool memory is allocated statically during Windows startup. Available pool memory depends on several factors to include boot switches such as /USERVA and /3GB, registry settings, and physical RAM.
Pool memory is not the amount of RAM on the system. It is a segment of the virtual memory or address space that Windows reserves on boot. These pools are finite because x86 OS can only address 2^32==4GB. By default, Windows uses 2GB for applications and 2GB for kernel.
These pools are used by either the kernel directly, indirectly by its support of various structures due to application requests on the system (CreateFile for example), or drivers installed on the system for their memory allocations made via the kernel pool allocation functions.
NonPaged means that this memory when allocated will not be paged to disk and thus resident at all times, which is an important feature for drivers. Paged conversely can be paged out to disk. In the end though, all this memory is allocated through a common set of functions, most common is ExAllocatePoolWithTag.
Page Tables are built for each process address space. The Page Table maps logical virtual addresses for a process to physical memory locations. System Page Table Entries (PTEs) are used to map system pages such as I/O space, Kernel stacks, and memory descriptor lists. Tuning the memory by using the USERVA switch in conjunction with the /3GB switch can often stave off PTE depletion issues.
A boot.ini switch used for more precise tuning of user and kernel virtual memory space in the Windows Server 2003. Use this switch in conjunction with /3GB switch in the Boot.ini file to tune reduce the User-mode space, allowing the difference to be returned to Kernel mode. Standard configuration is to use /USERVA=3030. However there might be several situations where you need to allow for more PTEs to become available. On an Exchange server, the value of USERVA should not be lower than 2970 without consulting Windows Performance team.
A boot.ini switch that allocates 1 GB to the kernel and 3 GB to the User-mode space. Using this switch reduces the memory available for Nonpaged Pool, Paged Pool, & System Page Table Entries (PTEs).
A Boot.ini switch. When more than 4GB of physical memory is used on the system, the process of paging memory to the disk increases dramatically, and performance may be negatively impacted. The Windows memory managers use PAE to provide more physical memory to the operating system. This reduces the need to swap the memory in and out of the page file and results in increased performance. All the memory management and allocation of the PAE memory is handled by the memory manager independently of the running programs. It is now a best practice to allow the PAE kernel to load on an Exchange 2003 server.
A pool tag is a four-byte character that is associated with a dynamically allocated chunk of pool memory. The tag is specified by a driver when it allocates the memory. The routine ExAllocatePoolWithTag is called to allocate pool memory. Pool tags are useful for identifying which drivers are allocating nonpaged and paged pool memory.
By the Numbers
Review the chart at the bottom of this document to identify the default boot up of Paged Pool, NonPaged Pool, and Free System PTE as it applies to your system configuration.
A typical large-scale Exchange 2003 server should use no more than 200 MB of paged pool memory under typical conditions. Paged pool memory use of more than 220 MB requires immediate attention. Under standard load, there should be approximately 50 MB of available paged pool memory. If you have less than 30 megabytes free, you should take immediate steps to reduce the load on the server.
In general, a system should always have around 10,000 free System PTE’s. If the value gets below 5,000, then the system could hang temporarily.
64-bit resolves this, right?
Not quite. While 64-bit programs can take advantage of the 16-TB tuning model (8 terabytes User and 8 terabytes Kernel) that 64-bit OS has, 32-bit programs still use the 4-GB tuning model (2 GB User and 2 GB Kernel). This means that 32-bit processes that run on 64-bit versions of Windows run in a 4-GB tuning model. So by running 32-bit programs on 64-bit OS, you could still run into limitations.
Troubleshooting Memory Depletion
Here are some basic troubleshooting steps to identifying the problem with Pool Memory depletion on Exchange 2000/2003 server.
- Is the system showing any symptoms of pool memory depletion (event log, hangs, etc)?
- Check system performance using Task Manager and performance monitor (Memory Counters: Available Mbytes, Free System Page Table Entries, Pool Nonpaged Bytes, & Pool Paged Bytes)
- Use Poolmon to capture the system’s memory allocations of paged and non-paged kernel pools. See the section below on using Poolmon. (poolmon –n –b –g)
- Review the poolmon output and identify any tags that are consuming a high amount of Paged or NonPaged memory. Compare the output to your baseline or another system.
- To search for a driver that uses a specific tag, use the findstr /m /l /s xxxx *.sys command from the cmd prompt (where xxxx is the tag name).
- If you suspect a tag or driver, research that driver (KB, Internet, etc) for any known issues and latest hotfixes.
- If needed, review the pool information using debugging tools (see debugging section below)
Common Causes of Pool Depletion
- Problem with a driver or application taking more than needed (leak)
- With Exchange 2000/2003, having more than 4 GB of physical RAM (Each byte of physical RAM that is installed in a server requires some kernel memory to address and manage it. The more RAM that is installed, the more kernel address space must be reserved for it. Address space may be borrowed from paged pool memory to satisfy this demand.)
- Having a Public Folder store mounted on a mailbox server that has many mailboxes. If mailboxes are connecting to the PF store, the additional connections require additional system resource to be consumed.
- User token size (http://support.microsoft.com/kb/912376)
NOTE: Size of the PF store does not matter – it is the connections to the store
If experiencing symptoms, collect the following data:
- From a remote location, take a performance monitor capture of the system
- Capture EXBPA, MPSReports, and Poolmon data from the server
- If needed, capture a user dump of the system (see debugging section below)
Common Best Practices
The following are some best practices for managing paged and nonpaged pool issues on Exchange 2000/2003.
NOTE: Before performing these steps, run Poolmon on the server at difference intervals to keep track of system changes
- Run Exchange Best Practice Analyzer (EXBPA) and resolve all of the issues that it identifies. EXBPA identifies many of the common configuration settings for improving memory.
- Check for proper Boot.ini configuration (/3GB, /USERVA, /basevideo, /PAE, etc)
- Within the System BIOS, disable Hot-Add memory -and- set the DynamicMemory registry key to a value of 1 (http://technet.microsoft.com/en-us/library/aa996104.aspx)
- Disable Windows Scalable Networking Pack and TCP Chimney on the server
- Disable Checksum Offloading by setting the DisableTaskOffload regkey to a value of 1 (http://support.microsoft.com/kb/904946)
- Check the Network card settings and make sure that “Number of Receive Descriptors” is set to default (NIC Properties > Advanced)
- Disable or remove unwanted Applications or drivers from the system
- Distribute heavy-connection users evenly across multiple servers. Heavy-connection users are likely to be those who have multiple computers or devices and those who are mobile users. This includes mobile, telephony, archiving, and mail client connections.
- A Public Folder store can impact paged and nonpaged memory. Consider using a dedicated PF server for large environments.
- Identify user TOKEN (TOKE) size (http://support.microsoft.com/kb/912376). If this is high, refer to KB 912376 to reduce token size or distribute users with large tokens across multiple servers.
- Apply relevant hotfixes & updated drivers to the system
- Restrict unauthorized clients and applications from accessing Exchange server. This can be done by isolating the subnet between clients and servers and by setting the Outlook connection range value on the server (also see KB 288894)
- Using terminal services in application server mode on an Exchange 2000/2003 server can drastically impact performance on the server. Do not use terminal services in application mode!
- Reduce File Cache Lifetime to 5 minutes (300 decimal) (http://support.microsoft.com/kb/267551)
- Set POP3/IMAP Protocol Logging to 0 (http://support.microsoft.com/kb/299778)
- Implement MaxPercentPoolThreads and AdditionalPoolThreadsPerProc (http://technet.microsoft.com/en-us/library/aa997700.aspx)
- Move the EXIFS free list (Flst) and auxiliary free list (AuxL) to paged pool memory
- Set “AuxFreeListInPagedPool” = REG_DWORD 0x00000001
- Set “FreeListInPagedPool” = REG_DWORD 0x00000001
- Set EnableAggressiveMemoryUsage for IIS to 1 (http://support.microsoft.com/kb/820129)
NOTE: There are other settings and adjustments that can be made to the system to potentially improve Paged and NonPaged Pool performance. Those adjustments should only be made after completing the above steps and consulting with Microsoft Services.
Poolmon is Windows Support Tool that displays data that the operating system collects about memory allocations from the system paged and non-paged kernel pools, and the memory pools used for Terminal Services sessions. The data is grouped by pool allocation tag.
NOTE: Pool tagging is permanently enabled on Windows Server 2003. However, in earlier OS versions (including XP), you must use Gflags.exe to enable pool tagging.
1. Open a command prompt and view the available options: Poolmon /? -s Display session pool -n [Logfile] Take a pool snapshot (Logfile maybe specified, default is poolsnap.log) -c [LocalTagFile] Display driver information using LocalTagFile -g [PoolTagFile] Display driver information using PoolTagFile -itag Include the tag -xtag Exclude the tag -e Display totals -t Sort by tags -a Sort by allocs -u | -b Sort by Bytes -f Sort by free -d Sort by diff -m Sort by each -l Highlight -p First turns on nonpaged, second turns on paged -( | -) Increase parenthesis -r Print memory summary information 2. Copy the Pooltag.txt file from the Windows Debugging tools directory to the system running poolmon. This will allow you to map the driver to the tag. 3. Next run Poolmon -n –b -g Review the poolsnap.log. The tags should be sorted by byte usage.
- Pool Tags List - http://alter.org.ua/en/docs/win/pooltag/index1.php
- Introduction to Pool Tags - http://blogs.technet.com/askperf/archive/2008/04/11/an-introduction-to-pool-tags.aspx
- Memory Management - Understanding Pool Resources - http://blogs.technet.com/askperf/archive/2007/03/07/memory-management-understanding-pool-resources.aspx
- Who's Using the Pool? - http://www.microsoft.com/whdc/driver/tips/PoolMem.mspx
- Understanding Pool Consumption - http://blogs.msdn.com/ntdebugging/archive/2006/12/18/Understanding-Pool-Consumption-and-Event-ID_3A00_--2020-or-2019.aspx
- How to find pool tags that are used by third-party drivers - http://support.microsoft.com/kb/298102
- Exchange Server 2003 Processor and Memory Scalability - http://technet.microsoft.com/en-us/library/aa996184.aspx
- CPU and memory scalability for Exchange Server 2003 and for Exchange 2000 Server - http://support.microsoft.com/kb/827281
- How to monitor and troubleshoot the use of paged pool memory in Exchange Server 2003 or in Exchange 2000 Server - http://support.microsoft.com/kb/912376
- Ruling Out Memory-Bound Problems - http://technet.microsoft.com/en-us/library/aa998384.aspx
- KB 177415 How to use Memory Pool Monitor (Poolmon.exe) to troubleshoot kernel mode memory leaks
- Poolmon on Technet - http://technet.microsoft.com/pt-pt/library/cc737099.aspx