Using the /3GB Switch with SBS 2003

I’d like to address a question that was asked during last week's MVP Summit, and periodically comes up on the various SBS mailing lists and newsgroups:  What is the definitive statement regarding the use of the /3GB switch on SBS servers?  I’m addressing SBS specifically, but these recommendations generalize to any DC running Exchange.   

Our official answer is contained in KB 815372 (emphasis mine):

For Exchange Server computers which are at the same time Active Directory Domain Controllers or Global Catalog servers we also do not recommend setting the /3GB switch in boot.ini. We recommend having dedicated Active Directory Domain Controllers or Global Catalog servers.

Because Microsoft Small Business Server installs as an integrated domain controller and Exchange server, we also do not recommend setting the /3GB switch when you run Exchange Server 2003 in a Small Business Server environment. However, the other concepts and settings that are described in this article apply equally to Exchange Server 2003 in a Microsoft Small Business Server environment

So, if you're not experiencing any problems with your SBS server, and performance is otherwise acceptable, most people can stop reading here.  The more intrepid, fearless admins and consultants will want to know why we don't recommend using the /3GB switch except for an extremely rare case detailed below.

More detailed explanations:

Mike Lee of the Exchange team has an excellent article describing the inner workings of Exchange memory management available at https://msexchangeteam.com/archive/2005/12/07/415733.aspx.

If you really want to get in to the nth level of detail on memory management, Mark Russinovich's Windows Internals book covers the topic in excrutiating detail in chapter 7.

Why this is important

The reasoning behind the KB article above is that you can actually starve your Active Directory for memory if Exchange (or potentially, other apps) experience heavy stress.  The underlying architecture of Exchange relies heavily on the GC for lookups, which can be quite intensive when, for instance, the server is processing a very large amount of mail (think mailing lists, mailmerges, mailbox-level backups).  In rare instances, you can get in to a scenario where AD is out of resources, even though Exchange still happily has enough user mode memory.  So, the long and the short of it is that, by using the /3GB switch on a DC, you are unlikely on a production SBS server to ever make full use of the additional gig of RAM.

The rare exception

So, let me address that extremely rare scenario where an SBS server may actually benefit from the /3GB switch.  I hesitate to do this because I know that human nature dictates that, even if I say that this will affect one in 10,000 SBS installs, there is an inclination to think "Well, maybe one of my SBS servers could be that one unlucky one".  Here's how to definetively determine that.

First, read https://msexchangeteam.com/archive/2005/12/28/416551.aspx.  The article is titled "Large Security Tokens and Kernel Memory Exhaustion", but it also applies more generally to PTE exhaustion in general.

It's then time to monitor memory and PTE usage on the server.  The reference for this is here: 

https://www.microsoft.com/technet/prodtechnol/exchange/guides/TrblshtE2k3Perf/7a44b064-8872-4edf-aac7-36b2a17f662a.mspx?mfr=true

Specifically, you'll want to monitor these performance counters:

Kernel memory space Performance Monitor counter Performance Monitor triggers with default boot.ini Performance Monitor triggers with boot.ini options /3GB and /USERVA = 3030

Paged Pool

MemoryPool Paged Bytes

“Warning” when the Pool Paged Bytes counter exceeds 300 MB

“Critical” when the Pool Paged Bytes counter exceeds 320MB

“Warning” when the Pool Paged Bytes counter exceeds 200 MB

“Critical” when the Pool Paged Bytes counter exceeds 220 MB

Nonpaged Pool

MemoryPool Nonpaged Bytes

“Warning” when the Pool Nonpaged Bytes counter exceeds 200 MB

“Critical” when the Pool Nonpaged Bytes counter exceeds 220 MB

“Warning” when the Pool Nonpaged Bytes counter exceeds 100 MB

“Critical” when the Pool Nonpaged Bytes counter exceeds 110 MB

System PTEs

MemoryFree System Page Table Entries *

“Warning” when Free System Page Table Entries is less than 8,000

“Critical” when Free System Page Table Entries is less than 5,000

“Warning” when the Free System Page Table Entries is less than 8000

“Critical” when the Free System Page Table Entries is less than 5,000

In most cases, the problem you are most likely to see first will be PTE exhaustion.  The error you will see logged in Event Viewer is:

Event Type: Error
Event Source: MSExchangeIS
Event Category: Content Engine
Event ID: 12800
Date: Date
Time: Time
User: N/A
Computer: Computer_Name
Description:
Message processing failed because there is not enough available memory (8007000E-82000387).

If you see this error, the next step is to verify that you are truly running out of memory and/or PTEs.  To do this, configure Perfmon to capture data on the counters above.  The link to the white paper above describes in great detail how to configure PerfMon and the various symptoms and resolutions to these problems.

In a nutshell

The vast majority of SBS servers will not benefit from use of the /3GB switch.  In some instances, performance may actually degrade.  In my straw poll of support escalation engineers on site here, we have never seen an SBS go over 2 GB of Exchange memory usage due to legitimate use (i.e. users actually sending large amounts of email to stress the store).  In virtually every instance we have seen here of the Store using massive amounts of memory, the root cause turned out to be:

  • by far the most common cause of memory problems is an improperly sized pagefile.  Administrators often think that adding an 8gb pagefile will help them in terms of available memory.  What actually happens is that this puts an increased strain on the server due to the number of page file handles we must track.  https://support.microsoft.com/default.aspx?scid=kb;EN-US;237740 covers how to correctly set pagefile size based on your RAM.
  • a corrupt database
  • Exchange add-ins
  • client misconfiguration (e.g. a client-side Outlook add-in that was auto-forwarding itself massive CAD files)
  • administrators who implement the various Exchange KBs without taking into account that Exchange is on a DC (for example, modifying heapdecommitfreeblockthreshold, systempages, etc.) 

In each of these cases, using the /3GB switch would actually make the problem worse, since Exchange would continue to exhaust system resources.