Large Message Processing in Exchange, Part 1: Prevention and Planning


EDIT 10/14/2009: We updated the section that recommends settings for environments that need to support messages larger than default 10MB message size.

Already an expert on large messages? You'll still want to check out the updated "Best Practices" section below to harden your server against possible outages.

Overview

Although email is not always the best way to share files, the method is frequently used. As an administrator, you probably have to allow messages to be sent with attachments. Sometimes these attachments are relatively large. But you also have to balance this business requirement with making sure that your server hardware does not become overly utilized or that some users are denied service while others are processing super large messages.

In Customer Support Services we see a lot of critical server unresponsive type issues caused by someone trying to attach a really large file, say perhaps someone trying to share a DVD home video .ISO with their friends and coworkers.

Although we've attempted to harden Exchange out of the box, there are still a few things that you should consider doing to further limit the possibility of something like this happening.

Reasonable limits

It all begins with "reasonable" size limits. So the question everyone asks is "what is reasonable?" The answer is that your mileage will certainly vary with your hardware, drive space, number of users, availability requirements, etc. For Exchange 2007, the typical size limits we see are in the range of 10-30 MB. There are business cases where larger messages are required. However, not all hardware is equally equipped to handle it. Think about the following cases:

Maximum allowed msg size

Maximum allowed number of recipients per message

Typical messages per second

Possible worst case peak scenario

10MB

5000

10 msg/sec

500GB/sec

30MB

5000

10 msg/sec

1.5TB/sec

100MB

5000

10 msg/sec

5TB/sec

Do you have that kind of space available? Do you want to pay for it? Can your disks keep up with that amount of I/O? Hub transports? Mailbox databases? Transaction log files? Will your archiving and retention systems be able to keep up?

Ok, so obviously this is an extreme case - every message sent is not going to be as large as the maximum limit, and not every message is going to be sent to as many recipients as possible. But you can easily see that message size is an important piece of planning for server sizing and availability. You must plan for both the average case, but also consider the peak/worst case. Even if the same chart were done with averages instead of worst cases, you can imagine that even if just a few large messages make it out per day, the average size goes up considerably. Also, your server has to be able to handle those few per day in a reasonable amount of time. Remember that it is possible to provide limits for one set of users and still allow the occasional large message to be sent by more important folks, say company executives.

Furthermore, if the possibility exists that someone can send a DVD .ISO to everyone in the company, you can bet that it will happen at some point. Will your server(s) handle it gracefully? Or will business stop because of this incident?

Our capacity and planning tools will help you make the decision, although they are designed around the average case - don't forget to at least consider that raising message sizes will not only affect the average case, it will also raise the peak load case. For more discussion on this, see this post.

Back Pressure

Exchange 2007 introduces a concept within Transport servers called Back Pressure. You can read all about it here. Suffice it to say, if your server becomes too busy, it will stop accepting new messages, and allow itself time to gracefully recover. It does this to protect itself from the extreme cases.

In short, Back Pressure is Exchange 2007's way of monitoring available disk space, memory and uncommitted messages. When any of those resources exceed their corresponding thresholds for a sustained period the HUB server stops accepting anonymous submissions (medium threshold) or all submissions (high threshold). For example:

Event Type: Warning
Event Source: MSExchangeTransport
Event Category: ResourceManager
Event ID: 15004
Description:
Resource pressure increased from Medium to High.

Resource utilization of the following resources exceed the normal level:

Version buckets = 213 [High] [Normal=80 Medium=120 High=200]

Back pressure caused the following components to be disabled:
Inbound mail submission from Hub Transport servers
Inbound mail submission from the Internet
Mail submission from the Pickup directory
Mail submission from the Replay directory
Mail submission from Mailbox servers
Mail delivery to remote domains

With large messages, you have the possibility that a database transaction to commit the message into the database will take some time to complete. During that time, the database is tracking the commit with what is called version buckets or version store. So with large messages, you can guess that version buckets will often be the measure of how the mail queue database is keeping up. A few seconds of back pressure a few times per day is fine, but if your server(s) spend a lot of time in back pressure, then there's the possibility that other messages aren't being processed in a timely fashion.

Let's also take a look at the graph below (Message Size (MB) versus Version buckets). This graph charts the size of the message sent along with the corresponding Version Buckets Allocated while the message was transmitted. The yellow line shows the default medium threshold for Back pressure resource monitoring of 120, and the red line is the default High Threshold of 200. As the message increases, the version buckets allocated continue to increase as well. When the message size reaches approximately 100MB the version buckets allocated have done beyond the medium threshold value of 120. This will cause inbound anonymous SMTP traffic to temporarily be suspended (Internal Hub to Hub and Mailbox submissions will not be affected. This chart shows that Exchange 2007 has the ability for transport to support a single message as large as 150MB without causing internal mail flow disruption - but again, remember, this is an isolated single message going to a single recipient.

Please note that these measurements were taken in the absence of processing a high rate of other smaller messages and it was performed on a server configuration that was I/O constrained (single disk).

For more information about the impact of specific messages on hardware, check out this study.

And for information about the impact of large messages on your mailbox stores, see this post.

Size Limits

Although, you've probably seen this, let's quickly review the common places that size limits are set and discuss possible ways to harden your server(s):

  • Global/organizational size limits - these settings control the absolute maximum size message that can be submitted to store with Outlook, and accepted by the categorizer throughout the organization. Although it is "global" in scope, there can be exceptions to this limit.
  • Global/organizational MaxRecipientEnvelopeLimit - this controls how many recipients can be on a single message. It is probably a good idea to consider lowering this from the default value of 5000 once Exchange 2003 is no longer in your organization as Exchange 2003 calculates this after distribution groups are expanded. But Exchange 2007 and later will consider a DG as a single recipient. The nice thing here is that you can setup the larger DGs for broadcast emailing and restrict who is allowed to send to them. You could also lower the value if say you only had 2000 mailboxes in your organization - no need to keep the default value.
  • Connector limits (send, receive, foreign, routing group, AD site link) - these are useful for what can pass through a particular connector. For example, you may want to allow user SMTP submissions to be 15 MB, but only 10 MB for application servers, and only 5 MB for anonymous "Internet" email. You should always have a reasonable size limit on your Internet facing receive connector(s).

Yawn. You've seen this before, right? Well what about these limits you may not remember?

  • OWA - By default, OWA users can upload attachments of up to 30MB for submission. This value can be modified.
  • DAV clients like older versions of Entourage - KB 935848 talks about the issue. You can set the MaxRequestEntityAllowed value on the w3svc/1/ROOT/Exchange key in the metabase to a value similar to the maximum size posting you want to allow from DAV clients.
  • Public folders - Although public folder posts do not go through transport, they can be equally dangerous to the server. If someone posts a gigantic message, it could cause the public folder store to consume system resources and run out of space. In addition, if the folder is replicated, the problem can spread throughout the organization (replication messages are system messages and bypass most other checks). You can set limits for replication messages, but large single posts will bypass this limit. Contrary to popular belief, this setting is not an upper limit on replication message size - it's more like a lower limit. When we have multiple changes to pack up, we will keep packing each change into the same message until we reach at least the replication message size limit. After that, we send the message, and start packing additional changes into the next replication message. Let's say your replication message size limit is 300k. If you post 500 1k items in a folder, 300 of them will be packed into the first replication message, and then 200 will be packed into the next one and sent. If you post 500 1MB items, each individual item exceeds the message size limit, so that just means that we won't pack any additional changes into each replication message - each item will be sent in its own 1 MB replication message. If you post a 1 GB item, the same applies - you get a 1 GB replication message. We don't split up a large item between replication messages. The only way to control this is to set reasonable item size limits, particularly on replicated folders. You can set these at the database and folder level.

Best Practices

An ounce of prevention is worth a pound of cure. So here are the best practices we recommend to protect your server(s) from large messages that might cause outages.

  • Install SP1 RU8. This rollup update contains an extremely important fix that should not be missed. KB 960775 is the fix that you need, particularly if you allow Outlook 2003 clients prior to SP3 to connect to your server. These clients will not ask for the maximum limits before synching and submitting a large message to the server. This can easily cause transaction log file growth and performance problems on the Mailbox server. But, worse, the store-generated DSN messages are then submitted to Transport and the problem can spread. This fix eliminates the possibility of Hub servers being affected. Regardless of this fix, it may still be a best practice to update your clients to SP3 and block legacy (unsupported) clients to limit the damage that can happen on the Mailbox server.
  • Run ExBPA. Although BPA does not know what's reasonable for your organization, it can make sure that at least size limits are in place. ExBPA can check all of your servers quickly.
  • Set reasonable size limits for your organization based on planning. See above section for commonly missed size limits. You can use the detail output from the BPA to make certain the limits are where you think that they are.
  • Particularly if you're supporting anything larger than the default 10MB message size, make sure that you've updated your edgetransport.exe.config file to the latest guidance for your version of Exchange. At the time of this publishing, the Exchange 2007 guidance when running the latest service pack is as follows:
    • The ESE cache size should be 512MB on any server with more than 4GB of RAM -
  • <add key="DatabaseMaxCacheSize" value="536870912" />

    For servers with 8GB of RAM or more, particularly if they are dedicated Hub role with transport dumpster enabled, you can set the value as high as 1GB:

    <add key="DatabaseMaxCacheSize" value="1073741824" />

    • The version bucket thresholds should be as follows -

    <add key="VersionBucketsHighThreshold" value="200" />
    <add key="VersionBucketsMediumThreshold" value="120" />
    <add key="VersionBucketsNormalThreshold" value="80" />

    • The checkpoint depth should be approximately half of the DatabaseMaxCacheSize -

    <add key="DatabaseCheckPointDepthMax" value="268435456" />

    Or, with  DatabaseMaxCacheSize set to 1GB -

    <add key="DatabaseCheckPointDepthMax" value="536870912" />

    • The Queue Database Logging Buffer Size should be as follows -

    <add key="QueueDatabaseLoggingBufferSize" value=" 5242880" />

  • Consider hardening and isolating Internet-facing receive connectors such that spam processing and virus scanning processes for inbound "unclean" message streams are not impacting the rest of mail flow. Set reasonable receive connector limits. This obviously transcends the large message discussion, but this is especially true if you allow larger messages.
  • Make sure that proper exclusions are set for file-based Antivirus software and that temporary locations are also located on drives with adequate space and speed. Temporary files can be created while converting large messages. Scanning these temporary files can cause problems - use proper Exchange Antivirus for protecting the messages and file-level scanning to protect the server(s).

Conclusion

Message size limits will protect your servers and make sure they stay happily running, but there is not any "one-size-fits-all" guidance. Nevertheless, setting reasonable message limits and following the best practices can save you a great deal of trouble.

- Scott Landry, Mohammad Nadeem and Bill Long

Comments (13)
  1. I was confronted face to face with the problem which almost stopped my Exchange farm. After I saw KB960775 in UR8 I sighed with relief, since the UR2 in our organization, we are faced with the problem of "clients will not ask for the maximum limits before synching and submitting a large message to the server" that has often led to stop the whole mail system, I wrote about this in the newsgroup

    microsoft.public.exchange.admin, "Exchange Server 2007, Message size and EdgeTransport.exe problem"

    and

    microsoft.public.exchange.admin, "EdgeTransport.exe took 98% CPU time"

    to which representatives of Microsoft had not received a clear answer that this BUG or not, only after so much time in UR8 it was fixed. The good news is that ends well and finally it was fixed!

    Little real life admin story :)

  2. Gary says:

    Hi,

    Interesting article. We are still running Exchange 2003 and Outlook 2003 (SP3), and wondered if there were any extra tips relevant to that version of Exchange, as we sometimes still get issues.

    We have a Global limit of 35 MB set for maximum message size, but on occasion we have seen that users have submitted larger messages than that (300 MB avi’s for example), and the problem seems to be that Antigen will still try and scan these messages and cause a server outageversion store issue. I’d have thought that as these messages are over the limit we have, they’d be rejected straight away, but it seems not ?

    Any ideas would be appreciated.

  3. Scott Landry says:

    @Arman – I’m glad to hear SP1 RU8 has addressed your issues.  I’m sorry I missed your newsgroup post — I generally hang out in the Transport forum.

    @Gary – E2K3 + O2K3 SP3 is a very good combination.  The version store issue in 2003 is discussed in this posting:

    http://msexchangeteam.com/archive/2006/04/19/425722.aspx

    In 2003, the best solution is to monitor disk activity on your database and logs, since most of the heavy activity still happens inside the mailbox store (2007 moves a lot of this to the Transport role).

  4. shaun says:

    Do we still have to conisder base64 encoding when setting limits in Exchange 2007? We did in previous versions and it added around 33% onto the size of a message.

  5. Scott Landry says:

    @Shaun – base64 encoding is not used inside of Exchange anymore (including E2K3).  In addition, Exchange 2007 will now see if the receiving SMTP server can accept 8bit or not.  If the server can, then there is no need for base64.  However, if the server cannot accept 8bit payload, then yes, base64 is used.  Remember, however, with the exception of transport rules, the size limits are for message sizes, not attachment sizes.

  6. Scott Landry says:

    @Gary – Sorry I wasn’t clearer.  What I was trying to say is that E2K3 w/ O2K3SP3 should not do this — the client and store should be blocking submission (though perhaps not non-submission e.g. cached mode sync to Drafts folder?).  Are you certain that is the source of the message?  Do you block all other clients including pre SP3, DAV, and OWA?

  7. mike says:

    you mentioned that there can be exceptions to the global transport settings?  what did you mean?  i have set the global limit to 15MB for example, and on a specific user set max send size for a specific user and noticed the 15MB was enforced anyway.

  8. Petri says:

    "Global/organizational size limits" … "Although it is "global" in scope, there can be exceptions to this limit…"

    Do you mean that finally we are able to set global limit to normal level, and set for certain users higher than global limit? In E2003, we were forced to put limits as high as highest limit should be and then on the user level set the lower limits.

  9. Scott Landry says:

    Mike & Petri:  Yes, there are a number of postings and articles that discuss this.  Even in Exchange 2003, an individual limit could exceed a global limit.  But remember, the sender and the recipient are both checked against their limit, and the smaller limit is the one that applies.  So if userA and userB need to send emails to each other of 30MB vs. global limit of 10MB, you can certainly easily do this.  Also, rememember that Internet senders and recipients always fall under the global limit unless you have them as a contact in your organization.

  10. mike says:

    so if internet recipients are always enforced by the global limit that makes it pretty much impossible to "override".  a common scenario is executives need to larger messages outside of the org.  This is what we would like to see become possible.  right now setting the executives limit to 30 when the global is 15 is a pointless setting.

  11. Scott Landry says:

    I agree that this particular scenario is not immediately intuitive.  But think for a moment if Internet senders/recipients were not held to the global limit.  What about the scenario where a particular relay is allowed — for example an Internet user is sending to a contact in your organization (that exists elsewhere) or even a non-authoritative domain (which doesn’t require a contact object).  In this case, it would be difficult for us to determine the administrator’s intended size limit if the globals did not equally apply.  I’ll be happy to take suggestions of how to handle that type of case — perhaps we could only apply the limit if the domain is non-authoritative and the recipient is not a mailbox?  Problem is that just makes it more complex…

    Your scenario is easy to achieve by doing the override in reverse — set everyone’s limit lower than the global except for the CEO.  With the Exchange shell, this is easier than ever.

  12. Petri says:

    Scott,

    I have a feeling that I did not understood your point, sorry :-)

    If the recipient is either the mailbox or the contact in the AD, administrators could set the limits for that recipient. If the recipient is beyond non-authoritative contact, the SMTP connector could set the limitations. And if the case is pure relay, then that hits to your global limits.

    And about CEO scenario, E-shell does not make this easier. Quite often you do have some others who needs higher limits, so you have to think about updating users activate and deactivate higher limits. This has been done by people who are family with GUI. Think about how many human mistakes this might cause?

    So basically, could you tell why I have to modify every single mailbox in Exchange organization to make one small group to work? Instead of modifying this tiny group and rest of people are working automatically since the first step?

  13. Scott Landry says:

    Petri, This is a scenario that we need to continue to focus on improving — and rest assured that we will.  One problem with connectors is that you cannot easily control the send connector which is used by a particular person, another area we are continuing to evaluate improving.  On the receive connector size limit, if you increase it too dramatically, you can open yourself to a world of problems — so I do have concerns there.

    That said, there are several ways to address this besides the possibilities suggested, but these things are not easy changes and there may be tradeoffs and folks who have come to depend on it working a certain way.  In the meantime, my point is that there are solutions and individual size limits do indeed override.

    I would be happy to have an offline conversation with you further if you’d like via email.  Maybe we could discuss some specific scenarios you’ve attempted to achieve — details you probably don’t care to share through the blog.  I will look forward to hearing from you.

Comments are closed.

Skip to main content