Are large messages slowing your Exchange server down?


As Ewan mentioned a little while ago, there are some pretty extreme things going on in some our customer’s Exchange environments.  One of the things that I could definitely relate to was the customer who sent a message that was 2.4GB that was successfully delivered.  While Exchange is quite capable of transferring huge messages, using it to do so usually results in severe performance issues.  In fact, one customer that I worked with was experiencing major problems due to a 6 GB message traveling through their Exchange server.  To make things even more interesting, the 6 GB message that we found was actually a non-delivery report (NDR) that was generated to tell a user that the 6 GB message that they had sent was over the maximum message size allowed for the organization.


 


Once we had the server back up there were a number of questions to be answered…


 


Why did the Exchange Information Store accept the message in the first place if it exceeded the configured size limits?


The Exchange Information Store accepts the message since it can’t tell whether a message exceeds configured size limits until it receives the entire message.


 


Why can’t we just modify Exchange’s behavior to stop accepting data as soon as it knows that the message is too big?


Doing this would actually result in a bigger problem – Outlook would just try to send the data again and again, resulting in a never-ending stream of data being sent to the Exchange server.  In addition to the obvious performance problems, this would result in a large amount of transaction log files being generated in a manner similar to the problem that Jeremy Kelly describes here.


 


Why did the Exchange Information Store pass it off to the categorizer?


Prior to Exchange 2003 SP1 and the Exchange 2000 Post-SP3 August 2004 hotfix roll-up the Information Store only performed size checks against per-user limits and not the global limit.  This greatly exaggerated the performance impact of large messages being sent through the system since the check against the global limit was performed much later in the message submission process.


 


With current versions of Exchange the check is now performed by the Information Store immediately upon message submission.  What this means is that Outlook clients in online mode will receive an immediate failure upon trying to send a message that is too large.  If Outlook is in cached mode, offline mode, or configured to connect to an Exchange server and use a PST as its delivery location then an NDR will be generated by the Information Store; since the message goes no further the impact on server performance will be much less.


 


Won’t that large NDR for non-online mode clients cause performance issues on the client, server, and network?


Yes, it will.  So, like all good software companies, we listened to our customers and have done a significant amount of work in both Outlook and Exchange to address this problem.  Here are the changes we made….


 


The first change we needed to make was to provide Outlook with a way to ask the Exchange server what limits have been configured.  We added this functionality to Exchange and you can get it today by installing any of the following:


 


          Exchange 2003 SP2


          The most recent Exchange 2003 post-SP1 update available from Microsoft Update.  This update contains the specific Exchange 2003 post-SP1 hotfix for this issue


          This post-SP3 hotfix for Exchange 2000.


 


The second part of the solution was for Outlook to use this newly exposed functionality in Exchange to determine whether a message is over configured size limits before sending it.  This work has also been completed, and you can get it today by installing Office 2003 SP2.


 


Once your Exchange servers and your Outlook clients have been updated you will start seeing the new behavior.  Specifically, if a message is sent that is over configured size limits, either per-mailbox or global, an error will be displayed as when Outlook goes to send the message to the server; no additional data will be sent across the wire and no load will be placed on the Exchange server.


 


As an added bonus, Outlook will also check to make sure that the mailbox is not over its quota before sending the message.  This keeps data from being sent to the server if Outlook knows that the message would be rejected with an NDR as soon as it is received by the server.  In addition, if your mailbox is over its quota then you will be presented with mailbox cleanup dialog to help you get it back under quota.


 


Where can I find more information?


Read KB 894795 for detailed information on the exact messages that users may see with this new functionality, a full description of the exact order of checks performed, and a description of how global vs. per-user limits are evaluated.


 


Dave Mills


Comments (11)
  1. RB says:

    Great work but…

    1) It requires Outlook 2003 + SP2.

    2) Exchange still attaches the original message + original attachments to the NDR

    Is there a way to prevent Exchange to send NDR’s above a certain size?

    Or how do you prevent Exchange adding the original attachment to the NDR?

  2. AG says:

    HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesSMTPSVCQueuingMaxDSNSize allows you to specify a maximum msg size to NDR the whole message.

    Take a look at http://support.microsoft.com/?id=308303

  3. Setesh says:

    "Once your Exchange servers and your Outlook clients have been updated you will start seeing the new behavior."

    We have implemented the same functionality on Exchange 2003 SP1 with Outlook XP by implementing the size restriction on the SMTP Virtual server.

    Which makes this Q and A just WRONG !

    "Why can’t we just modify Exchange’s behavior to stop accepting data as soon as it knows that the message is too big?

    Doing this would actually result in a bigger problem"

    When anyone outside out organisation attempts to send us an oversized message their exchange server (or equivalent) generates the NDR, not ours.

    These Blogs are normally first class, but this one is so inaccurate !

  4. also think that this is great work. But:

    Why wasn’t this considered upon development of the first release of Exchange 2000? This behavior is nothing "exotic"…when I think about efficient data transmission, this is "standard".

  5. mike says:

    Great work. Excellent blog entry…

  6. Dave Mills says:

    RB:

    As AG stated, take a look at http://support.microsoft.com/kb/308303 for information on how to limit the size of NDRs generated by SMTP. Be aware that this setting does not affect NDRs generated by the information store, so NDRs generated on Exchange 2003 SP1 or Exchange 2000 with the latest rollup for cached mode clients will still contain the original message and all its attachments. There is no way to change this behavior.

    Setesh:

    This blog entry is about Outlook clients that use MAPI to send messages to the Exchange server that exceed configured size limits. It sounds like the functionality you are referring to is the ability to block large messages coming in via SMTP. Prior to the updates that I described in this blog entry there was no way to keep Outlook in MAPI mode from sending messages that exceed size limits to the server. I apologize for any confusion and I hope this clears things up a bit.

    Christian:

    That’s a good question.

    The first thing to note is that this issue has technically existed in the product ever since Exchange 4.0.

    The second thing to keep in mind is that hard drives have increased in size quite rapidly in the last decade. For consumer class hard drives, the top capacity was about 2 GB by the end of 1995, 10 GB by the end of 1997, and 20 GB by the end of 1999. These relatively small hard drive sizes meant that not a lot of people had huge files on their hard drives, and if they did they usually weren’t the type of files that were likely to be sent as e-mail attachments.

    So, to answer your question, even at the time of Exchange 2000’s development this behavior actually was much more "exotic" than it is today in the world of 100 MB PPT files, multi-GB MP3 collections, and 500 GB hard drives.

  7. Dave,

    thanks for sharing your thoughts… It’s great to get feedback!

  8. RB says:

    Dave & AG: thank you for your replies.

    The article sounds promising, I can save resources on the SMTP side when sending NDRs to external recipients.

  9. Setesh says:

    "Setesh:

    This blog entry is about Outlook clients that use MAPI to send messages to the Exchange server that exceed configured size limits."

    As stated, we currently use Outlook XP (MAPI) to send messages via Exchange 2003 SP1. If a user attempts to send a mail that is over the size limit they instantly get a popup message that states "The message being sent exceeds the message size established for this user."

  10. A_N_IV says:

    I appreciate the new functionality, but would like to ask a question: Is there a product available that can integrate with Exchange and intercept large attachments, transfering those files in another fashion, say via HTTPS, and spawning an e-mail to the recipient with a link that allows them to download the file?

    I am currently looking at products like HyperSend and Tumbleweed secure file transfer, but they do not integrate into Exchange, which poses the problem of getting the user base to adopt the technology, and then to actually use it.

    If not, that may be a function MS may want to start trying to develop as an add-on product for exchange, as we all know, attachments just keep getting bigger and bigger…

  11. Dave Mills says:

    Setesh:

    The error popup that you are seeing only happens in online mode after all the data has been uploaded to the server. You can observe this for yourself by composing a new message, attaching a large file to it, and then watching for transaction logs being generated on the server prior to the message being sent. In addition, the fix described in this blog entry is specific to cached mode, offline mode, and Outlook configured with a PST as its delivery location, so even with Outlook 2003 in online mode you will still experience the same behavior.

    A_N_IV:

    I believe that Outlook 2003 does have some level of integration with SharePoint to do something like what you are asking for. As for third party solutions, you might be able to find what you’re looking for on our partner’s page for archiving and compliance at http://www.microsoft.com/exchange/partners/archivingandcompliance.asp.

Comments are closed.