MAPI Messaging Benchmark Being Retired


MAPI Messaging Benchmark (MMB3/MMB4) has been the benchmark for measuring the performance and scalability of computers running Microsoft Exchange Server.  While MMB has served its purpose, it also has limitations in helping our customers compare solutions.  Among the most significant limitations, MMB does not take into account variation in end-user usage profiles, modification and updates to storage solutions, differences in customer topologies, and impacts of other features or applications on performance.  MMB compares single servers, but does not compare solutions.

As the era of benchmarking changes to adapt to different types of clients, user profiles and other variables, the current methods are becoming impractical.  With that said, Microsoft will be closing down the MMB3 site as well as the Performance Benchmark result site, in the first quarter of 2008.  Although the MAPI Messaging Benchmark results will not be posted by Microsoft, we will continue providing the tools necessary to allow Microsoft partners to perform their own performance and scalability benchmark analysis for Exchange Server.  These tools include: Exchange Server JetStress, Exchange Load Generator, and Exchange Server Stress and Performance (this tool is in the process of being merged into LoadGen).

In place of the current MMB3 site, we will work with Partners to provide links to their respective sites where customers can find real world performance and scalability studies which assess actual customer scenarios and solutions.

Q&A

Q: What is MAPI Messaging Benchmarks?

A: MAPI Messaging Benchmarks (MMB) has been the benchmarking standard for measuring the performance and scalability of computers running Microsoft Exchange Server.  MMB evaluates the messaging performance of a server by measuring how it scales up when adding concurrent load that is modeled after typical corporate e-mail use. An operating environment with higher MMB results than another is able to hold more users per server, although the exact number is not the MMB score.  The MMB3 site gives a good overview of what MMB is designed to do, while the performance benchmarks for computers running Exchange 2003, gives results.

Q: Why is Microsoft Exchange no longer investing in MAPI Messaging Benchmarks?

A: MMB helps compare single servers, but does not do a good job comparing solutions.  MMB has a number of limitations in helping customers compare solutions:

  • MMB does not take into account variation in user profiles > There are many differences in average messages sent/received per day, mailbox size, e-mail client used, etc… which are not factored in with MMB.
  • MMB does not take into account variation in storage solutions > MMB factors in the hardware, but not the storage.  Adding a huge impractical storage system will give totally different results and therefore comparison is no longer with just the hardware, but the storage solution as well (Note: regarding the importance of storage to Exchange Server solutions, see the Exchange Solution Reviewed Program website)
  • MMB does not take into account variations in customer topologies > Differences exist in the number and size of hub and branch office sites, network bandwidth between sites, and other factors not measured with MMB.
  • MMB does not take into account other features or applications that impact performance > Including Microsoft Forefront, other Internet protocols, Outlook cached mode, or desktop search products.

Q: Where can I go to investigate partner solutions?

A:  The MMB3 site and results page are still available, into 2008. These pages will be removed and replaced with links to Partner sites for their Exchange solutions.

Jim Lucey


Share this post :

Comments (4)
  1. Tony, Zhan says:

    So LoadSim will not be recommended as load testing tools?

  2. Nick says:

    Yes, but when will Exchange 2007 SP1 be out?  We’re doing an upgrade to Exchange 2007 at the end of next week, and I’d love to have it in place before that happens.  Please tell me you can beat Thanksgiving?

  3. Jim Lucey says:

    LoadSim is the load tool for legacy versions of Exchange (2000, 2003, etc..).  Load Generator (LoadGen) is the tool for Exchange 2007 and future versions.

  4. Steve Tramack says:

    I view this as a positive move; MMB had certainly outgrown its usefulness in terms of deployment planning or even simulating the type of load that one might expect from a production Exchange deployment.  While there is still a need and validity in the load driving tools and some form of testing, the methodology and workload associated with the benchmark served little purpose beyond some level of server comparison.

    HP currently provides sizing tools, reference configurations and best practices guides focused on assessing server, storage and deployment architecture needs for a given Exchange workload.  These are freely available on our ActiveAnswers solutions portal (http://www.hp.com/solutions/activeanswers/exchange).  Microsoft also has the Exchange Solutions Reviewed Program (as referenced above), which is effectively a storage solutions benchmark, which is a much better approach to providing prescriptive guidance for a given configuration in support of a specified workload.  HP actively contributes ESRP configurations as well.

Comments are closed.