Servicing Exchange 2007


Exchange Server 2007 is going to be serviced by Microsoft in a unique, new way that will be a substantial improvement for customers in terms of system reliability, release predictability and overall quality of software patches. Our goal with servicing Exchange 2007 is to respond to mission critical problems as quickly as possible with high quality fixes, and here’s how we are going to do it.

Background

Exchange 2007 has been engineered from the start in a different way than previous versions of Exchange. One of the big investments we’ve made in the product is actually something customers will never see – a revolutionary test product that builds alongside the shipping code which allows an unprecedented amount of functional tests, validations, and scenario checking. Because of this tight integration and checking, our build system for Exchange 2007 is capable of building ship-quality maintenance releases on a regular basis.

A simple way to think about this is that for every code defect found in the shipping product, a corresponding test case gets checked into and built within the test product. They’re then run against each other and results are analyzed for any residual problems. The end result of this integration is that instead of shipping over time a large number of small, contained spot fixes for individual problems, we will continually build these into the product and ship cumulative rollup patches that correct all customer-reported and fixed issues on a predictable, ongoing basis.

Advantages

To understand the advantages of our new servicing model, it’s important to understand some limitations of the previous model (Exchange 2003 and prior). Consider this example:

After a given amount of time, we have had 10 separate reported customer problems and have shipped 10 separate hotfix packages for Exchange 2003. As an Exchange administrator, you’ve been trained to scan Knowledgebase articles to see if any fixes are particularly applicable to your deployments. In looking over the list, you decide that you could benefit from applying two of them. After procuring the fixes from Microsoft, you install them on your Exchange 2003 systems and all is well. In a later run of ExBPA (Exchange Best Practices Analyzer) however, it recommends installing a third fix for your systems, so you download and install it. Your system now consists of Exchange Server 2003 Service Pack 2 with three additional hotfixes.

Manageable, but not simple.

Here’s how that same situation will look for Exchange 2007:

After a given amount of time, we have had 10 separate reported customer problems with Exchange Server 2007. We fix them, we produce a single build containing all 10 of these fixes, and you as the Exchange administrator simply download the single update from Microsoft Update (your choice of course as to whether this happens automatically or not) and your system is now patched with fixes to all 10 reported problems.

One common reaction to this model has been the reply of “but I only needed three fixes, and you’re making me take 10 with this approach.”

This is absolutely true that the individual fixes are no longer singly available, but the real objection to this comes from the fact that hotfixes are not historically tested to the level of released software. We run all available tests (regression, build verification, etc.) against the individual fix, but the complete end-to-end system testing in our legacy model comes before service packs only. Therefore, hotfixes were perceived to be more risky, and not to be applied unless they were critical to your business.

With Exchange 2007, however, our cumulative builds are tested end-to-end, and the overall reliability of these releases should be even higher than the original shipped product, due to the additional fixed defects contained in the rollup.

Release Information

Our goal is to release our rollup hotfix packages publicly on a regular schedule of every six to eight weeks.

We had a lot of internal debate on the frequency of release, but it essentially comes down to a balance between two factors – as infrequently as possible allows for higher uptime for the servers with less service interruptions and less maintenance, and as frequentlyas possible maximizes the reliability of the overall system and chances of avoiding potential problems. We settled on the six to eight week timeframe as a compromise between these two factors. We anticipate that since most system administrators need to update their servers once per month to apply critical security patches anyway that they would take advantage of this window and apply the latest shipped Exchange cumulative update at the same time, thereby taking advantage of the opportunity to avoid running into any potential problems we have already fixed in the product.

We’ll ship our cumulative rollups via both Microsoft Update and the Microsoft Download Center on Technet, so you can take advantage of applying these using your current patch management solution. Since they will be regularly and predictably released, your overall Exchange hotfix management process will be much smoother than it has ever been for prior Exchange products.

Package Details

Our update rollup packages will be cumulative, in that they will contain every sustained engineering fix done to the product since the original release to manufacturing (RTM). They’re also cumulative with respect to each other, so if a particular customer decides to skip applying one rollup for any reason (call it RU1) they can apply RU2 when it is released several months later and it’ll contain every fix present in both RU1 and RU2. They will be completely up to date.

Private Fixes

If you happen to be one of the unlucky few Exchange 2007 customers who actually run into a critical product defect, the fix process that we work through with you will be similar to what you experience today. After working with your Customer Support Specialist and/or Escalation Services personnel, they’ll escalate a request to the Exchange Customer Experience (CXP) team and we’ll fix your specific problem and provide you with a Private Fix to keep your systems operational until the next scheduled cumulative patch is released. These Private fixes will benefit from some of the same testing that our cumulative patches go through, but not as comprehensive.

We’ll deliver these fixes to you within the same Service Level Agreements that we have today, with the additional agreement that once the official cumulative rollup is released, you should upgrade to the final released bits just to ensure that your system configuration is back to a known, quantifiable state.

Another advantage to this servicing model is that should your system ever have to be debugged by Microsoft personnel, symbol alignment will be much simpler, as you’ll be on easily identifiable releases (RTM, or RTM +RU1, etc.)

Internal Process

Here’s a quick (simplified) glimpse into our internal workflow process for Private fixes:

  1. Defect is reported to Exchange Customer Experience team
  2. Exchange developer codes a Private fix for customer
  3. Exchange tester runs automated and manual tests on Private fix to ensure it fixes the specific problem being reported
  4. Support personnel give Customer Private Fix for validation
  5. Once both Customer and Tester are satisfied with Private Fix, it is checked into the official build tree
  6. At next scheduled interval, we fork our build tree to begin process of shipping the next cumulative rollup
  7. Test validates and signs off on cumulative rollup
  8. Microsoft IT deploys cumulative rollup on internal Microsoft Exchange 2007 servers to begin a real-world deployment test
  9. Once it passes our internal ship criteria, the cumulative rollup is shipped via Microsoft Update and available for all customers to deploy at their convenience

So, not only will each of these rollups have to pass our internal test pass consisting of tens of thousands of automated test cases (plus any additional test cases we add to test specific fixes contained in that rollup), it’ll also have to be successfully deployed in our own IT environment, just to ensure that it works on real-world Exchange servers, as well as it does in our test servers. The end result of all this validation is that each cumulative rollup will be stable, quality releases that only increase the reliability and experience with Exchange Server 2007. It’s an innovative way to service what we are confident is an innovative product.

Feedback

I welcome your feedback on our approach to servicing Exchange 2007. You can either leave comments here on the blog, or send them to me directly (kurtp AT microsoft DOT com).

Kurt Phillips

Comments (11)
  1. Flemming Riis says:

    Sounds Perfect , if a patch is good enough for MS-IT i know its good enough for ones own server.

  2. Josh Maher says:

    This is an interesting approach that should help to solve a lot of problems. I like the idea of having each fix tested to the same standards as the RTM builds.

    The only thing that is troubling is that each individual patch can not be installed/uninstalled separately. Historically, this individualization has been critical to fix integration problems in Windows and in Exchange.

    As an example….

    I have an Exchange server running RTM+RU1 and a 3rd party product that MS does not use internally. When RU2 is released it is stated that it has 1 critical database patch and 9 non-critical patches. I am concerned about my mailbox databases, so I install RU2. Unfortunately there was an incompatibility with my 3rd party product and one of the 9 non-critical patches that MS was unable to verify (as they do not run this 3rd party product). In this case I have three options:

    1. un-install RU2 and leave the databases exposed

    2. un-install the 3rd party product (which could be difficult if it were the only anti-virus product in the environment)

    3. Spend several days getting the 3rd party vendor or MS to provide a private patch

    The option I no longer have is to un-install the single offending patch and continue operations. This is unfortunate as there are a lot of 3rd party applications that a lot of businesses depend on. Without an enhanced level of early integration, there is no guarantee this new approach will truly provide stability and flexibility.

    So again, I really like the approach; however, it would be beneficial to add the individualization that was there previously.

  3. Tony Woodruff says:

    I do not see a downside to this new approach.  If Microsoft were to split out the patches as Josh suggests, it would make for an untested environment – something I do not want for my production servers.

    Microsoft certainly cannot be held responsible for third-party functionality – in my mind they are only obligated to adhere to their own published APIs – or give adequate notice that something needs to change.  Overall compatibility is part of the calculus when deciding to implement third party software in the first place.  Often it is well worth the trade off to get a better price or better features than is available from Microsoft directly.

    Keeping the number of released builds to a finite and manageable level makes complete sense.  I would offer a "thank you" to Microsoft for continuing to improve processes – not just software.

  4. gblinckmann says:

    This is one of the best ideas I’ve heard of in a long time! Knowing what hotfixes to keep up with is a major pain. I really hate running into the problem and then finding that a patch was available for it months ago. Having all these patches regression tested also will make me feel a lot better about the patches. We can just schedule updates ahead of time. Perfect!

  5. This is professionalism to it´s core. I gladly lift my hat to salute you. Well done!

  6. Richard Siddaway says:

    Brilliant idea.  If only all systems operated this way.  Personally I would prefer to see the release cycle synchronise with the normal Micorosoft patching cycle rather than be concerned with tracking 2 seperate cycles

  7. shannon says:

    You had me at "cumulative."  :-)  

  8. Anonymous says:

    In some ‘Better Late Than Never’ news, Microsoft finally released Exchange 2007 Rollup 1 to the download

  9. Anonymous says:

    In the very cool department, today sees the first Update to Microsoft Exchange Server 2007 posted to

Comments are closed.