Servicing Exchange 2007

[The_Exchange_Team]

Exchange Server 2007 is going to be serviced by Microsoft in a unique, new way that will be a substantial improvement for customers in terms of system reliability, release predictability and overall quality of software patches. Our goal with servicing Exchange 2007 is to respond to mission critical problems as quickly as possible with high quality fixes, and here’s how we are going to do it.

Background

Exchange 2007 has been engineered from the start in a different way than previous versions of Exchange. One of the big investments we've made in the product is actually something customers will never see – a revolutionary test product that builds alongside the shipping code which allows an unprecedented amount of functional tests, validations, and scenario checking. Because of this tight integration and checking, our build system for Exchange 2007 is capable of building ship-quality maintenance releases on a regular basis.

A simple way to think about this is that for every code defect found in the shipping product, a corresponding test case gets checked into and built within the test product. They're then run against each other and results are analyzed for any residual problems. The end result of this integration is that instead of shipping over time a large number of small, contained spot fixes for individual problems, we will continually build these into the product and ship cumulative rollup patches that correct all customer-reported and fixed issues on a predictable, ongoing basis.

Advantages

To understand the advantages of our new servicing model, it's important to understand some limitations of the previous model (Exchange 2003 and prior). Consider this example:

After a given amount of time, we have had 10 separate reported customer problems and have shipped 10 separate hotfix packages for Exchange 2003. As an Exchange administrator, you've been trained to scan Knowledgebase articles to see if any fixes are particularly applicable to your deployments. In looking over the list, you decide that you could benefit from applying two of them. After procuring the fixes from Microsoft, you install them on your Exchange 2003 systems and all is well. In a later run of ExBPA (Exchange Best Practices Analyzer) however, it recommends installing a third fix for your systems, so you download and install it. Your system now consists of Exchange Server 2003 Service Pack 2 with three additional hotfixes.

Manageable, but not simple.

Here’s how that same situation will look for Exchange 2007:

After a given amount of time, we have had 10 separate reported customer problems with Exchange Server 2007. We fix them, we produce a single build containing all 10 of these fixes, and you as the Exchange administrator simply download the single update from Microsoft Update (your choice of course as to whether this happens automatically or not) and your system is now patched with fixes to all 10 reported problems.

One common reaction to this model has been the reply of "but I only needed three fixes, and you're making me take 10 with this approach."

This is absolutely true that the individual fixes are no longer singly available, but the real objection to this comes from the fact that hotfixes are not historically tested to the level of released software. We run all available tests (regression, build verification, etc.) against the individual fix, but the complete end-to-end system testing in our legacy model comes before service packs only. Therefore, hotfixes were perceived to be more risky, and not to be applied unless they were critical to your business.

With Exchange 2007, however, our cumulative builds are tested end-to-end, and the overall reliability of these releases should be even higher than the original shipped product, due to the additional fixed defects contained in the rollup.

Release Information

Our goal is to release our rollup hotfix packages publicly on a regular schedule of every six to eight weeks.

We had a lot of internal debate on the frequency of release, but it essentially comes down to a balance between two factors – as infrequently as possible allows for higher uptime for the servers with less service interruptions and less maintenance, and as frequentlyas possible maximizes the reliability of the overall system and chances of avoiding potential problems. We settled on the six to eight week timeframe as a compromise between these two factors. We anticipate that since most system administrators need to update their servers once per month to apply critical security patches anyway that they would take advantage of this window and apply the latest shipped Exchange cumulative update at the same time, thereby taking advantage of the opportunity to avoid running into any potential problems we have already fixed in the product.

We'll ship our cumulative rollups via both Microsoft Update and the Microsoft Download Center on Technet, so you can take advantage of applying these using your current patch management solution. Since they will be regularly and predictably released, your overall Exchange hotfix management process will be much smoother than it has ever been for prior Exchange products.

Package Details

Our update rollup packages will be cumulative, in that they will contain every sustained engineering fix done to the product since the original release to manufacturing (RTM). They're also cumulative with respect to each other, so if a particular customer decides to skip applying one rollup for any reason (call it RU1) they can apply RU2 when it is released several months later and it'll contain every fix present in both RU1 and RU2. They will be completely up to date.

Private Fixes

If you happen to be one of the unlucky few Exchange 2007 customers who actually run into a critical product defect, the fix process that we work through with you will be similar to what you experience today. After working with your Customer Support Specialist and/or Escalation Services personnel, they'll escalate a request to the Exchange Customer Experience (CXP) team and we'll fix your specific problem and provide you with a Private Fix to keep your systems operational until the next scheduled cumulative patch is released. These Private fixes will benefit from some of the same testing that our cumulative patches go through, but not as comprehensive.

We'll deliver these fixes to you within the same Service Level Agreements that we have today, with the additional agreement that once the official cumulative rollup is released, you should upgrade to the final released bits just to ensure that your system configuration is back to a known, quantifiable state.

Another advantage to this servicing model is that should your system ever have to be debugged by Microsoft personnel, symbol alignment will be much simpler, as you'll be on easily identifiable releases (RTM, or RTM +RU1, etc.)

Internal Process

Here's a quick (simplified) glimpse into our internal workflow process for Private fixes:

1. Defect is reported to Exchange Customer Experience team
2. Exchange developer codes a Private fix for customer
3. Exchange tester runs automated and manual tests on Private fix to ensure it fixes the specific problem being reported
4. Support personnel give Customer Private Fix for validation
5. Once both Customer and Tester are satisfied with Private Fix, it is checked into the official build tree
6. At next scheduled interval, we fork our build tree to begin process of shipping the next cumulative rollup
7. Test validates and signs off on cumulative rollup
8. Microsoft IT deploys cumulative rollup on internal Microsoft Exchange 2007 servers to begin a real-world deployment test
9. Once it passes our internal ship criteria, the cumulative rollup is shipped via Microsoft Update and available for all customers to deploy at their convenience

So, not only will each of these rollups have to pass our internal test pass consisting of tens of thousands of automated test cases (plus any additional test cases we add to test specific fixes contained in that rollup), it'll also have to be successfully deployed in our own IT environment, just to ensure that it works on real-world Exchange servers, as well as it does in our test servers. The end result of all this validation is that each cumulative rollup will be stable, quality releases that only increase the reliability and experience with Exchange Server 2007. It’s an innovative way to service what we are confident is an innovative product.

Feedback

I welcome your feedback on our approach to servicing Exchange 2007. You can either leave comments here on the blog, or send them to me directly (kurtp AT microsoft DOT com).

Kurt Phillips