INFO: Update Rollup 4 for Exchange Server 2007 Service Pack 1


For a brief period of time on August 9, 2008, a pre-release version of Update Rollup 4 for Exchange Server 2007 Service Pack 1 (KB952580) was inadvertently made available to Microsoft Update, the Microsoft Update Catalog, and Windows Software Update Services (WSUS) servers for download. While we quickly removed the update from Microsoft Update within a short period of time, some servers using these distribution methods might have detected, downloaded and/or installed this version of the update.

Known issues exist with this pre-release version of Update Rollup 4, including issues with Exchange Web Services (EWS) that creates the potential for a continuous crashing cycle, an uninstall issue where the EWS web.config is reset, and an issue with backup validation. The final release version of Update Rollup 4 will be released in the upcoming weeks, and until then, we recommend that customers who have not already installed Update Rollup 4 wait to do so. If you have already installed Update Rollup 4, we recommend uninstalling it.

The following known issues exist in the pre-release (version 8.1.311.1) of Update Rollup 4 for Exchange server 2007 Service Pack 1 (KB952580). Also included are recommended steps to resolve this for customers who encounter any of the issues.




























Symptom Validation Workaround Resolution

Failure observed in backup applications which result in the following events (or similar ones in the event log). The E00 in event ID 305 is the current log file for the storage group where the backup is performed. So it can be E0n if the backup is being performed against a different storage group.

Event Type:        Error
Event Source:    Storage Group Consistency Check
Event Category:                Termination
Event ID:              401
Description: Instance 1: The physical consistency check has completed, but one or more errors were detected. The consistency check has terminated with error code of -106 (0xffffff96).

Event Type:        Error
Event Source:    Storage Group Consistency Check
Event Category:                Log File Validation
Event ID:              305
Description: Instance 1: One or more errors were detected while validating the transaction log files in ‘d:\database\’ with a base name of ‘E00’. Transaction log file validation failed with error code -1811 (0xfffff8ed).

Check if the server has Update Rollup 4 for Exchange server 2007 Service Pack 1 (KB952580) v1 installed. (See instructions below.) None

1) Uninstall the rollup from “View Installed Updates” in Windows Server 2008 or “Add/Remove programs” in Windows Server 2003.

2) If the server also has the CAS role, open the web.config file located at <Exchange Install folder>\Client Access\exchweb\ews using an editor like Notepad.exe.

3) Replace all instances of %ExchangeInstallDir% with the actual path of the Exchange installation folder. E.g. If you have installed Exchange in D:\Program Files\Microsoft\Exchange Server replace %ExchangeInstallDir% with D:\Program Files\Microsoft\Exchange Server\.

Here is a line in the web.config file.

Before

<codeBase version=”0.0.0.0″ href=”file:///%ExchangeInstallDir%bin\Microsoft.Exchange.Common.IL.dll“/>

After

<codeBase version=”0.0.0.0″ href=”file:///D:\Program Files\Microsoft\Exchange Server\bin\Microsoft.Exchange.Common.IL.dll“/>

Restart W3SVC service


Intermittent crashes in w3wp process on the CAS servers when Exchange Web Services (EWS) GetItem operations are processed. The following event is also logged in the event log.

Log Name: Application
Source: MSExchange Common
Event ID: 4999
Task Category: General
Level: Error
Keywords: Classic
Description: Watson report about to be sent to dw20.exe for process id: 9364, with parameters: E12IIS, RTL-AMD64, 08.01.0240.006, WS, M.E.Services, M.E.S.C.PerfCounterReader.GetCPUPercent, System.InvalidOperationException, d177, 08.01.0311.001. ErrorReportingEnabled: True

Event Xml:
<Event xmlns=”
http://schemas.microsoft.com/win/2004/08/events/event“>
<System>
<Provider Name=”MSExchange Common” />  <EventID Qualifiers=”16388″>4999</EventID>
<Level>2</Level>
<Task>1</Task>
<Keywords>0x80000000000000</Keywords>  <TimeCreated SystemTime=”2008-09-11T00:49:11.000Z” /> <EventRecordID>1262</EventRecordID>  <Channel>Application</Channel>  <Computer>test.contoso.com</Computer>  <Security />
</System>
<EventData>
<Data>9364</Data>
<Data>E12IIS</Data>
<Data>RTL-AMD64</Data> <Data>08.01.0240.006</Data>  <Data>WS</Data>  <Data>M.E.Services</Data>  <Data>M.E.S.C.PerfCounterReader.GetCPUPercent</Data>  <Data>System.InvalidOperationException</Data>
<Data>d177</Data>  <Data>08.01.0311.001</Data>  <Data>True</Data>
</EventData>
</Event>


Check if the server has Update Rollup 4 for Exchange server 2007 Service Pack 1 (KB952580) v1 installed. (See instructions below.)


1) Restart the W3SVC service.

2) Run your EWS application immediately to ensure that EWS is the first instance of w3wp process.

Use Task Manager to double check that only 1 w3wp process is present. If there are more than 1 w3wp process, then redo the steps from the beginning. If there is only 1 EWS is being serviced by the first w3wp process. As long this w3wp process does not crash the CAS server will not run into this issue. are more  (which may be launched from a connection like OWA or ActiveSync).


1) Uninstall the rollup from “View Installed Updates” in Windows Server 2008 or “Add/Remove programs” in Windows Server 2003.

2) Open the web.config file located at <Exchange Install folder>\Client Access\exchweb\ews using an editor like Notepad.exe.

3) Replace all instances of %ExchangeInstallDir% with the actual path of the Exchange installation folder. E.g. If you have installed Exchange in D:\Program Files\Microsoft\Exchange Server replace %ExchangeInstallDir% with D:\Program Files\Microsoft\Exchange Server\.

Here is a line in the web.config file.

Before

<codeBase version=”0.0.0.0″ href=”file:///%ExchangeInstallDir%bin\Microsoft.Exchange.Common.IL.dll“/>

After

<codeBase version=”0.0.0.0″ href=”file:///D:\Program Files\Microsoft\Exchange Server\bin\Microsoft.Exchange.Common.IL.dll“/>

4) Restart W3SVC service


Exchange Web Services (EWS) operations fail with the following response.

HTTP/1.1 302 Found
Cache-Control: private
Content-Length: 175
Content-Type: text/html; charset=utf-8
Location: /ews/GenericErrorPage.htm?aspxerrorpath=/ews/Exchange.asmx
Server: Microsoft-IIS/7.0
X-AspNet-Version: 2.0.50727
X-Powered-By: ASP.NET
Date: Fri, 12 Sep 2008 01:00:02 GMT
<html>
<head>
   <title>Object moved</title>
</head>
<body>
<h2>Object moved to <a href=”/ews/GenericErrorPage.htm?aspxerrorpath=/ews/Exchange.asmx”>here</a>.</h2>
</body>
</html>


1) On the CAS server, open the web.config file located at <Exchange Install folder>\Client Access\exchweb\ews using an editor like Notepad.exe.

2) Look for the presence of the text %ExchangeInstallDir% in the file. E.g.

<codeBase version=”0.0.0.0″ href=”file:///%ExchangeInstallDir%bin\Microsoft.Exchange.Common.IL.dll“/>

None

1) Open the web.config file located at <Exchange Install folder>\Client Access\exchweb\ews using an editor like Notepad.exe.

2) Replace all instances of %ExchangeInstallDir% with the actual path of the Exchange installation folder. E.g. If you have installed Exchange in D:\Program Files\Microsoft\Exchange Server replace %ExchangeInstallDir% with D:\Program Files\Microsoft\Exchange Server\.

Here is a line in the web.config file.

Before

<codeBase version=”0.0.0.0″ href=”file:///%ExchangeInstallDir%bin\Microsoft.Exchange.Common.IL.dll“/>

After

<codeBase version=”0.0.0.0″ href=”file:///D:\Program Files\Microsoft\Exchange Server\bin\Microsoft.Exchange.Common.IL.dll“/>

3) Restart W3SVC service


During uninstall of Rollup 4, the admin is prompted for a request for Source.

Rollup 4 is being uninstalled via ‘Add/Remove Programs’ and there is a UI requesting the original source media. Still being looked at. Looks to be an MSI Installer bug that has been fixed in a future version of MSI that we hit when changing resetsearchindex.ps1. When prompted for the source, point to the original ExchangeServer.MSI that came with media that you installed with. This will allow you to continue with the uninstall.

Identification of Rollup version via Add/Remove Programs:


To identify the version of the Rollup that you have installed, press the “Click here for support information” link for the update, under Add or Remove Programs (or Programs and Features) applet in Control Panel:



Identification of the rollup version via the Registry:


To identify if the rollup is installed via the registry, the “DisplayVersion” value shown below will be 1. The final release version of the rollup will have a value data of 2 or higher.


HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\KB952580 
  Name:            DisplayVersion
  Type:            REG_SZ
  Data:            1


We apologize for any inconvenience this may have caused and we are working to make sure this does not happen again.


Scott Roberts

Comments (102)
  1. Andy says:

    I am sorry but this is absolutely unacceptable.  

    1. Rollups should not be automatically installed, they should be optional.

    2. If a prerelease patch can get into Microsoft update, you leave me no choice but to disable automatic updates.

  2. Chris says:

    It is always recommended on servers to configure the Microsoft Update Agent to automatically download, but never to automatically install.  I would suggest this approach in conjunction with proper testing procedures in a lab environment instead of disabling automatic updates altogether.  :)

  3. EdH says:

    Glad it was caught. LIke Chris, I only have my servers set to download but not install automatically. Patches only get installed once I’ve reviewed, or at least scanned, the information on the patch. I doubt there would have been an MS08-xx for this out yet and that alone would have prevented me from installing it until I knew more.

  4. bday says:

    This didn’t happen to us, but what happens if the client downloaded it already even if it wasn’t applied? When you release the new one, will the client download that one and override the pre-release one?

  5. Scott Roberts (Exchange) says:

    BDAY-

    If your client only downloaded it and is online to the internet, wait 24 hours and the client will force a rescan. Offline clients will be able to install it. I am understanding the meaning of offline as those clients that cannot fully connect to Microsoft Update at the the time of the next scan. In most cases, the 24 hours has already passed.

    FYI – Rollup 4 isn’t not classified as a Security Update.

    Moving forward, Exchange Rollups will be released on the second and fourth Tuesday of the month due to ‘may cause a reboot’.

  6. GeeB says:

    The screenshots and registry don’t seem to apply to Vista (using Exch Mgmt Console)

    The version i see in the console is 08.01.0311.001. What will the new version be?

  7. GeeB says:

    Also, the download of the wrong verion is still available via the download link in an MBSA scan.

  8. Andy says:

    So you are saying I should not trust any update released by Microsoft until I test it?  This is a great way for full time Exchange Admins to spend their time, but my customers are not going to pay me to do Microsoft’s job.  

    I go into small businesses and setup an Exchange server.  During the configuration I set Microsoft update to automatically install patches.  You are saying I should charge my customers to test each patch released by Microsoft then manually install the patch on each server I have setup.  Talk about job security.

  9. Scott Roberts (Exchange) says:

    Thanks GeeB –

    1. We will update the blog with the text of what you will see in Vista and Windows 2008. The new version when released will be 08.01.0311.0xx where xx will be higher than 01. I will probably update this blog once we know the official build number

    2. The MBSA Scan is probably due to the MU process that uses it. I beleive the link will not work. MBSA, if I remember correctly uses Microsoft Update regluarly to determine if a Security Update should be offered. It should now offer Rollup 3. I will get back to you on this one

    Scott (Exchange)

  10. KennG says:

    Andy –

    It sounds like what Scott is saying is that it is always a good idea to test  something before installing it (and this doesn’t just apply to Microsoft code, either, but to any code that you EVER install).

    Although I’m pretty sure that Microsoft tests everything they put out, I doubt that they have the time or resources to test it with every single configuration known to man.  Just because it works with (let’s say) the 99% of hardware/software configurations they DO test — it is not a good idea for you to be that 1% that they don’t test AND which crashes the system.

    Change Control Processes are ALWAYS a good idea — even in smaller businesses.

  11. Steve Shockley says:

    I’ve already installed the patch on my management workstation.  If I leave it alone, will it upgrade to the new version (via WU) when it’s released?

  12. Exchange says:

    Steve – yes… if you have the machine set to get the MU updates automagically, it will. If you installed it manually, you can just install it on top of what you have now.

  13. Scott Roberts (Exchange) says:

    Ok folks,

    I want to point out that there is an uninstall case that we just uncovered that people may hit. Although the cases where customers have had to uninstall this particular version of Rollup 4 is low, I want to share this with everyone.

    – On install, the EWS Web.config file is installed with ‘placeholder’ information that is modified outside of the MSI (installer) while Exchange Setup is running.
    – When we uninstall Rollup 4, the EWS web.config (example: D:Program FilesMicrosoftExchange ServerClientAccessexchwebewsweb.config) file is restored to the RTM version without the post-processing.
    – On startup, EWS tries to read it and does not function properly.

    For those who encounter this condition, the fix is to open the web.config file manually and Find + Replace the information it needs. I should have the blog updated with the official info shortly with ways to detect this and the steps to fix this.

    I will have more later this evening as I write up the official blog post.

    Scott

  14. Andy,

    I wouldn’t say you shouldn’t trust the updates, but as an experienced administrator, you should know never allow your servers to install patches automatically. You should be scheduling routine maintenance cycles for your systems. I would say this about ANY server environment– Windows, Linux, Mac, or otherwise. This is especially true if you’re doing work as a consultant… what would be worse for your customers– paying you for an hour of routine maintenance, or paying you for several hours of troubleshooting a botched patch update?

    In any case, what would you prefer Microsoft do here? Not even admit the problem exists, leave the updated posted, and then silently replace it with the correct version?

  15. MEC2 says:

    <i>In any case, what would you prefer Microsoft do here? Not even admit the problem exists, leave the updated posted, and then silently replace it with the correct version?</i>

    I would rather Microsoft have a little better QC in place than to get pre-releases rolled out to the auto-update system. This is why you’ll never see Microsoft Life Support 1.0 – who can afford the risk?

    And I concur – auto-updating is a recipe for disaster. The history of Microsoft and patches has notable problems (service pack 6a, anyone?), and having your system download and install a leap of faith patch from Microsoft is an activity that will, eventually, cost you.  

  16. BD says:

    MEC2

    Service Pack 6a?  what is this, 1996?

  17. Dr4g0nf1y says:

    As I write this I’m on the phone with Microsoft for the 8th Hour trying to fix 2 production servers that downloaded this Patch and installed it via WSUS.   It has been a nightmare.   And the fix?   Wipe and reload the servers.

  18. Scott Roberts (Exchange) says:

    Dr4g0nfly,

    What are your symptoms?

    An update to the Uninstall web.config issue is:

    Symptom:

    Exchange Web Services (EWS) operations fail with the following response.

    HTTP/1.1 302 Found

    Cache-Control: private

    Content-Length: 175

    Content-Type: text/html; charset=utf-8

    Location: /ews/GenericErrorPage.htm?aspxerrorpath=/ews/Exchange.asmx

    Server: Microsoft-IIS/7.0

    X-AspNet-Version: 2.0.50727

    X-Powered-By: ASP.NET

    Date: Fri, 12 Sep 2008 01:00:02 GMT

    <html>

     <head>

       <title>Object moved</title>

     </head>

     <body>

       <h2>Object moved to <a href="/ews/GenericErrorPage.htm?aspxerrorpath=/ews/Exchange.asmx">here</a>.</h2>

     </body>

    </html>

    Troubleshooting steps to validate root cause:

    1) On the CAS server, open the web.config file located at <Exchange Install folder>Client Accessexchwebews using an editor like Notepad.exe.

    2) Look for the presence of the text %ExchangeInstallDir% in the file. E.g.

    <codeBase version="0.0.0.0" href="file:///%ExchangeInstallDir%binMicrosoft.Exchange.Common.IL.dll"/>

    Issue:

    The issue happened because Update Rollup 4 for Exchange server 2007 Service Pack 1 (KB952580) v1 was previously installed on the server and subsequently uninstalled

    Workaround (if the rollup cannot be uninstalled immediately): None

    Resolution:

    1) Open the web.config file located at <Exchange Install folder>Client Accessexchwebews using an editor like Notepad.exe.

    2) Replace all instances of %ExchangeInstallDir% with the actual path of the Exchange installation folder. E.g. If you have installed Exchange in D:Program FilesMicrosoftExchange Server replace %ExchangeInstallDir% with D:Program FilesMicrosoftExchange Server.

    Here is a line in the web.config file.

    Before

    <codeBase version="0.0.0.0" href="file:///%ExchangeInstallDir%binMicrosoft.Exchange.Common.IL.dll"/>

    After

    <codeBase version="0.0.0.0" href="file:///D:Program FilesMicrosoftExchange ServerbinMicrosoft.Exchange.Common.IL.dll"/>

    3) Restart W3SVC service

  19. MEC2 says:

    Re: SP6a and 1996

    No, it’s 2007, but what’s changed, other than an automated way to distribute updates that break your system? My point… thanks for making it. The fact they had to actually add a revision increment to a service pack just drives it home.

    If you want something more recent, look up the XP SP3 auto update on AMD systems with OEM XP – I had to deal with this zero-hour and it was pretty damn aggravating. Others abound, in fact had one today, computer displays no fonts or text after autoupdates – restore to prior day – fine. The point is, auto updating makes you the beta tester. Wait, and download after the patch has matured, and let the pioneers take the arrows…

    MEC2

  20. Piethein Strengholt says:

    Many Thanks to Scott Roberts! We had the same problem. Thx.

  21. Mike Baker says:

    the fault lies with the WSUS Team – automatic deployment of applications and patches occurs because some items are bulk classified and the wsus team refuses to change their architecture to accomodate more classification to prevent autoinstalls.

    we had the same type of problems with FCS deploying to clients – on top of symantec AV.

  22. Steen Pedersen says:

    After uninstalling Rollup 4 I have a lot of Event ID 1309 and noone has access to Out Of Office Assistant. I tried to change the  %ExchangeInstallDir% to the actual install dir and restart W3SVC service – but it doesnt seem to help. What to do now?

  23. Steen Pedersen says:

    Forget what I just wrote :-) Forgot a in the end of the path to the exchangedir. Tried again with notepad (replace all) and then it worked.

    Thanks for the quick help.

  24. GeeB says:

    To Scott Roberts (Exchange),

    fyi…MBSA scan now no longer detects the need of UR4 as of this morning, however, on any previous reports the download link still works to the bad version:

    http://www.download.windowsupdate.com/msdownload/update/software/uprl/2008/08/exchange2007-kb952580-x86-en_becbcf2833533544c251695dfaff7e02da081d1a.cab

  25. "Not aware of many customers being affected" – let’s consider who was most likely to be affected: first, the timing of the patch on automatic update meant that it was night or evening in Europe and the Middle East, but still day in North America. Since most would have automatic updates set to go off outside business hours, not so many will have caught the patch in America, but more in time zones east of the Atlantic, and they’re less likely to have reached the attention of the Exchange team in the USA. Second the sort of customer using automatic update is likely to be the small to medium size, with a small IT staff or IT contacted out. These people are less likely to have support contracts and less likely to be able to reach the right people at Microsoft. Are you sure it was "not many customers"?

  26. Ted Forest says:

    The patch pointed to a KB which wasn’t available, so I was wary of it and didn’t install it on any of our management boxes–which patch automatically. I patch all our servers manually.

    1. Yes. It was unacceptable that this was released, but sometimes planes crash and bridges fail. They’ll fix it.

    2. KB should be live and say: "This is in testing. Do not install in a production environment". Pull the trigger on the KB at the same time you do the update.

    3. I run clean, pristine, simple boxes with grown-up *current* hardware and software: Dell 2950/MD1000, the surprisingly superb Trend ScanMail (and therefore unfortunately: Java), Symantec SEP, BESR, RAWS. And that’s it. I do this so that I can rely on the multibillion dollar corporation to adequately test against a standard configuration, because it’s going to do it better than I am.

  27. Tak Lai says:

    After uninstalling rollup 4 on my cas/hub server, owa login screen (form based) does not display properly, with red x for the images.  Can someone please help?  Thanks.

  28. M says:

    I could swear my updates are set to download and not install and it still installed the freaking update.  

    Everything is working fine but I am getting event id 401 on storage group 1

    Instance 1: The physical consistency check has completed, but one or more errors were detected. The consistency check has terminated with error code of -106 (0xffffff96).

    My backup has stopped because of this but the server is still running.  I think I am going to leave it alone.  

  29. Jason says:

    Holy crap you guys cannot be serious.  I just tried to uninstall rollup 4 and it wouldn’t let me because it needed the files from rollup 3.  I have applied all updates via WSUS and have no idea where to go about getting thos files.  This totally sucks.  

  30. Scott Roberts (Exchange) says:

    Jason,

    Install Rollup 3 ontop of Rollup 4. While I don’t know exactly what is happening for you, this should restore the cached files to the box for Rollup 3 and then you can try to uninstall Rollup 4. If that fails, let me know and I will get you the command line command to do the uninstall with MSI verbose logging turned on.

    I think that I might have someone with a local repro of the problem and hopefully can look at it today.

    Thanks

    Scott

  31. Scott Roberts (Exchange) says:

    Jason,

    While I don’t know your specific scenario, reinstall Rollup 3 which should put locally the MSP in your installer cache. Then you should be able to uninstall rollup 4.

    If this doesn’t work for you please let me know here. I can give you the uninstall string so that you can turn on verbose MSI logging and I will need that.

    Thanks

    Scott

  32. M says:

    What happens if I leave it alone?  I mean everything is workign fine now?

    thanks

  33. M says:

    Scott, thanks for your help on this … however, should I leave it alone for now or remove it?  The server is runing fine I am only getting event id 401 and 305  Consistency check

  34. Tak Lai says:

    I’ve resolve my issue.  I re-installed rollup 3 again, this time running the msp file manually instead of using Windows update.  I’m not sure why there’s a difference.

  35. Jamie Schwinn says:

    I was one of the unfortunate who happened to install the update Tuesday night. I followed Scott Roberts’ instructions for fixing the EWS web.config, but I had an additional anomaly. Throughout the entire web.config were randomly scattered NULL characters which had replaced valid characters. So for some people, a find and replace on %ExchangeInstallDir% might not completely solve the issue. Try opening the web.config up in a better text editor that is capable of displaying non-text ASCII characters. I used Notepad++ for this. You will still have to find and fix the NULL characters by hand.

    Jamie Schwinn

    http://www.systmsny.net

  36. John says:

    One of or CAS servers got the update…we happened to be patching/updating  the server for the Scheduled Microsoft Tuesday patches.

    I uninstalled Rollup4.

    I manually edited the web.config file as seen above.

    I rebooted.

    So far, no errors!  I looks OK.

  37. Scott Roberts (Exchange) says:

    Jamie,

    Thanks for poining out the whitespace issue.

    Tak, I am thinking that you might have hit the MAX Disk limit for the MSI Installer cache. I don’t know yet the side effects of hitting this so I am checking on it. It is 10% of the current system drive I believe.

  38. M says:

    so an answer ? should I leave it alone since my exchange si working fine?

    Thanks

  39. Michael Rychter says:

    We were also hit by the automatic update.  (Exchange 2007)

    We found all Exchange services were disabled.  By checking another Exchange 2007 system we changed all matching services to Automatic and restarted.  Now we have Exchange running but OWA is dead!  Not a severe limitation (iow, we can get by for a short term albeit with some inconvenience to me personally!).

    I tried to uninstall the rollup but it required the original Exchange 2007 CD which I couldn’t find readily.  In the meantime I will download another copy off Technet.

    I’ve elected to do nothing further pending Microsoft giving me a reliable method of repair.  From what I read in the blog – that degree e of reliability has not been reached yet.

    Can I suggest to Scott, that the revised update install over the top of Rollup 4 and fix it automatically for us???

  40. M says:

    Why most of you are uninstalling Rollup 4? What problems are you guys facing? Is it complete email failure?

  41. Scott Roberts (Exchange) says:

    Michael,

    I don’t know what is happening with your OWA based on the available information. I believe that reinstalling Rollup 3 will rerun the OWA VDIR powershell and set up OWA Properly for you. It runs for every rollup. If that doesn’t solve the problem, we should escalate the issue and get you support.

    We have also reproduced the issue with the uninstall that requires the original Exchange 2007 MSI but I don’t know exactly why yet. Should know soon. I have the MSI log file and currently reading it. Pointing the uninstall UI to the original MSI will get you around this.

    You will be able to install the newer rollup on top of the older Rollup 4. It will solve most of the uninstall issues. If you uninstall the newer Rollup 4 and then uninstall the older Rollup 4, you will still get into the same situation as described above.

    M, I would uninstall the rollup. I am researching your problem and should have an answer. You do not have to do what I recommend of course. It just might save you some trouble later on if you have to handle an issue where you need to uninstall the rollup.

  42. Exchange says:

    M,

    Ultimately – if you are not experiencing any of the issues mentioned in our blog post or this comment thread – I’d leave the patch in place. The server works fine, so why not. You will be offered the updated version of this Rollup when it releases and then apply that, and it should be business as usual.

  43. M says:

    Scott to help you out, here are the 2 errors

    Errors as follows

    Event ID 305

    Instance 1: One or more errors were detected while validating the transaction log files in ‘\?GLOBALROOTDeviceHarddiskVolumeShadowCopy6EisnerMstorage2log’ with a base name of ‘E01’. The operation failed with error code -1811 (0xfffff8ed).

    For more information, see Help and Support Center at
    http://go.microsoft.com/fwlink/events.asp.

    Then I got event ID 401

    Instance 1: The physical consistency check has completed, but one or more errors were detected. The consistency check has terminated with error code of -106 (0xffffff96).

    For more information, see Help and Support Center at
    http://go.microsoft.com/fwlink/events.asp.

    Yes it sux.. however, I ran an NTbackup on my exchange and it went through no issues.. I believe my database is fine and that this is a bug on Microsoft end.  However I found this article  

    http://msexchangeteam.com/archive/2008/09/11/449787.aspx?CommentPosted=true#commentmessage

    I am honestly very worried about uninstalling rollup4 as this is a production environment and I am worried it will crash my server.  It’s working now so I am leaving it alone and crossing my fingers it will be okay.  However, if you tell me that this will cause
    me serious issues very soon then I will take my chances and try to remove it then install roll up 3 to fix OWA.

    Thanks

  44. M says:

    I reviewd my log againa nd found this event from WED only

    Watson report about to be sent to dw20.exe for process id: 5516, with parameters: E12, c-RTL-AMD64, 08.01.0291.001, edgetransport, unknown, M.E.D.S.U.StreamHandler.CopyStreamData, System.OutOfMemoryException, 164e, unknown.  ErrorReportingEnabled: False

    For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

    event id 4999

  45. Gary says:

    I to have attempted to uninstall Rollup 4 and had the uninstall fail out needing files from the Rollup 3.  I then tried to reinstall Rollup 3 by doing a manual download of the patch from the Microsoft.Com web site.  That install also failed out and now all of my exchange services are set to disabled.  Can someone provide me with the a list of the services that I need to set back to automatic.  I do remember that several of them were not automatic but I do not know which ones they were.

  46. M says:

    see thats why I did not want to remove mine.  I thought today will be a slow day for me and I get to relax after I have been working on their network for 4 months anyways… life= Microsoft :)

    here is what you need to have started

    ME active directory topology

    ME any spam update

    edgesync

    information store

    mail submission

    mailbox assitants

    pop3

    replication service

    search indexer

    service host

    system attendant

    transport

    log search

    ME Exchange

    good luck

  47. M says:

    Gary did it work?

  48. Gary says:

    Ended up calling Microsoft and worked with tech there to further resolve issues.  To make a long story short we ended up manually removing the Parch 4 and then reapplied patch 3.  Everything is now working but my suggestion to everyone is if your not having serious issues with the defective patch 4 I would wait for Microsoft to release a fix.  Three hours on the phone makes my head hurt.

    Gary

  49. M says:

    Glad to hear that things worked for you…

    Do we know when this fix will be released?  Rollup4 V2?

    Thanks

  50. Michael Rychter says:

    Thanks Scott…

    We are a small site (only 5 users) and the missing OWA is really only an inconvenience to me!

    I’m still getting my mail (and even to my HTC phone).  The others have access to their Outlook mail client.

    I’m going to sit this one out till MSFT release an update and if that doesn’t fix the issue, I’ll ask for technical assistance.

    Best regards from Down Under (Sydney. Australia)

  51. Mohammed says:

    regardless of the issue, do you know what’s my dream is now?

    I BEG YOU to fix the issue that the RU doesn’t update the exchange version and causes the recoverCMS to fail for a version mismatch, please i’m asking you now from RU1 time and you dont bother,. we have issues still from SP1 !! please, only a couple of lines to the code is what i’m asking for months!!

  52. Scott Roberts (Exchange) says:

    Mohammed,

    I believe that the CMS issue was fixed and that the next web release that is going out for EXBPA will have that fix. On Monday, I will verify this.

  53. M says:

    anynews when the new rollup4 is coming out?

  54. Jelle says:

    A short period of time?
    I just visited
    http://www.microsoft.com/downloads/details.aspx?FamilyId=63E7F26C-92A8-4264-882D-F96B348C96AB&amp;displaylang=en&displaylang=en and decided to download the file Exchange2007-KB949870-x64-EN.msp

    After extracting, this turns out to be Rollup 4 and still referring to KB952580.

    Why is this update still available?

  55. Ricardo Sanchez says:

    Isso é muito complicado meu servidor ficou parado 4 horas por causa disso.

    É dificil justificar esse tipo de problema já que são gastos

    mais milhares de reais com essas soluções e nos administradores

    vendemos como não vai dar problema e acontece isso.

    Uma simples desculpa não vai pagar o que a empresa perdeu

    nesse periodo e muito menos a credibilidade na minha pessoa

    quanto a escolha da melhor solução para a empresa.

    E como o amigo do primeiro post comentou: Vocês não me dão

    outra escolha a não ser desabilitar o Windows Update.

  56. Ricardo Sanchez says:

    Isso é muito complicado meu servidor ficou parado 4 horas por causa disso.

    É dificil justificar esse tipo de problema já que são gastos

    mais milhares de reais com essas soluções e nos administradores

    vendemos como não vai dar problema e acontece isso.

    Uma simples desculpa não vai pagar o que a empresa perdeu

    nesse periodo e muito menos a credibilidade na minha pessoa

    quanto a escolha da melhor solução para a empresa.

    E como o amigo do primeiro post comentou: Vocês não me dão

    outra escolha a não ser desabilitar o Windows Update.

  57. robotworx says:

    M, I am also getting those same errors on my Exchange Server when Backup Exec 12 runs on it. Have you solved this issue yet? I have run every check possible on the databases and log files and have not found a single error. Any ideas?

  58. M says:

    I am very frustrated.  I can’t backup my exchange DB via backup exec.. I called Symantec and of course they blamed Rollup4.

    I don’t want to uninstall it because of the horror stories I heard.  I am stuck and Microsoft is not doing anything so far but saying sorry.

    I spent over 50 grand in software alone this past 6 months with them and I get sorry for the answer.

  59. Robert says:

    How about being a little more exact about the times this incorrect version was available? "brief period" is a weasel phrase.

  60. robotworx says:

    M, I think I found the answer. I have Backup Exec installed on a different server than my Exchange Server so I have the remote agent installed on the Exchange Server. While looking in the Event Viewer during the Rollup 4 uninstall, it appeared that the remote agent, beremote.exe, was preventing the uninstaller from accessing certain Exchange files. Once I stopped and disabled the BE 12 services on the Exchange server, the uninstall worked perfectly. In fact, I didn’t even have to modify the web.config file. After a reboot, I ran the Rollup 3 (Exchange2007-KB949870-x64-EN.msp), reset the BE 12 services to automatic, and rebooted one last time. I just ran a full exchange backup and all of the errors went away.

  61. Scott Roberts (Exchange) says:

    M,

    I am preparing a revision to the original blog post that should address your backup issue, among other symptoms already described, that we know and understand. In addition, the exact ETA for the release of the newer version is not known and will be communicated as soon as it is. Our recomendation is to uninstall the rollup.

    Robotworx,

    The scenario that you describe is handled using the MSI build-in file-in-use detection logic for both the install and the uninstall scenario. Any file being consumed by a thrid-party app will force this to happen since the Rollup does not handle the file-in-use scenario and does not give an option for the user to stop those applications/services manually so the reboot can be avoided. With that said, I am currently having someone look at your scenario in the lab and trying to see if I might have not understood your explanation + expected results.

  62. M says:

    robotworx, thanks for the valued answer.  I am glad things worked well for you.  As we speak I am doing NT backup as its my only option now.  

    I wish if I could reboot my server during the day like you but I have a production environment and email cant be down for even 1 minute (its heavily used)

    I stopped the remote agent service and I am going to uninstall Rollup4 tonight, however, I am just wondering why did you have to reinstall rollup3?  did you face any other issues.

    Thanks

  63. jpd says:

    Finally found this post; Backup Exec failing since the 9th with the consistency check error events occuring. I suspect that this is a more common symptom than the Invalid Op crashes, which we haven’t seen.

    Going to temporarily disable BE services and uninstall Rollup 4. Also interested in why / whether a reinstall of Rollup 3 is required here as some are suggesting.

    Scott, the backup problems are serious, more information inre: backup issues needs to be posted ASAP – there is no search engine visibility for these Storage Group Consistency Check events. I worry that there are admins out there who aren’t paying attention to their backup job failures, and if a Rollup 4 v2 isn’t pushed quickly, they could end up without recent backups if disaster strikes.

  64. Ryan Richardson says:

    I too have a backup exec problem.  My Backup Exec 12 server started reporting the consistency check errors after this update was automatically applied to my Exchange 2007 Windows 2008 server.  

    I second jpd’s concerns about these backup issues.  I was pulling my hair out trying Symantec’s KB article saying I just needed to create a separate backup job for the information store with AOFO turned off (which didn’t fix it, of course).  I’m going to list the Symantec KB number to make it easier for others to google this article-

    V-79-57344-813

  65. M says:

    Its 8 PM and I am about to remove Rollup4 in few… Pray for me please

  66. robotworx says:

    M, I only reapplied Rollup 3 just in case uninstalling Rollup 4 left any residual…more of a superstitious reason than a scientific one. But no, I didn’t run into any other issues. The whole process took me about 30 minutes so hopefully you won’t have too much down time.

    Scott, I am certainly no expert on the matter. All I know is that the Event Viewer started reporting that beremote.exe was preventing some of the Exchange files from being changed and the uninstall of Rollup 4 seemed to have hung at that point. My solution was to stop and disable the BE services and shortly after the uninstall progress bar started moving again. On the other hand, it might have been a complete coincidence. I am not a guy who likes to wait on progress bars:)

  67. M says:

    its asking for Rollup3 installation package please help (while removing rollup4)

  68. M says:

    I fixed it.. long story I will type what I went through to help tomorrow.  

    It was breath taking

  69. lman says:

    Had an issue where OWA wouldn’t work after installing SP1 RU4. The login page was fine but after trying to login in a 403 error was displayed. Uninstalling RU4 and reinstalling RU3 didn’t work but reinstalling the CAS role did fix the problem.

  70. Sunny says:

    experiencing the same issue

  71. Sunny says:

    experiencing the same issue

  72. robotworx says:

    M, I found that it wasn’t actually asking for Rollup 3 during the Rollup 4 install but actually SP1.

  73. Tom says:

    I had the backup exec problem to – uninstalling SP1 Rollup 4 fixed this..

  74. Brian says:

    We are having the same problem with the backups, does uninstalling the rollup require a reboot afterwards?

  75. Jason says:

    Where is the revision to this post? If the Exchange team has more information about the backup and uninstall problems, it needs to be shared now. Admins are currently hacking up their Exchange installs trying to correct the damage done by this reckless update – how many more haven’t even clued in yet, or haven’t found this post since it doesn’t appear at first glance to correlate to their problems?

  76. Andrew_S says:

    Dear Scott
    I installed rollup 4 on to 4 servers – edge, UM, and two cas/hub/mbx servers.
    It uninstalled eventually after a very long time. The reason I installed it was because I has been told by MS support that it contained a fix for a documented bug in UM compatibility with OCS which we are currently deploying. After spending Thursday of last
    week with MS support in the UK and not getting much resolution, I then spent over 4 hours on the phone with MS support in the US Friday night. We do have mail flow, and most things now seem OK, but we do have a continuing problem around permissions it seems.
    If I run exchange mgmt shell on one of the CAS servers, and run the following I get an error:-

    [PS] C:UsersexadminDesktop>Get-WebServicesVirtualDirectory
    Get-WebServicesVirtualDirectory : Unable to create Internet Information Service
    s (IIS) directory entry. Error message is: Access is denied.
    . HResult = -2147024891.
    At line:1 char:31
    + Get-WebServicesVirtualDirectory <<<<
    If you run the command so that the focus is shifted to query the other CAS server (e.g not the local machine) it then works (and vice versa)

    [PS] C:UsersexadminDesktop>Get-WebServicesVirtualDirectory -server servenamehere

    Name                       Server                     InternalUrl
    EWS (Default Web Site)     DECOLEX01                  https://servernamehere.domain..

    In addition play on phone no longer works on the UM server.
    My call is still open with MS UK, but they are making very little progress and its getting frustrating, I am hoping to get this resolved tonight, but would welcome any input you can provide

    Fortunately my wife has a picture of me to remind her of what I look like.

    Andrew

  77. M says:

    Here is the story in detail and what happened during the uninstall.  

    First if you don’t have Exchange 2007 SP1 CD don’t attempt to remove Rollup4 as it will ask for it.  It never asked for Rollup3 file (that info was wrong) it’s asking for exchangeserver.msi file which can be found on the CD…

    As the uninstallation begins it will stop all your exchange services and disable them so be smart and write them down just in case.  Microsoft assured me a million time that removing rollup4 will not stop my services or effect my production environment (WRONG!!!) it will stop everything for about 3 minutes and it will ask for the CD and if you don’t have it be ready to have people breathing down your neck.  

    Well first I did not have the CD so as it asked for the file I started to think where can I get the file from.. I called Microsoft and of course it was a fight to get to support as they think this was not urgent matter they filed it under C level (heck I know what that means)  It means I cant send and receive emails and all services are disabled and I am dead in the water.  

    I cancelled the uninstall and tried to install Rollup3 on top no luck.  Now I have disabled services and nothing working.  I enabled all the services again (I wrote them down) and rebooted…

    Email worked but I could not receive external emails I could send and receive internally and send externally as well.   Now The support engineer was reading his notes with no answer and I am 100 degrees hotter.

    I put him on hold and I look for the CD..  as I pointed to it and the msi file everything worked the uninstallation went through and services came back again.  No reason to reboot.

    HOWEVER, OWA was dead… so Microsoft engineer still reading.  

    I go to services and I see that ISS is dead, WWW services are dead and all its dependencies are dead.  I enable all of them in the right order and OWA is WORKING.  

    I DANCE and kick the Microsoft guy off the phone…

    No need to reapply Rolup3 in any case you can’t because you have to remove it first and reboot and re-apply which is a 15 min process.

    Everything is working fine now but I get a warning

    ASP.NET 2.0.50727.0

    Web event

    Event ID 1309

    Event code: 3005

    Event message: An unhandled exception has occurred.

    Event time: 9/17/2008 11:08:13 AM

    Event time (UTC): 9/17/2008 6:08:13 PM

    Event ID: 2d89d91b88004e55b4428253a8c6e98c

    Event sequence: 43

    Event occurrence: 42

    Event detail code: 0

    Application information:

    I need this fixed.  

  78. Disgruntled says:

    EHLO? Is this thing on? (tap) (tap)

    A week out and still no accurate information, no updated info that reflects the other serious problems caused by this roll-up. Microsoft is allowing people to trial-and-error their way through unpatching production systems sabotaged by a not-ready-for-production patch.

  79. Michael Rychter says:

    I was impatient in spite of my previous position that I woud wait for the fix.

    So, I found the original Exchange 2007 CD and uninstalled Rollup4.

    Instead of the tails of woe from others, my uninstall was completely uneventful, with the services starting up correctly after the mandatory re-boot of the server.

    OWA is now working.

    I downloaded Rollup3 and reinstalled that.  Uneventful as well.

    So, my advice to others – be wary but follow the advice given by Scott and uninstall!

  80. M says:

    Michael Rychter,

    are you reciving the event ID I am reciving?  1309 ASP error?

    Thanks

  81. Michael Rychter says:

    Yes, I am getting a 1309 ASP Warning.

    Event code: 3005
    Event message: An unhandled exception has occurred.
    Event time: 18/09/2008 3:48:25 PM
    Event time (UTC): 18/09/2008 5:48:25 AM
    Event ID: f93ad4001027498ea1b235a629d6e9ad
    Event sequence: 96
    Event occurrence: 95
    Event detail code: 0

    Application information:
       Application domain: /LM/W3SVC/1/ROOT/EWS-1-128661515570156250
       Trust level: Full
       Application Virtual Path: /EWS
       Application Path: C:Program FilesMicrosoftExchange ServerClientAccessexchwebEWS
       Machine name: SERVER

    Process information:
       Process ID: 7864
       Process name: w3wp.exe
       Account name: NT AUTHORITYSYSTEM

    Exception information:
       Exception type: FileNotFoundException
       Exception message: Could not load file or assembly ‘Microsoft.Exchange.Diagnostics, Version=8.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35’ or one of its dependencies. The system cannot find the file specified.

    Request information:
       Request URL:
    https://server.ashpoint.local:443/EWS/Exchange.asmx
       Request path: /EWS/Exchange.asmx
       User host address: 10.10.20.11
       User:  
       Is authenticated: False
       Authentication Type:  
       Thread account name: NT AUTHORITYSYSTEM

    Thread information:
       Thread ID: 5
       Thread account name: NT AUTHORITYSYSTEM
       Is impersonating: False
       Stack trace:    at Microsoft.Exchange.Services.AuthorizationModule.Init(HttpApplication context)

      at System.Web.HttpApplication.InitModulesCommon()
      at System.Web.HttpApplication.InitInternal(HttpContext context, HttpApplicationState state, MethodInfo[] handlers)

      at System.Web.HttpApplicationFactory.GetNormalApplicationInstance(HttpContext context)

      at System.Web.HttpApplicationFactory.GetApplicationInstance(HttpContext context)

      at System.Web.HttpRuntime.ProcessRequestInternal(HttpWorkerRequest wr)

    Custom event details:

    For more information, see Help and Support Center at

  82. Jeff Anderson says:

    I was one of the unlucky few who installed this rollup as well.  I noticed problems when attempting to backup my Exchange 2007 server using ARCserve 12 (ARCserve generated: " AE9708 Check integrity failed.  Check the Exchange server status." and there were numerous errors in the application log file on the Exchange server with an event ID of 302 indicating that there were log file validation errors).  I spent several days trying to resolve this before coming across this posting.  Uninstalling the rollup resolved the issue; I did not have any problems uninstalling the rollup.

  83. M says:

    I solved the ASP problem following the article above… I locked my server from updates… No more updates FOREVER

  84. Michael Rychter says:

    M – Where is the article you followed to resolve the ASP problem?

  85. Ryan Richardson says:

    I uninstalled the Rollup without incident and my backups are running normally again.

  86. Philip says:

    Well that’s a relief :)

  87. David Godfrey says:

    I’m running Arcserve 12 here and had the same issues with backups. This morning I pulled exchange sp1 media and the update3, uninstalled update4 and rebooted, installed patch3 and rebooted.

    After a few hours of running my backups are successful. I have noticed the 1309 ASP Warning error too and that will be my next thing to fix…

  88. KCotreau says:

    I was also getting 1309 and 1310 errors.

    The 1309 errors are caused by %ExchangeInstallDir% not being replaced by C:Program FilesMicrosoftExchange Server in the web.config file. I had missed two of them due to nulls when I used Wordpad to do a global search and replace so I was still getting that error.

    The 1310 errors are caused by the null characters in the web.config. You can open web.config with Microsoft Wordpad. Each null has to be replaced with the correct character manually. The 1310 error shows which line to look in, but there will certainly be more nulls below that first one.

  89. KCotreau says:

    I have also been getting the following error since the patch. It happens periodically on its own and every time I run Acronis True Image Server. I was wondering if anyone else has seen this one too?

    Event Type: Error

    Event Source: ESE

    Event Category: ShadowCopy

    Event ID: 2007

    Date: 9/22/2008

    Time: 3:56:34 AM

    User: N/A

    Computer: EDITEDDC01

    Description:

    Information Store (3916) Shadow copy instance 7 aborted.

  90. KCotreau says:

    Sorry for the multiple posts: In Wordpad, the nulls appear as little boxes. I could not find a way to search for them with any program I have, so you just have to hunt them down line by line and using the 1310 errors as guidance for any you may miss (since they list the line in the web.config file where it bombs out).

  91. mossmak says:

    After installed rollup 3, my OWA didn’t work.  i updated to R4, no worky.

    I rolled back both, but now have an issue with Scheduled Free/Busy not working.

    I did the web.config fix, but still get the ASP.net error 1309.

    One hint is that the ASP.net returns the user host address as the LAN IP address of my ISA Firewall.  How do I fix the Free-busy issue?

  92. GeeB says:

    I did an uninstall of UR4 and had the original msi available and used it when prompted. However, these errors came up:

    Erorr reading from file D:SetupServerRolesXommonEnExBPA.Config.xml

    Verify that the file exists and you can access it.

    CANCEL  RETRY IGNORE

    D:SetupServerRolesXommonEnExBPA.PreReqs.xml

    Verify that the file exists and you can access it.

    CANCEL  RETRY IGNORE

    D:SetupServerRolesXommonEnExBPA.Readiness.xml

    Verify that the file exists and you can access it.

    CANCEL  RETRY IGNORE

    D:SetupServerRolesXommonEnExBPA.ResetSearch.ps1

    Verify that the file exists and you can access it.

    CANCEL  RETRY IGNORE

    Not sure where it is picking up this drive/path. Exchange files are not present there. They are on another drive. I choose IGNORE on all and the uinstall seem to work fine. I then re-installed UR3.

    Any ideas on errors?

  93. KCotreau says:

    I have since found a solution to the problem I mentioned regarding this error. There is a rollup patch for VSS here: http://support.microsoft.com/kb/940349

    Event Type: Error

    Event Source: ESE

    Event Category: ShadowCopy

    Event ID: 2007

    Date: 9/22/2008

    Time: 3:56:34 AM

    User: N/A

    Computer: EDITEDDC01

    Description:

    Information Store (3916) Shadow copy instance 7 aborted.

  94. Alex DuBois says:

    I applied the fix above (replaced all %ExchangeInstallDir% variables with the absolute path in web.config), but now I’m seeing a new error. OWA was running fine for ~30min before the error.

    Faulting application w3wp.exe, version 7.0.6001.18000, time stamp 0x47919ed8, faulting module davex.dll, version 8.1.278.0, time stamp 0x47f36909, exception code 0xc0000005, fault offset 0x00000000000d3be1, process id 0xbd4, application start time 0x01c923f12e7de329.

  95. Alex DuBois says:

    We followed the instructions below to fix the problem in my previous post, and the MSExchangeOWAAppPool has now run almost 24 hours vs. ~5-10min.

    http://blogs.technet.com/mblant/archive/2008/08/29/exchange-2007-owa-crash-issue-on-windows-2008-server-iis7.aspx

  96. ScottB says:

    Rollup 4 SP 1 was released yesterday…

    http://support.microsoft.com/?kbid=952580

    Oddly the page is version 1…

    Article ID : 952580

    Last Review : October 7, 2008

    Revision : 1.0

    The patch package is version 2…

    Exchange2007-KB952580-v2-x64-EN.msp

    And the actual patch is listed as minor version 3…

    Version: 8.1.311.3

    Date Published: 10/7/2008

    Has anyone tried this yet, is all good in the world of exchange again?

    I am in need of this rollup to fix several problems beside the ones the pre-release broke.

    Thank you for your time,

    Scott

  97. Joao Gazzoni says:

    Hello,

    OWA does not start anymore:

    Process aspnet_wp.exe (OWA) (PID=4600). An remote procedure call (RPC) request to the Microsoft Exchange Active Directory Topology service failed with error 5 (Error 5 from HrGetServersForRole). Make sure that the Remote Procedure Call (RPC) service is running.

  98. MannyG says:

    As an admin you should always select to download BUT NOT install the patches. Review the patches and then if there is not known issue you can install them. You have to do this every every time you need to update. Microsoft sends the updates every second Tuesday of the month. I still reviewing this updates and after reading this i deleted from the Exchange 07 update list. Read before you install.

Comments are closed.