Problem Solved: The WSUS Export Bug


If you are managing updates for Windows Server 2012, you might have noticed that your WSUS server has been synchronizing a greater than usual number of updates these days. Windows Server 2012 is not only Microsoft’s most powerful, but also Microsoft’s most secure operating system to date. Since RTM, we’ve published dozens of updates that improve the security, reliability, and  functionality of Windows. Most notably, the Defender team has been publishing virus and malware definition updates 4 times a day, to ensure that your PC is never left unprotected from even the latest threats.

We’ve also added many new products to WSUS, including updates to Adobe Flash Player, Skype, Lync Server, and Office 2013. There are more WSUS updates than ever before.

All these updates certainly can take a toll on a WSUS server, especially on servers that have auto-approval rules. Many administrators were recently surprised to see that exporting updates from WSUS servers was failing, resulting in zero-size output files. It became clear to us rather quickly that the issue was due to a limitation in the CAB file format, which is an uncompressed file size limit of 2 GB.

I am pleased to announce that, as of today, a fix for this issue is available from Microsoft. An article describing how to get this update is available at:

Many people have been asking on the forums and even through this blog for this fix for some time, and were wondering why it took more than 4 months to complete. To that end, I thought it would be interesting to share some of the engineering story as well.

One of the unique challenges of working on the WSUS team, compared to other server role teams in Windows Server, is that our role actually ships, “out-of box,” all the way back to Windows Server 2003 R2. We had to be very careful to design a solution that would work on all versions of Windows Server since then. To keep everything as simple as possible, we constrained ourselves on using compression algorithms that are already publicly available in the .NET framework or public Win32 API. Here is what we considered:

 

Framework

Max Size

Benefits

Drawbacks

GzipStream

.NET 4.0

Unlimited, 4 GB prior to .NET4

Built in CRC

Results in 2 files

No support for signtool

DeflateStream

.NET 4.0

Unlimited, not available until .NET4

Supports larger file sizes than CAB

Results in 2 files, harder to work with raw DEFLATE format vs. Gzip, no support for signtool

CAB

.NET 2.0

2 GB uncompressed file limit

Built in CRC

Same file format

Supported by signtool

Temp files needed (extra disk I/O), 2 GB limit per uncompressed file

Zip64

.NET 4.5

unlimited

Built in CRC

No CLR support for Windows Server 2003 R2; no support for signtool

We decided it wouldn’t be a great customer experience to require people to install .NET 4.5 on their servers. Plus, we wanted to preserve compatibility with Windows Server 2003 R2 if at all possible. .NET 4.5 is not available on Windows Server 2003 R2. We could have also adapted the CAB format to produce multiple output files, but we preferred to produce only a single output file, especially since the resulting files are comparatively small.  On systems running the latest version of .NET 4.0, the maximum file size is limited only by NTFS file size. While Gzip does store the length of the compressed content using 32 bits, the structure stores only the lowest 32-bits thereby allowing for larger file sizes, limited only by NTFS file size (prior to .NET 4.0, the length of the compressed content couldn’t exceed 32 bits, or about 4 GB). This does mean that there’s a limit of 4 GB on systems using .NET 3.5 (i.e., WSUS 3.0 SP2 (3.2) ). Windows Server 2012 is on .NET 4.5 and therefore not affected by the limit. Similar to CAB, GzipStream also includes built-in CRC error detection, which makes the export and import processes highly reliable. Best of all, the compressed output is generated on-the-fly, without the need for temporary uncompressed output files. This greatly decreases disk I/O and we found it reduces the time needed for import and export significantly.

Using our current export algorithm, the export process results in 2 files, a metadata.txt file and an update.cab file, which must be kept together. The CAB format archives these two files together. However, since Gzip is not an archive format, the end user would have needed to keep these files together manually. We would have preferred to produce only a single export file to ease distribution. Therefore, we changed the WSUS exported data XML schema to allow for both metadata and update data in a single compressed file.

In parallel, we also worked with our test engineers to formulate a test plan for the hotfix. Our test matrix is large, as we need to ensure that every supported version of Windows Server will be able to install this update. For Windows Server 2003 R2, that matrix is made even bigger by our support for both x86 and x64 architectures. We also worked with Customer Support Services (CSS) to gather validation data with regard to cases that were currently opened about this issue. And of course, nothing works perfectly the first time, and we’ve gone through several revisions to make sure that the latest WSUS update is more robust than ever!

At this point, we have a hotfix that we feel really good about releasing: well-informed by customer needs, developed by our esteemed software engineers, and thoroughly tested and validated. I hope that it will improve the experience of the many WSUS admins around the world, and I look forward to hearing your feedback about this update.

Thanks for being a valued Windows Server and WSUS customer!

 

The WSUS Team

 


Comments (58)

  1. Anonymous says:

    Servers will not have Flash installed unless you install the Desktop Experience feature, which is not installed by default. If Flash is installed via Desktop Experience, it will automatically be kept up-to-date via Windows Update.

  2. Anonymous says:

    @Eric-

    1) All WSUS 3.x patches are cumulative, so installing KB2828185 would have included all updates prior to that KB. This is clearly stated on the KB page. I'm not sure what the solution to that fileVersion problem is right now, but I would encourage you to post on the forums for help from our MVPs or open a case with Support who can look into this in your specific situation.

    2) Yes, even after installing the KB you can still export a cab file by specifying a filename ending in .cab. That code path remains unchanged and the 2 GB limit still applies.

  3. Anonymous says:

    @Chauncey, the WSUS forum is the best place. social.technet.microsoft.com/…/threads

  4. Anonymous says:

    Yes, the hotfix needs to be installed on the import server as well, or else the code that reads the new file format will not be available.

  5. Anonymous says:

    >Why not Windows 7 as well?

    The answer to that, unfortunately, would need to come from Adobe and/or the IE or Windows Client teams. But I would encourage you to ask them if they have a blog.

  6. Anonymous says:

    @Ase, Updates to Adobe Flash are available under the Windows 8 and Windows Server 2012 categories. They apply only to Windows 8 and Windows Server 2012 machines.

    >Any update to the WSUS 3.0 SP2 Patch?

    @SCC, @Jay, Check back later today or tomorrow!

  7. Anonymous says:

    Explanations into why decisions were made the way they were are always appreciated.  

  8. Ase says:

    Hi Guys,

    What cought my eyes was a part of the second chapter:

    "We’ve also added many new products to WSUS, including updates to Adobe Flash Player"

    I looked in the products and cannot find anything added from Adobe, did I miss it?

    //Ase

  9. Ase says:

    Thanks for your fast reply Ben,

    I am still staying with Win7 on the workstations, but servers are 2012 but I do not think that have any flash installed, trying to keep them as clean as possible. Should updates for flash just appear  under "All Updates" If I have the checkmark on Windows Server 2012 Product and all Classifications selected?

    //Ase

  10. Ase says:

    Why not Windows 7 as well?

  11. Gabriel says:

    Hi Ben,

    I am presently managing WSUS 3.0 SP2 (Windows Server 2003) on a disconnected network, which involves exporting updates and metadata from a WSUS server on a connected network and then importing all that information into the WSUS server on the disconnected network.

    Presently, I have successfully installed the hotfix on the export WSUS Server and have exported the GZIP file. But may I ask whether there is a need to install this Hotfix to the import server as I am receiving the following error message: "There is a problem with this Installer package. A problem runs as part of the setup did not finish as expected. Contact your support or package vendor", after installing it…

  12. Rhiannon says:

    I apologize.  Just read an article elsewhere that states to run:  wsusutil.exe export export.xml.gz export.log

    I am trying this now.  Hope this works.  Thanks for the information.  Much appreciated.

  13. Eric says:

    Hello, I greatly appreciated your explanation, but unfortunately, have run into several problems applying this patch.

    I have two questions….

    Question 1)

    We have a WSUS server connected to the internet, that then patches are provided to 7 disconnected networks we manage.

    We had a CAB file that was zero, so I applied patch KB2828185 to the connected server with no problems, and the export.xml.gz was created successfully.

    When I installed KB2828185 on two of the disconnected WSUS servers (Windows 2008 SP2, 64-bit and Windows 2008 SP2, 32-bit), we got the same error  

     "An error occurred during the installation of assembly. Microsoft.UpdateServices.Utils,fileVersion ="3.1.7600.262" ,version="3.1.6001.001" ,culture="neutral" ,publicKeyToken="31BF3856AD364E35" ,processorArchitecture="MSIL". Please refer to Help and support for more information.

    What can be done?  I believe that installing KB2828185 on the two disconnected networks (one 64 bit, one 32 bit)  under the cover installed KB2720211.   KB2720211 was the nightmare of all nightmare costing my company at least 100 hours trying to recover…..requiring reinstalling WSUS on each network and making sure KB2720211 did not touch our network.

    Question 2)  once installing KB2828185….is it possible to export a CAB file?

    Thank you,

    Eric

  14. Chauncey Smith says:

    Can someone point me to where I can ask for help troubleshooting my WSUS 6 Windows Server 2012 Standard installation that suddenly went belly up?

  15. Rufus says:

    This problem is not solved for me.  After applying the update, I receive the following error.

    tinypic.com/view.php

    I find the error peculiar, because the error occurs after the file reaches 495 MBs, far shy of 4 GBs.

    Do you have any ideas?

  16. @Rufus, the *uncompressed* data must not be more than 4 GB on WSUS 3.2.

  17. Rufus says:

    Thanks for  you comment.  But I thought the application of KB2828185 and Gzip circumvents this restriction.

  18. Hi Rufus, with KB2828185 the limit, as stated in the table above, is "Unlimited, 4 GB prior to .NET4" — WSUS 3.x uses the .NET framework 3.5. To get around the limit, you can call the WSUS API from a .NET 4.5-based application.

  19. If you are using WSUS 6.x (on WS2012 with KB2828185, or WS2012 R2), you should not have any limitations on the exported data size.

  20. Rufus says:

    I get it.  The above solution cannot assist me in resolving my issue (the metadata exceeds 4GB).  However, I was able to find a workaround.  Perhaps, someone with a similar issue will find some utility from the workaround.

    1. Create a standalone (do not add this machine to the domain) virtual machine on the Internet-side of your network.

    2. Install WSUS and the WSUS KB2720211 patch.

    3. Select updates that you need.

    4. Copy the WSUS machine from the Internet-side to the closed side of your network.

    5. Add the machine to the domain.

    6. Configure WSUS clients to report to WSUS server.

    7. If clients receive a 800B0001 error, apply the System Update Readiness Tool to possibly fix the error.

  21. Nicholas Allen says:

    Hi — thanks for this patch, it’s been extremely helpful with advancing a disconnected WSUS environment. One thing that was important to note for me was that your WSUS environment may be working with remote SQL even if the registry is not configured correctly.
    We’ve got Remote SQL in place, but the "wYukonInstalled" value was still set to "1" (true) — despite no Windows Internal Database being installed on the box anymore. With that flag set to True, the patch looks for the WID to update and fails otherwise. Toggling
    the value to "0" allowed the patch to be installed successfully. Thought this might be useful for people with remote SQL installations 🙂

  22. Anonymous says:

    How to install windows updates in isolated environments? (WSUS or Configuration Manager)

    All required

  23. Gary says:

    Hi, I am having a problem since installing this update on my disconnected WSUS setup.

    Everything seems fine when doing the export, I get a xml.gz file which contains a 2.5gb XML file. when I try to import it I get the ‘Fatal Error: unable to uncompress package’. When I check the logfile there is no more information than that. Right now I am
    trying to import just the xml after extracting it using 7-zip…

    Any other ideas in case this doesn’t work?

  24. Gary says:

    Sadly that didn’t work either I still get the same error message.

    Need some help with this!

  25. SA says:

    I have Windows Server 2008 R2 update kb2828185 installed and .net 4.5.1. I’m still having issues with 4 GB. is the syntax the same? if not what is the new syntax?

  26. Rudresh says:

    Hi,

    I have done a few exports after applying the hotfix and tried to check the xml file. Seems like < is replaced by < and > is replaced by >. Wrote and application to replace the characters and the size of that xml reduced drastically. Is it possible to put that
    brackets ( since the xmls would usually have <> angled braces and not the representations there of ).

    Currently I am not able to export the file as it exceeds 4 GB and trying to install .NET 4 as suggested here. But still if we can replace those characters with braces it will surely take care of the size issue in my humble opinion.

    thanks & regards,
    ~ Rudresh

  27. Rudresh says:

    Please read :
    as Seems like < is replaced by "<" and > is replaced by ">" in my previous comment.

    thanks & regards
    ~ Rudresh

  28. Rudresh says:

    Again, the actual characters replaced.
    I meant that in the exported xml after the hotfix, the angled brackets are replaced by "&LESSTHAN (_<_)" and "&GREATERTHAN (_>_)" .Wrote and application to replace them and the xml size reduced drastically. If we can do something about that, lot of issues related
    with size will be taken care of IMHO.
    Sorry for too many comments back to back.

    thanks & regards
    ~ Rudresh

  29. sa says:

    rudresh seems no one replies back to my question and maybe you could help me. I have .net 4.5.1 and the hotfix installed but am still having issue with the 4 gb limitation. Is there a new syntax when doing the export of metadata?

    best regards,
    SA

  30. Rudresh says:

    Hi SA,

    Seems like it has something to do with your %TEMP% size also. If you can cleanup that directory , then exporting might work. Please let me know. Good luck.

    thanks & regards,
    ~ Rudresh

  31. SA says:

    Rudresh for replying. Tried cleaning using the wsus server cleanup wizard and the temp folder is clean.
    I have 2008 r2 sp1 with 4.5.1 and the hotfix. Should I be able to export metadata > 4GB?

    best regards,
    sa

  32. Rudresh says:

    Hi SA,

    I have hit the 4 GB limit. I am also now trying to install .net 4.0 / .NET 4.5 and see for myself.

    It should be able to generate the metadata.
    Will keep you posted.

    thanks & regards,
    ~ Rudresh

  33. SA says:

    Rudresh Let me know if you are able to solve it. I’ve been trying to get the answer and it seems there isn’t any aside from upgrading to w2012 with wsus 6. According to Ben they have fix it. I have the patch and the hotfix installed. So maybe i’m missing
    a step? Yeah let me know..

    v/r,
    SA

  34. Rudresh says:

    Hi SA,

    I have tried various .net versions ( 4 and 4.5 ) but still no luck in getting the metadata exported. I am posting my question in wsus forum. May be someone might have some idea.

    thanks & regards,
    ~ Rudresh

  35. John says:

    I’ve been dealing with this for more than a year now. I build a WSUS server, it will export the first time. Then never again. But now it will not export the first time after a new build. Whats very puzzling is that I pulled all the updates from the upstream
    server, let them completely download, then ONLY approved the CRITICAL and Security updates for December 2014, and only for the operating systems for the disconnected network. The WSUSCONTENT is only 2.2GB!! I’ve cleaned the database, and run the cleanup script
    on the SQL Database. I tried leaving the unapproved updates alone, and DECLINING them. The export always fails with the 4GB error. In the log it says the same thing. There has to be an explanation. Going to Server 2012 is NOT an option.

  36. John says:

    So before anyone asks, it’s Server 2008 R2 SP1, I have the KB2828185 installed. I’ve changed the environment variable so TEMP goes to a somewhere else since we have some restrictions on running from TMP/TEMP. I have .Net 4.5.1. All available updates have
    been applied from the primary WSUS. I’ve given the NETWORK SERVICE the necessary permissions.

  37. SA says:

    I guess we’re all having the same issue. Notice that even Ben who started out by saying Problem Solved don’t even give clear instructions on how to go about. Maybe this is just a big hype and there is no solution.

  38. KM says:

    I was running into the same issue (server 2008 reached the 4gb limit). I can’t take credit for this fix, but I did test it and it worked for me.
    1) Open the "update servicestools" folder and create a new file called "wsusutil.exe.config".

    2) Paste the following into this new config file. This will force wsusutil to use .NET4.

    3) Try the export command again.

    4) Once done, delete the config file until next time.

  39. KM says:

    Won’t let me post the code…should have known

  40. One last try says:

    configuration (put <> around it)
    startup (put <> around it )
    supportedRuntime version="v4.0.30319" (< /> around it)
    startup ( around it)
    configuration ( around it)

  41. SA says:

    KM and One last try thank you!!!!!.
    I went as far as removing and reinstalling wsus with the same result.

  42. John says:

    That doesn’t work for me. Still says it’s more than 4GB. Mind you, This is new server with only Decembers Critical and security updates approved. Everything else has been declined. the bottom startup and configuration, should they be enclosed in parenthesis
    () or brackets <> or < /> ??

  43. KM says:

    The last two lines should start with a less than sign followed by a forward slash / and they should end with a greater than sign.

  44. John says:

    KM, you da man!!
    it took me some time to decode the XML syntax. After restarting the server I was getting an immediate "side by side XML configuration" error. Thank you!!

  45. Rudresh says:

    Hi KM ,

    These .config files are for c# applications. I tried putting those changes you suggested and it didnt work. wsusutil.exe doesnt seem to be a c# app rather appears to be a c/c++ app.

    thanks & regards,
    ~ Rudresh

  46. Rudresh says:

    Hi,

    I am trying in MS wsus forum to see if some one was able to generate metadata.
    ( Forums > Windows Home Server > Windows Home Server Software )
    Title : wsusutil.exe fails to create metadata even after hotfix is applied.

    thanks & regards,
    ~ Rudresh

  47. KM says:

    I’m not sure what else to tell you. I followed the steps exactly as posted and I was able to generate metadata. When I unzipped the xml.gz file after the export ran, the XML file was 4.3GB, so I know it worked. I gave the fix to a few other people I work
    with and it worked for them too. Maybe something isn’t translating well since the forum won’t let me post it the way it needs to be posted.

  48. PJ says:

    Go to C:WindowsMicrosoft.NETFramework64 and check the framework version listed. The value listed in the config file needs to match the folder name exactly. i.e. supportedRuntime version="[FolderName]"

    C:WindowsMicrosoft.NETFramework64v4.0.30319 needs the following supportedRuntime version="v4.0.30319"

    Filename: wsusutil.exe.config

    ********

    ********
    https://social.technet.microsoft.com/Forums/windowsserver/en-US/8d97a976-2851-48f5-a594-3685d69fe5a6/gzip-stream-issue?forum=winserverwsus

  49. John says:

    I just ran the fix against the new MS updates this morning, and worked like a charm. PJ, maybe a dumb question, do you have KB2828185 installed? Also the fix, I had to put a space at the beginning of lines 2 and 4 before <, and 2 spaces in front of line
    3

  50. John says:

    OK, next problem. While I can get the updates to EXPORT successfully, now when I run the IMPORT on the disconnected WSUS, I’m getting the same 4GB error (IMPORT failed….). I tried using the config file, but no luck. The servers are identical, Server
    2008 R2. I verified the .NET versions were the same as well. When I performed this process earlier this month, I didn’t get the error. Any suggestions?

  51. John says:

    The error I get when importing, with this config file in place is "side by side is incorrect". If I remove the file, the IMPORT fails.

  52. PJ says:

    Sorry, I’ve been tied up with deploying the latest JRE updates. What a pain Java is!

    I haven’t done any imports, we just make them available to other departments with standalone networks. Haven’t heard of any issues importing.

    Yes we do have KB2828185 installed.

  53. MPW says:

    Export worked like a charm using the wsusutil.exe.config

    Make sure you put it in the same folder with WSUSUTIL and that you match your folder described above in PJ’s post to the supportedRuntime version

  54. MPW says:

    wsusutil.exe.config

    _configuration_
    _startup_
    _supportedRuntime version="v4.0.30319"/_
    _/startup_
    _/configuration_

    Replace left side underscores with less than characters and right side underscores with greater than characters. Hope this posts – no preview button 🙁

  55. Ravi R says:

    Hi Expers,

    In my case, I am planning to upgrade WSUS form 2008 R2 to 2012 R3 & I have applied KB2828185 patch on my Upstream server.

    But when I tried to export Metata, it got failed in between and I got a Fatal Error : The gzip stream can’t contain more than 4GB data.

    But actual export file size is only 550 MB.

    .Net 4.0 is installed on my upstream server
    Please suggest on this error.

  56. Thankfull says:

    Thank you KM!!!
    You saved me! KB2828185 + wsusutil.exe.config did the trick.

  57. Thankfull says:

    Thank you KM!!!
    You saved me! KB2828185 + wsusutil.exe.config did the trick.

  58. Peta says:

    It would seem that the WSUS team can’t export to a 64bit system. They are stuck in XP mode.