High CPU/High Memory in WSUS following Update Tuesdays


Updated 10/11/2017 - updated hotfix information.

Recently, we’ve seen an increase in the number of high CPU/High Memory usage problems with WSUS, including WSUS in a System Center Configuration Manager environment – these have mostly corresponded with Update Tuesdays.

Microsoft support has determined that the issue is driven primarily by the Windows 10 1607 updates, for example KB4022723, KB4022715, KB4025339, etc. See here for the list of Windows 10 1607 updates.

These updates have large metadata payloads for the dependent (child) packages because they roll up a large number of binaries. Windows 10, versions 1507 (Windows 10 RTM) and 1511 updates can also cause this, though to a lesser extent.  Windows 10, version 1703 is still recent enough that the metadata is not that large yet (but will continue to grow).

Symptom

The symptoms include

  • High CPU on your WSUS server - 70-100% CPU in w3wp.exe hosting WsusPool
  • High memory in the w3wp.exe process hosting the WsusPool – customers have reported memory usage approach 24GB
  • Constant recycling of the W3wp.exe hosting the WsusPool (identifiable by the PID changing)
  • Clients failing to scan with 8024401c (timeout) errors in the WindowsUpdate.log
  • Mostly 500 errors for the /ClientWebService/Client.asmx requests in the IIS logs

Cause

Microsoft support has determined that the issue is driven primarily by the Windows 10 1607 updates, for example KB4022723, KB4022715, KB4025339, etc. See here for the list of Windows 10 1607 updates.

These updates have large metadata payloads for the dependent (child) packages because they roll up a large number of binaries. Windows 10, versions 1507 (Windows 10 RTM) and 1511 updates can also cause this, though to a lesser extent. Windows 10, version 1703 is still recent enough that the metadata is not that large yet (but will continue to grow).

How to determine if the 1607 Updates are the cause

To determine if WSUS is affected by this problem, decline the Windows 10 updates (including the latest cumulative update). If both CPU and memory quickly drop back to normal, then the issue is likely the result of metadata size from the Windows 10 updates. They can be reapproved after you have determined if the updates are causing this issue, assuming you want to deploy them.

If declining the Windows 10 updates does not help, then the problem may be due to too many superseded updates in the WSUS server. Take the steps outlined in The Complete Guide to Microsoft WSUS and Configuration Manager SUP maintenance to decline the superseded updates. If, after doing this you are still having problems, read on.

This blog post may help alleviate some of these problems, but is not a magic bullet. After these changes are made, you will still see high CPU and memory until the system stabilizes as I explain further down.

WSUS Caching

WSUS has a caching mechanism whereby the first-time update metadata is requested by any client WSUS will store it in memory. Further requests for the same update revision will retrieve the update metadata from memory instead of reading it from the database. Some of the metadata in the database is compressed, so not only must it be retrieved, it must be decompressed into memory, which is an expensive operation.

You can monitor the current number of updates stored in the cache via Performance Monitor with the counter WSUS: Client Web Service/Cache size and instance spgetcorexml. Keep in mind that this counter provides the number of cached items, not the amount of memory consumed by cached metadata. w3wp.exe process memory can be used as a proxy for the amount of space consumed by the metadata cache.

The Problem

For large metadata packages and many simultaneous requests, it can take longer than ASP.NET’s default timeout of 110 seconds to retrieve all of the metadata the client needs. When the timeout is hit, ASP.NET disconnects the client and aborts the thread doing the metadata retrieval. If you look at Program Files\Update Services\LogFiles\SoftwareDistribution.log, the abort looks like this:

System.Threading.ThreadAbortException: Thread was being aborted.
   at System.Buffer.__Memcpy(Byte* dest, Byte* src, Int32 len) 
   at System.Buffer._Memcpy(Byte* dest, Byte* src, Int32 len)  
   at System.Buffer.Memcpy(Byte* dest, Byte* src, Int32 len)  
   at System.String.CtorCharPtrStartLength(Char* ptr, Int32 startIndex, Int32 length)    
   at Microsoft.UpdateServices.Internal.CabUtilities.ExpandMemoryCabToString(Byte[] src)   
   at Microsoft.UpdateServices.Internal.DataAccess.ExecuteSpGetCoreUpdateXml(Int32[] revisionIds) 
   at Microsoft.UpdateServices.Internal.DataAccessCache.GetCoreUpdateXml(Int32[] revisionIds, DataAccess da, Int64 maxXmlPerRequest)
   at Microsoft.UpdateServices.Internal.ClientImplementation.GetSyncInfo(Version clientProtocolVersion, DataAccess dataAccess, Hashtable stateTable, Hashtable deploymentTable, Boolean haveGroupsChanged, Boolean driverSyncNeeded, Boolean doChunking)
   at Microsoft.UpdateServices.Internal.ClientImplementation.SoftwareSync(DataAccess dataAccess, UnencryptedCookieData cookieData, Int32[] installedNonLeafUpdateIds, Int32[] leafUpdateIds, Boolean haveGroupsChanged, Boolean expressQuery, Guid[] filterCategoryIds, Boolean needTwoGroupOutOfScopeUpdates)
   at Microsoft.UpdateServices.Internal.ClientImplementation.SyncUpdates(Cookie cookie, SyncUpdateParameters parameters)
   at Microsoft.UpdateServices.Internal.ClientImplementation.SyncUpdates(Cookie cookie, SyncUpdateParameters parameters)

Note: What you are looking for is a ThreadAbortException with ExecuteSpGetCoreUpdateXml on the stack (ThreadAbortExceptions could happen for other reasons as well – we are concerned with this specific scenario).

When the thread abort happens, all of the metadata that has been retrieved to that point is discarded and is not cached. As a result, WSUS enters a continuous cycle where the data isn’t cached, the clients can never complete the scan and continue to rescan.

Another issue that can occur is the WSUS application pool keeps recycling because it exceeds the private memory threshold (which it is very likely to do if the limit is still the default of 1843200). This recycles the app pool, and thus the cached updates, and forces WSUS to go back through retrieving updates from the database and caching them.

Solution

A WSUS update is now available that includes improvements for update metadata processing. This update should be applied to all WSUS servers in your environment.

Windows Server 2016 (KB4039396)

Windows Server 2012 R2 (KB4041693)

Windows Server 2012 (KB4041690)

WSUS 3.0 SP2 (KB4039929)

In addition to applying the applicable update(s) noted above, it is recommended that routine maintenance of WSUS be performed. See The Complete Guide to Microsoft WSUS and Configuration Manager SUP maintenance for more info.

If you still occasionally experience thread abort exceptions, you can increase ASP.NET's default timeout.

Increase the ASP.NET timeout

  • Make a copy of \Program Files\Update Services\WebServices\ClientWebService\Web.Config.
  • Open \Program Files\Update Services\WebServices\ClientWebService\Web.Config.
  • Find the element “<httpRunTime”. It will look like this (in an unmodified web.config):
<httpRuntime maxRequestLength="4096" />
  • Modify httpRunTime by adding an executionTimeout attribute:
<httpRuntime maxRequestLength="4096" executionTimeout="3600" />
  • Save the web.config to a different location and copy the modified one into the directory.
  • From an elevated command prompt, run IISReset to restart IIS.

Monitoring WSUS Metadata Caching

Open Windows Performance monitor and add the following counters

  • WSUS: Client Web Service | Cache Size counter for spgetcorexml instance.
  • Process | Private Memory counters.
    • If there is more than one w3wp.exe, add them all – the one with the highest memory usage is probably the WSUSPool, but you can also add Process | ID Process to determine which worker process should be monitored.

Monitor the cache size counter – it should increase and eventually reach a peak value that does not change. This indicates all metadata that clients need is cached. It can take several hours for this to stabilize, so be patient.


Comments (32)
  1. Jason says:

    What’s the real fix? Design changes? Smaller updates? This issue had us running in circles especially after NotPetya!

    1. Hi Jason,
      we are working on a binary fix for this, which should be releasing ‘soon.’ We will update the blog post with the update package once it is released.

  2. Matthias Gysin says:

    Thanks. In my case the issue was not solved applying this configuration. I extended to 48 CPUs and 128 GB RAM. Furthermore I have a primary Site with ETA 45 Secondary Sites with SUP. Would be great when a hotfix is relased which can be applied using the console update. It is not practicable to do this per Server. And as I said ,,, issue not solved.

    1. rza49311 says:

      +1 Recommended steps DO NOT WORK.

  3. shimbesh says:

    Hi,

    I have tried all steps suggested in article however still w3wp.exe utilizing high CPU. it comes down when i stop wsuspool. it is causing server failure and delay in servicing client machines.
    MS must put more effort on this issue and release hotfix asap. thanks.

    1. Michael Nielsen says:

      Same here, does not work! I dont understand, that there isn’t a better testing from MS before sending updates out to millions of companies?
      MS should work 24/7 to provide a hotfix for this!

  4. We’re not seeing this resolved. We don’t see high CPU and RAM. Our clients just don’t download the updates, just sit there at 0 percent. This is only recent and we’re a little over the monthly issues we see with updates, you guys suck at testing things!

    1. Out clients seeing this issue are 1607. We have some 1511 clients but haven’t checked them yet.

      1. _MelQ says:

        With regard to client update downloads sitting at 0%, this could be completely unrelated to what you are seeing, but a month ago we had turned on “Enable installation of Express installation files on clients” (Administration / Site Configuration / Client Settings Properties then Software Updates on the left and see setting on right). August updates would not get past “Download 0%” on any of our clients until we turned that back off. We have not yet figured out why having that feature turned on causes that problem for us. A few others commenting here: reported the same.

  5. SimonT says:

    On my side, my issue is with Windows Server 2016. None of the servers who I install August 2016 Cumulative Update will report to WSUS anymore. And they all report 8024401c. But my Windows 10 device has no issue. I have a very small WSUS setup of about 100 workstations/servers.

    I’ll try your solution in my environment tomorrow.

    1. Hi Simon,
      it appears you have a different problem than what is described here. Please open a support case so we can troubleshoot it and get to the bottom of it.

  6. For those that this isn’t working for, please open a support case with us (Microsoft).

  7. RowdyRandale says:

    That worked for me!
    Especially changing httpRuntime maxRequestLength and executionTimeout.

    One has to be really Patient after IISReset and force some clients to contact WSUS so that cache is rebuild. After cache size is stable it is working!

    Thank you!

  8. Lochness says:

    Jarret, I also tried your above solutions and they don’t work. What is the status of this?

  9. Michael says:

    In my case I changed already this settings but the issue with the high CPU still occurs. What helped in my case:

    1. Re-index SUSDB, run WSUS Cleanup Wizard, Decline Superseded updates
    https://blogs.technet.microsoft.com/configurationmgr/2016/01/26/the-complete-guide-to-microsoft-wsus-and-configuration-manager-sup-maintenance/
    2. Uninstall SCUP and remove all content
    3. Perform a WSUS reset
    4. Perform a full sync
    5. Re-index SUSDB, run WSUS Cleanup Wizard, Decline Superseded updates

    If I install SCUP I have the same issue on the next Microsoft patchday. I hope Microsoft work hard on a real fix!

  10. RichardF says:

    The fixes above did NOT resolve the issue on my system. I have the same symptoms as SimonT in that all my Win10 clients can update successfully, but my Server 2016 cannot. They all report error 8024401c. Agree with the poster above that M$ should pull their fingers out and come up with a proper fix ASAP as this is a serious issue.

    1. Richard, you are having a different problem – please open a support case.

      1. Mikael says:

        I have the exact same problem on multiple Windows Server 2016 installations, after the latest updates are applied they can no longer talk to the WSUS server.

    2. Note that the cumulative updates in some cases are 1GB , increase the time in the tab Maximun Run Time on the KB from 5min to 20min and check again

  11. Matthias Gysin says:

    Question to Microsoft. We have a lot customer currently planned to implement CM 1707 CM. We cannot deliver due to on this issue. (Honestly we cannot provide a project and suggest customer do not update for the moment). Some of them are Missional Critical and have Premier Contracts. Is an internal Hotfix available which can be requested via CSS ? We have already customer escalations … Matthias.Gysin@softwareone.com

  12. I have updated this blog post with Windows Server and WSUS 3.0 SP2 hotfix information. Please review the KB articles for these updates.

    Thanks
    Yvette

    1. Robert Spinelli says:

      Yvette, does MS ever plan on hiring back a good QA team? How does something like this get missed? I know MS is focused on Intune and the cloud so they can get monthly fees from people, but the amount of hotfixes that have come out with bugs has been unacceptable.

      I’m assuming you have outsourced QA to India or there is no QA at all anymore? I know it’s not your fault; you’ve been around updating tech documents for almost as long as I have been doing SCCM and it would be great if you brought the frustration we feel back to management. Stop focusing on the cloud and support the products that corporations are currently using in the field.

    2. alexandr says:

      Yesterday I was changed my primary site according to
      IIS – Appools WsusPool
      change Private memory limit to 0
      Regular time out interval 0
      Ping Enable False
      Reset IIS

      and some other changes, you`ve deleted this information after added hotfix information?

      As I am understanding now, these updates making changes on IIS, instead of doing it manually, if not, according which article I have made changes on my primary site? )

      1. The hotfixes do not make these changes – it only updates susnativecommon.dll. The app pools should still be set to 0 private memory limit. regular time out 0, and ping enable false to prevent them from recycling.

        With the fix in place. the internal cache will build MUCH faster, so recycling isn’t as much of a concern, but you still don’t want it to happen.

        For reference, 6000 items will give a roughly 4-5GB stable-state size for w3wp once all the items are in cache; however, more memory is needed while the cache is being built – up to double or triple the steady-state size.

  13. JasonE says:

    In case you are looking for a solution for Server 2008 R2, use the WSUS 3.0 SP2 (KB4039929) that is posted here. This fixed our WSUS Server 2008 R2 server. Tried to limit WSUSPool in IIS under Application Pools to 20% CPU, but eventually had to just stop it to load the fix. Rebooted for good measure, then started WsusPool again. Was busy for an hour or so, but it was not at a constant 100% CPU utilization. Settled down and things are good.

  14. Ron says:

    Hi
    We also have the high CPU issue on a SCCM 2012 build 1702 environment. We are in the middle of a migration project so this issue is really annoying.
    The high CPU on the WSUS server starts as soon as we install the SCCM client on a Server 2016 machine. All other server versions don’t cause high CPU. I look in the wsuspool and there I see the IP address of the server I just installed. As soon as I shut down this server the CPU on the WSUS server drops down.
    I already followed the steps of the WSUS and configuration Manager SUP maintenance but this didn’t resolve anything.
    So the question is what can I do next, because a lot of solutions are mentioned above but a lot of people on this forum mention that it didn’t help.
    So when is Microsoft providing a solution.

    Regards,

    Ron

    1. Hello Ron

      Have you installed the applicable hotfix? If yes, then please open a case with our support team to assist you.

      Thanks
      Yvette

      1. Ron says:

        Hello Yvette
        Yesterday we installed the KB4039396 hotfix on the WSUS server.
        After the reboot the high CPU was gone on the WSUS server.
        Also the error 8024401c was gone on the server 2016 machine.
        So for the moment we have solved the issue.

        Thanks for the help.

        Regards,

        Ron.

  15. Russell says:

    We didn’t see this problem until today, even though we had already processed the August updates weeks ago. But the hotfix solved it immediately and the multiple w3wp.exe processes are using very little CPU when just one was sustaining 90%+ before the fix.

    Thanks!
    Russell

  16. Nick says:

    What if I’m having this problem on a 2008R2 server?

    1. Hello Nick

      If you have Windows Server 2008 R2, then I’m assuming that you are using WSUS 3.0 SP2. Please use the hotfix for that version https://support.microsoft.com/help/4039929.

      Thanks
      -Yvette

Comments are closed.

Skip to main content