Support Tip: ConfigMgr 2012 update scan fails and causes incorrect compliance status


~ Larry Mosley | Senior Escalation Engineer

FIX

Update The hotfix for this issue has released under KB 3112343. You can find the KB here: https://support.microsoft.com/en-us/kb/3112343

=====

Hi everyone, we are seeing a few cases where a System Center 2012 R2 Configuration Manager update scan is failing so I wanted to mention it here in case any of you happened to run across it. Typically the scenario is that a ConfigMgr 2012 R2 client is requesting an update scan but the Windows Update Agent on 32-bit Windows 7 computers fails to return the scan results to Configuration Manager. This causes the Configuration Manager client to report incorrect compliance status as a result. It is further evident that the updates fail to install on the Windows 7 32-bit clients when ConfigMgr requests the update cycle, however if you use the Windows Update control panel applet the updates will usually install.

You will also notice a message similar to the following in WindowsUpdate.log:

WARNING: ISusInternal::GetUpdateMetadata2 failed, hr=8007000E

At its core this is a memory allocation issue, thus 64-bit Windows 7 computers will not see this error because the address space is effectively unlimited. They will, however, exhibit high memory and high CPU usage, possibly affecting performance. Note that x86 clients will also exhibit high memory usage (around 1.2-1.4GB).

We have also had reports of this affecting System Center Endpoint Protection updates deployed by Configuration Manager as well as a few reports of Windows 8 x86 clients being affected.

A hotfix for the Windows Update Agent is currently in development. The update will change how the update metadata is loaded into memory. Pending the results of final testing, this fix should be available late in the 2nd quarter of CY2015. This post will be updated with more information as it becomes available.

Root Cause

Because Configuration Manager does compliance reporting, it has extremely broad scan criteria to determine what is applicable. This is different from a typical scan performed by the Windows Update agent which requires a smaller subset of criteria. If an update is on WSUS and in a non-declined state, the Configuration Manager scan criteria will cause the Windows Update agent to try to evaluate it.

Windows x86 clients have a much smaller addressable memory range than 64-bit clients. With enough updates, the data returned to Configuration Manager is large enough that RPC can’t allocate the shared section used to return the data. This is because overall memory usage of the svchost.exe process hosting the Windows Update Agent process is extremely high.

The result is that there is either not enough memory available to satisfy the request or there is not a large enough contiguous block of free memory to satisfy the request.

Workarounds

Please note that the workarounds listed below may not resolve the issue in every environment, thus It’s possible that some machines or applications will continue to have problems even after these workarounds have been implemented. It is important to test these workarounds in a testing or lab environment before deploying into production.

1. Move wuauserv (Windows Update Agent) to its own SVCHost.exe instance by completing the following:

a. On the client, open a Command Prompt and run sc config wuauserv type= own
b. Stop and then start wuauserv.

2. Decline any unnecessary updates on the WSUS server. This may avoid the problem because declined updates do not get offered to clients during scans. Unneeded updates include superseded updates, updates for products and/or classifications that are not present in the client environment, as well as expired updates. You can manually decline the updates within the WSUS console or use the script documented in the Cleaning up WSUS with a Scriptsection below.

NOTE Always backup the WSUS database (SUSDB) prior to performing any changes such as those described here.

After declining unneeded updates, re-index the susdb, then run the WSUS Server Cleanup Wizard to remove unneeded updates as appropriate. Note that this will remove the updates from any Configuration Manager software update groups of which it is part.

3. Set user VA to 3072 MB. This is done by running bcdedit /set IncreaseUserVA 3072at a Command Prompt on the client. This will free up another 1GB of memory in the user address space. Note that this requires a restart of the client computer.

Please be aware that when attempting to address this problem it may be necessary to use all of the steps above. Option 1 should be strongly considered because wuauserv (the Windows Update agent service) is in a shared svchost process and stopping wuauserv does not unload the svchost process. If the failures are due to memory fragmentation, the memory of the svchost process will remain fragmented because wuauserv is reloaded into that instance.

Cleaning up WSUS with a Script

We have provided an example of a script that can be used to clean up WSUS that will allow scripted declining of superseded updates in your WSUS environment. Updates need to be declined at the top-level WSUS instance and replicated to downstream WSUS instances that are configured for replica mode. You will need to run the script on any WSUS instance running in Autonomous mode.  To use the script you must rename it to Decline-SupersededUpdates.ps1. As always, it is important to test this script in a lab environment before deploying into production.

image

If there are too many updates in WSUS, the script may fail to get the updates and time out. The exact number of updates will vary greatly depending on many environmental variables (e.g. the number of operating systems present, the number of versions of Office deployed, the number of Internet Explorer versions, etc.). In the event that the script times out, you will have to resort to manually declining updates from the WSUS console. Instructions to manually decline updates are included in the section titled Cleaning up WSUS from the WSUS console below.

NOTE The default WSUS server port is 80, but if you have installed WSUS to a custom IIS site it is probably using a different port. You will need to determine what port WSUS is using and change the –Portparameter in the examples below to that port.

IMPORTANT Make sure to take a backup of the susdb before declining the updates!

The argument –DeclineLastLevelOnly declines only those updates that do notsupersede any other update. If you omit this argument, any update that is superseded will be declined, leaving only updates that are not superseded in a state other than ‘declined.’

First, run the script with the -SkipDeclineswitch to see how many superseded updates are in WSUS. For example, to do a test run against WSUS Server without SSL you would use the following command:

Decline-SupersededUpdates.ps1 -UpdateServer SERVERNAME -Port 80 -SkipDecline

Next, you can decline only the updates that are superseded but do not supersede updates (leaf-level updates):

Decline-SupersededUpdates.ps1 -UpdateServer SERVERNAME -Port 80 -DeclineLastLevelOnly

This next command will decline allsuperseded updates:

Decline-SupersededUpdates.ps1 -UpdateServer SERVERNAME -UseSSL -Port 8080

Cleaning up WSUS from the WSUS console

If you have to (or want to) decline updates manually, the WSUS console has an icon that will tell you if an update is superseded.

1. Open the Windows Update Services MMC.

2. Select the All Updates View : Set the display to show the Approval status of 'Any except Declined' with a Status of 'Any'. Click Refresh.

3. Display the Supersedence Column : Right-click the column headers and select Supersedence.

4. Sort by Supersedence : Left-click on the Supersede Column.

5. Select and decline the superseded updates.

The updates to be declined have one of two particular flowchart symbols for their updates as shown below. Select the correct updates and decline them by either right-clicking the selected updates and clicking Decline, or by pressing the Decline button in the action pane.

IMPORTANT You must select only the updates that have one of the two icons below. There are a total of 3 icons that may appear here and you are to select the following 2 only.

image- This means the update is superseded by another update and it supersedes another update.

image

- This means this update is superseded by another update. Below is a snapshot that shows updates that have been superseded by other updates.

image

As mentioned earlier, a hotfix for this issue is currently in development and is scheduled to be available late in the 2nd quarter of CY2015. This post will be updated with more information as it becomes available.

Larry Mosley | Senior Escalation Engineer | Microsoft GBS Management and Security Division

Get the latest System Center news on Facebook and Twitter:

clip_image001 clip_image002

System Center All Up: http://blogs.technet.com/b/systemcenter/

Configuration Manager Support Team blog: http://blogs.technet.com/configurationmgr/ 
Data Protection Manager Team blog: http://blogs.technet.com/dpm/ 
Orchestrator Support Team blog: http://blogs.technet.com/b/orchestrator/ 
Operations Manager Team blog: http://blogs.technet.com/momteam/ 
Service Manager Team blog: http://blogs.technet.com/b/servicemanager 
Virtual Machine Manager Team blog: http://blogs.technet.com/scvmm

Microsoft Intune: http://blogs.technet.com/b/microsoftintune/
WSUS Support Team blog: http://blogs.technet.com/sus/
The RMS blog: http://blogs.technet.com/b/rms/
App-V Team blog: http://blogs.technet.com/appv/
MED-V Team blog: http://blogs.technet.com/medv/
Server App-V Team blog: http://blogs.technet.com/b/serverappv
The Surface Team blog: http://blogs.technet.com/b/surface/
The Application Proxy blog: http://blogs.technet.com/b/applicationproxyblog/

The Forefront Endpoint Protection blog : http://blogs.technet.com/b/clientsecurity/
The Forefront Identity Manager blog : http://blogs.msdn.com/b/ms-identity-support/
The Forefront TMG blog: http://blogs.technet.com/b/isablog/
The Forefront UAG blog: http://blogs.technet.com/b/edgeaccessblog/

Decline-SupersededUpdates.txt


Comments (57)
  1. Anonymous says:

    Hello together, the Update seems now to be released:
    https://support.microsoft.com/en-us/kb/3050265

  2. Anonymous says:

    The issues with the script should now be fixed – sorry about the inconvenience.

  3. Anonymous says:

    I found my first 64bit desktop suffering from this problem while trying to do a test deployment of this months patches.

    Even though the post says they are not affected the symptoms were identical.

    This was on a computer with 8GB memory however it hadn’t been rebooted for a month and had VMs and a lot of apps running as well. I closed everything down and tried another Software Update Scan Cycle and still got the "WARNING: ISusInternal::GetUpdateMetadata2
    failed, hr=8007000E" error.

    I rebooted and tried the Scan again and it completed fine and I was able to apply the patches.

    The Blog post says it fails if "not a large enough contiguous block of free memory to satisfy the request" so I guess in theory that could apply 64bit as well?

    We are in a SCCM 2007 R3 environment but I guess the WSUS component is the same.

  4. Anonymous says:

    Hi Larry,
    We have WSUS installed on SCCM 2012 site server. Please suggest if we need to install KB3050265 on WSUS server along with Win 7 clients. Or we just need to install hardening hotfix 2938066 on WSUS server?

    Regards
    Dalip

  5. Great post. Thank you.

  6. Anonymous says:

    Any update on a potential hotfix patch?

  7. Anonymous says:

    Is there any further update on the availability of the hot-fix? We have 3,000+ machines, mainly Win 7 64-bit that are managed with SCCM 2012 R2 that are experiencing high CPU and high disk I/O which makes the machines unusable during scans.

  8. Justin K says:

    Interesting bug. Question: Must a user go through WSUS and decline the update or is it enough to remove it from SUGs (I'm guessing no because it may still be in the manifest)? I ask because I use an Orchestrator runbook to clean superseded updates out
    of SUGs, but in hindsight it seems targeting the WSUS server directly may be a better model.

  9. Steve says:

    Glad to see this post. Too bad I get HTTP 404 when trying to download script and view pictures.

  10. Henri says:

    Thank you for the script – it is quite useful. I'm looking forward to the hotfix, as this issue has been a bit of a pain to resolve.

    Please note that the script (at least the one I downloaded) looks to be missing an " character at the end of the script.
    Write-Host "
    ->
    Write-Host ""

  11. Larry Mosley says:

    Justin K,
    you have to decline the update in WSUS. The Windows Update agent doesn't know anything about SCCM or SUGs and doesn't ever talk to SCCM. It only communicates with WSUS (or Microsoft Updates of course), and that is where it gets metadata for updates.

  12. anony.muos says:

    Please update the broken links to the graphics. The statement "You must select only the updates that have one of the two icons below"… is not helpful when we cannot see the icons referred to.

    Thank you.

  13. André Boutet says:

    Hi JC, can we use powershell « Invoke-WSUSserverCleanup -DeclineSupersededUpdates » instead of the script ?

  14. MM says:

    Please check out the important pictures at the bottom of the article. I can’t see them (File or directory not found).
    Thank you.

  15. ian_pick says:

    Any updates on the release date?

  16. Joe says:

    I downloaded the script today and it was failing to run. Henri pointed out that it was missing an " on the last line, so I immediately checked that. Now it has too many ".

    At C:supportwsusDecline-SupersededUpdates.ps1:203 char:12
    + Write-Host """
    + ~~~
    The string is missing the terminator: ".
    + CategoryInfo : ParserError: (:) [], ParseException
    + FullyQualifiedErrorId : TerminatorExpectedAtEndOfString

    On the very last line, I just removed one of the quotes — it had three of them.

  17. steelie34 says:

    Mr. Mosley, I wonder if you could chime in on this technet forum post:

    https://social.technet.microsoft.com/Forums/windows/en-US/4a782e40-bbd8-40b7-869d-68e3dfd1a5b4/windows-update-scan-high-memory-usage?forum=w7itproperf

    This has become a very hot topic, and has linked to this blog post for a possible upcoming hotfix addressing the issue. However, this blog post is specifically scoped towards SCCM, but the forum post is pertaining to the WU client itself. Can you provide any
    detail on whether or not this will apply to all Win7 clients? TIA.

  18. Adam G. says:

    same thing here with 7 ENT x64 windows update service brings computers to a stand still. will there be a hotfix for this?

  19. ryan says:

    Is there a better ETA available on this patch yet, besides 2Q CY15?

  20. steelie34 says:

    I’m interested in how this will impact the conversation currently taking place over here:

    https://social.technet.microsoft.com/Forums/windows/en-US/4a782e40-bbd8-40b7-869d-68e3dfd1a5b4/windows-update-scan-high-memory-usage?forum=w7itproperf

    There is a significant number of users who are experiencing issues with wuaserv.exe process, and we are hoping this update will address the issue. Can anyone comment on the relevance of this planned update in regards to the mentioned Technet thread?

    Thanks

  21. Matt says:

    Also a huge problem for us. Our machines are hideously out of date as we are only just implementing a WSUS server but it renders machines useless whilst checking for updates (100% svchost | Memory).

    Have had to block all machines from contacting the WSUS server until this is sorted. The couple of 32-bit machines I’ve seen have not experienced this issue.

  22. Lourens N says:

    Any update on this. I’ve got avout 4000+ workstations with this problem. Seems like a big problem just to keep quiet about it.

  23. Mr. Anderson says:

    Does this also occur with systems that do not use SCCM?

  24. Tim says:

    We are running Server 2012 R2 Essentials with approx 10 clients, all Win 7 x64 with latest patches installed. All clients are experiencing this issue and it is bringing our computers to a crawl. The Server dashboard list all the pc’s as being up to date
    and the issue continues throughout the entire day.. Please fix this soon.

  25. ian_pick says:

    Any updates on the patch?

  26. Alan Dooley says:

    I’ve got about 5,000 machines suffering from this issue, most are x64 with 4GB but it still cripples them for about 20 minutes each time a scan runs. I’ve run through option 2 as a workaround and am awaiting feedback. A hotfix is needed ASAP.

  27. Andy Perkins says:

    In my environment of approx 10,000 x64 machines workaround 2 has taken the svchost.exe memory usage down to approx 1GB which is better than the 1.8GB it was hitting but this is still enough for the machine to start paging. We run SCEP and as workstations
    typically get 2 updates a day we are seeing this at least twice a day on every machine. Not keen to deploy option 1 to all machines.

  28. Mat says:

    Hello, does the System Center 2012 R2 Configuration Manager SP1 address any of the issues found?

  29. søren says:

    when is hotfix available? I see +2300 machines with this error

  30. Tim Girard says:

    found another pc with this problem.. Its not joined to a domain and not connecting to a WSUS server, its Win 7 home premium x64 system, this issue is wide spread.. Fix the Windows update service already.. the only solution right now is to disable it completely.

  31. Lourens N says:

    I've logged a call with Microsoft as we're experiencing the problem as well. After our remote session I'll supply an update here in the comments.

  32. Dave W says:

    Is there anyupdate on this PLEASE

  33. RC says:

    I have the same Problem affecting about 1000 Clients. A ticket at MS is open but no solution so far. We cannot update our clients since 3 month. All Workarounds have been made but without success. I hope there will be a working solution soon.

  34. Bob Hyatt says:

    This should be a priority patch since it is preventing critical security updates from being installed.

  35. S B says:

    Has anyone noticed that this happens to Windows Server 2008 R2 environments as well?

    We have the exact same thing happen to our Windows 7 SP1 Enterprise clients being fed by a Server 2008 R2 WSUS server.

  36. jim says:

    Thanks Lourens N. we are also experiencing the same issue. I look forward to hearing your resolution.

  37. Leon g says:

    Any update on this?

  38. tom s. says:

    Does anybody have an updated status on this issue? i am experiencing it also

  39. Frank P says:

    So what is the status on this fix? We are having this issue.

  40. Geoff P says:

    Looks like a patch has been released to fix this issue will try this out and hopefully it resolve this issue.

    https://support.microsoft.com/en-us/kb/3050265

  41. Steve says:

    Windows Update Agent June 2015 hot fix released:
    https://support.microsoft.com/en-us/kb/3050265

  42. Mat says:

    KB3065987 is the fix

  43. 3050265 says:

    An update to fix the issue has been rolled out:

    support.microsoft.com/en-us/kb/3050265

    Thank you all for your patience.

  44. Shaiju Ashraf says:

    Hi All,
    Did anyone got a prompt solution after logging call with Microsoft? Please reply.
    Thanks

  45. Andrew Laurence says:

    Will this post be updated to show that the update has now been released?
    https://support.microsoft.com/en-us/kb/3050265

  46. florian says:

    The Update seems to be released:
    https://support.microsoft.com/en-us/kb/3050265

  47. Alan Dooley says:

    Looks like today’s update to Windows Update should resolve this…

    https://support.microsoft.com/en-us/kb/3050265

  48. Nathan T says:

    The PS script worked well in my test environment. I have a couple of basic questions:
    1. In SCCM 2007 SP2 SUP’s are not supposed to be set to replica mode. This means that the Decline Superseded Updates PS script will need to be run on every SUP, correct?
    2. I don’t recall where exactly I’ve seen them but there were statements made years ago about never managing software updates in the WSUS console when it is part of an SCCM 2007 environment. Is this no longer taboo? If not, should this script be part of a normal
    maintenance routine?

  49. Lourens N says:

    I can confirm from my side the update resolved the problem.

  50. Jair Real says:

    We seem to be experiencing a similar issue although there is no error message logged in WindowsUpdate.log. Background: 2 software updates were removed manually from a workstation that has the ConfigMgr client and it reports back Compliant although missing
    these updates. After many scans (even forced full scans) with no Compliance status sent back to supersede the old Compliant message, only an client uninstall>Delete client from SCCM database>reboot>client reinstall – then the client reported back correct compliance.
    I’m still wondering what happened here.

  51. DirkGent says:

    You need to install KB3050265 on the Windows 7 clients and it is wise to install them on the servers also as the memory footprint will be lower when scanning for updates.

  52. Vishal Jaitly says:

    I am facing the same issue with Windows 8.0 (x86) machines. When trying to same hotfix on machines it pops up with the error "Update is not Applicable". Any Solution for this?

  53. Amiel says:

    After installing the superseded July Hotfix KB3065987 to our problematic machines (approximately 12-15%) we have had some success, but still noticing failures – note that we have NOT installed KB2938066 on our SCCM server with the WSUS role. Is anyone
    else still experiencing this problem?

  54. Henrik says:

    We are seeing this issue again. Anyone else seeing this as well ?

  55. Henrik says:

    Yes, we are having the same issue again.

  56. Andre says:

    Same issue… Win7 32bit… Fix don't resolve this problem….

    1. Glen says:

      For those of us at organisations, that are still running legacy systems: Windows Server 2008 SP2 x86, (deep sigh) this same issue occurs. Sadly there is no hotfix for WS 2008 SP2, however ” bcdedit /set IncreaseUserVA 3072″ seems to have resolved the issue for the systems I have tested it on.

Comments are closed.

Skip to main content