Windows Server 2016 – WUSA Event ID 3: “The referenced assembly could not be found”

___________________________________________________________________________________________________________________________

IMPORTANT ANNOUNCEMENT FOR OUR READERS!

AskPFEPlat is in the process of a transformation to the new Core Infrastructure and Security TechCommunity, and will be moving by the end of March 2019 to our new home at https://aka.ms/CISTechComm (hosted at https://techcommunity.microsoft.com). Please bear with us while we are still under construction!

We will continue bringing you the same great content, from the same great contributors, on our new platform. Until then, you can access our new content on either https://aka.ms/askpfeplat as you do today, or at our new site https://aka.ms/CISTechComm. Please feel free to update your bookmarks accordingly!

Why are we doing this? Simple really; we are looking to expand our team internally in order to provide you even more great content, as well as take on a more proactive role in the future with our readers (more to come on that later)! Since our team encompasses many more roles than Premier Field Engineers these days, we felt it was also time we reflected that initial expansion.

If you have never visited the TechCommunity site, it can be found at https://techcommunity.microsoft.com. On the TechCommunity site, you will find numerous technical communities across many topics, which include discussion areas, along with blog content.

NOTE: In addition to the AskPFEPlat-to-Core Infrastructure and Security transformation, Premier Field Engineers from all technology areas will be working together to expand the TechCommunity site even further, joining together in the technology agnostic Premier Field Engineering TechCommunity (along with Core Infrastructure and Security), which can be found at https://aka.ms/PFETechComm!

As always, thank you for continuing to read the Core Infrastructure and Security (AskPFEPlat) blog, and we look forward to providing you more great content well into the future!

__________________________________________________________________________________________________________________________

 

Hi this is Michael Koeppl from the Support for Mission Critical team again. This time I wanted to walk you through a recent troubleshooting scenario I had at my customer.

14 out of 2150 Server 2016 systems reported and error when my customer tried to install the latest monthly rollup hotfix.

Normally, to be honest with you, we wouldn’t even bother as the number of systems is so low that it can almost fly under the radar. The Customer usually re-installs the machines in question but this time as all of them showed the same error they send me the list of affected machines and I started to investigate.

As I knew there’s an issue with Patch installation, I instantly turned to DISM an checked the CBS Store for any kind of corruption:

C:\>dism /online /cleanup-image /scanhealth

Deployment Image Servicing and Management tool

Version: 10.0.14393.0

Image Version: 10.0.14393.2457

[==========================100.0%==========================] No component store corruption detected.

The operation completed successfully.

Repair a Windows Image

https://docs.microsoft.com/de-de/windows-hardware/manufacture/desktop/repair-a-windows-image

Okay no joy, what next?

I manually tried to install the January rollup (KB4480977) as this is the one my customer tried to install as well. For a list of all available monthly rollup packages visit: https://support.microsoft.com/en-us/help/4000825 for S2016

Here’s the error message we were getting

Log Name:      Setup

Source:        Microsoft-Windows-WUSA

Date:          26.03.2019 12:47:41

Event ID:      3

Task Category: None

Level:         Error

Keywords:

User:          Domain\AdminUser

Computer:      MyPC.domain.de

Description:

Windows update “Update for Windows (KB4480977)” could not be installed because of error 2147956481 “The referenced assembly could not be found.” (Command line: “”C:\Windows\system32\wusa.exe” “C:\tmp\windows10.0-kb4480977-x64.msu” “)

Event Xml:

<Event xmlns=”http://schemas.microsoft.com/win/2004/08/events/event“>

  <System>

    <Provider Name=”Microsoft-Windows-WUSA” Guid=”{09608C12-C1DA-4104-A6FE-B959CF57560A}” />

   <EventID>3</EventID>

    <Version>0</Version>

    <Level>2</Level>

    <Task>0</Task>

    <Opcode>0</Opcode>

    <Keywords>0x8000000000000000</Keywords>

    <TimeCreated SystemTime=”2019-03-26T11:47:41.355709400Z” />

    <EventRecordID>109</EventRecordID>

    <Correlation />

    <Execution ProcessID=”4824″ ThreadID=”9044″ />

    <Channel>Setup</Channel>

    <Computer>MyPC.domain.de</Computer>

    <Security UserID=”S-1-5-21-4188125556-1267690402-888888790447-0000″ />

  </System>

  <EventData>

    <Data Name=”UpdateTitle”>”Update for Windows (KB4480977)”</Data>

    <Data Name=”ErrorCode”>2147956481</Data>

    <Data Name=”ErrorString”>The referenced assembly could not be found.</Data>

    <Data Name=”CommandLine”>”C:\Windows\system32\wusa.exe” “C:\tmp\windows10.0-kb4480977-x64.msu” </Data>

  </EventData>

</Event>

Okay, let’s have a look at the CBS.log file which can be found at \windows\logs\cbs

2019-03-26 12:40:12, Error                 CSI    0000001a@2019/3/26:11:40:12.185 (F) onecore\base\wcp\componentstore\csd_locking.cpp(200): Error STATUS_SXS_ASSEMBLY_MISSING originated in function CCSDirectTransaction::LockComponent expression: (null)

[gle=0x80004005]

2019-03-26 12:40:12, Info                  CBS    Added C:\Windows\Logs\CBS\CBS.log to WER report.

2019-03-26 12:40:12, Info                  CBS    Added C:\Windows\Logs\CBS\CbsPersist_20190326072401.log to WER report.

2019-03-26 12:40:12, Info                  CBS    Added C:\Windows\Logs\CBS\CbsPersist_20190324034222.log to WER report.

2019-03-26 12:40:12, Info                  CBS    Added C:\Windows\Logs\CBS\CbsPersist_20190324001230.log to WER report.

2019-03-26 12:40:12, Info                  CBS    Added C:\Windows\Logs\CBS\CbsPersist_20190323234238.log to WER report.

2019-03-26 12:40:12, Info                  CBS    Added C:\Windows\Logs\CBS\CbsPersist_20190323230433.log to WER report.

2019-03-26 12:40:12, Info                  CBS    Not able to add pending.xml to Windows Error Report. [HRESULT = 0x80070002 – ERROR_FILE_NOT_FOUND]

2019-03-26 12:40:12, Info                  CBS    Not able to add pending.xml.bad to Windows Error Report. [HRESULT = 0x80070002 – ERROR_FILE_NOT_FOUND]

2019-03-26 12:40:12, Info                  CBS    Not able to add poqexec.log to Windows Error Report. [HRESULT = 0x80070002 – ERROR_FILE_NOT_FOUND]

2019-03-26 12:40:12, Error                 CSI    0000001b (F) STATUS_SXS_ASSEMBLY_MISSING #1760302# from CCSDirectTransaction::OperateEnding at index 0 of 1 operations, disposition 2[gle=0xd015000c]

2019-03-26 12:40:12, Error                 CSI    0000001c (F) HRESULT_FROM_WIN32(ERROR_SXS_ASSEMBLY_MISSING) #1760150# from Windows::ServicingAPI::CCSITransaction::ICSITransaction_PinDeployment(Flags = 0, a = cd4484f3ed27edafc9d99e613bb62654, version 10.0.14393.2457, arch amd64, nonSxS, pkt {l:8 b:31bf3856ad364e35}, cb = (null), s = (null), rid = ‘Package_7763_for_KB4462917~31bf3856ad364e35~amd64~~10.0.1.7.4462917-16739_neutral’, rah = ‘3’, manpath = (null), catpath = (null), ed = 0, disp = 0)[gle=0x80073701]

2019-03-26 12:40:12, Info                  CBS    Failed to pin deployment while resolving Update: Package_7763_for_KB4462917~31bf3856ad364e35~amd64~~10.0.1.7.4462917-16739_neutral from file: (null) [HRESULT = 0x80073701 – ERROR_SXS_ASSEMBLY_MISSING]

2019-03-26 12:40:12, Info                  CBS    Failed to bulk stage deployment manifest and pin deployment for package:Package_9131_for_KB4480977~31bf3856ad364e35~amd64~~10.0.1.7 [HRESULT = 0x80073701 – ERROR_SXS_ASSEMBLY_MISSING]

2019-03-26 12:40:12, Info                  CBS    CommitPackagesState: Started persisting state of packages

2019-03-26 12:40:12, Info                  CBS    CommitPackagesState: Completed persisting state of packages

2019-03-26 12:40:12, Info                  CSI    0000001d@2019/3/26:11:40:12.294 CSI Transaction @0x27526657710 destroyed

2019-03-26 12:40:12, Info                  CBS    Perf: Resolve chain complete.

2019-03-26 12:40:12, Info                  CBS    Failed to resolve execution chain. [HRESULT = 0x80073701 – ERROR_SXS_ASSEMBLY_MISSING]

2019-03-26 12:40:12, Error                 CBS    Failed to process single phase execution. [HRESULT = 0x80073701 – ERROR_SXS_ASSEMBLY_MISSING]

2019-03-26 12:40:12, Info                  CBS    WER: Generating failure report for package: Package_for_RollupFix~31bf3856ad364e35~amd64~~14393.2759.1.7, status: 0x80073701, failure source: Resolve, start state: Absent, target state: Installed, client id: WindowsUpdateAgent

2019-03-26 12:40:12, Info                  CBS    Not able to query DisableWerReporting flag.  Assuming not set… [HRESULT = 0x80070002 – ERROR_FILE_NOT_FOUND]

2019-03-26 12:40:12, Info                  CBS    Added C:\Windows\Logs\CBS\CBS.log to WER report.

2019-03-26 12:40:12, Info                  CBS    Added C:\Windows\Logs\CBS\CbsPersist_20190326072401.log to WER report.

2019-03-26 12:40:12, Info                  CBS    Added C:\Windows\Logs\CBS\CbsPersist_20190324034222.log to WER report.

2019-03-26 12:40:12, Info                  CBS    Added C:\Windows\Logs\CBS\CbsPersist_20190324001230.log to WER report.

2019-03-26 12:40:12, Info                  CBS    Added C:\Windows\Logs\CBS\CbsPersist_20190323234238.log to WER report.

2019-03-26 12:40:12, Info                  CBS    Added C:\Windows\Logs\CBS\CbsPersist_20190323230433.log to WER report.

2019-03-26 12:40:12, Info                  CBS    Not able to add %windir%\winsxs\poqexec.log to WER report. [HRESULT = 0x80070002 – ERROR_FILE_NOT_FOUND]

2019-03-26 12:40:12, Info                  CBS    Not able to add %windir%\winsxs\pending.xml to WER report. [HRESULT = 0x80070002 – ERROR_FILE_NOT_FOUND]

2019-03-26 12:40:12, Info                  CBS    Not able to add %windir%\winsxs\pending.xml.bad to WER report. [HRESULT = 0x80070002 – ERROR_FILE_NOT_FOUND]

2019-03-26 12:40:12, Info                  CBS    Reboot mark cleared

2019-03-26 12:40:12, Info                  CBS    Winlogon: Simplifying Winlogon CreateSession notifications

2019-03-26 12:40:12, Info                  CBS    Winlogon: Deregistering for CreateSession notifications

2019-03-26 12:40:12, Info                  CBS    FinalCommitPackagesState: Started persisting state of packages

2019-03-26 12:40:12, Info                  CBS    Reporting package change for package: Package_for_RollupFix~31bf3856ad364e35~amd64~~14393.2759.1.7, current: Absent, pending: Default, start: Absent, applicable: Installed, target: Installed, limit: Installed, hotpatch status: StillGoing, status: 0x0, failure source: Resolve, reboot required: False, client id: WindowsUpdateAgent, initiated offline: False, execution sequence: 161, first merged sequence: 161, reboot reason: REBOOT_NOT_REQUIRED, RM App session: -1, RM App name: N/A, FileName in use: N/A, release type: Update, release quality: final, OC operation: False, download source: 0, download time (secs): 4294967295, download status: 0x0 (S_OK), Express download: False, Download Size: 0

2019-03-26 12:40:13, Info                  CBS    SQM: Package change report datapoints not populated because SQM is not initialized or not running online.

2019-03-26 12:40:13, Info                  CBS    Reporting package change completion for package: Package_for_RollupFix~31bf3856ad364e35~amd64~~14393.2759.1.7, current: Absent, original: Absent, target: Installed, status: 0x80073701, failure source: Resolve, failure details: “(null)”, client id: WindowsUpdateAgent, initiated offline: False, execution sequence: 161, first merged sequence: 161, pending decision: InteractiveInstallFailed, primitive execution context: Interactive Flight: False

2019-03-26 12:40:13, Info                  CBS    The store corruption status report is incomplete. [HRESULT = 0x80070002 – ERROR_FILE_NOT_FOUND]

2019-03-26 12:40:13, Info                  CBS    Resolve time performance datapoint is invalid. [HRESULT = 0x80070490 – ERROR_NOT_FOUND]

2019-03-26 12:40:13, Info                  CBS    Stage time performance datapoint is invalid. [HRESULT = 0x80070490 – ERROR_NOT_FOUND]

2019-03-26 12:40:13, Info                  CBS    Execute time performance datapoint is invalid. [HRESULT = 0x80070490 – ERROR_NOT_FOUND]

2019-03-26 12:40:13, Info                  CBS    SQM: Package change report datapoints not populated because SQM is not initialized or not running online.

2019-03-26 12:40:13, Info                  CBS    FinalCommitPackagesState: Completed persisting state of packages

2019-03-26 12:40:13, Info                  CBS    Enabling LKG boot option

2019-03-26 12:40:13, Info                  CBS    Exec: Processing complete.  Session: 30729156_3451904436, Package: Package_for_RollupFix~31bf3856ad364e35~amd64~~14393.2759.1.7 [HRESULT = 0x80073701 – ERROR_SXS_ASSEMBLY_MISSING]

2019-03-26 12:43:05, Info                  CBS    Trusted Installer is shutting down because: SHUTDOWN_REASON_AUTOSTOP

2019-03-26 12:43:05, Info                  CBS    TiWorker signaled for shutdown, going to exit.

2019-03-26 12:43:05, Info                  CBS    CbsCoreFinalize: ExecutionEngineFinalize

2019-03-26 12:43:05, Info                  CBS    Ending the TiWorker main loop.

Ohhhhh, okay well…..so lot’s of ASSEMBLY errors as well. Well, that at least matches what we saw in the event log …but where to start?

It’s particularly hard to read here as we do have more line breaks as in the original cbs.log file but I’ve marked the interesting things above.

Look at the package name / number we tried to install

Failed to bulk stage deployment manifest and pin deployment for package:Package_9131_for_KB4480977~31bf3856ad364e35~amd64~~10.0.1.7 [HRESULT = 0x80073701 – ERROR_SXS_ASSEMBLY_MISSING]

This is the January KB we tried to install (KB4480977) and the error is ERROR_SXS_ASSEMBLY_MISSING

Okay, but can you spot an identical error just before?

2019-03-26 12:40:12, Error                 CSI    0000001b (F) STATUS_SXS_ASSEMBLY_MISSING #1760302# from CCSDirectTransaction::OperateEnding at index 0 of 1 operations, disposition 2[gle=0xd015000c]

2019-03-26 12:40:12, Error                 CSI    0000001c (F) HRESULT_FROM_WIN32(ERROR_SXS_ASSEMBLY_MISSING) #1760150# from Windows::ServicingAPI::CCSITransaction::ICSITransaction_PinDeployment(Flags = 0, a = cd4484f3ed27edafc9d99e613bb62654, version 10.0.14393.2457, arch amd64, nonSxS, pkt {l:8 b:31bf3856ad364e35}, cb = (null), s = (null), rid = ‘Package_7763_for_KB4462917~31bf3856ad364e35~amd64~~10.0.1.7.4462917-16739_neutral’, rah = ‘3’, manpath = (null), catpath = (null), ed = 0, disp = 0)[gle=0x80073701]

2019-03-26 12:40:12, Info                  CBS    Failed to pin deployment while resolving Update: Package_7763_for_KB4462917~31bf3856ad364e35~amd64~~10.0.1.7.4462917-16739_neutral from file: (null) [HRESULT = 0x80073701 – ERROR_SXS_ASSEMBLY_MISSING]

As you can see, the very same error is reported for an already installed patch, in this case the October 2018 (KB4462917) rollup.

 

The solution for those of you who are eager to know was to uninstall the patch which reported the error first – KB4462917 this can be done either through the control panel à view installed updates à uninstall or in our case we used DISM again to automate this.

1st get a list of all installed packages

dism /online /get-packages /format:table

2nd from this list find the package name for the one you want to uninstall

DISM.exe /Online /Remove-Package /PackageName:Package_for_KB2870699~31bf3856ad364e35~amd64~~6.2.1.1 /quiet /norestart

After that we were able to install any of the monthly rollup patches again without any issues.

Other related but not purposeful geek trivia for you to know.

I first had the idea that just some RegKey is missing or corrupt, so I checked the referenced key from the CBS.log.

Registry :

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Component Based Servicing\Packages\Package_7763_for_KB4462917~31bf3856ad364e35~amd64~~10.0.1.7

As you can see the LastError “0x80073701” is matching with the one from the CBS.log and the CurrentState which is 112 dec means installed.

Applicable/Current State

Hex

Dec

Absent 0 0
Uninstall Pending 0x5 5
Resolving 0x10 16
Resolved 0x20 32
Staging 0x30 48
Staged 0x40 64
Superseded 0x50 80
Install Pending 0x60 96
Partially Installed 0x65 101
Installed 0x70 112
Permanent 0x80 128

For a complete list and more information’s visit https://blogs.technet.microsoft.com/tip_of_the_day/2015/10/12/tip-of-the-day-cbs-servicing-states-chart-refresher/

This didn’t help at the end and the only one and proper solution is to remove the patch which is reporting the assembly error first and either re-install that one or if available a newer one.

Thanks for reading

Michael