This week’s servicing issue


It’s been asked several times that I start to write more about how to troubleshoot servicing issues.  Several things go into working on these issues, many of which I have written about on here at one point or another.  The best way I can think about “teaching” someone how to work on servicing cases is to show them.  So, on that note, my plan is to present the problem that was presented to me and then what we did to resolve the issue (with log snippets).

So, onto this week’s problem.  The issue the user was experiencing was that they couldn’t install the IIS components on a Windows 7 Enterprise installation.  Commonly when we see an issue of this nature, the first thing we do is check to see if anything else will install properly (like Telnet client or Games) just to prove that the servicing stack is indeed functioning the way we expect.  If those updates work, then we know that there is potentially an issue with the specific packages, catalogs, etc for the update that isnt installing properly. In this case we installed the Games component and it failed.  The log snippet looked like this:

2010-11-23 15:15:13, Info                  CBS    Exec: Resolving Package: Microsoft-Windows-Shell-InboxGames-Package~31bf3856ad364e35~amd64~de-DE~6.1.7600.16385, Update: More Games

2010-11-23 15:15:13, Info                  CBS    Exec: Resolving component from existing package; passing NULL manifest path to PinDeployment and hoping things haven’t been scavenged.

2010-11-23 15:15:13, Info                  CBS    Exec: Resolving Package: Microsoft-Windows-Shell-InboxGames-Package~31bf3856ad364e35~amd64~de-DE~6.1.7600.16385, Update: More Games, PinDeployment: amd64_microsoft-windows-s..oyment-languagepack_31bf3856ad364e35_6.1.7600.16385_de-de_08eb0375be2e567f

2010-11-23 15:15:13, Error                 CSI    0000000c@2010/11/23:23:15:13.896 (F) d:\w7rtm\base\wcp\componentstore\csd_locking.cpp(324): Error STATUS_SXS_ASSEMBLY_MISSING originated in function CCSDirectTransaction::LockComponent expression: (null)

[gle=0x80004005]

2010-11-23 15:15:16, Error                 CSI    0000000d (F) STATUS_SXS_ASSEMBLY_MISSING #59554# from CCSDirectTransaction::OperateEnding at index 0 of 1 operations, disposition 2[gle=0xd015000c]

2010-11-23 15:15:16, Error                 CSI    0000000e (F) HRESULT_FROM_WIN32(ERROR_SXS_ASSEMBLY_MISSING) #59439# from Windows::ServicingAPI::CCSITransaction::ICSITransaction_PinDeployment(Flags = 0, a = Microsoft-Windows-Shell-InboxGames-MoreGames-Deployment-LanguagePack, Version = 6.1.7600.16385, pA = PROCESSOR_ARCHITECTURE_AMD64 (9), Culture = [l:10{5}]"de-DE", VersionScope = 1 nonSxS, PublicKeyToken = {l:8 b:31bf3856ad364e35}, Type neutral, TypeName neutral, PublicKey neutral, cb = (null), s = (null), rid = [97]"Microsoft-Windows-Shell-InboxGames-Package~31bf3856ad364e35~amd64~de-DE~6.1.7600.16385.More Games", rah = (null), manpath = (null), catpath = (null), ed = 0, disp = 0)[gle=0x80073701]

From looking over this snippet,  we were able to see that the error when attempting this was: 0x80073701 which is the (ERROR_SXS_ASSEMBLY_MISSING) you see above.  So, because the assembly seems to be missing, another thing we would commonly do is run the CheckSUR utility to see if there were other issues around this component.  Running CheckSUR resulted in the following log snippets:

Checking Component Store
(f)    CSI Missing Winning Component Key    0x00000000    amd64_microsoft-windows-p..cemanager.resources_31bf3856ad364e35_6.1.7600.16385_ja-jp_2c493d3ffdc1b57f       
(fix)    CSI Missing Winning Component Key    CSI Registry Item Repaired    Deleted winners map value 6.1.7600.16385
(f)    CSI Missing Winning Component Key    0x00000000    amd64_prnca00z.inf-languagepack_31bf3856ad364e35_6.1.7600.16385_ja-jp_9a44400edd3bd265       
(fix)    CSI Missing Winning Component Key    CSI Registry Item Repaired    Deleted winners map value 6.1.7600.16385
(f)    CSI Missing Winning Component Key    0x00000000    x86_microsoft-windows-eudcedit.resources_31bf3856ad364e35_6.1.7600.16385_de-de_faad07b2e5533b64       
(fix)    CSI Missing Winning Component Key    CSI Registry Item Repaired    Deleted winners map value 6.1.7600.16385
(f)    CSI Missing Winning Component Key    0x00000000    amd64_microsoft-windows-i..filercore.resources_31bf3856ad364e35_8.0.7600.16385_de-de_3146b49a9601486e       
(fix)    CSI Missing Winning Component Key    CSI Registry Item Repaired    Deleted winners map value 8.0.7600.16385
(f)    CSI Missing Winning Component Key    0x00000000    x86_microsoft-windows-d..tshow-dmo.resources_31bf3856ad364e35_6.1.7600.16385_fr-fr_0aa7683de3386960       
(fix)    CSI Missing Winning Component Key    CSI Registry Item Repaired    Deleted winners map value 6.1.7600.16385
(f)    CSI Missing Winning Component Key    0x00000000    amd64_microsoft-windows-t..writerqfe.resources_31bf3856ad364e35_6.1.7600.16385_ja-jp_cbb69d5d9cd32768       
(fix)    CSI Missing Winning Component Key    CSI Registry Item Repaired    Deleted winners map value 6.1.7600.16385

Summary:
Seconds executed: 478
Found 14209 errors
Fixed 14197 errors
  CSI Missing Winning Component Key Total count: 14209
  Fixed: CSI Missing Winning Component Key.  Total count: 14197

As you can see, there were over 14000 problems found, the above snippet was repeated over and over in the log like the ones above.  The interesting thing about this when you look at the log snippets is that both of them point to references to alternate language packs being installed on the system.  We can see references to the German language pack in the Games snippet and several other language packs on the CheckSUR snippet.

From here, we looked in the System log to see if there were other servicing related snippets and here’s an example of what we saw:

Log Name:      System

Source:        Microsoft-Windows-LanguagePackSetup

Date:          5/4/2010 11:46:52 AM

Event ID:      1000

Task Category: Running the lpksetup Wizard

Level:         Error

Keywords:     

User:          SYSTEM

Computer:      computername

Description:

CBS Client initialization failed. Last error: 0x80080005

Event Xml:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">

  <System>

    <Provider Name="Microsoft-Windows-LanguagePackSetup" Guid="{7237FFF9-A08A-4804-9C79-4A8704B70B87}" />

    <EventID>1000</EventID>

    <Version>1</Version>

    <Level>2</Level>

    <Task>30</Task>

    <Opcode>0</Opcode>

    <Keywords>0x8000000000000000</Keywords>

    <TimeCreated SystemTime="2010-05-04T18:46:52.121169000Z" />

    <EventRecordID>1555</EventRecordID>

    <Correlation />

    <Execution ProcessID="2820" ThreadID="2904" />

    <Channel>System</Channel>

    <Computer>computername</Computer>

    <Security UserID="S-1-5-18" />

  </System>

  <EventData>

    <Data Name="Error">0x80080005</Data>

  </EventData>

</Event>

Here we can see further evidence that the language pack was failing due to an access denied issue. 

Uninstalling the language packs on this machine enabled the installation of IIS and the other updates to install properly.

Hope that helps.

–Joseph


Comments (3)

  1. Anonymous says:

    I dont think anyone, myself included, is saying that the current servicing model is perfect.  It's software, nothing is perfect.  However, I can tell you personally from a supportability perspective we see less overall issues with the way we do things now than we did in the "glory days".  And this is not conjecture, this is just me basing it on 13yrs of taking phone calls at Microsoft and seeing whats coming in on the phones.  

    It's not to say we dont get servicing problems, because we do.  But, there arent as many overall and the calls we get we are generally able to resolve for customers.  I think many people forget that there was plenty of rebuilding and "big copy" type repairs on systems prior to Vista as well.  Sometimes releasing a hotfix would actually force a rebuild of a machine because other operations hadnt completed and we had no good way to tell the user they needed to do that.  We have that now and I think things will get even better in future Windows releases.  Mostly because of comments like these.  Thanks for adding it.

    –Joseph

  2. anonymous says:

    Honestly I feel the Vista/7 servicing stack doesn't do properly what it was designed for – reliable servicing. Just the other day, I had to repair a reinstall a Vista system from scratch because it stopped booting while installing Service pack 1 from Automatic Updates (don't know how it corrupted itself but nothing would woirk, no tools from WinRE/WinPE would help. Every few months, I get some component like the in-built games crashing on Vista or 7 and the event log says something to the effect of  "The image doesn't match the one stored in the CBS store" or something similar. Never before Vista did I have built-in minor components of Windows like games and accessories crashing. I bet my ass that many systems are going to stop booting or getting stuck in a loop when users trying to install the upcoming Windows 7 Service Pack 1.

  3. Susan says:

    But in this era of online-ism, we don't call, we bingle.  So to say that you don't get the calls, is that a fair measuring stick?  Do you have automatic feedback (CEIP?) that gives you data?

    In my testing of Win7 I could not get it installed if I downloaded but I could if I used the Windows update bit flip.