WMI: Repository Corruption, or Not?

  • Article

Scenario

Windows Management Instrumentation failing due to repository being corrupted

The WMI Repository (%windir%System32\Wbem\Repository) is the database that stores meta-information and definitions for WMI classes; in some cases the repository also stores static class data as well. If Repository becomes corrupted, then the WMI service will not be able to function correctly.

Before grabbing that preverbal hammer approach and just rebuilding your repository, ask yourself, “Is the WMI repository OK?”

Common symptoms that lead to this question are: provider load failure, access denied, class not found, invalid namespace, and namespace not found to mention a few.

If you suspect WMI or repository corruption, rebuilding repository is the last thing you should do without verifying this is truly the case. Deleting and rebuilding the repository can cause damage to Windows or to installed applications. Other steps should be taken first to eliminate other possibilities or to confirm we have repository corruption. Noting here that having a repository too large also creates problems; an issue that can sometimes be interpreted as a corrupt repository, which is not always the case. If issues are due to a large repository, rebuilding the repository is currently the only method available to reduce it back to a working size.

Since I mentioned “large repository”, let me set some guidelines up front. There is no hard fast number per say as to when you will start feeling performance problems with a large repository. As a guideline, if the objects.data file, located in (%windir%System32\Wbem\Repository, is 1 GB or larger, then I would recommend rebuilding your repository to reduce it back down to a working and manageable size. If the size is between 600-900 MB, and you are not feeling any noticeable performance issues, then I would recommend against rebuilding the repository.

If WMI is corrupted, you can receive various errors and symptoms, depending on what activity was being done at the time. Below is a few errors and symptoms that could indicate that the repository is corrupted:

  1. Unable to connect to root\default or root\cimv2 namespaces. Fails returning error code 0x80041002 pointing to WBEM_E_NOT_FOUND.
  2. When we open Computer Management and Right Click on Computer Management (Local) and select Properties, you get the following error: "WMI: Not Found" or it hangs trying connect
  3. 0x80041010 WBEM_E_INVALID_CLASS
  4. Trying to use wbemtest, it hangs
  5. Schemas/Objects missing
  6. Strange connection/operation errors (0x8007054e):

get-cimclass : Unable to complete the requested operation because of either a catastrophic media failure or a data structure corruption on the disk.

At line:1 char:1

+ get-cimclass -Namespace root\cimv2\TerminalServices

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ CategoryInfo : NotSpecified: (:) [Get-CimClass], CimException

+ FullyQualifiedErrorId : HRESULT 0x8007054e,Microsoft.Management.Infrastructure.CimCmdlets.GetCimClassCommand

Check the Windows Application log for events in the past week where Source = Microsoft-Windows-WMI.  Look for any of the following WMI event IDs: 28, 65, 5600, 5601, 5614. Any of these could indicate a WMI repository issue or core infrastructure problem.

If you do not find any of these events logged, your next action is to use the built in repository checker. From an elevated command prompt run "winmgmt /verifyrepository". If the repository has an issue, it will respond "repository is not consistent".

If repository check comes back as “consistent”, then look at my other Ask Perf blogs for applicability:

WMI: Missing or Failing WMI Providers or Invalid WMI Class (COMING SOON)

WMI: High Memory Usage by WMI Service or Wmiprvse.exe (COMING SOON)

How to troubleshoot High CPU Usage by WMI Components (COMING SOON)

WMI Self-Recover

When the WMI service restarts or detects Repository corruption, the self-recovery procedure will trigger automatically in two approaches (one or the other):

  1. AutoRestore: if the VSS backup mechanism enable snapshot the timestamp backup images In the system (ex. Win7 feature: previous fileversion), WMI will apply the AutoRestore approach to restore backup valid images in version queue. (if possible)
    • EVT: 65 (start restore) / 66 (succeed recovered with VSS Path)
  1. AutoRecovery: the rebuild process to generate fresh images of Repository based on registered mofs( listed @ HKLM\Software\Microsoft\WBEM\CIMOM: AutoRecover Mofs).
    • EVT: 5616 (complete recovery), eventually, there are lots of EVT:63 for mof warning about Localsystem registration of providers.

Note: Under almost no circumstance should you use the script that rebuilds the WMI repository from the MOF files

The script is inherently flawed, for 2 reasons:

    1. If you navigate to the %systemroot%\system32\wbem folder, and list the MOF files, you see find MOFs named (some provider name)_uninstall.mof. When you mofcomp those, they remove the classes in the MOF. The script mofcomps everything, so it can very easily install then immediately uninstall the classes, resulting in the class not being accessible.
    2. Replaying mofs is often sequence dependent. Example: classes in mof1 can depend on or have associations with classes in mof2. If they aren't present, MOFCOMP will not insert the classes. It's extremely difficult to know what is / is not the right sequence, so any script that simply MOFCOMPs everything is not going to be fully successful.

In addition to causing damage to your system that's almost impossible to fix correctly, if you take that approach you will blow away all information that could be used to determine the root-cause.

If the repository check (winmgmt /verifyrepository) comes back as inconsistent, your first action is to run “winmmgmt /salvagerepository” followed by running “winmgmt /verifyrepository” again to see if it now comes back as consistent.

If it is still comes back inconsistent, then you need to run “winmgmt /resetrepository.” Before running this, please read the important note below for Server 2012.

Force-recovery process -- rebuild based on the registry list of AutoRecover Mofs

 

  1. Check regkey value is empty or not @ HKLM\Software\Microsoft\WBEM\CIMOM: 'Autorecover Mofs' (** first line on some OSs is empty, review it in opening the regkey value )
  2. If above regkey is empty, copy/paste the regkey value from another machine of equal System/OS to the suspect machine
  3. Run the following command from command prompt with admin rights: “Winmgmt /resetrepositoy”
  4. If you get the error noted below, stop all dependency services on the WMI service by running following command: 

net stop winmgmt /y

then run

Winmgmt /resetrepositoy

WMI repository reset failed
Error code:     0x8007041B
Facility:       Win32
Description:    A stop control has been sent to a service that other running services are dependent on

NOTE: Applies to Server 2012

We have encountered some issues when running the mofcomp command on Windows Server 2012 which has caused the Cluster namespace to be removed due to the cluswmiuninstall.mof contained in the c:\windows\system32\wbem folder. It has also caused unregistering class names for Cluster Aware Updating (CAU) because of its cauwmiv2.mof also in the wbem folder. Those could also affect other namespace that have an uninstall type .mof in the wbem folder beyond the two mentioned above.

Furthermore, the uninstall .mof files for servers running Microsoft Cluster are also part of the autorecover folder that is used when you run the winmgmt /resetrepository command which will end up having the same effect of first installing the Cluster namespace, then uninstalling it just as if you had run a script to rebuild the repository that contained the “for” command to recompile all of the MOFs in the WBEM folder.

Take the following actions to confirm if the uninstall problem for this scenario exists on your server. If it doesn’t, then you can run the winmgmt /reset repository, otherwise follow my directions below for manually accomplishing rebuild.

  1. Open regedit, and navigate to hklm\software\microsoft\wbem\cimom, and open “Autorecover MOFS”
  2. Copy the data from that string value, and paste it into notepad
  3. Do a search for ClusWmiUninstall.mof. If the cluster provider uninstall has autorecover, it will be listed here
  4. If found, then continue to manual rebuild below, if not found then go ahead and use the winmgmt /resetrepository command

How to manually rebuild repository on Server 2012 Cluster machine when cluster provider uninstall has an autorecover

First ensure you have run winmgmt /verifyrepository to ensure that it is “inconsistent” and that you have tried winmgmt /salvagerepository to see if it resolves your issue.

Change the startup type for the Window Management Instrumentation (WMI) Service to disabled.

 

  1. Stop the WMI Service, you may need to stop services dependent on the WMI service first before it allow you to successfully stop WMI Service
  2. Rename the repository folder:  C:\WINDOWS\system32\wbem\Repository to Repository.old
  3. Open a CMD Prompt with elevated privileges
  4. Cd windows\system32\wbem
  5. Run following command to re-register all the dlls: for /f %s in ('dir /b /s *.dll') do regsvr32 /s %s
  6. Set the WMI Service type back to Automatic and restart WMI Service
  7. cd /d c:\   ((go to the root of the c drive, this is important))
  8. Run following command specifically adopted for 2012 Clustered servers to recompile the MOFs: “dir /b *.mof *.mfl | findstr /v /i uninstall > moflist.txt & for /F %s in (moflist.txt) do mofcomp %s”
  9. Restart WMI service

As a final note, if you run into a reoccurring corruption issue in your environment with WMI, try to exclude the WBEM folder and all subfolders under it from AV scanning. AV scanning is known to cause corruption and other issues in WMI.

Other repository recovery solutions:

Note: in following solutions (1 & 2), if the backup images (repository) are in large size (>100MB), restoring the repository will take some time.

  1. Apply WMI AutoRestore story in the system to recover the repository image quickly and keep it in sync with previous state.
  2. Enable VSS backup-related features for storing image snapshots
    • ex.  Volume Shadow Copy (VSS), or check any valid copies listed Local Disk(C:) Properties >> Shadow copies
  3. Make sure registry key has following setting: HKLM\Software\Microsoft\WBEM\CIMOM: AutoRestoreEnabled=1
  4. Frequently snapshot the restore-points of the system (if needed, refer to the following PowerShell scripts)

$filterStmt = [String]::Format("DriveLetter='{0}'",$env:SystemDrive)

# Get systemdrive Volume info

$vlm = gcim win32_Volume -Filter $filterStmt

# create a shadowcopy

$res = icim win32_ShadowCopy -MethodName Create -Arguments @{Volume=$Vlm.DeviceID}

if ($res.ReturnValue -eq 0)

{ gcim win32_ShadowCopy -Filter ("ID='"+$res.ShadowID+"'") } # **

else

{ $res | fc }

    • AutoRestore only searches the top 3 queued snapshots for the latest valid backup, if no valid one is found, AutoRecovery will apply.
    • To restore others in the queue of snapshots ( manually )
      1. In Server sku, looking at 'previous version' tab of the repository folder to find the expected backup path 
        • ex. "\\localhost\C$\@GMT-2014.03.13-01.02.49Windows\system32\wbem\repository"
        • to check whether the snapshot image is valid or not, from command prompt do:
          • winmgmt /verifyrepository "\\localhost\C$\@GMT-2014.03.13-01.02.49Windows\system32\wbem\repository"
      2. stop WMI service: Net stop winmgmt /y
      3. Replace all of the files in %windir%\system32\wbem\Repository folder with the files from the backup path found in step 1

Note The WMI Service has auto start setting, and if it comes back alive, you will not be able to replace the files.  The service needs to be in a stopped state (if WMI service is alive at the time, repeat step: 2~3)

ex. Directory of \\localhost\C$\@GMT-2014.03.13-01.02.49Windows\system32\wbem\repository

03/11/2014  11:53 AM    <DIR>          .
03/11/2014  11:53 AM    <DIR>          ..
03/12/2014  05:30 PM         4,759,552 INDEX.BTR
03/12/2014  05:30 PM            90,640 MAPPING1.MAP
03/12/2014  03:26 PM            90,640 MAPPING2.MAP
03/12/2014  05:24 PM            90,640 MAPPING3.MAP
03/12/2014  05:30 PM        27,541,504 OBJECTS.DATA

4. Run following wmic command to bring WMI service back to life: wmic os

You should set up a regular Scheduled Task to backup the latest repository:

      • winmgmt /backup <backup image path>
      • tracing EVT: 67,68

You could also schedule restores as necessary

  • winmgmt /restore <above backup image path> 1
  • tracing EVT: 65,66

If the issue is not a repository issue, and the objects are not retrievable:

  • Re-install the product. This is the first place to start.
  • If there is a specific provider that is not showing up, you can re-run mofcomp of a single provider. See Ask the Performance Team Blog article WMI: Missing or Failing WMI Providers or Invalid WMI Class (COMING SOON)

If the issue persists or keeps returning, then at this point you will now need to open a Support Incident Case with Microsoft for further assistance.

Reference this blog when you open the Support Incident Case with Microsoft as it will help the engineer understand what actions have been taken or followed and well help us track the effectiveness of the blog.

Next up: WMI: Missing or Failing WMI Providers or Invalid WMI Class