Network Discovery not working in SCOM due to DLL corruption (debugging DLL loader component – LDR)


So, wow, this was nice! ... a super one-off scenario related to SCOM. This is very interesting though from a point of view of debugging and can theoretically happen in any application, so we are going to have a look at how to debug (and automate debugging) of the DLL Loader component (LDR) from ntdll.dll.

Let's start with the actual issue first 😀

SCOM Network Discovery component is not working and the error we can see being logged in the Operations Manager event log on the Management Server which is configured to run the Network Discovery rule, is Error event ID 157:

Creation of module with CLSID "{6620A03E-9E85-4BEA-947C-D508CDB540FE}" failed with error "%1 is not a valid Win32 application." in rule "discovery<SOME_NUMBER>.<SOME_GUID>" running for instance "<SOME_NAME>" with id:"{<SOME_GUID>}" in management group "<SOME_MG_NAME>".

Now the error "not a valid Win32 application" means that we are trying to load an EXE or DLL file and it has one of 2 issues:

  1. we either are trying to load a 32bit binary file from a 64bit process (or the other way around - very uncommon ... but not impossible)
  2. or there is something directly wrong with the binary file contents (some corruption at the file structure level / header): https://en.wikipedia.org/wiki/Portable_Executable

But what file (most probably DLL) is this?! We have the CLSID in the error which is: 6620A03E-9E85-4BEA-947C-D508CDB540FE

So how do we figure out which DLL it is? By searching in the registry (on a SCOM Management Server) by that CLSID and somewhere under HKEY_CLASSES_ROOT\CLSID\, we should find this GUID - and after a search, we do and under: HKEY_CLASSES_ROOT\CLSID\{6620A03E-9E85-4bea-947C-D508CDB540FE}\InprocServer32, we find the value called (Default) which contains the full path to the DLL: C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\NetworkDiscoveryModules.dll

 

Pah! Super simple, right? It looks like the NetworkDiscoveryModules.dll is somehow corrupted and we should just replace it with a working one from a SCOM Management Server with the exact same SCOM version and Update Rollup installed - right?!

Well, we did that and it did not work ... so whaaat is going on here?!

 

Well, this clearly is not a "SCOM" issue, but some file/OS related issue that can happen with any software - but, because I am a curious man ...

 

First things first - let's totally separate our problem from SCOM and let's manually try to reproduce the problem by just loading the DLL directly - we don't need to complicate ourselves with writing native code, we can just do this in PowerShell directly - like this:

Add-Type @"
using System;
using System.Runtime.InteropServices;
public static class NativeMethods {
   [DllImport(@"$($env:ProgramFiles)\Microsoft System Center 2012 R2\Operations Manager\Server\NetworkDiscoveryModules.dll")]
   public static extern void DllCanUnloadNow();
}
"@
[NativeMethods]::DllCanUnloadNow()

Extra info for the other curious people out there about this script:

  • it will create a C# file which will get compiled in a temporary directory that will do the actual native (C/C++) DLL loading/import
  • after this, we will call a standard DLL function (usually included in every DLL - verified that we have it in NetworkDiscoveryModules.dll because I can look at the code :p - but you can usually rely on the fact that almost all DLLs have this) - DllCanUnloadNow()
  • this function - DllCanUnloadNow() - will not return anything from this PowerShell script on a computer where there is no issue with the DLL - however, it will return an error if there is something wrong with it - just like this one below which we got when running the script on the affected server:

So the error we are getting is actually the same (only more descriptive): An attempt was made to load a program with an incorrect format.

 

Ok cool! So now we can reproduce the issue totally outside of SCOM, by just trying to load that DLL and calling a function from it (because only when we make an actual call will the DLL be truly loaded for code execution).

So, now we can do a live debug by attaching to the PowerShell.exe process which we will use to execute that script: https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/

Everything I will show next in the debugger, will be using Public Symbols, so everyone can follow the steps (don't need any internal private symbols): https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/microsoft-public-symbols

The Windows LDR works like this:

  1. we want to load DLL whatever.dll
  2. we go verify if the binary file has the correct bitness and if it's not corrupt in any way
  3. if all goes well, we check the import section and for each DLL which it is dependent on, we do the exact same thing one by one - steps 1 to 3 for each of them
  4. if any of the DLLs in the import section cannot be correctly loaded (not correct bitness, corrupt in some way, etc.) we will fail and throw the error describing why the DLL cannot be loaded - however, we throw the error for the initial DLL we were trying to load, we don't specify if there actually was a failure in the loading of one of the dependent DLLs

So we need to track down which dependent DLL has the actual corruption ...

Where do we break to check what is wrong? Well, we have these 2 nice function calls where we will break and check what is going on:

  1. kernelbase!LoadLibraryExW which is used to load DLLs by Windows OS (Ex comes from "Extended" which means being able to pass in more "options" - parameters and W comes from the fact that the UNICODE string version of it is being used) => we are just starting here in order to continue with  the investigation with other breakpoints only when we reach the point where we are trying to load the DLL we are interested in troubleshooting (in this case NetworkDiscoveryModules.dll)
  2. ntdll!LdrpFindOrMapDll (not documented on MSDN, should not directly be used) which will get called by the DLL loader component (LDR) to try to find the DLL in different places, verify binary file structures/header, load all the DLL that it is dependent on (verifying consistency of each as well) and in the end load the actual DLL we call in order to be able to execute from it => this will be important because this is where we can get the DLL file name that we are currently trying to load
  3. ntdll!ZwCreateSection which goes into Kernel and does the actual file read from disk and checks the required information (is it execution code, does it have a valid "MZ" start, is the header ok?, does it have the correct bitness?, etc.) => this is the actual function call we are interested in the end, because this is the one that will return the actual error in case that we have an issue loading the binary file to execute from it

Let's set the first breakpoint and go in WinDbg:

0:000> bp kernelbase!LoadLibraryExW

0:000> g
Breakpoint 0 hit
kernelbase!LoadLibraryExW:
00007ffe`36b48d10 48895c2410 mov qword ptr [rsp+10h],rbx ss:0000000c`301d4b38=0000000c301d91a8

So, sure enough, we will break on the first/next execution of kernelbase!LoadLibraryExW and we can check what DLL is currently being loaded.

Just as an "FYI" we can/will also have a look at the stack because we are curious people:

0:001> k
Child-SP RetAddr Call Site
0000000c`301d4b28 00007ffe`310ae4b4 kernelbase!LoadLibraryExW
0000000c`301d4b30 00007ffe`310ae421 clr!CLRLoadLibraryExWorker+0x54
0000000c`301d4b80 00007ffe`31098aff clr!CLRLoadLibraryEx+0x51
0000000c`301d4bd0 00007ffe`31098a28 clr!LoadedImageLayout::LoadedImageLayout+0xca
0000000c`301d4e90 00007ffe`313f6a84 clr!PEImageLayout::Load+0x3d
0000000c`301d4ed0 00007ffe`313f69de clr!PEImage::GetLayoutInternal+0x361930
0000000c`301d4f20 00007ffe`3107ee8f clr!PEImage::GetLayout+0x36195e
0000000c`301d4fa0 00007ffe`3107ed53 clr!RuntimeOpenImageInternal+0xcf
0000000c`301d5080 00007ffe`3107ec7b clr!GetAssemblyMDInternalImportEx+0xcd
0000000c`301d5100 00007ffe`310d3cc9 clr!CreateMetaDataImport+0x1b
0000000c`301d5140 00007ffe`310d3c1d clr!BindResult::Init+0x69
0000000c`301d51b0 00007ffe`3118867b clr!BindResult::CreateFromPath+0xc9
0000000c`301d5220 00007ffe`31188891 clr!NATIVE_BINDER_SPACE::VerifyNativeCandidate+0x55f
0000000c`301d58b0 00007ffe`31086b40 clr!NATIVE_BINDER_SPACE::CreateVerifiedNICandidate+0x10a
0000000c`301d5bc0 00007ffe`31085f39 clr!NATIVE_BINDER_SPACE::BindToNativeDependency+0x2ac
0000000c`301d5de0 00007ffe`31085803 clr!BindToNativeAssembly+0xed
0000000c`301d5f00 00007ffe`3108799f clr!BindToNativeImage+0x1bb
0000000c`301d6230 00007ffe`31082a6b clr!CAsmDownloadMgr::RecordInfoAndProbeNativeImage+0x106
0000000c`301d62a0 00007ffe`3108303b clr!CAsmDownloadMgr::PreDownloadCheck+0x518
0000000c`301d6ae0 00007ffe`31082e45 clr!CAssemblyDownload::PreDownload+0xb7
0000000c`301d6b80 00007ffe`3108251a clr!CAssemblyName::BindToObject+0x419
0000000c`301d6cd0 00007ffe`310822c2 clr!FusionBind::RemoteLoad+0x1aa
0000000c`301d6e10 00007ffe`31082047 clr!AssemblySpec::LoadAssembly+0x19a
0000000c`301d6f30 00007ffe`31081cf5 clr!AssemblySpec::FindAssemblyFile+0x113
0000000c`301d76d0 00007ffe`3108a4ae clr!AppDomain::BindAssemblySpec+0xef7
0000000c`301d8950 00007ffe`310ab18b clr!AssemblySpec::LoadDomainAssembly+0x1ec
0000000c`301d8f00 00007ffe`3118c9e8 clr!AssemblySpec::LoadAssembly+0x1b
0000000c`301d8f40 00007ffe`2f6ebde2 clr!AssemblyNative::Load+0x304
0000000c`301d92b0 00007ffe`304c4326 mscorlib_ni!System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(System.Reflection.AssemblyName, System.Security.Policy.Evidence, System.Reflection.RuntimeAssembly, System.Threading.StackCrawlMark ByRef, IntPtr, Boolean, Boolean, Boolean)+0xd2
0000000c`301d9360 00007ffe`31209f51 mscorlib_ni!System.Reflection.RuntimeAssembly.LoadWithPartialNameInternal(System.Reflection.AssemblyName, System.Security.Policy.Evidence, System.Threading.StackCrawlMark ByRef)+0xd66ad6
0000000c`301d93e0 00007ffe`31209e4f clr!ExceptionTracker::CallHandler+0xc5
0000000c`301d9480 00007ffe`312081c7 clr!ExceptionTracker::CallCatchHandler+0x7f
0000000c`301d9510 00007ffe`396c347d clr!ProcessCLRException+0x2e6
0000000c`301d95f0 00007ffe`39685405 ntdll!RtlpExecuteHandlerForUnwind+0xd
0000000c`301d9620 00007ffe`31208260 ntdll!RtlUnwindEx+0x385
0000000c`301d9d00 00007ffe`3120821c clr!ClrUnwindEx+0x40
0000000c`301da220 00007ffe`396c33fd clr!ProcessCLRException+0x2b2
0000000c`301da300 00007ffe`39684847 ntdll!RtlpExecuteHandlerForException+0xd
0000000c`301da330 00007ffe`39683a6d ntdll!RtlDispatchException+0x197
0000000c`301daa00 00007ffe`36b495fc ntdll!RtlRaiseException+0x18d
0000000c`301db1c0 00007ffe`31209864 kernelbase!RaiseException+0x68
0000000c`301db2a0 00007ffe`314bb7ef clr!RaiseTheExceptionInternalOnly+0x2fe
0000000c`301db3a0 00007ffe`3118cb37 clr!UnwindAndContinueRethrowHelperAfterCatch+0x80
0000000c`301db3f0 00007ffe`2f75d99e clr!AssemblyNative::Load+0x453
0000000c`301db760 00007ffe`2ff13666 mscorlib_ni!System.Reflection.RuntimeAssembly.LoadWithPartialNameInternal(System.Reflection.AssemblyName, System.Security.Policy.Evidence, System.Threading.StackCrawlMark ByRef)+0x14e
0000000c`301db890 00007ffe`1933970f mscorlib_ni!System.Reflection.Assembly.LoadWithPartialName(System.String)+0x46
0000000c`301db8e0 00007ffe`193394e1 system_management_automation_ni!System.Management.Automation.ExecutionContext.LoadAssembly(System.String, System.String, System.Exception ByRef)+0x15f
0000000c`301db9a0 00007ffe`193211f2 system_management_automation_ni!System.Management.Automation.ExecutionContext.AddAssembly(System.String, System.String, System.Exception ByRef)+0x21
0000000c`301db9f0 00007ffe`19316887 system_management_automation_ni!Microsoft.PowerShell.Commands.ModuleCmdletBase.LoadBinaryModule(System.Management.Automation.PSModuleInfo, Boolean, System.String, System.String, System.Reflection.Assembly, System.String, System.Management.Automation.SessionState, ImportModuleOptions, ManifestProcessingFlags, System.String, Boolean, Boolean, Boolean ByRef, System.String, Boolean)+0x452
0000000c`301dbbf0 00007ffe`1931b82b system_management_automation_ni!Microsoft.PowerShell.Commands.ModuleCmdletBase.LoadModuleNamedInManifest(System.Management.Automation.PSModuleInfo, Microsoft.PowerShell.Commands.ModuleSpecification, System.String, Boolean, System.String, System.Management.Automation.SessionState, ImportModuleOptions, ManifestProcessingFlags, Boolean, Boolean, System.Object, Boolean ByRef, System.String)+0x977
0000000c`301dbe20 00007ffe`19316ab5 system_management_automation_ni!Microsoft.PowerShell.Commands.ModuleCmdletBase.LoadModuleManifest(System.String, System.Management.Automation.ExternalScriptInfo, System.Collections.Hashtable, System.Collections.Hashtable, ManifestProcessingFlags, System.Version, System.Version, ImportModuleOptions ByRef, Boolean ByRef)+0x4d0b
0000000c`301dc9d0 00007ffe`1931fa1b system_management_automation_ni!Microsoft.PowerShell.Commands.ModuleCmdletBase.LoadModuleManifest(System.Management.Automation.ExternalScriptInfo, ManifestProcessingFlags, System.Version, System.Version, ImportModuleOptions ByRef)+0xd5
0000000c`301dca70 00007ffe`1932386c system_management_automation_ni!Microsoft.PowerShell.Commands.ModuleCmdletBase.LoadModule(System.Management.Automation.PSModuleInfo, System.String, System.String, System.String, System.Management.Automation.SessionState, System.Object, ImportModuleOptions ByRef, ManifestProcessingFlags, Boolean ByRef, Boolean ByRef)+0xc4b
0000000c`301dcda0 00007ffe`193241d8 system_management_automation_ni!Microsoft.PowerShell.Commands.ImportModuleCommand.ImportModule_LocallyViaName(ImportModuleOptions, System.String)+0x4ac
0000000c`301dcef0 00007ffe`193dbb0e system_management_automation_ni!Microsoft.PowerShell.Commands.ImportModuleCommand.ProcessRecord()+0x198
0000000c`301dcfd0 00007ffe`19382b55 system_management_automation_ni!System.Management.Automation.CommandProcessor.ProcessRecord()+0x29e
0000000c`301dd080 00007ffe`1937953e system_management_automation_ni!System.Management.Automation.CommandProcessorBase.DoExecute()+0xe5
0000000c`301dd0f0 00007ffe`193666d0 system_management_automation_ni!System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(System.Object, System.Collections.Hashtable, Boolean)+0x17e
0000000c`301dd1e0 00007ffe`1936581f system_management_automation_ni!System.Management.Automation.Runspaces.LocalPipeline.InvokeHelper()+0x750
0000000c`301dd400 00007ffe`19365241 system_management_automation_ni!System.Management.Automation.Runspaces.LocalPipeline.InvokeThreadProc()+0x2af
0000000c`301dd550 00007ffe`193348d5 system_management_automation_ni!System.Management.Automation.Runspaces.LocalPipeline.StartPipelineExecution()+0x321
0000000c`301dd5d0 00007ffe`19363fe4 system_management_automation_ni!System.Management.Automation.Runspaces.PipelineBase.CoreInvoke(System.Collections.IEnumerable, Boolean)+0x305
0000000c`301dd650 00007ffe`193df48e system_management_automation_ni!System.Management.Automation.Runspaces.PipelineBase.Invoke(System.Collections.IEnumerable)+0x24
0000000c`301dd690 00007ffe`193deff0 system_management_automation_ni!System.Management.Automation.PowerShell+Worker.ConstructPipelineAndDoWork(System.Management.Automation.Runspaces.Runspace, Boolean)+0x2de
0000000c`301dd770 00007ffe`193d9fbe system_management_automation_ni!System.Management.Automation.PowerShell.CoreInvokeHelper[[System.__Canon, mscorlib],[System.__Canon, mscorlib]](System.Management.Automation.PSDataCollection`1<System.__Canon>, System.Management.Automation.PSDataCollection`1<System.__Canon>, System.Management.Automation.PSInvocationSettings)+0x350
0000000c`301dd820 00007ffe`193d9aca system_management_automation_ni!System.Management.Automation.PowerShell.CoreInvoke[[System.__Canon, mscorlib],[System.__Canon, mscorlib]](System.Management.Automation.PSDataCollection`1<System.__Canon>, System.Management.Automation.PSDataCollection`1<System.__Canon>, System.Management.Automation.PSInvocationSettings)+0x4de
0000000c`301dd9b0 00007ffe`193efca0 system_management_automation_ni!System.Management.Automation.PowerShell.CoreInvoke[[System.__Canon, mscorlib]](System.Collections.IEnumerable, System.Management.Automation.PSDataCollection`1<System.__Canon>, System.Management.Automation.PSInvocationSettings)+0x1ba
0000000c`301dda60 00007ffe`193ed6c2 system_management_automation_ni!System.Management.Automation.CommandDiscovery.AutoloadSpecifiedModule(System.String, System.Management.Automation.ExecutionContext, System.Management.Automation.SessionStateEntryVisibility, System.Exception ByRef)+0x2c0
0000000c`301ddb00 00007ffe`19376326 system_management_automation_ni!System.Management.Automation.CommandDiscovery.TryModuleAutoDiscovery(System.String, System.Management.Automation.ExecutionContext, System.String, System.Management.Automation.CommandOrigin, System.Management.Automation.SearchResolutionOptions, System.Management.Automation.CommandTypes, System.Exception ByRef)+0x312
0000000c`301ddc10 00007ffe`19375ed1 system_management_automation_ni!System.Management.Automation.CommandDiscovery.LookupCommandInfo(System.String, System.Management.Automation.CommandTypes, System.Management.Automation.SearchResolutionOptions, System.Management.Automation.CommandOrigin, System.Management.Automation.ExecutionContext)+0x3d6
0000000c`301ddcf0 00007ffe`193dd8ce system_management_automation_ni!System.Management.Automation.CommandDiscovery.LookupCommandProcessor(System.String, System.Management.Automation.CommandOrigin, System.Nullable`1<Boolean>)+0x31
0000000c`301ddd50 00007ffe`193dcfd1 system_management_automation_ni!System.Management.Automation.ExecutionContext.CreateCommand(System.String, Boolean)+0x6e
0000000c`301dddb0 00007ffe`19335255 system_management_automation_ni!System.Management.Automation.PipelineOps.AddCommand(System.Management.Automation.Internal.PipelineProcessor, System.Management.Automation.CommandParameterInternal[], System.Management.Automation.Language.CommandBaseAst, System.Management.Automation.CommandRedirection[], System.Management.Automation.ExecutionContext)+0x3f1
0000000c`301ddef0 00007ffe`1a0a036e system_management_automation_ni!System.Management.Automation.PipelineOps.InvokePipeline(System.Object, Boolean, System.Management.Automation.CommandParameterInternal[][], System.Management.Automation.Language.CommandBaseAst[], System.Management.Automation.CommandRedirection[][], System.Management.Automation.Language.FunctionContext)+0x135
0000000c`301ddfb0 00007ffe`1938591b system_management_automation_ni!System.Management.Automation.Interpreter.ActionCallInstruction`6[[System.__Canon, mscorlib],[System.Boolean, mscorlib],[System.__Canon, mscorlib],[System.__Canon, mscorlib],[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Run(System.Management.Automation.Interpreter.InterpretedFrame)+0x19e
0000000c`301de050 00007ffe`1938591b system_management_automation_ni!System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(System.Management.Automation.Interpreter.InterpretedFrame)+0x16b
0000000c`301de160 00007ffe`1938575b system_management_automation_ni!System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(System.Management.Automation.Interpreter.InterpretedFrame)+0x16b
0000000c`301de270 00007ffe`193752af system_management_automation_ni!System.Management.Automation.Interpreter.Interpreter.Run(System.Management.Automation.Interpreter.InterpretedFrame)+0x7b
0000000c`301de2f0 00007ffe`19383925 system_management_automation_ni!System.Management.Automation.Interpreter.LightLambda.RunVoid1[[System.__Canon, mscorlib]](System.__Canon)+0x11f
0000000c`301de380 00007ffe`193832f8 system_management_automation_ni!System.Management.Automation.DlrScriptCommandProcessor.RunClause(System.Action`1<System.Management.Automation.Language.FunctionContext>, System.Object, System.Object)+0x405
0000000c`301de510 00007ffe`19382fce system_management_automation_ni!System.Management.Automation.DlrScriptCommandProcessor.Complete()+0x1a8
0000000c`301de5c0 00007ffe`19382d01 system_management_automation_ni!System.Management.Automation.CommandProcessorBase.DoComplete()+0x16e
0000000c`301de680 00007ffe`19379551 system_management_automation_ni!System.Management.Automation.Internal.PipelineProcessor.DoCompleteCore(System.Management.Automation.CommandProcessorBase)+0xb1
0000000c`301de720 00007ffe`193666d0 system_management_automation_ni!System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(System.Object, System.Collections.Hashtable, Boolean)+0x191
0000000c`301de810 00007ffe`1936581f system_management_automation_ni!System.Management.Automation.Runspaces.LocalPipeline.InvokeHelper()+0x750
0000000c`301dea30 00007ffe`192e7001 system_management_automation_ni!System.Management.Automation.Runspaces.LocalPipeline.InvokeThreadProc()+0x2af
0000000c`301deb80 00007ffe`2f6d2d45 system_management_automation_ni!System.Management.Automation.Runspaces.PipelineThread.WorkerProc()+0x31
0000000c`301debb0 00007ffe`2f6d2ab9 mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0x285
0000000c`301ded10 00007ffe`2f6d2a97 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0x9
0000000c`301ded40 00007ffe`2f6ea161 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+0x57
0000000c`301ded90 00007ffe`3103afb3 mscorlib_ni!System.Threading.ThreadHelper.ThreadStart()+0x51
0000000c`301dede0 00007ffe`3103ae9e clr!CallDescrWorkerInternal+0x83
0000000c`301dee20 00007ffe`3103b632 clr!CallDescrWorkerWithHandler+0x4a
0000000c`301dee60 00007ffe`31146bd9 clr!MethodDescCallSite::CallTargetWorker+0x251
0000000c`301df010 00007ffe`3103c8e1 clr!ThreadNative::KickOffThread_Worker+0x105
0000000c`301df270 00007ffe`3103c868 clr!ManagedThreadBase_DispatchInner+0x2d
0000000c`301df2b0 00007ffe`3103c7d9 clr!ManagedThreadBase_DispatchMiddle+0x6c
0000000c`301df3b0 00007ffe`3103c91b clr!ManagedThreadBase_DispatchOuter+0x75
0000000c`301df440 00007ffe`31146aba clr!ManagedThreadBase_FullTransitionWithAD+0x2f
0000000c`301df4a0 00007ffe`31153656 clr!ThreadNative::KickOffThread+0xd2
0000000c`301df570 00007ffe`38f113d2 clr!Thread::intermediateThreadProc+0x7d
0000000c`301df7b0 00007ffe`396454e4 kernel32!BaseThreadInitThunk+0x22
0000000c`301df7e0 00000000`00000000 ntdll!RtlUserThreadStart+0x34

 

Ok, cool, so now how do we figure out which is the DLL that is currently being loaded? Well, from MSDN, we know that the first parameter passed to the kernelbase!LoadLibraryExW function is the name of the DLL (lpFileName).

Hmm ... first parameter ... so where is that? Well, we know that on x64 calling convention (fastcall), the first parameter is passed on into the RCX register: https://msdn.microsoft.com/en-us/library/ms235286.aspx

So good! Let's have a look at that - as it should be a UNICODE string, we can look at it like this:

0:001> du @rcx
0000000c`2f4dd240 "C:\Windows\assembly\NativeImages"
0000000c`2f4dd280 "_v4.0.30319_64\Microsoft.P521220"
0000000c`2f4dd2c0 "ea#\ab32ec62ecc4b90efc07602a8353"
0000000c`2f4dd300 "8c65\Microsoft.PowerShell.Comman"
0000000c`2f4dd340 "ds.Utility.ni.dll"

Pff ... well this is lame, not the correct DLL ... so we go again further until the next breakpoint with the "g" command and verify again and go again and verify and so on, until we finally reach the DLL we are interested in (in this case NetworkDiscoveryModules.dll):

0:001> g
Breakpoint 0 hit
kernelbase!LoadLibraryExW:
00007ffe`36b48d10 48895c2410 mov qword ptr [rsp+10h],rbx ss:0000000c`301dcbd8=0000000c301dddf8
0:001> du @rcx
0000000c`301dd078 "C:\Program Files\Microsoft Syste"
0000000c`301dd0b8 "m Center 2012 R2\Operations Mana"
0000000c`301dd0f8 "ger\Server\NetworkDiscoveryModul"
0000000c`301dd138 "es.dll"

Perfect! Now we can put breakpoints on the other functions that will get called next to have a look (dependency) DLL by DLL to figure out which is corrupt and then go (continue execution):

0:001> bp ntdll!LdrpFindOrMapDll
0:001> bp ntdll!ZwCreateSection
0:001> g
Breakpoint 2 hit
ntdll!LdrpFindOrMapDll:
00007ffe`396a61c0 4053            push    rbx

So, we break into the next call of ntdll!LdrpFindOrMapDll. Aaaaaand how exactly do we know which dependent DLL is being currently loaded? The function is not documented on MSDN ... ok, so a little bit of internals - this function gets the DLL name as a pointer to a string in the format of the ntdll!_UNICODE_STRING structure - which is actually documented on MSDN and which we can see using Public Symbols.

First parameter? Ok, so again, still in RCX register - let's dump it out using "dx" (Debugger Data Model functionality - which can be extended using JavaScript as well - but a little later on extending it):

0:001> dx ((ntdll!_UNICODE_STRING *)@rcx)
((ntdll!_UNICODE_STRING *)0x00000000c301dc948) : 0xc301dc948 : "C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\NetworkDiscoveryModules.dll" [Type: _UNICODE_STRING *]
[<Raw View>] [Type: _UNICODE_STRING]

Ha, ok, now we're taking, we are going through the verification/load of our DLL of interest directly in ntdll!LdrpFindOrMapDll. Cool - so let's continue execution and we should break in ntdll!NtCreateSection next which we can use to verify the DLL consistency check status and see if there is any error (corruption in this case) - let's also have a look at the stack after we break:

0:001> g
Breakpoint 1 hit
ntdll!NtCreateSection:
00007ffe`396c0b50 4c8bd1 mov r10,rcx

0:001> k
Child-SP RetAddr Call Site
0000000c`301dc538 00007ffe`396a6921 ntdll!NtCreateSection
0000000c`301dc540 00007ffe`3965b2b5 ntdll!LdrpFindOrMapDll+0x761
0000000c`301dc8b0 00007ffe`39698d69 ntdll!LdrpLoadDll+0x295
0000000c`301dcae0 00007ffe`36b48dda ntdll!LdrLoadDll+0x99
0000000c`301dcb60 00007ffe`310ae4b4 kernelbase!LoadLibraryExW+0xca
0000000c`301dcbd0 00007ffe`310ae421 clr!CLRLoadLibraryExWorker+0x54
0000000c`301dcc20 00007ffe`31197dfd clr!CLRLoadLibraryEx+0x51
0000000c`301dcc70 00007ffe`3140bf1e clr!LocalLoadLibraryHelper+0x31
0000000c`301dcca0 00007ffe`310af2a4 clr!NDirect::LoadLibraryModule+0x35c7de
0000000c`301dd4f0 00007ffe`310af50a clr!NDirect::NDirectLink+0x80
0000000c`301dd820 00007ffe`310af469 clr!NDirect::GetStubForILStub+0x4a
0000000c`301dd870 00007ffe`310af54b clr!GetStubForInteropMethod+0x65
0000000c`301dd8b0 00007ffe`310a9696 clr!MethodDesc::DoPrestub+0xca7
0000000c`301dda80 00007ffe`3103226a clr!PreStubWorker+0x3d6
0000000c`301ddd90 00007ffd`d1a40a1c clr!ThePreStub+0x5a
0000000c`301dde60 00007ffe`2cb64c1c DynamicClass.CallSite.Target(System.Runtime.CompilerServices.Closure, System.Runtime.CompilerServices.CallSite, System.Type)+0xfc
0000000c`301ddee0 00007ffe`1938b088 system_core_ni!System.Dynamic.UpdateDelegates.UpdateAndExecute1[[System.__Canon, mscorlib],[System.__Canon, mscorlib]](System.Runtime.CompilerServices.CallSite, System.__Canon)+0x39c
0000000c`301ddff0 00007ffe`1938591b system_management_automation_ni!System.Management.Automation.Interpreter.DynamicInstruction`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Run(System.Management.Automation.Interpreter.InterpretedFrame)+0x58
0000000c`301de050 00007ffe`1938591b system_management_automation_ni!System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(System.Management.Automation.Interpreter.InterpretedFrame)+0x16b
0000000c`301de160 00007ffe`1938575b system_management_automation_ni!System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(System.Management.Automation.Interpreter.InterpretedFrame)+0x16b
0000000c`301de270 00007ffe`193752af system_management_automation_ni!System.Management.Automation.Interpreter.Interpreter.Run(System.Management.Automation.Interpreter.InterpretedFrame)+0x7b
0000000c`301de2f0 00007ffe`19383925 system_management_automation_ni!System.Management.Automation.Interpreter.LightLambda.RunVoid1[[System.__Canon, mscorlib]](System.__Canon)+0x11f
0000000c`301de380 00007ffe`193832f8 system_management_automation_ni!System.Management.Automation.DlrScriptCommandProcessor.RunClause(System.Action`1<System.Management.Automation.Language.FunctionContext>, System.Object, System.Object)+0x405
0000000c`301de510 00007ffe`19382fce system_management_automation_ni!System.Management.Automation.DlrScriptCommandProcessor.Complete()+0x1a8
0000000c`301de5c0 00007ffe`19382d01 system_management_automation_ni!System.Management.Automation.CommandProcessorBase.DoComplete()+0x16e
0000000c`301de680 00007ffe`19379551 system_management_automation_ni!System.Management.Automation.Internal.PipelineProcessor.DoCompleteCore(System.Management.Automation.CommandProcessorBase)+0xb1
0000000c`301de720 00007ffe`193666d0 system_management_automation_ni!System.Management.Automation.Internal.PipelineProcessor.SynchronousExecuteEnumerate(System.Object, System.Collections.Hashtable, Boolean)+0x191
0000000c`301de810 00007ffe`1936581f system_management_automation_ni!System.Management.Automation.Runspaces.LocalPipeline.InvokeHelper()+0x750
0000000c`301dea30 00007ffe`192e7001 system_management_automation_ni!System.Management.Automation.Runspaces.LocalPipeline.InvokeThreadProc()+0x2af
0000000c`301deb80 00007ffe`2f6d2d45 system_management_automation_ni!System.Management.Automation.Runspaces.PipelineThread.WorkerProc()+0x31
0000000c`301debb0 00007ffe`2f6d2ab9 mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0x285
0000000c`301ded10 00007ffe`2f6d2a97 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0x9
0000000c`301ded40 00007ffe`2f6ea161 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+0x57
0000000c`301ded90 00007ffe`3103afb3 mscorlib_ni!System.Threading.ThreadHelper.ThreadStart()+0x51
0000000c`301dede0 00007ffe`3103ae9e clr!CallDescrWorkerInternal+0x83
0000000c`301dee20 00007ffe`3103b632 clr!CallDescrWorkerWithHandler+0x4a
0000000c`301dee60 00007ffe`31146bd9 clr!MethodDescCallSite::CallTargetWorker+0x251
0000000c`301df010 00007ffe`3103c8e1 clr!ThreadNative::KickOffThread_Worker+0x105
0000000c`301df270 00007ffe`3103c868 clr!ManagedThreadBase_DispatchInner+0x2d
0000000c`301df2b0 00007ffe`3103c7d9 clr!ManagedThreadBase_DispatchMiddle+0x6c
0000000c`301df3b0 00007ffe`3103c91b clr!ManagedThreadBase_DispatchOuter+0x75
0000000c`301df440 00007ffe`31146aba clr!ManagedThreadBase_FullTransitionWithAD+0x2f
0000000c`301df4a0 00007ffe`31153656 clr!ThreadNative::KickOffThread+0xd2
0000000c`301df570 00007ffe`38f113d2 clr!Thread::intermediateThreadProc+0x7d
0000000c`301df7b0 00007ffe`396454e4 kernel32!BaseThreadInitThunk+0x22
0000000c`301df7e0 00000000`00000000 ntdll!RtlUserThreadStart+0x34

Hmm, ok good! But what now? How do we figure out what the ntdll!NtCreateSection function will return? Because it will actually do a return of the status (which can be an error value). We know that by conversion, compilers will use the RAX register as return value just before finishing execution of a function (unless specifically coded otherwise in assembly).

Let's go at the end of the function execution of ntdll!NtCreateSection using "pt" and have a look at the value of the RAX register:

0:001> pt
ntdll!NtCreateSection+0xa:
00007ffe`396c0b5a c3 ret

0:001> r @rax
rax=0000000000000000

Ok, so RAX is 0 and 0 is good! 0 means ERROR_SUCCESS status. Don't get confused by the name - it actually meas that the operation completed successfullyhttps://msdn.microsoft.com/en-us/library/windows/desktop/ms681382(v=vs.85).aspx

So, we don't throw an error and we validate this DLL (currently NetworkDiscoveryModules.dll) successfully. Ok, so next, we should go through each dependent DLL and try to validate/load each on of them. So, let's step through each one of these and check the status (return value in RAX) of ntdll!NtCreateSection each time.

So after checking each one (ntdll!LdrpFindOrMapDll for the DLL name & ntdll!NtCreateSection for the return value) we finally reach the DLL that is throwing the error (binary file validation error):

0:001> g
Breakpoint 2 hit
ntdll!LdrpFindOrMapDll:
00007ffe`396a61c0 4053 push rbx

0:001> dx ((ntdll!_UNICODE_STRING *)@rcx)
((ntdll!_UNICODE_STRING *)@rcx) : 0xc301dc4b0 : "dmboot.dll" [Type: _UNICODE_STRING *]
[<Raw View>] [Type: _UNICODE_STRING]

0:001> g
Breakpoint 1 hit
ntdll!NtCreateSection:
00007ffe`396c0b50 4c8bd1 mov r10,rcx

0:001> pt
ntdll!NtCreateSection+0xa:
00007ffe`396c0b5a c3 ret

0:001> r @rax
rax=00000000c000012f

Ha, nice! So return status for ntdll!NtCreateSection for the dmboot.dll  DLL is not 0 (ERROR_SUCCESS) anymore - it is: 00000000c000012f

Guess what? We also have a command from a build-in extension (!error) for WinDbg which we can use to "resolve" the actual error name/description by the HRESULT (00000000c000012f) value:

0:001> !error 00000000c000012f
Error code: (NTSTATUS) 0xc000012f (3221225775) - The specified image file did not have the correct format, it did not have an initial MZ.

 

Well ... what do you know? It looks like the dmboot.dll DLL is the one that is actually corrupt and it is being loaded because it is a dependency DLL of our NetworkDiscoveryModules.dll DLL.

We got the dmboot.dll DLL from a healthy SCOM Management Server with the same SCOM version and Update Rollup and replaced this corrupt one - aaand ... there you go, issue solved! 😀

 

 

How did this file (DLL) get corrupt in the first place?! Well now, that is the real question! Unfortunately, it cannot really be explained after it already happened without any type of "file-change-auditing", but it has too many possible causes:

  • some filter driver (like anti-virus software or anything else like "my-super-cool-awesome-disk-speed-up.sys" and so on ...
  • computer power-failure in the middle of specific type of disk IO (direct access) - very unlikely though ...
  • network (possibly filter) driver while copying a file from some other computer (or some network device in between that somehow ends up modifying bits)
  • super cool awesome storage controller that directly manipulates (RAM) memory through the mother board (not through OS Kernel)
  • some totally other random weird issue like a CPU bit-flip while the file was being written to disk ?! ...

 

Yes - this gets much easier if Private Symbols are available 😉 But, as this should (I hope) ideally help anyone trying to debug such a scenario, I have documented everything using Public Symbols 😀

But hey ... why not automate this process by creating a cool JavaScript extension for WinDbg and just let the computer do the work for me next time this ever happens with ... whatever ... other software? 😀

Here is the documentation about creating extensions in JavaScript for the Data Model: https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/javascript-debugger-scripting & also check out this Channel9 video from the Defrag Tools series about this, with cool info from Andrew Richards, Andy Luhrs and Bill Messmerhttps://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-170-Debugger-JavaScript-Scripting

This could have easily been just a series of functions in the scripts, but I complicated it a little bit because I wanted to show it as an extension to the process object - I might extend this in the future and create a more complex extension around the LDR component. I highly doubt that it will be public though :p because without public symbols, I have to rely on calling conversions (and get directly directly from registers when possible - and it's not always possible) as I cannot resolve any locals or parameters (by calling DX's getModuleSymbol), which would over-complicate things.

"use strict";

class __DllLoadStatus
{
   get Ldr()
   {
      return new __DllLoadFailure(this);
   }
}

class __DllLoadFailure
{
   constructor(process)
   {
      this.__process = process;
   }

   Initialize()
   {
      var ctl = host.namespace.Debugger.Utility.Control;
      ctl.ExecuteCommand("bp /1 ntdll!ZwCreateSection;g");
      ctl.ExecuteCommand("pt");

      var address = host.currentThread.Registers.User.rip;
      ctl.ExecuteCommand("bp ntdll!LdrpFindOrMapDll \"dx @$curprocess.Ldr.GetDllName();gc\"");
      ctl.ExecuteCommand("bp " + address.toString(16) + " \"dx @$curprocess.Ldr.GetLoadStatus();gc\"");

      host.diagnostics.debugLog("Ok, we are all set! Run \"g\" to start automated debugging.\n");
   }

   GetDllName()
   {
      var ctl = host.namespace.Debugger.Utility.Control;
      var address = host.currentThread.Registers.User.rcx;

      var pDllName = host.createPointerObject(address, "ntdll.dll", "_UNICODE_STRING *");
      var dllName = host.memory.readWideString(pDllName.dereference().Buffer).toString();

      host.diagnostics.debugLog(dllName, "\n");
   }

   GetLoadStatus()
   {
      var ctl = host.namespace.Debugger.Utility.Control;
      var status = ctl.ExecuteCommand("!error @rax").First().toString();

      host.diagnostics.debugLog(status, "\n");
   }
}

function initializeScript()
{
   return [
      new host.namedModelParent(__DllLoadStatus, "Debugger.Models.Process")
   ];
}

Alright, so now we have this script saved somewhere (ex. C:\DbgScripts\LdrExtensionSample.js). So let's load the JsProvider (!load jsprovider) and afterwards using the full path of the script as parameter, load the script using the .scriptload command:

0:004> !load jsprovider
0:004> .scriptload C:\DbgScripts\LdrExtensionTest.js
JavaScript script successfully loaded from 'C:\DbgScripts\LdrExtensionTest.js'

So, let's have a go at our script and see the output of the automated analysis by running "dx @$curprocess.Ldr.Initialize()" and then letting the process execute with the "g" command:

0:000> dx @$curprocess.Ldr.Initialize()
Ok, we are all set! Run "g" to start automated debugging.
@$curprocess.Ldr.Initialize()
 
0:001> g
C:\Windows\system32\rsaenh.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
C:\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\NetworkDiscoveryModules.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
MSVCR100.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
KERNEL32.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
USER32.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
ole32.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
OLEAUT32.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
ADVAPI32.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
sm-server.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
sm-discovery.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
sm-auto-discovery.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
MSVCP100.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
ntdll.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
RPCRT4.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
SHLWAPI.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
HealthServiceRuntime.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
KERNEL32.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
MSVCP100.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
MSVCR100.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
KERNEL32.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
ADVAPI32.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
sm-clsapi.dll
@$curprocess.Ldr.GetDllName()
Error code: (Win32) 0 (0) - The operation completed successfully.
@$curprocess.Ldr.GetLoadStatus()
dmboot.dll
@$curprocess.Ldr.GetDllName()
Error code: (NTSTATUS) 0xc000012f (3221225775) - The specified image file did not have the correct format, it did not have an initial MZ.
@$curprocess.Ldr.GetLoadStatus()

 

Pfff ... well, that was much easier 😀

 

Have fun developing WinDbg JS Extensions & Scripts! 🙂

 

 

 

Comments (0)

Skip to main content