Troubleshooting a DLL Redirection and SideBySide Assemblies issue

Hey folks, it’s Sakthi Ganesh again.  Today we’re going to talk a little bit about an interesting issue I recently worked on with one of our customers related to MMC Launch failures.  The problem itself had little to do with the MMC’s themselves and more to do with SideBySide Assemblies.  Before we dig into the issue though, let’s set the stage appropriately …

Most IT Administrators are familiar with the terms “DLL Hell” or “Dependency Hell”.  The terms refer to the different complications that can occur when dealing with multiple versions of the same Dynamic Link Library (DLL) file.  The whole point behind DLL’s is that you can write a single container object that is shared by any number of other components that need that DLL’s functions.  This saves hard disk space as well as memory space since the DLL only has to map into memory once.  It also allows you to change functionality or even fix bugs in a single location.  This however opens you up to unintended consequences if a DLL is replaced with an older or sometimes even newer version. When this occurs, you might get errors like this:

  • "The procedure entry point Foo couldn’t be located in Bar.dll"
  • "Ordinal 123 could not be located in the Dynamic Link Library Example.dll"

These errors simply mean that the app was calling an entry point that the DLL did not recognize because it moved or has been removed.  In some scenarios, a program may fail to check what version of a DLL (for example MFC42.DLL) is already installed on a system before it simply overwrites a newer version with an older one.  This creates all sorts of problems – the new program may work fine, but all the other programs on the system that relied on the updated functionality of the new DLL no longer work.  A well written installer should always check the version of shared files to as opposed to simply overwriting them.  Another scenario may arise when a newer version of a DLL has some functionality changes that result in compatibility issues.  In theory newer DLL’s should be fully backwards-compatible, but in actuality – issues may arise.  All of this is prior to the introduction of System File Protection and SideBySide Assemblies which by and large prevent an application from simply overwriting core files with whatever version it wants to.  System File Protection monitors a list of protected operating system files and makes sure they don’t get replaced.  If an application tries to overwrite one of our protected files, we allow it to succeed, and then immediately come in behind it and replace it with the version stored in the DLLCACHE directory.  The application never even knows this happened and in the vast majority of instances, the new application will function with no issues.

However, if you are the developer of an application and you need to use your own version of one of our DLL’s, that is fine too.  That is where DLL Redirection and SideBySide Assemblies come in.  DLL Redirection is actually a fairly simple concept.  When you install your application, place the DLL’s you need in the same directory as the .EXE for the app, and then create a new empty file in that location with the .LOCAL extension.  For instance, you have an app named Test.exe, so you create a Test.exe.local file in the same location.  Then, when you run the Test.exe app, it will use the local directory as it’s LoadLibrary search path, thus using the DLL’s in that directory in preference of the ones stored by the system.

SideBySide Assemblies are Win32 assemblies described by manifests and created so that multiple versions can run at the same time without conflicts. SideBySide Assemblies store information about binding and COM activation in their manifests. Developers can use SideBySide assemblies provided by Microsoft, other publishers or themselves. This allows an app vendor to call the specific version of a DLL that they need to work properly. By calling a SideBySide assembly built into Windows for instance, a third-party application can take advantage of the visual styles included with Windows XP or Windows Vista without changing anything in their application. Much more information about DLL Redirection and SideBySide Assemblies is available at the MSDN pages listed below.

Now, let’s turn to that issue I was working on.  As I mentioned, the problem behavior was that no 32-bit MMC would launch on a 64-bit server.  In the Event Logs we found error messages such as the one below:

Event Type: Error
Event Source: SideBySide
Event Category: None
Event ID: 59
Date: (Date)
Time: (Time)
User: N/A
Computer: (Computer Name)
Generate Activation Context failed for C:\WINNT\SysWOW64\mmc.exe. Reference error message: The referenced assembly is not installed on your system.

Event Type: Error
Event Source: SideBySide
Event Category: None
Event ID: 59
Date: (Date)
Time: (Time)
User: N/A
Computer: (Computer Name)
Resolve Partial Assembly failed for Microsoft.Windows.Common-Controls. Reference error message: The referenced assembly is not installed on your system.

There are a number of different ways to troubleshoot a problem like this.  In this particular scenario, we elected to launch MMC.EXE from within the Windows Debugger on a working machine and see what was going on.  The debug output is below:

Microsoft (R) Windows Debugger Version 6.8.0004.0 X86
Copyright (c) Microsoft Corporation. All rights reserved. 

CommandLine: C:\Windows\SysWOW64\mmc.exe
Symbol search path is: srv*c:\symbols*\\symbols\symbols
Executable search path is: 
ModLoad: 00db0000 00f68000   mmc.exe 
ModLoad: 77b00000 77c50000   ntdll.dll
ModLoad: 77040000 77150000   C:\Windows\syswow64\kernel32.dll
ModLoad: 760d0000 7618f000   C:\Windows\syswow64\ADVAPI32.dll
ModLoad: 6bdd0000 6be35000   C:\Windows\SysWOW64\ODBC32.dll
ModLoad: 74480000 74506000   C:\Windows\WinSxS\\COMCTL32.dll
ModLoad: 764f0000 76fbe000   C:\Windows\syswow64\SHELL32.dll 

d: 74840000 7486c000   C:\Windows\SysWOW64\apphelp.dll
(1618.15f4): Break instruction exception - code 80000003 (first chance)
eax=00000000 ebx=00000000 ecx=c1400000 edx=00000000 esi=fffffffe edi=77be0094
eip=77b10004 esp=002af7e4 ebp=002af814 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
77b10004 cc              int     3
0:000> g
ModLoad: 77390000 773f0000   C:\Windows\SysWOW64\IMM32.DLL
ModLoad: 76420000 764e7000   C:\Windows\syswow64\MSCTF.dll
ModLoad: 77ae0000 77ae9000   C:\Windows\syswow64\LPK.DLL
ModLoad: 75fc0000 7603d000   C:\Windows\syswow64\USP10.dll
ModLoad: 75720000 758b4000   C:\Windows\WinSxS\\comctl32.dll
ModLoad: 71e10000 71e48000   C:\Windows\SysWOW64\odbcint.dll
ModLoad: 773f0000 77474000   C:\Windows\syswow64\CLBCatQ.DLL
ModLoad: 6adc0000 6afd4000   C:\Windows\SysWOW64\mmcndmgr.dll

Based on the information in the Event Logs we surmised that there was a problem with the Common Control SideBySide Assemblies.  The two paths above (in red) are the only ones that refer to the WinSxS folder, so the first thing to do is to take a look to see if the referenced files are actually there (since the Event Log message indicates missing files).  In this particular instance, the files themselves were missing.  Now that we know exactly what the problem is, the resolution itself was relatively simple.  We replaced the relevant folders on the problem machine with folders from the working machine … and all was well again!

With that, it’s time to wrap up this post.  Thanks for stopping by!

Additional Resources:

– Sakthi Ganesh

Share this post :