Case in the corner series: The crashing Add User Wizard

[Today’s post comes to us courtesy of Damian Leibaschoff from Commercial Technical Support ]

The symptoms for this case were that after a few seconds of launching the Add User Wizard, the wizard would crash, without any user input.

clip_image001

When we face a user mode crash like this we usually try to get our hands on the dump that is generated so we can review what triggered the crash. Depending on the crash, the associated wizard log might also contain valuable information (this was not the case here). Furthermore, we might also rely on tools such as Sysinternal’s Process Monitor.

For this case we launched the application within our Windows Debugger to see what was taking place, here is a summary of what we found and how we found it:

  1. Launched Windbg (elevated rights)

  2. Verified that the public symbols were configured properly.

  3. Selected File, Open Executable, and picked C:\Program Files\Windows Small Business Server\Bin\adduser.exe

  4. Once it loaded, proceeded to let the program run while connected to the debugger by issuing the GO ‘g’ command.

    clip_image003

  5. The Debugger will break on the second chance exception, on this case a CLR exception, pointing us to an issue while running .Net Framework code.

    clip_image005

  6. At this point, we need to load the proper extensions to handle .Net Framework debugging so we can take a peek into what is going on. You do that by loading the SOS extension:

    .loadby sos mscorwks
    To view the exception, you can now use:
    !printexception or
    !pe
    The output for this (key lines highlighted):
    0:006> .loadby sos mscorwks
    0:006> !pe
    Exception object: 00000000027f30a0
    Exception type: System.ArgumentException
    Message: An entry with the same key already exists.
    InnerException: <none>
    StackTrace (generated):
    SP IP Function
    000000001B2AEF80
    000007FEF69C4BD3System_ni!System.ThrowHelper.ThrowArgumentException(System.ExceptionResource)+0x2ad1d3
    000000001B2AEFC0 000007FEF62AC16DSystem_ni!System.Collections.Generic.SortedList`2[[System.Int32, mscorlib],[System.__Canon, mscorlib]].Add(Int32, System.__Canon)+0x5d
    000000001B2AF030 000007FF001AF6EF
    AdminCommon!Microsoft.WindowsServerSolutions.Admin.Common.Utils.GetPrimarySmtpWorker(System.Object)+0x89f
    000000001B2AF130 000007FEF6DC175C
    mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+0x9c
    000000001B2AF180 000007FEF6E36112
    mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(System.Threading._ThreadPoolWaitCallback)+0x62
    000000001B2AF1D0 000007FEF6E35F50
    mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(System.Object)+0x50

    StackTraceString: <none>
    HResult: 80070057

  7. We can easily tell by looking at the exception that we are getting a duplicate entry in a list of what should be unique items. By looking at the CLR call stack that led to the exception we can also see our code trying to find the Primary SMTP address from Exchange. From this, we rapidly found the lines of code that we were using when we hit the exception, confirmed what they were doing and quickly found the cause of the problem. Before we go into the details of what was wrong, let’s continue this scenario as if we did not have access to the source code, how would we have found what was wrong going the long way.

    When we look at a working Add User Wizard, we can see where we are using the SMTP address in plain sight:

    clip_image006

    Once we see that, it is just a matter of understanding how Exchange 2007 manages SMTP addresses to find the place where the problem lies, in our case, the E-mail Address Policies <More information> . On normal Exchange server in SBS, they would look like this:

    clip_image008

    Yet on the system where we had the crash, they looked like this:

    clip_image010

    You can also get the output from Exchange Management Shell
    [PS] C:\>Get-EmailAddressPolicy

    Name Priority RecipientFilter
    ---- -------- ---------------
    Default Policy Lowest Alias -ne $null
    Windows SBS Email Address Lowest Alias -ne $null
    Policy

    Can you spot what is wrong?
    ..
    ..
    ..

    Both policies have the SAME priority, not only that, they have the default LOWEST priority. This priority is a special case, where its actual value in active directory is 2147483647, this is the highest this value can be, thus the lowest priority. Our code queries for all AD objects that are of objectClass=msExchRecipientPolicy and the one with the lowest msExchPolicyOrder is the primary, we should not have duplicates.

  8. To fix this we need to correct the priority of the “Windows SBS Email Address Policy” as that is clearly incorrect. Unfortunately we cannot change it using the Exchange native tools as we get an error:

    [PS] C:\>Set-EmailAddressPolicy "Windows SBS Email Address Policy" -Priority 1
    Set-EmailAddressPolicy : The operation cannot be completed on the default e-mail address policy.
    At line:1 char:23
    + Set-EmailAddressPolicy <<<< "Windows SBS Email Address Policy" -Priority 1

    To get this fixed we had to use ADSIEDIT.MSC and manually edit the AD object:

    clip_image012

    clip_image013

    Changing the attribute to the correct value was all that was needed to regain functionality.

This was a very strange situation, the most likely cause was that the original Default Policy had been manually renamed to the Windows SBS Email Address Policy and a subsequent service pack for Exchange tried to recreate the missing Default Policy or it could have been as simple as some incorrect manipulation of AD objects..