.NET Application Hosting Web Browser Control Freezes / Hangs During Start Up


Background

Recently, I ran into few incidents where .NET Application hosting web browser control was hanging during start up. When I investigated dump file, I found out that the hang has nothing to do with the .NET or JavaScript or Web Browser Control. In fact, it was related to a bug introduced in one of the MSXML security fixes MS15-084. In order to mitigate the issue, you may want to install KB3076895.

Analysis

Open the dump file using WinDbg (ships with Debugging Tools for Windows).

  • Run !locks command and you'll get output similar to the following:

    CritSec ntdll!LdrpLoaderLock+0 at 77af20c0
    WaiterWoken        No
    LockCount          50
    RecursionCount     1
    OwningThread       2ea8
    EntryCount         0
    ContentionCount    bd
    *** Locked

    CritSec bcrypt!g_csLoaderLock+0 at 74844060
    WaiterWoken        No
    LockCount          1
    RecursionCount     1
    OwningThread       d5c
    EntryCount         0
    ContentionCount    1
    *** Locked

    CritSec +18631a44 at 18631a44
    WaiterWoken        No
    LockCount          0
    RecursionCount     2
    OwningThread       2ea8
    EntryCount         0
    ContentionCount    0
    *** Locked

    Scanned 4335 critical sections

  • Run ~~[threadid]kb command for the call stacks. In our case, it would be ~~[d5c]kb and ~~[2ea8]kb.

    ~~[d5c]kbL Output or call stack for thread with id d5c
    CritSec +18631a44 at 18631a44
    # ChildEBP RetAddr Args to Child
    00 0dffee7c 77a2b4b4 00000078 00000000 00000000 ntdll!NtWaitForSingleObject+0x15
    01 0dffeee0 77a2b398 00000000 00000000 0dffef48 ntdll!RtlpWaitOnCriticalSection+0x13e
    02 0dffef08 77a202a9 77af20c0 7a06f0f2 7483275c ntdll!RtlEnterCriticalSection+0x150
    ...

    ~~[2ea8]kbL Output or call stack for thread with id 2ea8
    # ChildEBP RetAddr Args to Child
    00 1502def4 77a2b4b4 00001790 00000000 00000000 ntdll!NtWaitForSingleObject+0x15
    01 1502df58 77a2b398 00000000 00000000 00000000 ntdll!RtlpWaitOnCriticalSection+0x13e
    02 1502df80 74832ff8 74844060 00000000 177ccde4 ntdll!RtlEnterCriticalSection+0x150
    ...

  • As you can see, thread d5c owns critical section 74844060 (step #1) and it is waiting for critical section 77af20c0 (step #2).
  • Similarly thread 2ea8 owns critical section 77af20c0 (step #1) and it is waiting for critical section 74844060 (step #2).
  • In other words, they are in mutual deadlock. They both need exclusive access to the critical section resource.
  • The entire process is freezing because of a special critical section (loader lock). You can find more information here, here and so on.
    • Very simple and short explanation is that a loader lock is acquired during dll initialization to make sure global & static variables are initialized properly. Plus few other reasons. You can read more about what exactly loader lock does in this msdn article (Causes of loader lock).
  • If you want to go one step further, then you can take one more dump after few seconds / minutes and will see that the threads are not consuming any CPU.
  • To do so execute !runway command on both the dumps and compare the time taken by each thread. It lists the threads consuming time the descending order with the one that consumed more CPU at the top.

    User Mode Time
    Thread Time
    0:28dc 0 days 0:00:03.744
    10:2b84 0 days 0:00:00.390
    9:2408 0 days 0:00:00.156
    48:2d28 0 days 0:00:00.124
    37:2728 0 days 0:00:00.109
    33:2538 0 days 0:00:00.109
    15:1a34 0 days 0:00:00.109
    31:1ca8 0 days 0:00:00.093
    20:2298 0 days 0:00:00.078
    17:287c 0 days 0:00:00.062
    44:17b4 0 days 0:00:00.046
    40:888 0 days 0:00:00.046
    28:23e4 0 days 0:00:00.046
    27:d5c 0 days 0:00:00.046
    18:2c30 0 days 0:00:00.046
    56:dd4 0 days 0:00:00.031
    54:2940 0 days 0:00:00.031
    38:2db4 0 days 0:00:00.031
    22:2698 0 days 0:00:00.031
    14:2a08 0 days 0:00:00.031
    13:12dc 0 days 0:00:00.031
    51:2d2c 0 days 0:00:00.015
    49:22f4 0 days 0:00:00.015
    46:c10 0 days 0:00:00.015
    41:4e4 0 days 0:00:00.015
    34:1de8 0 days 0:00:00.015
    32:22ac 0 days 0:00:00.015
    19:2878 0 days 0:00:00.015
    16:2ee4 0 days 0:00:00.015
    99:2d4c 0 days 0:00:00.000
    98:1aec 0 days 0:00:00.000
    97:2df4 0 days 0:00:00.000
    96:2920 0 days 0:00:00.000
    ...

  • Check process up time – displayed when you open the dump file

    Debug session time: Fri Oct 23 11:56:12.000 2015 (UTC - 5:00)
    System Uptime: 0 days 1:48:54.485
    Process Uptime: 0 days 0:14:37.000

  • The process is running for over 14 minutes and 37 seconds. The application is still initializing and it hasn't used more than 4-5 seconds of time. So application must be hung otherwise it will consume some amount of CPU until it initializes completely.
  • Verify the version of MSXML files installed on the machine. To do so run lmvm msxml* command in WinDbg.

    73630000 73788000 msxml6 (deferred)
    Image path: C:\Windows\System32\msxml6.dll
    Image name: msxml6.dll
    Timestamp: Tue Jul 14 21:52:06 2015 (55A5CAD6)
    CheckSum: 0015F444
    ImageSize: 00158000
    File version: 6.30.7601.18923
    Product version: 6.30.7601.18923
    File flags: 0 (Mask 3F)
    File OS: 40004 NT Win32
    File type: 2.0 Dll
    File date: 00000000.00000000
    Translations: 0000.04b0
    CompanyName: Microsoft Corporation
    ProductName: Microsoft(R) MSXML 6.0 SP3
    InternalName: MSXML6.dll
    OriginalFilename: MSXML6.dll
    ProductVersion: 6.30.7601.18923
    FileVersion: 6.30.7601.18923
    FileDescription: MSXML 6.0 SP3
    LegalCopyright: Copyright (C) Microsoft Corporation. 1981-2008

 

So what is the fix?

The issue was first introduced in MS15-084 and was fixed in KB3076895.

Comments (3)

  1. You can also use tools like DebugDiag to detect deadlock. Here is an output from the latest version of DebugDiag
    Detected a serious critical section related problem in MyApplication.exe__PID__1084__Date__10_23_2015__Time_11_23_45PM__935__Manual Dump.dmp
    Lock at bcrypt!g_csLoaderLock owned by thread 27 is Deadlocked with lock at ntdll!LdrpLoaderLock owned by thread 50

    Impact analysis

    1% of threads blocked

    (Threads 50)

    The following functions are involved in the root cause
    bcrypt!LoadProvider+32

    The following modules are involved in the root cause
    C:WindowsSystem32bcrypt.dll from Microsoft Corporation

  2. You can also use sosex to detect the deadlock
    0:000> !dlk
    Examining SyncBlocks…
    Scanning for ReaderWriterLock instances…
    Scanning for holders of ReaderWriterLock locks…
    Scanning for ReaderWriterLockSlim instances…
    Scanning for holders of ReaderWriterLockSlim locks…
    Examining CriticalSections…
    Scanning for threads waiting on SyncBlocks…
    Scanning for threads waiting on ReaderWriterLock locks…
    Scanning for threads waiting on ReaderWriterLocksSlim locks…
    Scanning for threads waiting on CriticalSections…
    *DEADLOCK DETECTED*
    OS thread 0xd5c holds CriticalSection 74844060
    …and is waiting on CriticalSection 77af20c0
    OS thread 0x2ea8 holds CriticalSection 77af20c0
    …and is waiting on CriticalSection 74844060
    OS Thread 0xd5c is waiting at ntdll!LdrGetProcedureAddressEx+0x159
    OS Thread 0x2ea8 is waiting at bcrypt!LoadProvider+0x32 

    1 deadlock detected.

  3. anonymouscommenter says:

    Background Recently, I ran into few incidents where .NET Application hosting web browser control was

Skip to main content