Capturing an user mode crash on ISA Server – Part 2 of 2

1. Introduction


Back in May I wrote the first part of this blog where I explained how to use DebugDiag to capture a crash on ISA Server Service. Now is time to see how it looks like.


2. Got ya !!!


An administrator’s feeling when it catches a crash while the debugger is attached to the process that he was hunting for awhile is just super. Now with the crash (dump) files in hands it is time to decide what to and here it comes the experience and the knowledge that will drive to the answer. Sometimes, even when the administrator has strong debugging skills he could be limited due the fact that in some situations only the private symbols will really give you the full explanation about what happened during that time.


3. Reviewing the Crash


Assuming that you have the Debugging Tools installed and the public symbols configured, it is time to open your crash dump. When you open the crash dump, the first command that you probably will use (since this is a crash) is the !analyze-v. The result will vary, but you will have an output structured in the following parts:


0:019> !analyze -v


*                                                                       *

* Exception Analysis                                                    *

*                                                                       *



Debugger WatsonDb Connection::Open failed 80004005




1e301a7a f3a5 rep movs dword ptr es:[edi],dword ptr [esi]


EXCEPTION_RECORD: ffffffff — (.exr ffffffffffffffff)


//Address where the exception happened

ExceptionAddress: 1e301a7a (isamon2+0x00001a7a)


//Type of Exception, in this case access violation

ExceptionCode: c0000005 (Access violation)

ExceptionFlags: 00000000

NumberParameters: 2

Parameter[0]: 00000000

Parameter[1]: 20686000

Attempt to read from address 20686000




//Process (ISA Server Firewall Service Process)

PROCESS_NAME: wspsrv.exe


ERROR_CODE: (NTSTATUS) 0xc0000005 – The instruction at “0x%08lx” referenced memory

at “0x%08lx”. The memory could not be “%s”.


READ_ADDRESS: 20686000


LAST_CONTROL_TRANSFER: from 1e307a44 to 1e301a7a



WARNING: Stack unwind information not available. Following frames may be wrong.

01639674 1e307a44 20685fdf 1e31ecc4 00000029 isamon2+0x1a7a

01639698 1e30e96d 1e301b0b 1e3b3ff8 2734e7b8 isamon2!GetFilterVersion+0x16d4

0163969c 1e301b0b 1e3b3ff8 2734e7b8 0163fd38 isamon2!DllUnregisterServer+0x90d

016396b8 1e308292 20685b68 1e31e81c 00000004 isamon2+0x1b0b

016396ec 6472eb58 0163d96c 0000000b f4e03f1f isamon2!GetFilterVersion+0x1f22

01639710 01639750 1f486d50 1f486d38 1e3e8ec7 W3Filter!HTTP_FILTER::GetInfo+0x4d

01639740 1e30aa2a 1e3e8ebc 1e320d5c 1f495a6c 0x1639750

0163fc40 71c02679 0163fc64 63703493 275362c0 isamon2!GetFilterVersion+0x46ba

71b2199f 90909090 ffffff90 b356acff b356b571 ws2_32!WSARecv+0x77

71b219bf b3586cff b3587571 00000271 b3584500 0x90909090

71b219c3 b3587571 00000271 b3584500 b3584e71 0xb3586cff

71b219c7 00000000 b3584500 b3584e71 90909071 0xb3587571


// In the stack above, the function at the top of the stack is GetFilterVersion in the module isamon2.dll, the denotation is “file_name!function_name”. The “+0x16d4” next to the function name indicates how far into the code of that function the thread is. Notice that W3Filter.dll call uses the function HTTP_Filter right before the ISAMon2 takes over. 





1e301a7a f3a5 rep movs dword ptr es:[edi],dword ptr [esi]




MODULE_NAME: isamon2

IMAGE_NAME: isamon2.dll


SYMBOL_NAME: isamon2+1a7a

STACK_COMMAND: ~19s; .ecxr ; kb



Followup: MachineOwner


In this case you can see the ISAMon2 version by running the command lmvm isamon2 the result will be the following:


0:019> lmvm isamon2

start end module name

1e300000 1e3ab000 isamon2 C (export symbols) isamon2.dll

Loaded symbol image file: isamon2.dll

Image path: C:\Program Files\Microsoft ISA Server\isamon2.dll

Image name: isamon2.dll

Timestamp: Mon May 24 08:47:11 2004 (40B1FCDF)

CheckSum: 00000000

ImageSize: 000AB000

File version:

Product version:

File flags: 0 (Mask 3F)

File OS: 40004 NT Win32

File type: 2.0 Dll

File date: 00000000.00000000

Translations: 0409.04b0

CompanyName: GFI Software Ltd.

ProductName: GFI Web Monitor

InternalName: GFIWEBMON

OriginalFilename: IsaMon2.dll

ProductVersion: 2, 0, 0, 2

FileVersion: 2, 0, 0, 2

FileDescription: GFI Web Monitor

LegalCopyright: Copyright © 2003


This was a case where this filter was causing an access violation and crashing the Firewall Service.  An access violation occur when an application attempts to use memory located in another process address space. By referring the customer to the third party vendor for solution they were able to fix it by updating the version.


4. Conclusion


By examining and understanding a call stack like this when a server or application crashes makes you more comfortable to tells with authority what components were involved in the crash and what types of actions they were performing. Remember that this is not always the case, however this much information can often lead to root cause of a problem without any further debugging.



Comments (1)

  1. Anonymous says:

    1. Introduction This is another one of those cases where ISA Server Service mysterious crashes once a