Capturing an user mode crash on ISA Server – Part 2 of 2


1. Introduction


 


Back in May I wrote the first part of this blog where I explained how to use DebugDiag to capture a crash on ISA Server Service. Now is time to see how it looks like.


 


2. Got ya !!!


 


An administrator’s feeling when it catches a crash while the debugger is attached to the process that he was hunting for awhile is just super. Now with the crash (dump) files in hands it is time to decide what to and here it comes the experience and the knowledge that will drive to the answer. Sometimes, even when the administrator has strong debugging skills he could be limited due the fact that in some situations only the private symbols will really give you the full explanation about what happened during that time.


 


3. Reviewing the Crash


 


Assuming that you have the Debugging Tools installed and the public symbols configured, it is time to open your crash dump. When you open the crash dump, the first command that you probably will use (since this is a crash) is the !analyze-v. The result will vary, but you will have an output structured in the following parts:


 


0:019> !analyze -v


*************************************************************************


*                                                                       *


* Exception Analysis                                                    *


*                                                                       *


*************************************************************************


 


Debugger WatsonDb Connection::Open failed 80004005


 


FAULTING_IP:


isamon2+1a7a


1e301a7a f3a5 rep movs dword ptr es:[edi],dword ptr [esi]


 


EXCEPTION_RECORD: ffffffff — (.exr ffffffffffffffff)


 


//Address where the exception happened


ExceptionAddress: 1e301a7a (isamon2+0x00001a7a)


 


//Type of Exception, in this case access violation


ExceptionCode: c0000005 (Access violation)


ExceptionFlags: 00000000


NumberParameters: 2


Parameter[0]: 00000000


Parameter[1]: 20686000


Attempt to read from address 20686000


 


DEFAULT_BUCKET_ID: APPLICATION_FAULT


 


//Process (ISA Server Firewall Service Process)


PROCESS_NAME: wspsrv.exe


 


ERROR_CODE: (NTSTATUS) 0xc0000005 – The instruction at “0x%08lx” referenced memory


at “0x%08lx”. The memory could not be “%s”.


 


READ_ADDRESS: 20686000


BUGCHECK_STR: ACCESS_VIOLATION


LAST_CONTROL_TRANSFER: from 1e307a44 to 1e301a7a


 


STACK_TEXT:


WARNING: Stack unwind information not available. Following frames may be wrong.


01639674 1e307a44 20685fdf 1e31ecc4 00000029 isamon2+0x1a7a


01639698 1e30e96d 1e301b0b 1e3b3ff8 2734e7b8 isamon2!GetFilterVersion+0x16d4


0163969c 1e301b0b 1e3b3ff8 2734e7b8 0163fd38 isamon2!DllUnregisterServer+0x90d


016396b8 1e308292 20685b68 1e31e81c 00000004 isamon2+0x1b0b


016396ec 6472eb58 0163d96c 0000000b f4e03f1f isamon2!GetFilterVersion+0x1f22


01639710 01639750 1f486d50 1f486d38 1e3e8ec7 W3Filter!HTTP_FILTER::GetInfo+0x4d


01639740 1e30aa2a 1e3e8ebc 1e320d5c 1f495a6c 0x1639750


0163fc40 71c02679 0163fc64 63703493 275362c0 isamon2!GetFilterVersion+0x46ba


71b2199f 90909090 ffffff90 b356acff b356b571 ws2_32!WSARecv+0x77


71b219bf b3586cff b3587571 00000271 b3584500 0x90909090


71b219c3 b3587571 00000271 b3584500 b3584e71 0xb3586cff


71b219c7 00000000 b3584500 b3584e71 90909071 0xb3587571


 


// In the stack above, the function at the top of the stack is GetFilterVersion in the module isamon2.dll, the denotation is “file_name!function_name”. The “+0x16d4” next to the function name indicates how far into the code of that function the thread is. Notice that W3Filter.dll call uses the function HTTP_Filter right before the ISAMon2 takes over. 


 


FAULTING_THREAD: 00000a7c


FOLLOWUP_IP:


isamon2+1a7a


1e301a7a f3a5 rep movs dword ptr es:[edi],dword ptr [esi]


 


SYMBOL_STACK_INDEX: 0


FOLLOWUP_NAME: wintriag


MODULE_NAME: isamon2


IMAGE_NAME: isamon2.dll


DEBUG_FLR_IMAGE_TIMESTAMP: 40b1fcdf


SYMBOL_NAME: isamon2+1a7a


STACK_COMMAND: ~19s; .ecxr ; kb


FAILURE_BUCKET_ID: ACCESS_VIOLATION_isamon2+1a7a


BUCKET_ID: ACCESS_VIOLATION_isamon2+1a7a


Followup: MachineOwner


 


In this case you can see the ISAMon2 version by running the command lmvm isamon2 the result will be the following:


 


0:019> lmvm isamon2


start end module name


1e300000 1e3ab000 isamon2 C (export symbols) isamon2.dll


Loaded symbol image file: isamon2.dll


Image path: C:\Program Files\Microsoft ISA Server\isamon2.dll


Image name: isamon2.dll


Timestamp: Mon May 24 08:47:11 2004 (40B1FCDF)


CheckSum: 00000000


ImageSize: 000AB000


File version: 2.0.0.2


Product version: 2.0.0.2


File flags: 0 (Mask 3F)


File OS: 40004 NT Win32


File type: 2.0 Dll


File date: 00000000.00000000


Translations: 0409.04b0


CompanyName: GFI Software Ltd.


ProductName: GFI Web Monitor


InternalName: GFIWEBMON


OriginalFilename: IsaMon2.dll


ProductVersion: 2, 0, 0, 2


FileVersion: 2, 0, 0, 2


FileDescription: GFI Web Monitor


LegalCopyright: Copyright © 2003


 


This was a case where this filter was causing an access violation and crashing the Firewall Service.  An access violation occur when an application attempts to use memory located in another process address space. By referring the customer to the third party vendor for solution they were able to fix it by updating the version.


 


4. Conclusion


 


By examining and understanding a call stack like this when a server or application crashes makes you more comfortable to tells with authority what components were involved in the crash and what types of actions they were performing. Remember that this is not always the case, however this much information can often lead to root cause of a problem without any further debugging.


 


 

Comments (1)

  1. Anonymous says:

    1. Introduction This is another one of those cases where ISA Server Service mysterious crashes once a