Capturing an user mode crash on ISA Server – Part 2 of 2
1. Introduction
Back in May I wrote the first part of this blog where I explained how to use DebugDiag to capture a crash on ISA Server Service. Now is time to see how it looks like.
2. Got ya !!!
An administrator’s feeling when it catches a crash while the debugger is attached to the process that he was hunting for awhile is just super. Now with the crash (dump) files in hands it is time to decide what to and here it comes the experience and the knowledge that will drive to the answer. Sometimes, even when the administrator has strong debugging skills he could be limited due the fact that in some situations only the private symbols will really give you the full explanation about what happened during that time.
3. Reviewing the Crash
Assuming that you have the Debugging Tools installed and the public symbols configured, it is time to open your crash dump. When you open the crash dump, the first command that you probably will use (since this is a crash) is the !analyze-v. Theresult will vary, but you will have an output structured in the following parts:
0:019> !analyze -v
*************************************************************************
* *
* Exception Analysis *
* *
*************************************************************************
Debugger WatsonDb Connection::Open failed 80004005
FAULTING_IP:
isamon2+1a7a
1e301a7a f3a5 rep movs dword ptr es:[edi],dword ptr [esi]
EXCEPTION_RECORD: ffffffff -- (.exr ffffffffffffffff)
//Address where the exception happened
ExceptionAddress: 1e301a7a (isamon2+0x00001a7a)
//Type of Exception, in this case access violation
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000000
Parameter[1]: 20686000
Attempt to read from address 20686000
DEFAULT_BUCKET_ID: APPLICATION_FAULT
//Process (ISA Server Firewall Service Process)
PROCESS_NAME: wspsrv.exe
ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx" referenced memory
at "0x%08lx". The memory could not be "%s".
READ_ADDRESS: 20686000
BUGCHECK_STR: ACCESS_VIOLATION
LAST_CONTROL_TRANSFER: from 1e307a44 to 1e301a7a
STACK_TEXT:
WARNING: Stack unwind information not available. Following frames may be wrong.
01639674 1e307a44 20685fdf 1e31ecc4 00000029 isamon2+0x1a7a
01639698 1e30e96d 1e301b0b 1e3b3ff8 2734e7b8 isamon2!GetFilterVersion+0x16d4
0163969c 1e301b0b 1e3b3ff8 2734e7b8 0163fd38 isamon2!DllUnregisterServer+0x90d
016396b8 1e308292 20685b68 1e31e81c 00000004 isamon2+0x1b0b
016396ec 6472eb58 0163d96c 0000000b f4e03f1f isamon2!GetFilterVersion+0x1f22
01639710 01639750 1f486d50 1f486d38 1e3e8ec7 W3Filter!HTTP_FILTER::GetInfo+0x4d
01639740 1e30aa2a 1e3e8ebc 1e320d5c 1f495a6c 0x1639750
0163fc40 71c02679 0163fc64 63703493 275362c0 isamon2!GetFilterVersion+0x46ba
71b2199f 90909090 ffffff90 b356acff b356b571 ws2_32!WSARecv+0x77
71b219bf b3586cff b3587571 00000271 b3584500 0x90909090
71b219c3 b3587571 00000271 b3584500 b3584e71 0xb3586cff
71b219c7 00000000 b3584500 b3584e71 90909071 0xb3587571
// In the stack above, the function at the top of the stack is GetFilterVersion in the module isamon2.dll, the denotation is “file_name!function_name”. The “+0x16d4” next to the function name indicates how far into the code of that function the thread is. Notice that W3Filter.dll call uses the function HTTP_Filter right before the ISAMon2 takes over.
FAULTING_THREAD: 00000a7c
FOLLOWUP_IP:
isamon2+1a7a
1e301a7a f3a5 rep movs dword ptr es:[edi],dword ptr [esi]
SYMBOL_STACK_INDEX: 0
FOLLOWUP_NAME: wintriag
MODULE_NAME: isamon2
IMAGE_NAME: isamon2.dll
DEBUG_FLR_IMAGE_TIMESTAMP: 40b1fcdf
SYMBOL_NAME: isamon2+1a7a
STACK_COMMAND: ~19s; .ecxr ; kb
FAILURE_BUCKET_ID: ACCESS_VIOLATION_isamon2+1a7a
BUCKET_ID: ACCESS_VIOLATION_isamon2+1a7a
Followup: MachineOwner
In this case you can see the ISAMon2 version by running the command lmvm isamon2 the result will be the following:
0:019> lmvm isamon2
start end module name
1e300000 1e3ab000 isamon2 C (export symbols) isamon2.dll
Loaded symbol image file: isamon2.dll
Image path: C:\Program Files\Microsoft ISA Server\isamon2.dll
Image name: isamon2.dll
Timestamp: Mon May 24 08:47:11 2004 (40B1FCDF)
CheckSum: 00000000
ImageSize: 000AB000
File version: 2.0.0.2
Product version: 2.0.0.2
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 2.0 Dll
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: GFI Software Ltd.
ProductName: GFI Web Monitor
InternalName: GFIWEBMON
OriginalFilename: IsaMon2.dll
ProductVersion: 2, 0, 0, 2
FileVersion: 2, 0, 0, 2
FileDescription: GFI Web Monitor
LegalCopyright: Copyright © 2003
This was a case where this filter was causing an access violation and crashing the Firewall Service. An access violation occur when an application attempts to use memory located in another process address space. By referring the customer to the third party vendor for solution they were able to fix it by updating the version.
4. Conclusion
By examining and understanding a call stack like this when a server or application crashes makes you more comfortable to tells with authority what components were involved in the crash and what types of actions they were performing. Remember that this is not always the case, however this much information can often lead to root cause of a problem without any further debugging.