ISA Server Firewall Service crashed…but why?

1. Introduction

 

When question that I always receive when working with Firewall Service crashing is: why is it crashing? When the answer is: due a third party application…then the next question is: how is that? I thought each process was running independently and one couldn’t crash the other, right? That’s correct; however you need to remember how things work on ISA core architecture. Let’s step back and review the following diagram:

 

Figure 1 – ISA Server architecture (from ISA Server 2006 Firewall Core Document)

 

Notice that Firewall Service (wspsrv.exe) runs in User Mode while Firewall Engine (fweng.sys) runs in kernel mode. While is true that each process has its own address space, security token, etc, it is also true that each process is composed by threads, where each thread can be executing a different set of instructions and interacting with different components. ISA Server 2006 allows third party application to build their proprietary Web Filter (ISA Server supports ISAPI filter development) and by doing so it will somehow interfere in the way that Web Proxy Filter acts by default.

 

2. Digging in

 

If you use Process Explorer (or ProcMon) to open the properties of wspsrv.exe process you will see that there are many threads in execution as shown Figure 2:

 

 

Figure 2 – Threads running in the context of wspsrv.exe process.

 

If you select one of those threads and click Stack you will see the stack content and the modules in use. A stack is a region of the memory that is used to temporarily store data; it is added and removed in a last-in-first-out base. When you choose the thread and click on the stack you can see what it is in execution on that thread on that moment. Having this foundation understanding let’s take a look in the following diagram to understand how wspsrv.exe process can be affected by a third party filter:

 

 

Figure 3 – Firewall service process and the threads that belongs to it.

 

As you can see in this diagram there are some threads within the wspsrv.exe process and I’m using the stack of two of them as example. First stack from thread 1988 just have Microsoft modules and for the purpose of this example let’s focus on the stack that belongs to the thread 1920 which has a third party module (MyWebFilter.dll) loaded into it.

 

If this module for some reason execute an operation that cause an unhandled exception we might compromised the whole thread and possible crash the process. If you do not have a debugger attached to the process you will not get a dump for the wspsrv.exe, the only thing that will happen is that Firewall Service will crash (process quits from the memory) and an event is registered in the event viewer saying that the Firewall Service crashed. If you want to catch this type of crash you need a debugger attached to the process, to do that you can use an article that I wrote some time back about that, check it out here.

 

3. Access Violation

 

For the purpose of this example let's assume that this fake third party filter module did cause Firewall Service to crash and since I did have DebugDiag attached to wspsrv.exe process I was able to catch the second chance crash. In this case here it is the result for this crash by a partial output from !analyze –v command:

 

0:040> !analyze -v

*******************************************************************************

* *

* Exception Analysis *

* *

*******************************************************************************

FAULTING_IP:

ntdll!KiUserExceptionDispatcher+e

7c82857e 0ac0 or al,al

EXCEPTION_RECORD: 102cf8cc -- (.exr 0x102cf8cc)

ExceptionAddress: 10161a50 (MyWebFilter.dll+0x00001a50)

   ExceptionCode: c0000005 (Access violation)

  ExceptionFlags: 00000000

NumberParameters: 2

   Parameter[0]: 00000000

   Parameter[1]: 10192068

Attempt to read from address 10192068

DEFAULT_BUCKET_ID: STATUS_STACKOVERFLOW

PROCESS_NAME: wspsrv.exe

ERROR_CODE: (NTSTATUS) 0xc00000fd - A new guard page for the stack cannot be created.

READ_ADDRESS: 1016caac

NTGLOBALFLAG: 0

APPLICATION_VERIFIER_FLAGS: 0

IP_MODULE_UNLOADED:

MyFilter+1a50

10161a50 ?? ???

CONTEXT: 102cf8e8 -- (.cxr 0x102cf8e8)

eax=102cfe44 ebx=00000000 ecx=10192048 edx=f9b10046 esi=10192048 edi=102cfe38

eip=10161a50 esp=102cfbb4 ebp=102cfbdc iopl=0 nv up ei pl nz na pe nc

cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010206

MyWebFilter.dll+0x1a50:

10161a50 ?? ???

Resetting default scope

RECURRING_STACK: From frames 0x7 to 0xa

LAST_CONTROL_TRANSFER: from 102cfe38 to 10161a50

IP_ON_STACK:

+102cfe38

102cfe38 5c pop esp

FRAME_ONE_INVALID: 1

STACK_TEXT:

WARNING: Frame IP not in any known module. Following frames may be wrong.

102cfbb0 102cfe38 00000006 00000000 10192048 MyWebFilter.dll+0x1a50

102cfc2c 776bf813 77796898 00327277 00000000 0x102cfe38

102cfc30 77796898 00327277 00000000 00000000 ole32!COleStaticMutexSem::Release+0x1a

102cfe6c 7c83ac6c 00000000 0efd5400 0efd54a8 ole32!gComLock+0x18

102cfec8 7c83ca92 62251dc0 00000000 0efd5400 ntdll!RtlpWaitOrTimerCallout+0x74

102cfeec 7c83a857 0efd54a8 7c88b080 0ef88528 ntdll!RtlpAsyncWaitCallbackCompletion+0x37

102cff44 7c83aa3b 7c83ca5b 0efd54a8 00000000 ntdll!RtlpWorkerCallout+0x71

102cff64 7c83aab2 00000000 0efd54a8 0ef88528 ntdll!RtlpExecuteWorkerRequest+0x4f

102cff78 7c839f90 7c83a9fa 00000000 0efd54a8 ntdll!RtlpApcCallout+0x11

102cffb8 77e6482f 00000000 00000000 00000000 ntdll!RtlpWorkerThread+0x61

102cffec 00000000 7c839f2b 00000000 00000000 kernel32!BaseThreadStart+0x34

Let see our registers:

 

0:040> r

eax=00000000 ebx=00000000 ecx=1016caac edx=7c828786 esi=00000000 edi=00000000

eip=1016caac esp=102911b0 ebp=102911d0 iopl=0 nv up ei pl zr na pe nc

cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00010246

MyWebFilter.dll+0xcaac:

1016caac ?? ???

 

Now let’s look at the EIP register (which points to where in the program the processor was currently executing the code):

 

0:040> r eip

eip=1016caac

 

Let’s dump it:

 

0:040> dd eip

1016caac ???????? ???????? ???????? ????????

1016cabc ???????? ???????? ???????? ????????

1016cacc ???????? ???????? ???????? ????????

1016cadc ???????? ???????? ???????? ????????

1016caec ???????? ???????? ???????? ????????

1016cafc ???????? ???????? ???????? ????????

1016cb0c ???????? ???????? ???????? ????????

1016cb1c ???????? ???????? ???????? ????????

Well, it doesn’t looks good since it is pointing to a bunch of question mark (either invalid or not accessible memory). Let’s see what memory address EIP was pointing to:

 

0:040> !address eip

    10160000 : 10160000 - 00040000

                    Type 00000000

                    Protect 00000001 PAGE_NOACCESS

                    State 00010000 MEM_FREE

                    Usage RegionUsageFree

 

What this PAGE_NOACCESS means? Let’s see the definition from MSDN:

 

“Pages in the region become guard pages. Any attempt to read from or write to a guard page causes the operating system to raise the STATUS_GUARD_PAGE exception and turn off the guard page status. Guard pages thus act as a one-shot access alarm. The PAGE_GUARD flag is a page protection modifier. An application uses it with one of the other page protection flags, with one exception: it cannot be used with PAGE_NOACCESS. When an access attempt leads the operating system to turn off guard page status, the underlying page protection takes over. If a guard page exception occurs during a system service, the service typically returns a failure status indicator.”

From: https://msdn.microsoft.com/en-us/library/aa450977.aspx

 

One strong hypothesis here (since we don’t have the code for the third party application to debug) is that this module tried to access an invalid memory address and therefore corrupted the stack causing the access violation. This was enough to cause the whole process (wspsrv.exe) to crash.