ISA Server Stop Answering Requests and Firewall Service Hangs

The problem that this post is going to discuss was related to a random issue where certain times of the day the ISA Server was stopping answering requests and when the firewall administrator tried to restart the firewall service the service didn’t start. The only event that we have prior to the issue happens was the one below:


Event Type: Error

Event Source:     Microsoft ISA Server Web Proxy

Event Category:   None

Event ID:   14172

Date:       13/3/2009

Time:       18:37:43

User:       N/A

Computer:   ISASRV


The cache was not properly initialized.  caching will be disabled (internal code 503. Identify the specific reason for the failure from previous relevant event logs. Fix the problem, and then restart the Firewall service to enable caching.


Doing a quick assessment I could see that the Antivirus was scanning all folders, including ISA Folders (not good at all). As a troubleshooting step I disabled the AV but the issue persisted. Using ProcMon I could see that when ISA Storage process (ISAStg.exe) was trying to read a value in register the AV filter drive was still present in kernel mode and intercepting the request.  Here it is the sequence:


ISASTG process:


34408 2:23:05.8643957 PM      isastg.exe  3904  RegEnumValue      HKLM\SOFTWARE\Microsoft\Fpc\Storage\Array-Root\Arrays\{0A8D8F99-6862-47B9-9388-12890728AF1A}\Servers\{B622A644-418A-40E1-988F-C1182B246652}\Proxy-Cache-Directories\Proxy-Cache-Directory1  SUCCESS     Index: 3, Name: msFPCDirectoryName, Type: REG_SZ, Length: 34, Data: D:\urlcache\Dir1


The stack for this process shows the AV filter drive (klif.sys):


0      ntoskrnl.exe  ntoskrnl.exe + 0x17859f    0x8097859f    C:\WINDOWS\system32\ntoskrnl.exe

1      ntoskrnl.exe  ntoskrnl.exe + 0x146c3c    0x80946c3c    C:\WINDOWS\system32\ntoskrnl.exe

2      klif.sys      klif.sys + 0xfa1c    0xf685fa1c    C:\WINDOWS\system32\drivers\klif.sys

3      ADVAPI32.dll  ADVAPI32.dll + 0x12530     0x77f62530    C:\WINDOWS\system32\ADVAPI32.dll

4      isastg.exe    isastg.exe + 0x8352  0x408352      D:\Program Files\Microsoft ISA Server\isastg.exe

5      isastg.exe    isastg.exe + 0x9054  0x409054      D:\Program Files\Microsoft ISA Server\isastg.exe

6      RPCRT4.dll    RPCRT4.dll + 0x30193 0x77c80193    C:\WINDOWS\system32\RPCRT4.dll

7      RPCRT4.dll    RPCRT4.dll + 0x933e1 0x77ce33e1    C:\WINDOWS\system32\RPCRT4.dll

8      RPCRT4.dll    RPCRT4.dll + 0x935c4 0x77ce35c4    C:\WINDOWS\system32\RPCRT4.dll

9      RPCRT4.dll    RPCRT4.dll + 0x2ff7a 0x77c7ff7a    C:\WINDOWS\system32\RPCRT4.dll

10     RPCRT4.dll    RPCRT4.dll + 0x3042d 0x77c8042d    C:\WINDOWS\system32\RPCRT4.dll

11     RPCRT4.dll    RPCRT4.dll + 0x30353 0x77c80353    C:\WINDOWS\system32\RPCRT4.dll

12     RPCRT4.dll    RPCRT4.dll + 0x311dc 0x77c811dc    C:\WINDOWS\system32\RPCRT4.dll

13     RPCRT4.dll    RPCRT4.dll + 0x312f0 0x77c812f0    C:\WINDOWS\system32\RPCRT4.dll

14     RPCRT4.dll    RPCRT4.dll + 0x38678 0x77c88678    C:\WINDOWS\system32\RPCRT4.dll

15     RPCRT4.dll    RPCRT4.dll + 0x38792 0x77c88792    C:\WINDOWS\system32\RPCRT4.dll

16     RPCRT4.dll    RPCRT4.dll + 0x3872d 0x77c8872d    C:\WINDOWS\system32\RPCRT4.dll

17     RPCRT4.dll    RPCRT4.dll + 0x2b110 0x77c7b110    C:\WINDOWS\system32\RPCRT4.dll

18     kernel32.dll  kernel32.dll + 0x24829     0x77e64829    C:\WINDOWS\system32\kernel32.dll


Later on we fail to create the file:


34838  2:23:05.9429702 PM   mspadmin.exe  612    CreateFile    D:\urlcache   SUCCESS       Desired Access: Read Attributes, Read Control, Write DAC, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, Impersonating: S-1-5-21-2611182321-852623426-2620623114-500, OpenResult: Opened


34839  2:23:05.9430612 PM   mspadmin.exe  612    QueryBasicInformationFile  D:\urlcache   SUCCESS       CreationTime: 2/13/2009 1:51:15 PM, LastAccessTime: 2/13/2009 2:23:04 PM, LastWriteTime: 2/13/2009 1:51:15 PM, ChangeTime: 2/13/2009 1:51:15 PM, FileAttributes: D


34840  2:23:05.9431081 PM   mspadmin.exe  612    QuerySecurityFile    D:\urlcache   BUFFER OVERFLOW       Information: Owner, Group, DACL, 0x80000000


We uninstalled the AV and the issue didn’t happen anymore. Since his environment had a requirement to have AV installed on ever single Windows machine we implemented the correct folder exclusion following the article “Considerations when using antivirus software on ISA Server” and the environment got stabilized.


Interesting side of this story is that this article was published exactly one year ago, one year later we still have firewall administrators not following such recommendation and therefore having unexpected downtimes.

Comments (1)
  1. sohbet odalari says:

    Süper paylaşım, teşekkürler..

    My pleasure, glad you found the articles helpful, I found very useful, I wish working the success.

Comments are closed.

Skip to main content