ISA Server triggers lots of 14197 events
1. Introduction
This post is about a scenario where Firewall Administrator notices that the application log in the ISA Server computer is full of events 14197 saying:
Event ID 14197
Source Microsoft Web Proxy
Type Error
Description ISA Server failed to write content to cache file. The error code in the Data area of the event properties indicates the cause of the failure.
He notices that after some time with this event one of the following symptoms happen:
· The Firewall Service stops responding and when he tries to restarts it hangs with starting status for awhile before the status change to started.
· Sometimes the Firewall Service quits without any message.
· Sometimes it is not possible to restart firewall service because it hangs in stopping status.
I’m very familiar with those events and most of the time (9 out 10) this is caused by a third party software scanning or locking the cache file, the article https://support.microsoft.com/kb/887311 explains that in details. For this particular scenario the most common third party application that can cause this is the antivirus. When you have file scan antivirus installed on ISA Server (or TMG) you need to use the article Considerations when using antivirus software on ISA Server to create an exclusion list of the folders, files and process that shouldn’t be scanned by the antivirus.
So, if I follow the recommendations from that article I should be good, right? You should, as long as the exclusions are REALLY…. REALLY in place. I’m emphasizing the “really” because of some recent unfortunate experiences that I had; let me tell you a story to chill the bones about a thing that I saw…oops, that a song.
2. The Tales
Recently I worked in at least three different scenarios very similar where the firewall administrator created the exclusion list by following that article, however the exclusions were added though AV policy deployment located in the central antivirus server management. The reason why this policy was deployed via central location was because the local client AV installed on ISA didn’t allow local changes; everything was done by this AV Server. However even after making those exclusions the event 14197 was still happening. Of course I heard the most common argument in such scenarios: I told you that this was not a problem on our AV, this is an ISA issue for sure.
Ok, at that point I couldn’t really argue and had to think that every piece of software is innocent until we prove otherwise. So I followed this line of think I went to a path of collecting evidence to see who was doing that. Hunting time !!
3. Preparing the Environment
One way to collect evidences that there is a process other than ISA Server process is to audit the folder where the cache is located, by default %systemdrive%\urlcache. To do that you need to follow the steps below:
1. Enable local audit for object access (Start / Run / secpol.msc):
2. Enable Auditing for the URLCache folder for the following users:
Notice that the column access says special, this is because I’m choosing the following type of access:
The reason why I’m selecting those users is because:
· Most of the ISA Server services run under System account and Firewall Service runs under Network Service account.
· The Antivirus that I’m running on my lab (MS FCS) runs under System account.
· I’m selecting everyone because I want to catch manual attempts (from users) to access this folder.
After finishing that you just need to wait for the next occurrence.
3. Collecting Evidences
In order to show what you will see in the event viewer when a process try to access the URLCache folder I decided to build a lab and use Forefront Client Security without the ISA folder exclusions. After implementing the actions above here what I got when FCS scanned the URLCache folder:
Event Type: Success Audit
Event Source: Security
Event Category: Object Access
Event ID: 560
Date: 1/30/2010
Time: 5:00:22 AM
User: NT AUTHORITY\SYSTEM
Computer: ISACONTN1
Description:
Object Open:
Object Server: Security
Object Type: File
Object Name: C:\urlcache\Dir1.cdat
Handle ID: 1772
Operation ID: {0,6542847}
Process ID: 1772
Image File Name: C:\Program Files\Microsoft Forefront\Forefront System\Client\AntiMalware\MsMpEng.exe
Primary User Name: ISACONTN1$
Primary Domain: CONTOSO
Primary Logon ID: (0x0,0x3E7)
Client User Name: -
Client Domain: -
Client Logon ID: -
Accesses: ReadAttributes
Privileges: -
Restricted Sid Count: 0
Access Mask: 0x80
This is the evidence that you need in order to prove that there is another process accessing this file!!
At this point you probably are asking: OK, I got all that, but on your real scenarios, why the folder exclusion was not working? Well, that I don’t technically know since it was a third party AV that was failing to push the policy from the central location to the AV agent installed on ISA Server, hence the folder exclusion was not taking effect.