This is Keith Abluton with the Forefront Edge team at Microsoft. I had a case recently where a legacy application that was improperly coded was causing a critical situation for one of our customers. The application was used to send information to a server that was located on the Internet.
I won't bore you with all of the details but the bottom line is that the application was not following RFC 2616 (HTTP/1.1) and Forefront TMG was dropping some of the traffic coming back from the server. Since the RFC was not closely adhered to by the application, part of a response from the remote server was unexpected and TMG was dropping it.
Because of the mission critical nature of this application the customer didn't have time to go back to the legacy application developers and get them to fix their code. They needed a workaround to get the application working immediately.
Since the traffic in question was utilizing the HTTP protocol we needed to create a couple of rules on TMG to allow the traffic to pass without being evaluated by the Web Proxy Filter. Why would you want to bypass the Web Proxy Filter you ask? The filter will follow RFC and may drop, silently or otherwise, traffic that doesn’t follow RFC. You may find yourself in a similar situation so I wanted to lay out the instructions for doing this.
Here are the steps that we took to get this application working.
1.) In the ISA/TMG MMC click on the Firewall Policy branch. In the far right choose the Toolbox tab and click on Protocols. (See Figure 1)
2.) Under Protocols select New, then Protocol. This will start the New Protocol Definition Wizard. (See Figure 2)
2.) Under "Protocol Definition Name" give it something meaningful to you. I named mine HTTP_FilterUnbound. (See Figure 3)
3.) For the Primary Connection Information click on New and fill in the information as follows: Protocol Type: TCP , Direction: Outbound , Port Range: From: 80 To: 80 (See Figure 4)
5.) Click Next and choose "No" for the question "Do you want to use secondary connections". Choose Finish and then Apply.
6.) Now create an Access Rule that will use this new User Defined Protocol. Keep in mind that you want to make this rule as specific as you can so that it only applies to the traffic you want to bypass the Web Proxy Filter. In my example I created a Computer Object with the IP address of the server that this application was communicating with. By doing this, all other traffic will be evaluated by your normal Web Access rules and will have the Web Proxy Filter applied to it.
7.) Create a Deny Rule that is directly below the rule you just created in Step 6. The easiest way to do this is to copy the Access Rule and make a few changes to it. Change it to a Deny rule and change the Protocol to be the normal HTTP protocol. There is a specific reason for creating this Deny Rule. ISA/TMG will evaluate the rules in order from top to bottom. Unless it hits this Deny rule, it will continue down your rules list. If it finds another rule that matches the criteria AND uses the Web Proxy Filter, it will use that rule instead. Because we have the Deny Rule immediately below it, this will not occur. The rule you created in Step 6 will be applied to the traffic. If this rule were not in place, it would have continued down to my default Internet Access rule and the Web Proxy Filter would have been applied. The result would be that the application would still fail.
8.) I moved both these rules up to the top of the list to make sure they were being applied before my default Internet Access rule. (See Figure 5)
In this particular communications for a third party application that wasn’t following RFC for the HTTP Protocol was being dropped by TMG. The customer had an immediate need for a workaround. By creating a rule with a custom protocol that did not apply the Web Proxy Filter we were able to provide some relief to the customer. Keep in mind that there are some potential security ramifications using this approach and it should only be used sparingly and care should be taken to make sure the newly created rule is only being applied to the traffic for which it was intended. In any case, the best resolution would be to work with the vendor of the application to fix their coding to follow the RFC more strictly.
Author: Keith Abluton, Sr. Support Engineer, Forefront Edge Team, Microsoft
Reviewer: Brett Crane, Support Escalation Engineer, Forefront Edge Team, Microsoft