Why do I need a deny rule to make an allow rule for a custom protocol work correctly?

Protocol definitions include lists of primary connections, secondary connections, and associated application filters. Each primary connection includes a port range, which may cover one or more port numbers. When traffic is sent to an ISA Server computer, ISA Server uses the port on which it arrives to identify its protocol.

When a policy rule allows traffic for a certain protocol, the Firewall service checks the definition of the protocol and passes the traffic to all the application filters associated with the protocol definition for processing.

If traffic of a specific type is sent to a port corresponding to a predefined protocol that is associated with an application filter and you do not want the application filter to process this traffic, you can define a custom protocol that has the same primary and secondary connections as the predefined protocol but is not associated with the application filter. Then you can use your custom protocol in a policy rule that allows this traffic. Protocol definitions with primary connections that include the same port are called overlapped protocol definitions.

As an example, let’s say that you have an internal server to which VPN clients send nonstandard HTTP traffic through TCP port 80. When this traffic is allowed by a matching access rule that uses the predefined HTTP protocol and is configured to allow traffic from the VPN Clients network to the Internal network, the Web Proxy Filter is invoked and rejects this traffic because it does not comply with HTTP standards. To allow nonstandard HTTP traffic to reach your nonstandard HTTP server, you can create a custom protocol definition that has a primary connection for outbound traffic through TCP port 80 and is not associated with the Web Proxy Filter. We will refer to this protocol as the CustomHTTP protocol.

Note: The steps provided here may not allow client access to some nonstandard HTTP servers. This usually happens when ISA Server rejects responses from the HTTP server because the behavior of the HTTP server is not RFC-compliant. Analyzing the traffic between the ISA Server computer and the HTTP server should help to reveal the underlying cause.

To allow the nonstandard HTTP traffic, you need to create two access rules:

· An access rule that uses the CustomHTTP protocol and allows traffic from the VPN Clients network to the computer object representing the nonstandard HTTP server.

· An access rule that uses the predefined HTTP protocol and denies traffic from the VPN Clients network to the computer object representing the nonstandard HTTP server.

The new allow rule must come before your original rule that allows HTTP traffic from the VPN Clients network to the Internal network in the ordered list of policy rules, and the new deny rule should be placed immediately after the new allow rule.

The following table lists the rules that you should have to enable traffic in this scenario:

Order

Name

Action

Protocols

From

To

1

Allow CustomHTTP to Nonstandard HTTP Server

Allow

CustomHTTP

VPN Clients

Nonstandard HTTP Server

2

Deny HTTP to Nonstandard HTTP Server

Deny

HTTP

VPN Clients

Nonstandard HTTP Server

3

Allow HTTP to Internal Servers

Allow

HTTP

VPN Clients

Internal

4

Default rule

Deny

All Traffic

All Networks

All Networks

So why do I need the new deny rule (the second rule)? The short answer is that this rule is needed to prevent the third rule or any other rule from invoking the Web Proxy Filter for traffic that matches the first rule.

To understand why this rule is needed, you need to know how ISA Server processes traffic sent to a port that is associated with overlapped protocols. When traffic arrives at a port that is associated with overlapped protocols, the first policy rule that matches the traffic for each of the overlapped protocols (the HTTP and CustomHTTP protocols) is found, and the rule that is highest in the list of rules is applied. In our case, that would be the first rule with the CustomHTTP protocol, which allows traffic to the nonstandard HTTP server but does not invoke the Web Proxy Filter. In addition, all the rules for the overlapped protocols in the ordered list of rules are processed, their secondary connections are added to the session, and the application filters associated with them are invoked until an access rule that denies traffic is encountered. In our case, the second rule, which is a deny rule, stops this processing. Without the second rule, the third rule would be processed for traffic that matches the first rule, and the Web Proxy Filter would be invoked for it.

If the Web Proxy Filter would be invoked by the third rule, the Web Proxy Filter would discover that the traffic does not conform to HTTP standards. The Web Proxy Filter would then block the traffic and add an entry to the Web Proxy log indicating that the Allow HTTP to Internal Servers rule blocked the traffic.

Pesach Shelnitz

ISA Server Team