Server Publishing is Failing through ISA Server
Introduction
I occasionally see cases where a customer has attempted to publish a non Web Server (such as SMTP) through ISA and it is failing. This is almost always caused by improper routing within the network. Because I have seen a lot of these cases, one of the first places I look on is under the To Tab on the Server Publishing Rule. There is a section (Figure 1) which specifies how ISA Server forwards the requests to the published server. In a lot of cases changing the default setting of “Requests appear to come from the original client” to “Requests appear to come from the ISA Server” resolves the issue.
Figure 1
In this article I will show you why changing this setting from its default is just a workaround that masks a larger problem of incorrect routing on the network.
Scenario 1 (Wrong way and it fails)
Consider a scenario where a published SMTP Server has an IP address of 10.1.1.10 and is using a default gateway of 10.1.1.231 (Figure 2). There is an ISA Server 2006 which is being used to publish the SMTP Server. The ISA Server has an Internal IP address of 10.1.1.254 and its external interface has a public IP address.
Figure 2
In this scenario if the ISA Administrator took the default on the Server Publishing rule and left it at “Requests appear to come from the original client” the rule would not work. Using a network trace tool such as Netmon 3 gives us a clue as to why this is not working (Figure 3).
Figure 3
In frame 56 we see the external client (65.53.10.102) send a SYN packet. In frame 57 the SMTP Server (10.1.1.10) sends a SYN ACK back to the external client. We never see the final ACK so the TCP 3 way handshake is never completed. A closer look at frame 57 shows us why this never makes it back to the client. The destination MAC address is for the device which has an IP address of 10.1.1.231. This makes sense since it is the default gateway for the SMTP Server. When this device sees this traffic it is dropped since it is not expected. In my test environment this device was also an ISA Server and a look at the monitoring (Figure 4) shows us this traffic is denied. Any device that employs stateful inspection is going to drop this traffic because it never received the original SYN packet from the client.
Figure 4
Scenario 2 (Wrong way but it works)
Using the same network we used in Scenario 1 we can get around this by changing the setting in the Server Publishing rule to “Requests appear to come from the ISA Server” (Figure 5). This setting basically NATs the connection so that all traffic is seen as coming from the ISA internal interface.
Figure 5
Now the publishing rule works but a Netmon trace (Figure 6 ) taken on the SMTP server shows that every request is seen as coming from the ISA Server (10.1.1.254).
Figure 6
Although this now works it is NOT an ideal situation. There are some applications that may want the original client IP address in order to function properly. All that this is doing is “working around” an incorrect network setup.
Scenario 3 (Correct way)
In this scenario we have the same ISA Server and the same SMTP Server except now the default gateway is set to be the IP address of the ISA Server which is doing the publishing (Figure 7). (Note: The default gateway for the published server does not have to be the ISA Server but it must ultimately route out through the ISA Server or it will still fail.) Remember that the Published Server is basically a SecureNAT client to the ISA Server so the routing infrastructure will need to support that. For more on this please see the article
Setting up the default gateway for SecureNAT clients at Microsoft TechNet Library.
Figure 7
Now the publishing rule works and we can see the IP address of the client in the Netmon trace taken on the SMTP Server.
Figure 8
Conclusion
In this article I demonstrated several ways in which ISA Server could publish a Server and 1 common mistake made by ISA Administrators. Although ISA Server is very flexible and will allow you to publish servers even when the network is not necessarily set up the way that it should be this may not always be your best option.
Author
Keith Abluton
Security Support Engineer
Microsoft CSS Forefront Security Edge Team
Technical Reviewers
Billy Price
Security Support Engineer
Microsoft CSS Forefront Security Edge Team
Yuri Diogenes
Security Support Engineer
Microsoft CSS Forefront Security Edge Team