Isolating problems that seem to be related to ISA Server - Part I
1. Introduction
When you deal with a firewall, many times the issue seems to be related to the firewall itself. Sometimes during troubleshooting, all results that are gathered from analysis point to the firewall as a root cause of the whole problem, but after deep troubleshooting, the conclusion sometimes is that Microsoft® Internet Security and Acceleration (ISA) Server is not responsible.
The goal of this post is to show two situations where ISA Server seems to be the root cause of the problem, but it is not the culprit. The first scenario is published on this post, and the other one will be posted next week.
2. Scenario–Problems establishing a site-to-site VPN between ISA Server and a third-party firewall
The environment is presented from a customer as shown in the following figure.
Figure 1
In this scenario, the customer has an outside partner who needs to connect via a site-to-site virtual private network (VPN). The external partner has other companies that are already configured and running just fine. In the customer's environment, there is an ISA Server computer with two network interface cards (NICs), one connected to an internal switch and the other connected to a modem (DSL) using a dynamic IP address. The customer doesn't know whether this modem has firewall capability.
The VPN is using Point-to-Point Tunneling Protocol (PPTP), and the configuration on the customer's side was done using the article "Site-to-Site VPN in ISA Server 2004" located at Microsoft TechNet. On the partner's side, the configuration is correct, and the partner even showed us that another company that has ISA Server is using the same template.
3. Collecting and analyzing data
After some troubleshooting using Microsoft Network Monitor and ISA Server Monitoring (no relevant error on both of them), we couldn't determine the root cause. At that point, we decided to isolate the ISA Server computer and focus the troubleshooting on the PPTP connection itself. To do this, the following tools were used:
· PPTPCLNT. Client version used to send the PPTP packets.
· PPTPSRV. Server version used to receive the connection attempt.
Using these tools, the idea is to stop making a direct connection, where ISA Server will be the PPTP client and the remote server will be the PPTP server. What we expect to see on this test is basically two things:
· Protocol 47 validation using Generic Route Encapsulation (GRE)
· Test on port 1723
In a normal scenario, the results expected from this tool are shown in the following figures (192.168.3.2 is a test computer used for this example).
Figure 2—PPTP on the client side
Figure 3—PPTP on the server side
Note This tool is available at Windows XP SP2 Support Tools.
We started the Microsoft Network Monitor on the external NIC of the ISA Server computer and on the external NIC of the remote server. After that, we pinged the remote server using the PPTPCLNT tool. The result on the screen was clear, in another words, PPTPSRV did not receive the packet.
By analyzing the traffic captured using Network Monitor on the remote server, we have the explanation about what was failing:
· The first phase was the three-way handshake between ISA Server and the remote server:
192.168.3.4 192.168.3.8 TCP TCP: Flags=.S......, SrcPort=1289, DstPort=1723, Len=0, Seq=640629435, Ack=0, Win=65535 (scale factor not found)
192.168.3.8 192.168.3.4 TCP TCP: Flags=.S..A..., SrcPort=1723, DstPort=1289, Len=0, Seq=4099648897, Ack=640629436, Win=16384 (scale factor not found)
5 12:54:36.651 9.043754 192.168.3.4 192.168.3.8 TCP TCP: Flags=....A... , SrcPort=1289, DstPort=1723, Len=0, Seq=640629436, Ack=4099648898, Win=65535 (scale factor not found)
· After that, the communication starts to happen on TCP port 1723, and the answer happens just fine:
192.168.3.4 192.168.3.8 TCP TCP: Flags=...PA..., SrcPort=1289, DstPort=1723, Len=2, Seq=640629436 - 640629438, Ack=4099648898, Win=65535 (scale factor not found)
+ Ethernet: Etype = Internet IP (IPv4)
+ Ipv4: Next Protocol = TCP, Packet ID = 25870, Total IP Length = 42
- Tcp: Flags=...PA..., SrcPort=1289, DstPort=1723, Len=2, Seq=640629436 - 640629438, Ack=4099648898, Win=65535 (scale factor not found)
SrcPort: 1289
DstPort: 1723
SequenceNumber: 640629436 (0x262F3ABC)
AcknowledgementNumber: 4099648898 (0xF45BAD82)
+ DataOffset: 80 (0x50)
+ Flags: ...PA...
Window: 65535 (scale factor not found)
Checksum: 59567 (0xE8AF)
UrgentPointer: 0 (0x0)
PPTPPayload: Binary Large Object (2 Bytes)
192.168.3.8 192.168.3.4 TCP TCP: Flags=...PA..., SrcPort=1723, DstPort=1289, Len=46, Seq=4099648898 - 4099648944, Ack=640629438, Win=65533 (scale factor not found)
+ Ethernet: Etype = Internet IP (IPv4)
+ Ipv4: Next Protocol = TCP, Packet ID = 3079, Total IP Length = 86
- Tcp: Flags=...PA..., SrcPort=1723, DstPort=1289, Len=46, Seq=4099648898 - 4099648944, Ack=640629438, Win=65533 (scale factor not found)
SrcPort: 1723
DstPort: 1289
SequenceNumber: 4099648898 (0xF45BAD82)
AcknowledgementNumber: 640629438 (0x262F3ABE)
+ DataOffset: 80 (0x50)
+ Flags: ...PA...
Window: 65533 (scale factor not found)
Checksum: 64939 (0xFDAB)
UrgentPointer: 0 (0x0)
PPTPPayload: Binary Large Object (46 Bytes)
· The second part of the PPTP communication is done using GRE (protocol 47). The result is:
192.168.3.4 192.168.3.8 GRE GRE: Protocol = 0x6c6c, Flags = .RK.s........... Version 1
+ Ethernet: Etype = Internet IP (IPv4)
+ Ipv4: Next Protocol = GRE, Packet ID = 25873, Total IP Length = 52
- Gre: Protocol = 0x6c6c, Flags = .RK.s........... Version 1
- flags: .RK.s........... Version 1
C: (0...............) Checksum Absent
R: (.1..............) Offset Present
K: (..1.............) Key Present
S: (...0............) Sequence Number Absent
ssr: (....1...........) Strict
Source Route
Present
Recur: (.....000........) Recursion Control
A: (........0.......) Acknowledgment sequence number Absent
ReservedFlags: (.........1100...)
Version: (.............101) 5
NextProtocol: 0x6c6c
This packet is sent six times from the remote server to the ISA Server computer without an answer back. Reviewing the trace that was taken on the ISA Server computer, we could see that this packet never arrived there.
4. Conclusion
After this, we can conclude that the issue was not related to ISA Server. Because the GRE protocol was not arriving on the ISA Server computer, it was not possible to complete the PPTP connection. But what was blocking GRE? If you remember the information at the beginning of the article, the customer was uncertain whether the modem has firewall capability. After additional research, the customer did find out that the modem has this capability, but the modem firewall is so basic that it does not support protocol 47 (GRE).
One good resource to learn more about known issues with DSL modems when using with ISA Server is the article "How to use the Windows Server 2003 Routing and Remote Access Service or ISA Server 2006 or ISA Server 2004 with a DSL router for Internet Access" at Microsoft Help and Support.
Yuri Diogenes
Support Engineer – Latin America Team – Platforms
Microsoft