Error 10060 while browsing Internet through ISA Server 2006

1. Connection Timeout – you should believe.

 

It is very interesting how many people really don’t believe that when ISA triggers the error 10060 is because ISA didn’t receive an answer from the destination host. Most of the time the question after they receive this error is: Why ISA is timing out? How can I increase the timeout from ISA so it doesn’t show me this error? Why ISA is doing that?

 

Let’s demystify this and understand that ISA doesn’t do this, the timeout is a due the Windows Operating parameter under the TCP/IP stack or better saying the windows Sockets implementation called winsock. The parameters that controls this are: TcpMaxConnectRetransmissions , TcpMaxDataRetransmissions and TCPInitialRtt located under HKLM\SYSTEM\CCS\Services\Tcpip\Parameters. Also, for ISA Server there is a KB191143 that explains in more details this registry setting.

 

It is not recommended to change this default registry setting for this parameter until you really understand why this is happening. Remember that ISA Server is only responsible to externalize the result of the time out by showing the error below:

 

Figure 1 – The infamous 10060 error.

Let’s go under the hood and see what happens when ISA shows this error.

2. Demo Scenario

The scenario where the user is receiving this error has the following topology:

Figure 2 – Diagram used in this network.

The downstream server is using the following configuration for webchain:

Figure 3 – Web Chain.

The following network monitor trace was taken from the external interface of the ISA Server downstream server:

Downstream ISA sends the SYN for TCP port 8080 to 192.168.40.10 (upstream):

10:08:05.779 192.168.1.113 192.168.40.10 TCP TCP:Flags=......S., SrcPort=12493, DstPort=HTTP Alternate(8080), PayloadLen=0, Seq=1953064972, Ack=0, Win=65535 (scale factor 0) = 65535

...no answer and then it try again:

10:08:08.743 192.168.1.113 192.168.40.10 TCP TCP:[SynReTransmit #11]Flags=......S., SrcPort=12493, DstPort=HTTP Alternate(8080), PayloadLen=0, Seq=1953064972, Ack=0, Win=65535 (scale factor 0) = 65535

Notice that after approximately 3 seconds the downstream server re-transmits the SYN and since it doesn’t receive an answer it tries it again:

10:08:14.762 192.168.1.113 192.168.40.10 TCP TCP:[SynReTransmit #11]Flags=......S., SrcPort=12493, DstPort=HTTP Alternate(8080), PayloadLen=0, Seq=1953064972, Ack=0, Win=65535 (scale factor 0) = 65535

Those attempts keep going on and on until it times out. It is important to emphasize that the TCP retransmission is a standard mechanism documented in RFC 2988.

3. Conclusion

Although this post shows as an example a webchain scenario, it is important to emphasize that this could be caused in any scenario where the destination host does not answer in timely manner. What you need to do prior to change any setting is use at least network monitor to see if ISA it is or not receiving answer from the destination server.

Author

Yuri Diogenes

Security Support Engineer

Microsoft CSS Forefront Security Edge Team

Technical Reviewer

Thomas Detzner

Escalation Engineer

Microsoft CSS Forefront Security Edge Team