Error 64 can happen due many situations and I documented one of those situations last year and as you could see sometimes it is not easy to find out why this error happens. The issue that I’m about to describe here was identified while I was troubleshooting a third party application that uses TCP Port 80 to transmit files, but not using HTTP. What?? Yeah, I know. Although IANA has established port 80 for HTTP, anyone can create an application that uses port 80 to send whatever they want. This is fine, as long as you don’t try to use this application behind a Firewall that does application layer inspection and look to that traffic and say: what is that? This is not HTTP Protocol and it is using TCP Port 80…I shall block this traffic!
The firewall administrator was smart to understand that and what he did was, he created a custom protocol using port 80 and didn’t bind Web Proxy filtering to it. Fair enough, but doesn’t fully resolved this issue.
2. The Error
When the client (which had the 3rd party application installed on his computer) started to transmit the file to the destination it received an error and didn’t proceed. Using Logging feature the Firewall Administrator saw the error below:
Figure 1 – Error 64
On the netmon trace we could see that the TCP Handshake was established fine, but after the first HTTP Payload has being sent ISA Server 2006 didn’t like what he saw and the connection was reset.
Figure 2 – Connection reset right after first attempt to use TCP Port 80 (with a non compliance HTTP Protocol).
To resolve this problem what you need to do is not only create a custom protocol and an access rule to use this protocol, but also a deny rule right after this access rule to block the regular HTTP Protocol that has the Web Proxy Filter bind to it. The access rules will look like this:
Figure 3 – Access rule with a Deny to HTTP (with filter) Protocol.
Why do I have to do this? Read this post here and you will know the reason:
Why do I need a deny rule to make an allow rule for a custom protocol work correctly?