More than one L2TP VPN connection from behind a NAT device fails with error 809 when TMG 2010 has been configured as a VPN Server
Introduction
Consider a scenario where a TMG administrator has configured their TMG Server 2010 installed on a Windows Server 2008 R2 for inbound VPN connections. External VPN users (two or more users) are behind a NAT device, which NATs all outbound L2TP VPN traffic. When users try to connect using L2TP VPN connections, only one user from this network can connect at a time. Every connection attempt from another user fails with the following error:
Error message: error 809 The network connection between your computer and the VPN server could not be established because the remote server is not responding. This could be because one of the network devices (e.g., firewalls, NAT, Router etc.) between your computer and the remote server is not configured to allow VPN connections. Please contact your Administrator or your service provider to determine which device may be causing the problem. |
Troubleshooting
In this case, since users can connect to VPN as long as they are unique from behind a NAT device, the basic VPN configuration on the TMG Server would not have any problem.
To verify if the issue is specific to L2TP VPN, we configured the TMG VPN Server to allow PPTP. Tests from the client end revealed that we could successfully establish more than one PPTP VPN connections. So the test confirmed that the issue was specific to L2TP VPN connections.
Since in such a scenario, traffic would appear to come from the same external IP, it is normal to think that the TMG server might be dropping the connections as per its Flood Mitigation settings. We did notice some TCP/IP Connection Limit Exceeded errors on the TMG Server in this case. So we created an exception list for the incoming IP address in the Flood Mitigation Settings. Though we see the TCP/IP connection Limit errors go away, this did not resolve our issue.
TMG Live logging shows Initiated and Closed Connections without much detail. Network traces do not help much in this case as the traffic is L2TP and encrypted.
To isolate whether the issue was related to TMG or RRAS, we set up a parallel VPN setup with a Windows Server 2008 R2 machine, configured as RRAS. We were able to reproduce the issue successfully without installing TMG on this machine.
Resolution
The problem in our scenario turned out to be specific to L2TP VPN traffic from behind a NAT device to Windows Server 2008 R2 RRAS Server. This was identified as a problem in Windows Server 2008 R2 and the following KB article addresses this issue. The public hotfix included within the article below updates the Fwpkclnt.sys and Tcpip.sys files:
**Author**Niladri Dasgupta
Support Engineer
Microsoft CSS Forefront TMG Team
Technical Reviewers
Yuri Diogenes
Sr. Support Escalation Engineer
Microsoft CSS Forefront TMG Team
Mohit Kumar
Sr. Support Escalation Engineer
Microsoft CSS Forefront TMG Team