Windows 2012 R2 server fails to establish outbound connections


Hi there,

It's been a very long while since I have blogged something here and it's time to come back and continue sharing our field experiences with the IT community hoping to shed light for similar problems.

I was tasked to deal with a customer problem where the end users were reporting various problems like "cannot access the file server, getting authentication prompts" and the IT admins were also observing various problems like the server wasn't properly applying GPOs, Netlogon service complaining about DC access issues and etc. At times, they were even able to manually reproduce the issue by issuing a "telnet DC-IP 389" command from the affected server.

There might be a lot of reasons behind, so I decided to collect a number of logs while the issue was reproduced:

a) TCPIP ETL trace:

You can collect it with the below commands on a Windows client/server: (from an elevated command prompt)

netsh start trace capture=yes scenario=internetclient

<<repro>>

netsh trace stop

b) Network trace:

This could be collected in different ways like using the above command, Wireshark, Network Monitor, Message Analyzer,...

c) Handle outputs

This could be collected as follows:

Note: Handle tool could be downloaded from the following link: https://technet.microsoft.com/en-us/sysinternals/handle.aspx Handle v4.1

handle.exe -a -u >> %computername%_handledetails.txt

handle.exe -s >> %computername%_handlesummary.txt

ANALYSIS:

========

The logs were collected while doing a repro with telnet command on the server. After the logs were shared with us, I checked various things to understand why the outbound connection might be failing (by the way, the file server not being able to authenticate the incoming users was also a side effect of this issue since the file server wasn't able to verify the client credentials via Netlogon secure channel)

1) I first checked network traces, but there were no outgoing connection attempts (TCP SYNs sent to the target server) which means the issue is local to the server itself

2) Then I checked the TCPIP ETL trace and observed the root cause:

Note: You can open up the ETL file that is generated as a result of running netsh command in Network Monitor or Message Analyzer

[0]03E0.5214::01/04/18-15:07:37.5237622 [Microsoft-Windows-TCPIP/Diagnostic] TCP: endpoint (sockaddr=0.0.0.0) bind failed: port-acquisition status = The transport address could not be opened because all the available addresses are in use..

[0]58F0.4558::01/04/18-15:07:51.8242042 [Microsoft-Windows-TCPIP/Diagnostic] TCP: endpoint (sockaddr=0.0.0.0) bind failed: port-acquisition status = The transport address could not be opened because all the available addresses are in use..

[0]04D8.072C::01/04/18-15:07:52.0110322 [Microsoft-Windows-TCPIP/Diagnostic] TCP: endpoint (sockaddr=0.0.0.0) bind failed: port-acquisition status = The transport address could not be opened because all the available addresses are in use.. 1616260 [0]

...

Actually that clearly explained why the outbound connections were failing: PORT EXHAUSTION.

3) And the main reason behind the port failure was a socket leak caused by an outdated 3rd party AV software: (from handles.exe output)

Note: The process name was deliberately changed

92355 ABC.exe pid: 1148 NT AUTHORITY\SYSTEM

92517   144: File  (---)   \Device\Afd

92519   148: File  (---)   \Device\Afd

92627   220: File  (---)   \Device\Afd

92629   224: File  (---)   \Device\Afd

92633   22C: File  (---)   \Device\Afd

92635   230: File  (---)   \Device\Afd

92689   29C: File  (---)   \Device\Afd

92701   2B4: File  (---)   \Device\Afd

92703   2B8: File  (---)   \Device\Afd

92705   2BC: File  (---)   \Device\Afd

92707   2C0: File  (---)   \Device\Afd

92743   308: File  (---)   \Device\Afd

92755   320: File  (---)   \Device\Afd

92761   32C: File  (---)   \Device\Afd

92767   338: File  (---)   \Device\Afd

92771   340: File  (---)   \Device\Afd

92773   344: File  (---)   \Device\Afd

92779   350: File  (---)   \Device\Afd

92881   420: File  (---)   \Device\Afd

92897   440: File  (---)   \Device\Afd

92899   444: File  (---)   \Device\Afd

92927   47C: File  (---)   \Device\Afd

92929   480: File  (---)   \Device\Afd

92933   488: File  (---)   \Device\Afd

92935   48C: File  (---)   \Device\Afd

92941   498: File  (---)   \Device\Afd

92977   4E0: File  (---)   \Device\Afd

92993   500: File  (---)   \Device\Afd

93053   578: File  (---)   \Device\Afd

93073   5A0: File  (---)   \Device\Afd

93075   5A4: File  (---)   \Device\Afd

93077   5A8: File  (---)   \Device\Afd

93079   5AC: File  (---)   \Device\Afd

93093   5C8: File  (---)   \Device\Afd

93113   5F0: File  (---)   \Device\Afd

93145   630: File  (---)   \Device\Afd

93165   658: File  (---)   \Device\Afd

93167   65C: File  (---)   \Device\Afd

93175   66C: File  (---)   \Device\Afd

93195   694: File  (---)   \Device\Afd

93199   69C: File  (---)   \Device\Afd

93217   6C0: File  (---)   \Device\Afd

93219   6C4: File  (---)   \Device\Afd

93227   6D4: File  (---)   \Device\Afd

93239   6EC: File  (---)   \Device\Afd

93249   700: File  (---)   \Device\Afd

93253   708: File  (---)   \Device\Afd

93265   720: File  (---)   \Device\Afd

93269   728: File  (---)   \Device\Afd

93271   72C: File  (---)   \Device\Afd

93273   730: File  (---)   \Device\Afd

93275   734: File  (---)   \Device\Afd

93277   738: File  (---)   \Device\Afd

93281   740: File  (---)   \Device\Afd

93283   744: File  (---)   \Device\Afd

93285   748: File  (---)   \Device\Afd

93297   760: File  (---)   \Device\Afd

93299   764: File  (---)   \Device\Afd

93301   768: File  (---)   \Device\Afd

93305   770: File  (---)   \Device\Afd

93307   774: File  (---)   \Device\Afd

93313   780: File  (---)   \Device\Afd

93317   788: File  (---)   \Device\Afd

93321   790: File  (---)   \Device\Afd

93323   794: File  (---)   \Device\Afd

93327   79C: File  (---)   \Device\Afd

93329   7A0: File  (---)   \Device\Afd

93331   7A4: File  (---)   \Device\Afd

93333   7A8: File  (---)   \Device\Afd

93335   7AC: File  (---)   \Device\Afd

93339   7B4: File  (---)   \Device\Afd

93343   7BC: File  (---)   \Device\Afd

93355   7D4: File  (---)   \Device\Afd

93357   7D8: File  (---)   \Device\Afd

93359   7DC: File  (---)   \Device\Afd

93361   7E0: File  (---)   \Device\Afd

93365   7E8: File  (---)   \Device\Afd

93373   7F8: File  (---)   \Device\Afd

93383   810: File  (---)   \Device\Afd

93389   81C: File  (---)   \Device\Afd

 

RESOLUTION:

===========

So we advised the customer to update the 3rd party AV software. Apart from that, you can take the following actions to avoid possible port leak issues:

a) Please make sure that Windows OS runs with latest rollups/security updates

b) Please make sure that all 3rd party softwares are up to date (including Firewall, AV, backup or any kind of software that might have to frequently establish outbound connections)

c) Finally you may consider extending the port range for busy servers which are supposed to establish many outbound connections very frequently. The following is the maximum range that you can set, but you may extend the range in phases instead of maxing out at the very beginning: (from an elevated command prompt)

netsh int ipv4 set dynamicport tcp start=1025 num=64500

netsh int ipv4 set dynamicport udp start=1025 num=64500

and you can decrease the TCPTimedWaitDelay registry key on the servers: (you may lower it to 30 seconds)

https://technet.microsoft.com/en-us/library/cc757512(v=ws.10).aspx TcpTimedWaitDelay

The TcpTimedWaitDelay value determines the length of time that a connection stays in the TIME_WAIT state when being closed. While a connection is in the TIME_WAIT state, the socket pair cannot be reused. This is also known as the 2MSL state because the value should be twice the maximum segment lifetime on the network. To adjust the TcpTimedWaitDelay settings, you have to modify/create the registry settings as listed below:

 

Key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
Value: TcpTimedWaitDelay
Data Type: REG_DWORD
Range: 30-300 (decimal)
Default value: 0x78 (120 decimal)
Recommended value: 30
Value exists by default? No, needs to be added.

Note: This change requires a server reboot

 

Please note that the same techniques could be applied to virtually any Windows versions as of Windows 7/Windows 2008 R2 onwards easily.

 

Hope this helps

Thanks,

Murat

 

Comments (0)

Skip to main content