Detecting ephemeral port exhaustion


Symptoms

When Windows or Windows Server is out of ephemeral/outbound/dynamic network ports, it will not be able to establish any outbound network connections. This results in a lot of connection failures such as database and/or domain controller connections. If the system is not responding, then try increasing the port range (discussed below) – this change is effective immediately. If the system immediately recovers, then you know you are dealing with this issue. You can either increase the port range which may just delay the problem or you can try to identify the cause.

Detecting user mode port leaks

Ephemeral ports are range of ports that Windows and Windows Server use for outbound communications over the TCP/IP network protocol. When an outbound connection is finished, the port associated to the connection is put into a TIMED_WAIT state for two minutes by default. This allows any lingering packets on the network to be ignored. Windows Server 2008 and later use the IANA range which uses the ports between 49152 and 65535 providing 16,383 ports.

Some applications and services such as Microsoft Exchange Server CAS servers can be very “chatty” and might actually use all 16,383 ports within a two minute time period. The result is connection failures similar to “Couldn’t connect to X, due to no ports available from the end point mapper”.

If you suspect ephemeral port exhaustion, then consider running the following Powershell script called “Log-EphemeralPortStats.ps1”:
https://1drv.ms/f/s!AhuJirRUDDbmkotk33Jr-spEVrTOcA

Warning: This script is provided as sample code only. Please review it and use at your own risk.

Be aware that this only detects user mode port leaks. If a driver (kernel mode) is leaking ports, then a complete memory dump must be sent to Microsoft Support for analysis to know which driver is responsible. The memory dump must be taken when the system has leaked a significant number of ports.

This script is designed to run in an infinite loop of 1 minute sleep intervals and write to a log file called “EphemeralPortStats.log”. Here is an example of the output it produces:

Computer       DateTime             LocalAddress  #OfEPortsInUse Max#OfEPorts %EPortUsage #OfTcpListeningPorts #OfPids
——–       ——–             ————  ————– ———— ———– ——————– ——-
ETCHEDCHAMPION 8/9/2013 12:37:42 PM 127.0.0.1                  6        16384           0                   15      11
ETCHEDCHAMPION 8/9/2013 12:37:42 PM 172.18.96.192              3        16384           0                   15      10
ETCHEDCHAMPION 8/9/2013 12:37:42 PM 192.168.1.2               69        16384         0.4                   15      17

This script is intended to be ran from the console of the computer suspected to be running low on ephemeral ports and to leave it running. Periodically review the log to see if there was any ephemeral port exhaustion detected.

This script gets the port range from:

netsh int ipv4 show dynamicportrange tcp

Then, correlates this information with the output of:

netstat –ano –p tcp

PsExec can be potentially used to get this information from remote computers, but keep in mind that passwords used in PsExec are sent in the clear over the network.

My PFE colleagues and customers have used this script quit a bit and I hope it will help you as well.

Also, as a temporary work-around, the ephemeral/dynamic/outbound port range can be increased using the following command:

netsh int ipv4 set dynamicport tcp start=10000 num=55535

This change takes affect immediately. No need to restart. No need to reboot. This is about three times larger than the default. If this change has a positive affect when the problem occurs, then you know that this is the issue your system is dealing with.

To set the port range back to default, use the following command:

netsh int ipv4 set dynamicport tcp start=49152 num=16384

To check the port range:

netsh int ipv4 show dynamicportrange tcp

Once you have identify the process consuming the ports, contact the developer of the process for a fix.

Also, consider decreasing the TIMED WAIT delay using the TcpTimedWaitDelay registry key.

Detecting kernel mode port leaks

All of the above covers user mode port leaks – meaning leaks originating from processes. If a driver (kernel mode) leaks a port, then netstat will not be able to report it. Kernel mode port leaks can only be detected by inducing a complete memory dump and then analyzing the dump with the MEX debug extension. Load MEX into WinDBG and then run the following commands to help identify ports associated with drivers.

UPDATE – I have heard that Windows 8 and Windows Server 2012 have a version of NETSTAT that will detect kernel port leaks as well. I have not verified this yet.

!mex.afd -conn -report -verbose

!afd -endp -report

!tcpip -p

Once you identify the driver consumer the ports, contact the developer of the driver.

If you suspect a Windows Server 2008 R2 system might be running out of ephemeral ports, then apply the following hotfix:

Kernel sockets leak on a multiprocessor computer that is running Windows Server 2008 R2 or Windows 7
https://support.microsoft.com/en-us/kb/2577795

Windows Server 2012 and later are already patched with this fix.


Comments (20)

  1. Ah, it must be 4 minutes. My memory must be failing. I was thinking of TcpTimedWaitDelay at

    technet.microsoft.com/…/cc938217.aspx. It says 4 minutes.

  2. Thank you for the useful information.

  3. Brian says:

    Two minutes or four minutes?  I thought four minutes as that is 2xMSL, with each MSL being 120 seconds (2 minutes). TechNet says 4 minutes as well.

  4. Tilo Boehme says:

    Thanks Client awesome script. Very useful. Should be somewhere in Windows monitored or an Event log if running out of ports.
    Can you put it to
    http://gallery.technet.microsoft.com/scriptcenter/site/upload as well please. Thanks

  5. Greg S says:

    So what next? How to troubleshoot this? Owner of process is System

  6. Aneel says:

    To Clint Huffman – Can you please provide the powershell script source code as cannot download from the link provided –
    https://skydrive.live.com/redir?resid=E6360C54B48A891B!5328

  7. Nikhil says:

    Hi Clint: Good article. Can you please provide the powershell script source sode as cannot download from the link provided.

  8. Andrew S says:

    Thanks for the script. Should this list the actual detail of ports in use or just the number of ports? i.e. if ports start to get exhausted will we need to then run ‘netstat -ano -p tcp’ to get the detail?

    This is generally happening out of hours so we would not be on the console to check. By the time we do the ports are fine again.

  9. nick p says:

    Hi Clint,
    i’ve taken the liberty of adapting your script to list the number of ephemeral ports in ESTBLISHED, TIME_WAIT and SYN_RECEIVED states, and also listing the top 5 PIDs, with process name and number of ports assigned. any suggestions as to what i can do with
    it? do you have a listing on the script repository i could append it to?

    1. You can send it directly to me at clinth@microsoft.com and I’ll update the script. Thanks!

    2. Clint Huffman says:

      I don’t have a repository for this, but you are welcome to publish the script/code anyway you want.

  10. JohnF says:

    Hi, How do you fix the issue if you have this happening to an Exchange server?

    1. By increasing the port range with the following command:
      netsh int ipv4 set dynamicport tcp start=10000 num=55535

  11. Jon Tooth says:

    I see a lot of talk on how to detect this, but not how to troubleshoot the underlying cause. In my case it is also local SYSTEM and connecting to SQL. But we do not know which of the many services tools and tasks is behind it, as they are hidden behind
    system.exe

    1. Correct. This blog post only covers user mode port leaks. I’ll update it to include kernel mode.

  12. Phil says:

    For a quick check of total and used ephemeral ports of a single service (so in my case, a SQL server) do this:

    PS C:UsersRack> netsh int ipv4 show dynamicportrange tcp

    Protocol tcp Dynamic Port Range
    ———————————
    Start Port : 49152
    Number of Ports : 16384

    PS C:UsersRack> netstat -ano | findstr 1433 | measure

    Count : 910
    Average :
    Sum :
    Maximum :
    Minimum :
    Property :

    It will only count the MSSQL connections, but in my case those were his only ephemeral. You can also do this with a couple different ports to get numbers you can sum up.

  13. M S Ali says:

    Do you have the same script working for monitoring UDP port? I tried modifying the script for UDP but not luck.

    M S Ali

    1. Change line 156 to be:
      $oOutput = Invoke-Expression -Command ‘netstat -ano -p udp’

  14. Florian says:

    Once again, thanks Clint! This is exactly what i´m looking for!

  15. Sorry for the delay. I updated this blog post with a working link to the script.

Skip to main content