Windows Networking: Identifying Port Exhaustion on Windows Server 2008/Vista+ Operating Systems

  • Article

Hi all,

I wanted to publish this article for people that might be encountering a port exhaustion scenario with their servers. With the current existing tools, there is no nice and easy way to determine if your machine might be encountering a port exhaustion scenario and if so, what process is consuming the most ephemeral ports. Hopefully with this information, you should be able to quickly identify if you might be encountering a port exhaustion scenario with your servers and which process is responsible.

 

Symptoms of Port Exhaustion

System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send.

- or -

System.Net.WebException: Unable to connect to the remote server

 

Identify port exhaustion using ETL Tracing

  • On the affected machine, currently in state, open administrator command prompt and run the following command:
    • Netsh trace start scenario=netconnection capture=yes
  • Attempt to establish an outbound connection and reproduce the issue
  • Stop the trace using:
    • Netsh trace stop
  • Now that you have the trace captured from the machine, download and install network monitor 3.4 and load the trace.
  • Open the ETL trace in netmon, and then use the following display filter to identify any 'STATUS_TOO_MANY_ADDRESSES' which would be an indicator of port exhaustion
  • Wscore_MicrosoftWindowsWinsockAFD.AFD_EVENT_BIND.Status.LENTStatus.Code == 0x209

Ok so cool, now you know you are running out of ephemeral ports on your machine and that is the reason you can no longer establish outgoing connections. Now what?

Identify the Process(es) consuming ports using Process Explorer

The next step then at this point would be to identify the process that is consuming all the ephemeral ports. The easiest way to do this is to download and use process explorer from sysinternals.

  • Download the Process Explorer tool and extract the procexp.exe onto the affected machine that is currently in state
  • Launch Process Explorer as Administrator
    • Failing to launch as Administrator w/ UAC enabled will prevent the services from loading
  • Navigate to View and select 'Show Lower Pane'
  • Navigate to View > Select Columns and within the Process Performance tab, checkmark 'Handle Count'
  • Sort the processes by Handle Count to locate the services consuming the most handles on the OS
  • Within the Lower Pane, sort the results by Name
  • At this point, you will then want to examine the processes to locate \device\afd which is an indication that the process has an ephemeral port opened. Since we running into a ephemeral port exhaustion, you will want to locate a process where you see thousands of instances of \device\afd defined.

In my scenario above in the screenshot, I don't have port exhaustion but I wanted to make sure I provided an image for clarification purposes of what you are looking for. Once you have identified the process that is consuming the ephemeral ports, you will want to follow up with the necessary company that owns that component to have them investigate further to determine if there is a issue with the code or if you are encountering another environment factor.

Let me know if this is helpful!

Related Articles:

Process Explorer v16.04

https://technet.microsoft.com/en-us/sysinternals/bb896653.aspx

Network Monitor 3.4

https://www.microsoft.com/en-us/download/details.aspx?id=4865