Explaining Close_Wait

I have been working with a customer recently who has a print server that has had its spooler crashing after a 3rd Party service running on it was locking up and freezing and falling over.

One of the signs it was about to fall over was running the following command against the server and seeing lots and lots of Close_waits. This was observed by running the

Netstat –an command. Example output below.

So what do the “State” actually mean. And what is the significance of Close_Wait.

Understanding the TCP sequence of steps for socket closing

As the TCP conversation is a ports and sockets sequence, to understand how to troubleshoot it and carry out root cause analysis. This was an excellent blog that explains this tcp socket conversation very well

http://j2eedebug.blogspot.com/2008/12/difference-between-closewait-and.html

Also see below for the explanation of the different states sockets can enter into as part of that conversation.

State Description

http://support.microsoft.com/kb/137984

CLOSED
Indicates that the server has received an ACK signal from the client and the connection is closed

CLOSE_WAIT
Indicates that the server has received the first FIN signal from the client and the connection is in the process of being closed

So this essentially means that his is a state where socket is waiting for the application to execute close()

A socket can be in CLOSE_WAIT state indefinitely until the application closes it.
Faulty scenarios would be like filedescriptor leak, server not being execute close() on socket leading to pile up of close_wait sockets

ESTABLISHED
Indicates that the server received the SYN signal from the client and the session is established

FIN_WAIT_1
Indicates that the connection is still active but not currently being used

FIN_WAIT_2
Indicates that the client just received acknowledgment of the first FIN signal from the server

LAST_ACK
Indicates that the server is in the process of sending its own FIN signal

LISTENING
Indicates that the server is ready to accept a connection

SYN_RECEIVED
Indicates that the server just received a SYN signal from the client

SYN_SEND
Indicates that this particular connection is open and active

TIME_WAIT
Indicates that the client recognizes the connection as still active but not currently being used

So the explanation for a close_wait situation is as below;

CLOSE is an operation meaning "I have no more data to send." that is the

client/server has chosen to treat CLOSE in a simplex fashion. The user who CLOSEs

may continue to RECEIVE Until he is told that the other side has CLOSED also. Thus,

a program/application could initiate several SENDs followed by a CLOSE, and then

continue to RECEIVE until signalled that a RECEIVE failed because the other side has

CLOSED. We assume that the TCP will signal a user, even if no RECEIVEs are

outstanding, that the other side has closed, so the user can terminate his side

gracefully. A TCP will reliably deliver all buffers SENT before the connection was

CLOSED so a user who expects no data in return need only wait to hear the

connection was CLOSED successfully to know that all his data was received at the

destination TCP. Users must keep reading connections they close for sending until

the TCP says no more data.

 Adjusting Registry Settings

Registry keys to look at which can sometimes help to configure and adjust this conversation.

MaxUserPort

TcpTimedWaitDelay

http://support.microsoft.com/kb/137984

TCP Connection States and Netstat Output

http://support.microsoft.com/kb/328476

http://technet.microsoft.com/en-us/library/cc938196.aspx

HKEY_Local_Machine\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

• MaxUserPort

This entry makes more ports available. 

• TcpTimedWaitDelay

Reducing this value from its default setting of 240 seconds will make ports expire sooner. This parameter determines the length of time that a connection stays in the TIME_WAIT state when it is being closed. While a connection is in the TIME_WAIT state, the socket pair cannot be reused. This is also known as the 2MSL state because the value should be double the maximum segment lifetime on the network. See RFC 793 for more details.