Why doesn't Windows 2008 server negotiate TCP MSS smaller than 536 bytes?

  • Article

Hi,

In today's blog, I'll talk about an MTU issue that occurs on Windows Vista onwards (Vista/7/2008/2008 R2).

One of our customers reported that their SMTP server (running on Windows 2008) was failing to send e-mails to certain remote SMTP servers because e-mail delivery was disrupted at transport layer.

After analyzing the network trace collected on the source Windows 2008 Server, we found out that the remote system was offering a TCP MSS size of 512 bytes and Windows 2008 server kept sending the data packets with an MSS size of 536 bytes. As a result, those packets weren't succesfully delivered to the remote system. You can find more details about the problem and root cause below:

Note: IP addresses and mail server names are deliberately changed.

Source SMTP server: 10.1.1.1 - mailgateway.contoso.com
Target SMTP server: 10.1.1.5 - mailgateway2.contoso.com

a) Source SMTP server establishes TCP 3-way handshake with the target SMTP server. Source server suggests an MSS size of 1460 bytes and the target suggests an MSS size of 512 bytes:

No. Time Source Destination Protocol Info
1 0.000000 10.1.1.1 10.1.1.5 TCP 28474 > 25 [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=8
2 0.022001 10.1.1.5 10.1.1.1 TCP 25 > 28474 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=512
3 0.000000 10.1.1.1 10.1.1.5 TCP 28474 > 25 [ACK] Seq=1 Ack=1 Win=65392 Len=0

b) Then data starts flowing. Under normal circumstances, the minimum of MSS will be selected as the MSS of the given TCP session by both parties and it will be used throughout the session.

No. Time Source Destination Protocol Info
4 0.075005 10.1.1.5 10.1.1.1 SMTP S: 220 mailgateway2.contoso.com ESMTP Tue, 20 Apr 2010 15:18:42 +0200
5 0.001000 10.1.1.1 10.1.1.5 SMTP C: EHLO Mailgateway.contoso.com
6 0.021001 10.1.1.5 10.1.1.1 SMTP S: 250-mailgateway2.contoso.com Hello Mailgateway.contoso.com [10.1.1.1] | 250-SIZE 26214400 | 250-PIPELINING | 250 HELP
7 0.001000 10.1.1.1 10.1.1.5 SMTP C: MAIL FROM:<postmaster@contoso.com> SIZE=2616 | RCPT TO:<test@test.abc.com>
8 0.183011 10.1.1.5 10.1.1.1 SMTP S: 250 OK | 250 Accepted
9 0.000000 10.1.1.1 10.1.1.5 SMTP C: DATA
10 0.022001 10.1.1.5 10.1.1.1 SMTP S: 354 Enter message, ending with "." on a line by itself

c) Even though an MSS size of 512 should be commonly agreed by both parties, Windows 2008 server doesn't seem to be using that value and keeps sending data with an MSS of 536 bytes:

No. Time Source Destination Protocol Info
11 0.294017 10.1.1.1 10.1.1.5 SMTP C: Message Body, 536 bytes

d) Most likely the TCP segment with 536 bytes of data doesn't arrive at the target server and we don't get a TCP ACK back as a result so we start TCP packet retransmissions:

No. Time Source Destination Protocol Info
12 0.600034 10.1.1.1 10.1.1.5 SMTP [TCP Retransmission] C: Message Body, 536 bytes
13 0.190011 10.1.1.5 10.1.1.1 SMTP [TCP Retransmission] S: 354 Enter message, ending with "." on a line by itself
14 0.000000 10.1.1.1 10.1.1.5 TCP [TCP Dup ACK 12#1] 28474 > 25 [ACK] Seq=649 Ack=269 Win=65124 Len=0
15 1.010058 10.1.1.1 10.1.1.5 SMTP [TCP Retransmission] C: Message Body, 536 bytes
16 2.400137 10.1.1.1 10.1.1.5 SMTP [TCP Retransmission] C: Message Body, 536 bytes
17 4.800274 10.1.1.1 10.1.1.5 SMTP [TCP Retransmission] C: Message Body, 536 bytes

e) Finally the source server closes the TCP session as it fails to successfully deliver the 536 bytes TCP segment to the target system:

No. Time Source Destination Protocol Info
18 9.600550 10.1.1.1 10.1.1.5 TCP 28474 > 25 [RST, ACK] Seq=649 Ack=269 Win=0 Len=0

The same problem doesn't happen if the source server is a Windows 2003 server.

After explaining the problem, now let's try to understand the root cause:

This issue stems from the fact that Windows Vista onwards systems don't accept an MTU size lower than 576 bytes:

TCP/IP Registry Values for Microsoft Windows Vista and Windows Server 2008
https://www.microsoft.com/downloads/details.aspx?familyid=12AC9780-17B5-480C-AEF7-5C0BDE9060B0&displaylang=en

MTU
Key: Tcpip\Parameters\Interfaces\interfaceGUID
Value Type: REG_DWORD—number
Valid Range: From 576 to the MTU of the underlying network
Default: 0xFFFFFFFF
Description: This value overrides the default Maximum Transmission Unit (MTU) for a network interface. The MTU is the maximum IP packet size, in bytes, that can be transmitted over the underlying network. For values larger than the default for the underlying network, the network default MTU is used. For values smaller than 576, the MTU of 576 is used. This setting only applies to IPv4.
Note: Windows Vista TCP/IP uses path MTU (PMTU) detection by default and queries the network adapter driver to find out what local MTU is supported. Altering the MTU value is typically not necessary and might result in reduced performance.

Since minimum MTU that could be used by a Window Vista onwards system is 576 bytes, a TCP MSS (maximum segment size) should be 536 bytes at miminum so that's why Windows 2008 source server tries to send TCP segments with 536 bytes of data. TCP MSS value is calculated as follows:

TCP MSS = IP MTU - IP header size (20 bytes by default) - TCP header size (20 bytes by default)

Hope this helps

Thanks,
Murat