Why doesn’t Windows 2008 server negotiate TCP MSS smaller than 536 bytes?


Hi,

In today's blog, I'll talk about an MTU issue that occurs on Windows Vista onwards (Vista/7/2008/2008 R2).

One of our customers reported that their SMTP server (running on Windows 2008) was failing to send e-mails to certain remote SMTP servers because e-mail delivery was disrupted at transport layer.

After analyzing the network trace collected on the source Windows 2008 Server, we found out that the remote system was offering a TCP MSS size of 512 bytes and Windows 2008 server kept sending the data packets with an MSS size of 536 bytes. As a result, those packets weren't succesfully delivered to the remote system. You can find more details about the problem and root cause below:

Note: IP addresses and mail server names are deliberately changed.

Source SMTP server: 10.1.1.1 - mailgateway.contoso.com
Target SMTP server: 10.1.1.5 - mailgateway2.contoso.com

a) Source SMTP server establishes TCP 3-way handshake with the target SMTP server. Source server suggests an MSS size of 1460 bytes and the target suggests an MSS size of 512 bytes:

No.     Time        Source        Destination    Protocol   Info
      1 0.000000    10.1.1.1       10.1.1.5        TCP      28474 > 25 [SYN] Seq=0 Win=8192 Len=0 MSS=1460 WS=8
      2 0.022001    10.1.1.5       10.1.1.1        TCP      25 > 28474 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=512
      3 0.000000    10.1.1.1       10.1.1.5        TCP      28474 > 25 [ACK] Seq=1 Ack=1 Win=65392 Len=0

b) Then data starts flowing. Under normal circumstances, the minimum of MSS will be selected as the MSS of the given TCP session by both parties and it will be used throughout the session.

No.     Time        Source        Destination    Protocol   Info
      4 0.075005    10.1.1.5       10.1.1.1        SMTP     S: 220 mailgateway2.contoso.com ESMTP Tue, 20 Apr 2010 15:18:42 +0200
      5 0.001000    10.1.1.1       10.1.1.5        SMTP     C: EHLO Mailgateway.contoso.com
      6 0.021001    10.1.1.5       10.1.1.1        SMTP     S: 250-mailgateway2.contoso.com Hello Mailgateway.contoso.com [10.1.1.1] | 250-SIZE 26214400 | 250-PIPELINING | 250 HELP
      7 0.001000    10.1.1.1       10.1.1.5        SMTP     C: MAIL FROM:<
postmaster@contoso.com> SIZE=2616 | RCPT TO:<test@test.abc.com>
      8 0.183011    10.1.1.5       10.1.1.1        SMTP     S: 250 OK | 250 Accepted
      9 0.000000    10.1.1.1       10.1.1.5        SMTP     C: DATA
     10 0.022001    10.1.1.5       10.1.1.1        SMTP     S: 354 Enter message, ending with "." on a line by itself

c) Even though an MSS size of 512 should be commonly agreed by both parties, Windows 2008 server doesn't seem to be using that value and keeps sending data with an MSS of 536 bytes:

No.     Time        Source        Destination    Protocol   Info
     11 0.294017    10.1.1.1       10.1.1.5        SMTP     C: Message Body, 536 bytes

d) Most likely the TCP segment with 536 bytes of data doesn't arrive at the target server and we don't get a TCP ACK back as a result so we start TCP packet retransmissions:

No.     Time        Source        Destination    Protocol   Info
     12 0.600034    10.1.1.1       10.1.1.5        SMTP     [TCP Retransmission] C: Message Body, 536 bytes
     13 0.190011    10.1.1.5       10.1.1.1        SMTP     [TCP Retransmission] S: 354 Enter message, ending with "." on a line by itself
     14 0.000000    10.1.1.1       10.1.1.5        TCP      [TCP Dup ACK 12#1] 28474 > 25 [ACK] Seq=649 Ack=269 Win=65124 Len=0
     15 1.010058    10.1.1.1       10.1.1.5        SMTP     [TCP Retransmission] C: Message Body, 536 bytes
     16 2.400137    10.1.1.1       10.1.1.5        SMTP     [TCP Retransmission] C: Message Body, 536 bytes
     17 4.800274    10.1.1.1       10.1.1.5        SMTP     [TCP Retransmission] C: Message Body, 536 bytes

e) Finally the source server closes the TCP session as it fails to successfully deliver the 536 bytes TCP segment to the target system:

No.     Time        Source        Destination    Protocol   Info
     18 9.600550    10.1.1.1       10.1.1.5        TCP      28474 > 25 [RST, ACK] Seq=649 Ack=269 Win=0 Len=0

The same problem doesn't happen if the source server is a Windows 2003 server.

After explaining the problem, now let's try to understand the root cause:

This issue stems from the fact that Windows Vista onwards systems don't accept an MTU size lower than 576 bytes:

TCP/IP Registry Values for Microsoft Windows Vista and Windows Server 2008
http://www.microsoft.com/downloads/details.aspx?familyid=12AC9780-17B5-480C-AEF7-5C0BDE9060B0&displaylang=en

MTU
Key:  Tcpip\Parameters\Interfaces\interfaceGUID
Value Type: REG_DWORD—number
Valid Range: From 576 to the MTU of the underlying network
Default: 0xFFFFFFFF
Description: This value overrides the default Maximum Transmission Unit (MTU) for a network interface. The MTU is the maximum IP packet size, in bytes, that can be transmitted over the underlying network. For values larger than the default for the underlying network, the network default MTU is used. For values smaller than 576, the MTU of 576 is used. This setting only applies to IPv4.
Note: Windows Vista TCP/IP uses path MTU (PMTU) detection by default and queries the network adapter driver to find out what local MTU is supported. Altering the MTU value is typically not necessary and might result in reduced performance.

Since minimum MTU that could be used by a Window Vista onwards system is 576 bytes, a TCP MSS (maximum segment size) should be 536 bytes at miminum so that's why Windows 2008 source server tries to send TCP segments with 536 bytes of data. TCP MSS value is calculated as follows:

TCP MSS = IP MTU - IP header size (20 bytes by default) - TCP header size (20 bytes by default)

Hope this helps

Thanks,
Murat

Comments (6)

  1. MuratKa1 says:

    Hi James,

    Thanks for your feedback.

    Unfortunately there's no solution to this if the source server is Windows 2008 onwards. Such systems don't use any MTU values smaller than 576 bytes and this is hardcoded:

    MTU

    Key:  TcpipParametersInterfacesinterfaceGUID

    Value Type: REG_DWORD—number

    Valid Range: From 576 to the MTU of the underlying network

    Default: 0xFFFFFFFF

    Description: This value overrides the default Maximum Transmission Unit (MTU) for a network interface. The MTU is the maximum IP packet size, in bytes, that can be transmitted over the underlying network. For values larger than the default for the underlying network, the network default MTU is used. For values smaller than 576, the MTU of 576 is used. This setting only applies to IPv4.

    The best way forward would be to change the MTU size at the target system (provided that you have control over the target). If the target system is a public system, the best option would be to use Windows 2003 server where a smaller MTU size could be configured. (Please note that Windows 2003 also doesn't accept an MTU size smaller than 576 bytes if it's dynamically learnt via ICMP but at least you have the option to manually configure the MTU on the interface to a value smaller than 576 bytes). You can see the below links for further information:

    blogs.technet.com/…/do-you-still-set-enablepmtudiscovery-to-0.aspx

    support.microsoft.com/…/898060

    -> It's allowed to configuer the MTU as low as 68 bytes on Windows 2003: (From KB898060)

    Key: HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParametersInterfacesID for Adapter

    Value Type: REG_DWORD Number

    Valid Range: 68 to the MTU of the underlying network

    Default: 0xFFFFFFFF

    In short, there's no solution to this on Vista/7/2008/2008 R2 if you don't have a chance to make any modifications at the target system.

    Thanks,

    Murat

  2. Anonymous says:

    Hi Murat – Thanks for this very detailed explanation! I am experiencing a very similar issue, but with web traffic instead of SMTP traffic. Were you able to resolve this at all?

    Thanks,

    James

  3. sopolenius says:

    Why are the packets dropped? Who's dropping them in your case? We have the same problem since we migrated to W2k8, but we have no control over the end devices. It works in a controlled environment, but not in production, I suspect that some firewall is dropping them because of mss mistmatch.

  4. sopolenius says:

    Hi again,

    In our case, it was a policy in a load balancer that caused the packets to be dropped

  5. fred says:

    classic issue , the loadbalancer is probably filtering icmp type 3 code 4 , anyone expericed this issue on blue coat caches ?

  6. Bob Dobbs says:

    The RFCs defines the minimum segment size.

    This may be of assistance. http://tools.ietf.org/html/rfc879

Skip to main content