TCP Offloading/Chimney & RSS…What is it and should I disable it?


Having decided to start this blog to convey my experience with network analysis and troubleshooting, one subject instantly sprang to mind for my first post, TCP Chimney/Offloading.

I get asked about this so often I have a ready email of advice around what it is, and what I (note not Microsoft, although I think our official recommendation would likely mirror mine) recommend to customers around the use of it. My advice is based on thousands of customer cases I've handled over the years around this feature. I've therefore compiled what's hopefully a one-stop shop for all your TCP offloading needs. Apologies in advance if it's a bit wordy, but I've tried to convey everything I can around the subject for you.

So why do I get asked about it all the time? Well, let's start with what it is and what it does.

 

What is TCP Offloading/Chimney?

Starting when Windows Server 2003 SP1 was the current server OS, Microsoft released the Scalable Networking Pack http://support.microsoft.com/kb/912222/en-us

This turned on in the OS, three distinct things:

 

1.) RSS

Where multiple CPUs reside in a single computer, the Windows networking stack limits "receive" protocol processing to a single CPU. In essence, the old design of dealing with all incoming network traffic on a single processor core was starting to cause a bottleneck on newer multiprocessor systems. RSS resolves this issue by enabling the packets that are received from a network adapter to be balanced across multiple CPUs. In essence with RSS on, each incoming TCP connection is load balanced over the available cores, spreading the load and preventing a bottleneck from occurring. This is becoming a necessity as servers have to handle increasingly high loads of network traffic.

 

2.) TCP Chimney (sometimes referred to as TCP Offloading)

This feature is designed to take processing of the network such as packet segmentation and reassembly processing tasks, from a computer's CPU to a network adapter that supports TCP Chimney Offload. This has the effect of reducing the workload on the host CPU and moving it to the NIC, allowing both the Host OS to perform quicker and also speed up the processing of network traffic.

 

3.) Network Direct Memory Access (NetDMA)

http://technet.microsoft.com/en-us/library/gg162716(WS.10).aspx

The NetDMA interface provides generic access to direct memory access (DMA) engines that can perform memory-to-memory data transfers with little CPU involvement. Again, this is designed to take work away from the CPU by allowing the NIC to move data from receive buffers without using the CPU as much.

Why would I want to disable it then?

All these features sound brilliant, and only enabled with the installation of the Scalable Networking Pack, so why would you want to disable it?

Well, with the release of Service Pack 2 for Windows Server 2003, Microsoft decided to include this scalable networking pack and thus turn the features on. If a server has a NIC which supports these features, and it's enabled in the NIC properties (more on this later) then we'll use them. The problem with this approach was, that many NICs reported to the OS that they supported these features, which they indeed did, but many didn't perform these functions very well at all in reality. With 2003 this was an all or nothing effort, if offloading was turned on, we offloaded all TCP connections on a supported NIC, regardless of whether it would benefit or not.

We had a great number of issues through Microsoft support over the course of the next year or so which were caused by drivers misbehaving with these features which had a knock on effect on the network traffic. This in turn caused a wide range of weird and wonderful symptoms seen across the board from the Exchange team to SQL to BizTalk to IIS to ISA. Most of these landed on the lap of my colleagues and I in the Networking support team and as such we've probably seen issues numbering in the thousands caused by these features, where turning them off resolved the problem, so much so that turning off Offloading/RSS became an almost standard troubleshooting step with 2003 cases.

Most NIC vendors have released numerous updates over the years to resolve these issues as have Microsoft to improve the feature in the OS, however my general advice with Server 2003 is to disable these features.

In essence, with server 2003 this feature tends to cause more problems than it solves so I tend to recommend it is turned off. It doesn't really become effective until you get high speed (10gbps) networks and low latency and, as mentioned, many older NIC drivers don't implement this well and thus we get a lot of issues. If enabled, the bare minimum recommendation would be that the NIC and teaming drivers and firmware are the latest available and http://support.microsoft.com/kb/950224/en-us is applied.

 

How do I turn off these features in Server 2003?

If the features aren't needed then they should be turned off, we can do this in one of two ways.

The safest way is to do this in the registry by using method 3 in http://support.microsoft.com/kb/948496/en-us which sets the registry to configure these features to be off.

 

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

 

  • Right-click EnableTCPChimney, and then click Modify.
  • In the Value data box, type 0, and then click OK.
  • Right-click EnableRSS, and then click Modify.
  • In the Value data box, type 0, and then click OK.
  • Right-click EnableTCPA, and then click Modify.
  • In the Value data box, type 0, and then click OK.
  • Exit Registry Editor, and then restart the computer.

 

Alternatively you can disable the features in the NIC properties of most NICs, however the naming convention and exposure of these settings varies from NIC to NIC and also from driver to driver.

 

If the feature is disabled on either the NIC properties, or the registry, it's off, regardless if the other is on. This is why I recommend you use the registry as it won't be affected by driver updates etc and is much easier to control centrally.

 

So what about newer OS versions?

The first point of note is that Microsoft made a lot of effort in the newer OS versions to ensure the drivers were up to the job, and also a lot of improvements in the implementation of the features and TCP stack in the OS which makes the enabling of the features much safer in post 2003 OS versions.

 

Server 2008

In Windows Server 2008 (not R2) Offloading is turned off by default anyway. http://support.microsoft.com/kb/951037/en-us

You can enable it using the NIC properties or by using Netsh which is outlined in the link above. The offloading capabilities are more granular in 2008 than they were in 2003, we offload suitable network connections.

As before, it's always wise if using this feature that the latest NIC drivers and firmware are installed to ensure the NIC manufacturers latest updates are in place.

Ensure you have http://support.microsoft.com/kb/976035/en-us installed on top of SP2 to prevent an unexpected restart scenaro.

If the above steps are done, in my experience it's very safe to turn the feature on in 2008 if you feel it is needed.

To check the state of offloading you can run the following steps:

 

Run Netstat –t in a command prompt an you'll get the following output:

 

Active Connections

 

Proto Local Address Foreign Address State Offload State

TCP 127.0.0.1:52613 computer_name:52614 ESTABLISHED InHost

TCP 192.168.1.103:52614 computer_name:52613 ESTABLISHED Offloaded

 

InHost shows the connection is not offloaded and thus handled by the OS, Offloaded mean exactly that.

For those of you looking in memory dumps and wondering if these features are in use, you should be able to dump the registry keys used to set this by running x tcpip!*disable* in windbg for server 2003.

In this example, both RSS and TCP Chimney is disabled.

x tcpip!*disable*

b8f1a0d4 tcpip!DisableRSS = 1

b8f1a360 tcpip!DisableUserTOSSetting = 1

b8f1df34 tcpip!DisableMediaSenseEventLog = 0

b8f1a0d0 tcpip!DisableTCPChimney = 1

b8f1ae54 tcpip!DisableTaskOffload = 0

b8f1cdc0 tcpip!DisableLargeSendOffload = 0

b8f1a0b4 tcpip!DisableIPSourceRouting = 2

b8f1ae4c tcpip!DisableMediaSense = 0

b8f1a0ec tcpip!DisableUserTOS = 1

b8f01ff3 tcpip!DisableRouter (void)

b8f0c3b0 tcpip!IPDisableMediaSenseRequest (struct _IRP *, struct _IO_STACK_LOCATION *)

b8f106d6 tcpip!OlmDisableOffloadOnInterface (unsigned int)

b8f04d4b tcpip!IPDisableChimneyOffload (struct _IRP *, struct _IO_STACK_LOCATION *)

b8f048bf tcpip!IPDisableSniffer (struct _UNICODE_STRING *)

 

As these structures don't exist in 2008 + you'll need to use another command which im currently trying to confirm what the best method is and I'll update the blog with the info.

 

Server 2008 R2

With server 2008 R2 this feature is much more intelligent, it'll only offload when the conditions are right..as per http://technet.microsoft.com/en-us/library/gg162709(WS.10).aspx

Automatic. In automatic mode, TCP Chimney Offload considers offloading the processing for a connection only if the following criteria are met: the connection is established through a 10 Gbps Ethernet adapter, the mean round trip link latency is less than 20 milliseconds, and at least 130 KB of data has been exchanged over the connection. In automatic mode, the TCP receive window is set to 16 MB. Because the Windows stack has performance optimizations not found in Chimney-capable network adapters, automatic mode restricts offloads only to those connections that might receive the most benefit from it.

This is the default setting and I'd advise it's left as default. As always, ensure the latest NIC drivers/Firmware is installed to remove the risk of any known issues but in my experience taken from many thousands of customers, this feature is a real benefit to the OS. In fact I've seen multiple customers who have gotten into the habit of disabling these features in their OS build following the issues they experienced with Server 2003. I've been called out to look at performance issues and when we've re-enabled the features we notice a massive performance improvement.

If you are getting problems which are resolved by turning off TOE in 2008 R2, my first step would be to update the NIC driver and firmware as there are almost always updates for the NICs which resolve the majority of offloading issues I encounter.

If the problem persists, turning off Offloading is the wrong thing to do, raise a case with Microsoft and we'll help you get to the bottom of it, by having a policy of disabling these features, you are effectively restricting your Windows platform's network performance for the sake of one or two issues which could be investigated and resolved.

The performance improvement on certain connections is enormous and shouldn't be thrown away due to habit (i.e. the 2003 behaviour) or a few issues which haven't been fully investigated, quick fixes will eventually come round and bite you, in my personal experience.

To manage the settings in 2008 R2 the following KB gives more information on the Netsh commands available.

http://technet.microsoft.com/en-us/library/gg162682(v=ws.10).aspx

It's also advised to install http://support.microsoft.com/kb/2477730/en-us to resolve an issue with offloading in 2008 R2, this is non urgent so could be planned into your next change window.

Server 2012

Offloading in Windows Server 2012 works much as it does in server 2008 R2 so the same advice applies. RSS however becomes more important in this OS due to the fact SMB Multichannel relies on it.

http://support.microsoft.com/kb/2846837/en-us is a recommended hotfix for RSS in server 2012.

Additional points of note around offloading:

Large send offload and checksum offload

I've seen many references on the internet pointing to things around TCP task offloading, such as Checksum offloading and Large Send offload being related to TCP chimney. Its important to note, these are not related to the TCP Chinmey/offload described above. Checksum offload is where we allow the NIC to set the checksum on a packet when it leaves the machine (which is why Netmon and Wireshark often show "incorrect checksum" on packets as the driver which captures them sits above the NIC where the checksum is set). Large Send offload (LSO) allows the Application layer to dump down a packet which would be too big for transmission and allows the NIC to chop it up into transmittable sizes (which is why you can see packets > 1460 bytes of payload in Netmon/Wireshark).

These can be set in the NIC properties but are generally very very safe to leave on. You may want to disable LSO if you're sniffing traffic as you wont be seeing the packets as they are transmitted on the wire.

 

 

Network tracing offloaded connections:

Another reason you may want to disable TCP offloading is if you want to take a network trace. Both Netmon's filter driver and Wireshark's will show you only the three way handshake and the session tear down if offloading is being used. This is due to where the drivers sit, when offloading is used, the data bypasses these drivers so you'll only see the part of the session the OS is responsible for, the session setup and tear down.

http://blogs.technet.com/b/messageanalyzer/archive/2012/09/17/meet-the-successor-to-microsoft-network-monitor.aspx is a new tool from Microsoft which allows us to trace at different layers other than NDIS (where netmon sits) and thus may allow you to work round this issue depending on the scenario.

So for a short summary my recommendations for Offloading and RSS are:

Server OS version

RSS/Chimney On by default

Recommended setting

Methods to disable

Additional Recommendations

2003 SP2

Yes

Turn off unless needed

NIC properties or registry

Update NIC drivers and apply

http://support.microsoft.com/kb/950224/en-us

2008

No

Turn on only if needed

Nic properties or Netsh

Update NIC drivers and apply 

http://support.microsoft.com/kb/979614/en-us

http://support.microsoft.com/kb/967224/en-us

if enabled

2008 R2

Yes (Only offloads suitable connections)

Leave enabled

NIC properties or Netsh

Update NIC drivers and apply

http://support.microsoft.com/kb/2958399

http://support.microsoft.com/kb/2511305

2012 inc R2

Yes (Only offloads suitable connections)

Leave enabled

Nic properties or Netsh

Update NIC drivers and apply

http://support.microsoft.com/kb/2885978

Please note, the hotfixes may seem irrelevant but are the latest versions (as of 19 August 2014) of the relevant binaries which contain the code for handling RSS and Offloading and thus contain any hotfixes to this date which may help performance & functionality in this area.

  • Server 2003: Turn it off unless absolutely needed
  • Server 2008: Off by default, Turn on if needed after a NIC driver update and Windows hotfix.
  • Server 2008 R2/2012: On, Automatic mode by default, leave as default and update NIC drivers if possible. Install hotfix on 2008 R2 in next change window.

 

*Update.

 

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2055853 is a recommended fix if you're using RSS and VMware.

 

Hopefully this post has provided you with all you need to know around TCP offloading/RSS and has encouraged you to keep it turned on in newer OS versions.

 

Comments (17)

  1. Anonymous says:

    @Joe You are correct in saying NetDMA cannot be used with Chimney. If the OS detects Chimney and NetDMA are supported, it’ll disable Chimney and use NetDMA.

    In terms of it’s performance, it definitely can improve throughput and reduce CPU load on busy systems with high network utilisation. I have seen issues caused by it in my time at MS, but at a far lower rate than RSS and TCP Offloading. I suspect a lot of people
    just turn off all three technologies together without pinpointing which one was causing a problem though. There is also the fact that not all hardware supports NetDMA too.

    It is also only supported if the BIOS has it supported and enabled, the CPU supports it and the NIC does too. As always if you’re going to use it, make sure the latest firmware and drivers are installed (and BIOS) and test first.

    Server 2008 and R2 use a newer version than 2003 and it’s deprecated in 2012 (RDMA is used instead).

    Personally I’d say, if you’re not seeing throughput or CPU utilisation issues on 2003, then I’d turn it off. If you are seeing high CPU utilisation, and your hardware supports it, update the drivers and BIOS and test with it enabled to see if it alleviates
    the problem on the CPU.

    As for 2008 and 2008 R2, as with Chimney and RSS I’d leave it set to default if you have updated the NIC drivers and firmware & BIOS to the latest revision. It may need enabling in the BIOS though which RSS and chimney normally don’t.

    With 2012 we rely on RDMA for fast memory access from supported NICs so it’s not a concern. You also need specialist hardware for this to work but the throughput can be astonishingly good.

  2. Anonymous says:

    Hi Ian, I wasn’t sure on this so have checked with a colleague who’s an RDS Escalation Engineer and so deals with this more often than I. His (and my) opinion is that the article is based on Server 2003, so yes by all means turn off Offloading for that
    OS. For more modern OS versions i’d have to say you need to check with Citrix if there are any specific technical reasons for disabling offloading but as far as i’m aware it should work fine in tandem with Citrix. One point of note, don’t disable RSS at all
    if you can help it, as we’ve seen issues with bottlenecking caused by high DPC when we’re stuck to a single processor. Regards, Paul

  3. Anonymous says:

    Thanks for the great article and recommendations. It’s the best one I’ve found on the subject to date. I was curious however regarding recommendations for NetDMA. You only mention it briefly but I see many less in-depth articles on the net saying it should
    be turned it off (usually in combination with Chimney and RSS). The Technet description also states "You cannot use TCP Chimney Offload and NetDMA together". Can you give us further insight and/or recommendation on this subject?

  4. Anonymous says:

    Thank you

  5. Anonymous says:

    Hi Rhys, the term Chimney stems from the fact that when it’s enabled, network packets bypass the normal network stack and the application hands them off to the NIC driver to send on the network and vice versa . As if when packets arrive at the NIC driver,
    they are sent up a virtual chimney to the application layer at the top. Much like a chimney in a house takes smoke a directed route out. At least that’s my understanding of where the term came from! It’s also why, when enabled, you wont see frames in netmon
    other than the session setup and teardown, as the netmon filter driver sits in NDIS, which is bypassed with Chimney enabled.

  6. Anonymous says:

    Hi Craig, you’re correct in that key can disable task offload features such as IPSEC offload, checksum offload and large send offload. Although in newer OS versions, (Vista + if memory serves) that key is deprecated by ‘netsh int ip set global taskoffload=disabled’
    and check with ‘netsh in ip show offload a reboot is normally required to bring this into effect.
    ‘ I would strongly advise against disabling these settings for anything other than troubleshooting as it’s likely to have an impact on performance. As this is normally done on a single server/client basis, I normally recommend it is done by the NIC driver so
    we can disable individual elements for testing.

  7. Anonymous says:

    Great post.

  8. Ian Parsons says:

    Have you any experience with OS streaming products like Citrix Provisioning server? Citrix suggests disabling TCP offloading in http://support.citrix.com/article/ctx127549. Any comments on using offloading in this scenario? Thanks.

  9. Shiwani says:

    Excellent information keep up your good work thanks.

  10. Rhonin Powers says:

    Great article Paul, was browsing the net and stumbled upon this treasure. Everything in one place and on one page…nothing better than that!!!

  11. DrB says:

    DMA and dca aren’t supported after w8.this create a huge issue since ram and had drive were always the bottleneck to begin with,could ms change MSI or msix value from 1 per socket to one per core.also,what will happen with xbox one (since it was built
    from the ground up for dca,DMA .

  12. Craig says:

    According to the Windows Driver documentation "Using Registry Values to Enable and Disable Task Offloading" (https://msdn.microsoft.com/en-us/library/windows/hardware/ff571012(v=vs.85).aspx),

    "Setting [DisableTaskOffload] to one disables all of the task offloads from the TCP/IP transport. Setting this value to zero enables all of the task offloads," which I take to mean any of the other options in the article — which includes both checksum and
    Large Send Offload (LSO). I take this to mean that if one wanted to disable both checksum and LSO, one would only have to set DisableTaskOffload to REG_DWORD 0x1. Can you verify this? I mention it because you state to set this in the NIC properties, but setting
    the option via "reg add" is far quicker and less error prone than using in a driver dialog — especially when you have to configure many servers.

  13. Rhys says:

    Thanks for a great post. Really helpful. Where does the term ‘Chimney’ come from and what does it relate to?

  14. Rhys says:

    Thanks Paul. I had kind of guesed that, but wasn’t sure if there some hidden meaning I was missing. Just seems like an odd term. Anyway in offload mode you shoulnd’t need a chimney becuase the CPU won’t be smoking!!

  15. Zepfanman says:

    Thank you for this in-depth article. You have hotfix 2885978 listed for Server 2012 in the chart, but you mention 2846837 for RSS earlier in the article. Should both be applied? Neither hotfix mentions Server 2012 on the platform list on the Microsoft
    download page.

  16. Zepfanman says:

    (I just realized I can log in here.) For a little more background, in September we installed new servers according to the recommendations of a vendor whose software we are configuring (JustWare). We are getting the following error about once every 20 to
    60 minutes of use on the client end. The program will work, we just have to run the command again when we get the error. The software also runs slowly and we have frequent "Not Responding" delays at the top of the title bar. Here’s the error: Server was unable
    to process request. —> A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 – The semaphore timeout period has expired.) —> The semaphore timeout period has expired.

    Thanks again.

  17. Zepfanman says:

    RESOLVED! If anyone else happens to have the same problem we were getting on our server, the fix for us was to set up a link aggregate (LAG) on our switch. On our Meraki (Cisco) switch, it was as easy as going to the "Switch ports", check the two ports
    that were plugged into our JustWare server, and then click Aggregate.
    https://docs.meraki.com/display/MS/Switch+Ports#SwitchPorts-LinkAggregation

Skip to main content