Receive Side Scaling for the File Servers

Hello everyone,

Hubert here from the Microsoft Networking Escalation team. In the cause of investigating performance problems on file- as well as on DFS-Servers we often discover that Receive Side Scaling (RSS) is disabled.
Therefore I want to give some information what RSS is about and why it plays such a vital role for fileservers (and related workloads).

Receive Side Scaling as well as TCP Chimney Offload and NetDMA are integral part of the Windows Server Operating System since the Scalable Networking Pack for Windows Server 2003 SP1.

RSS distributes the network I/O to several CPUs.
Without RSS the whole network I/O takes place on one single CPU and thus limits the maximum network throughput the machine can deliver.

From my experience a state-of-the-art CPU, under best conditions, can deliver up to 7GBit/s pure TCP traffic (ntttcp.exe) to the network. Thus a 10GBit/s Ethernet link cannot be saturated without RSS.

In order to use RSS, the network card and its drivers have to support RSS.
However nowadays this is the case for pretty much all sever-adapters and also for all virtual network cards.
In order for RSS to work, RSS has to be activated and properly configured in the network card driver AND the operating system.

By default RSS is enabled in the Windows operating system (since 2008).
For the network card drivers there are differences in the defaults among manufacturers and even sometimes driver versions.

Besides the switch “On/Off”, many drivers and the OS support a granular configuration through the following settings:

  • RSS Base Processor Number  <- Determines at which CPU the adapter begins to count
  • Maximum RSS Processors  <- Determines how many CPUs beginning at Base Processor Number should be used for RSS on this Adapter
  • RSS Queues  <- Determines for many modern cards the amount of queues which the driver should uses und thus should match the number of RSS processors

With this more granular configuration it is for example possible to configure the first NIC to use RSS Base Processor 0 and Max Processor 4 thus working on the CPUs 0-3 and to configure the second NIC to use the CPUs 4-7.

Note: The exact terms of the settings can vary among manufacturers and some even support additional settings. Further information about RSS can be found at the sites of the respective network card manufacturers. Information about the operating system settings can be found here:

https://technet.microsoft.com/en-us/library/hh997036.aspx

https://msdn.microsoft.com/en-us/library/windows/hardware/ff567236(v=vs.85).aspx

However, why is RSS so important especially for file servers?

Let’s have a quick look into the past.

With the rise of multi-processor machines, long before RSS existed, a mechanism was needed to distribute file server load among several cores.
For this purpose, the LanmanServer got a thread that allocates the Requests to several WorkerQueues and thus loadbalances them among the several CPUs.

When the new SMB protocol (SMB2) was implemented with Windows 2008, RSS as technology was already available. Thus it was decided to get rid of the additional Overhead of the loadbalancing thread and instead stick with the loadbalancing decision already made by RSS thus handling Network I/O and Fileservice work on the same CPU.
This gain in efficiency is since implemented in all Windows Server Versions and also applies to our implementation of SMB3.

This means:
If RSS is disabled or configured to a single CPU, the whole network I/O AND the whole file service work for SMB2 and SMB3 is done on a single CPU (usually CPU0).

Or to say it quite frankly: Your big beefy 8-core server is without RSS a mere single-core file server.

Especially for CPU-intensive operations such as ABE-filtering and highly frequented DFS-Namespace and File servers, performance deficits are to be expected, if RSS is not activated.

Obviously RSS is compulsory for all servers that have to deliver a high network throughput or that have to hold many simultaneous connections.

Please take some time and check your file servers, SQL-servers, Exchange servers, etc. if RSS is enabled or not.
It makes sense to incorporate the configuration of RSS into the set-up procedures for new server systems.

Yours sincerely
Hubert Sachs
Support Escalation Engineer