Windows Server 2003, is your network bandwidth all going to 1 CPU? Here is how to fix it.

Article
11/23/2012

Applies to:

Windows Server 2003 Service Pack 1 + KB 912222

The Microsoft Windows Server 2003 Scalable Networking Pack release

Windows Server 2003 Service Pack 2

Symptoms:

Note: Mike Lagase has a screen shot of what task manager looks like here Processor 0 increased CPU utilization

History:

Back in the Windows NT 4.0 Server, Windows 2000 Server, Windows Server 2003 RTM/Service Pack 1, we had to use this registry key to load balance the incoming network i/o between the processors:

892100 The processor load is not distributed across multiple processors on a computer that is running Windows Server 2003, Windows 2000 Server, or Windows NT 4.0

When we launched Service Pack 2 for Windows Server 2003, there was this new feature called “Receive Side Scaling” (RSS) that would let the network traffic being received be dispersed amongst the different CPU’s. It was a part of the “Scalable Network Pack” (SNP).

What is Receive Side Scaling (RSS)?

Leverage multiple processors for greater parallelism in network receives.

Good for heavy receive processing with large numbers of connections.

i.e. SQL servers, Exchange servers, File servers, Sharepoint servers, Web servers, FTP servers, etc…

It was first available as a hotfix KB 912222 in Windows Server 2003 Service Pack 1. Which it was included with “Service Pack 2”.

For us and you the administrator, we saw a spike on network related issues. Which are documented in these blogs and articles:

We went ahead and pushed out KB 948496 via Windows Updates, one of those rare non-security hotfixes that are hosted there.

What we found afterwards, most of the issues were due to TCP Chimney and not RSS or TCPA.

So what is the recommendation today (as of at least Feb. 2009) for RSS?

WARNING: RSS is incompatible with NAT or with Network Load Balancing (NLB), so for those, please leave it turned off and use KB 892100

WARNING 2: Test, test, and test.

1) First, check in Task Manager, to see if you observe the symptom (i.e. Do you have high cpu utilization on processor 0 due to high incoming network load?).

Note: You might want to take a perfmon to see if certain scheduled jobs (i.e. Backups) make the network load and processor 0 utilization go up.

If no, nothing needs to be done.

If yes, continue to step 2 below.

2) Install:

950224 A Scalable Networking Pack (SNP) hotfix rollup package is available for Windows Server 2003
Note: You need to make sure that Step 3 is taken care of when you install this hotfix, since it’s a fix from us (Microsoft) and your NIC vendor.

3) Update the NIC driver with a version newer than Feb. 2009.

Note: You need to make sure that Step 2 is taken care of, since it’s a fix from us (Microsoft) and your NIC vendor.

Note 2: I would personally update to one that is at least within a year from today. Search the hardware vendors newsgroups for issues with different versions of the NIC driver.

4) Disable TCP Offload

904946 You experience intermittent communication failure between computers that are running Windows XP or Windows Server 2003

5) Re-enable RSS

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

EnableRSS (DWORD) 1 (hex)

6) Reboot

7) Monitor