Deploying Windows Server 2012 with SMB Direct (SMB over RDMA) and the Intel NetEffect NE020 card using iWARP – Step by Step
1) Introduction
We have covered the basics of SMB Direct and some of the use cases in previous blog posts and TechNet articles. You can find them at https://smb3.info.
However, I get a lot of questions about specifically which cards work with this new feature and how exactly you set those up. This is one in a series of blog posts that cover specific instructions for RDMA NICs. In this specific post, we’ll cover all the details to deploy the Intel NE020 adapters, the iWARP “flavor” of RDMA.
2) Hardware and Software
To implement and test this technology, you will need:
- Two or more computers running Windows Server 2012 Release Candidate
- One or more Intel NetEffect Ethernet Adapter (NE020) card for each server
- One or more 10GbE switches
- Two or more cables required for the NE020 (typically using SFP+ connectors)
Intel states support for Windows Server 2012 SMB Direct and Kernel-mode RDMA capabilities on the following adapter models:
- NetEffect™ Ethernet Server Cluster Adapter CX4 (Dover)
- NetEffect™ Ethernet Server Cluster Adapter SFP+SR (Argus)
- NetEffect™ Ethernet Server Cluster Adapter DA (Argus)
You can find more information about these adapters on Intel’s web site at https://www.intel.com/Products/Server/Adapters/Server-Cluster/Server-Cluster-overview.htm.
Important note: You should use NE020 cards with the chip version X2A1 (this is printed on the chip itself). If you have an old card with a X2A chip, it won't work with SMB Direct in Windows Server 2012.
3) Download and update the drivers
Windows Server 2012 includes an inbox driver for the Intel NE020. However, Intel provides an updated driver for download. You should be able to use the inbox driver to access the Internet to download the updated driver.
The latest Intel NE020 driver can be downloaded from: https://www.intel.com/support/go/network/adapter/ne020/win8. The driver is provided to you as a single ZIP file that you should extract to a specific folder. It will contain a few files, including one or more SYS files with the driver itself, an INF text file with the required information to install the driver and a few supporting files.
To update to the new driver, follow these steps:
- Open “Device Manager” and find the NE020 device, under “Network Adapters”
- Right click the device and select “Update Driver Software”
- Click on “Browse my computer for driver software”
- Point to the folder where you extracted the ZIP file you downloaded
- Follow the wizard to complete the installation
- Restart the computer to reload the drivers and services
Performance Note: The Intel NE020 driver has been found to benefit from using 4 SMB Direct connections per session, instead of the default 2 connections. While this increases the use of NIC resources, it has been found to provide significant IOPS improvement. To implement this optimization, run the following PowerShell cmdlet on all computers using the NE020 adapters:
Set-ItemProperty -Path "HKLM:SYSTEMCurrentControlSetServicesLanmanWorkstationParameters" ConnectionCountPerRdmaNetworkInterface -Value 4 –Force
4) Configure IP Addresses
After you have the drivers in place, you should configure the IP address for your NIC. If you’re using DHCP, that should happen automatically, so just skip to the next step.
For those doing manual configuration, assign an IP address to your interface using either the GUI or something similar to the PowerShell below. This assumes that the interface is called RDMA1, that you’re assigning the IP address 192.168.1.10 to the interface, that your default gateway is at 192.168.1.1 and that your DNS server is at 192.168.1.2.
Set-NetIPInterface -InterfaceAlias RDMA1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias RDMA1 -AddressFamily IPv4 -Confirm:$false
New-NetIPAddress -InterfaceAlias RDMA1 -IPAddress 192.168.1.10 -PrefixLength 24 -Type Unicast -DefaultGateway 192.168.1.1
Set-DnsClientServerAddress -InterfaceAlias RDMA1 -ServerAddresses 192.168.1.2
5) Configure the firewall
iWARP uses TCP/IP for communications, so you need to configure the Firewall to allow that traffic. You essentially need to add a firewall rule to the SMB Server to allow incoming traffic from the SMB Direct clients. In Windows Server 2012, SMB Direct with iWARP uses TCP port 5445, in addition to the traditional 445 port used for SMB.
Here’s how you would enable the built-in firewall rule using a PowerShell cmdlet on the SMB server to allow access by the clients:
Enable-NetFirewallRule FPSSMBD-iWARP-In-TCP
Note that the FPSSMBD-iWARP-In-TCP is preconfigured for all Windows Servers and it essentially allows incoming traffic on port 5445. It should only be enabled for systems with iWARP network interfaces with the proper drivers for SMB Direct usage.
If you have multiple SMB servers, you will need to enable this firewall rule on every server that will use SMB Direct with iWARP.
6) Allow cross-subnet access (optional)
One of the main advantages of iWARP RDMA technology is the ability to be routed across different subnets. While the most common setup is a single subnet (or maybe even single rack) deployment, you can use the Intel NE020 cards to connect computers across subnets. However, this capability is disabled by default on Windows Server 2012 since not everyone will require it.
To enable Network Direct (and therefor SMB Direct) in this fashion, you do need to configure every system (SMB Servers and SMB Clients) to allow routing RDMA across subnets. This is done using the following PowerShell cmdlet:
Set-NetOffloadGlobalSetting -NetworkDirectAcrossIPSubnets Allow
Disable-NetAdapter -InterfaceAlias RDMA1
Enable-NetAdapter -InterfaceAlias RDMA1
Note: Disabling and re-enabling the interface makes the settings change effective without a reboot.
We recommend that you apply the configuration change above before creating any shares. If you do happen to apply it (or make any other major network configuration changes), the SMB client will re-evaluate its connections when new interfaces are detected or every 10 minutes. You can also tell SMB to update its connections immediately by using the following cmdlet on the SMB clients:
Update-SmbMultichannelConnection
7) Verify everything is working
Follow the steps below to confirm everything is working as expected:
7.1) Verify network adapter configuration
Use the following PowerShell cmdlets to verify Network Direct is globally enabled and that you have NICs with the RDMA capability. Run on both the SMB server and the SMB client.
Get-NetOffloadGlobalSetting | Select NetworkDirect
Get-NetAdapterRDMA
Get-NetAdapterHardwareInfo
7.2) Verify SMB configuration
Use the following PowerShell cmdlets to make sure SMB Multichannel is enabled, confirm the NICs are being properly recognized by SMB and that their RDMA capability is being properly identified.
On the SMB client, run the following PowerShell cmdlets:
Get-SmbClientConfiguration | Select EnableMultichannel
Get-SmbClientNetworkInterface
On the SMB server, run the following PowerShell cmdlets:
Get-SmbServerConfiguration | Select EnableMultichannel
Get-SmbServerNetworkInterface
netstat.exe -xan | ? {$_ -match "445"}
Note: The NETSTAT command confirms if the File Server is listening on the RDMA interfaces.
7.3) Verify the SMB connection
On the SMB client, start a long-running file copy to create a lasting session with the SMB Server. While the copy is ongoing, open a PowerShell window and run the following cmdlets to verify the connection is using the right SMB dialect and that SMB Direct is working:
Get-SmbConnection
Get-SmbMultichannelConnection
netstat.exe -xan | ? {$_ -match "445"}
Note: If you have no activity while you run the commands above, it’s possible you get an empty list. This is likely because your session has expired and there are no current connections.
8) Review Performance Counters
There are several performance counters that you can use to verify that the RDMA interfaces are being used and that the SMB Direct connections are being established. You can also use the regular SMB Server and and SMB Client performance counters to verify the performance of SMB, including IOPs (data requests per second), Latency (average seconds per request) and Throughput (data bytes per second). Here's a short list of the relevant performance counters.
On the SMB Client, watch for the following performance counters:
- RDMA Activity - One instance per RDMA interface
- SMB Direct Connection - One instance per SMB Direct connection
- SMB Client Shares - One instance per SMB share the client is currently using
On the SMB Server, watch for the following performance counters:
- RDMA Activity - One instance per RDMA interface
- SMB Direct Connection - One instance per SMB Direct connection
- SMB Server Shares - One instance per SMB share the server is currently sharing
- SMB Server Session - One instance per client SMB session established with the server
9) Review the connection log details (optional)
SMB 3.0 now offers a “Object State Diagnostic” event log that can be used to troubleshoot Multichannel (and therefore RDMA) connections. Keep in mind that this is a debug log, so it’s very verbose and requires a special procedure for gathering the events. You can follow the steps below:
First, enable the log in Event Viewer:
- Open Event Viewer
- On the menu, select “View” then “Show Analytic and Debug Logs”
- Expand the tree on the left: Applications and Services Log, Microsoft, Windows, SMB Client, ObjectStateDiagnostic
- On the “Actions” pane on the right, select “Enable Log”
- Click OK to confirm the action.
After the log is enabled, perform the operation that requires an RDMA connection. For instance, copy a file or run a specific operation.
If you’re using mapped drives, be sure to map them after you enable the log, or else the connection events won’t be properly captured.
Next, disable the log in Event Viewer:
- In Event Viewer, make sure you select Applications and Services Log, Microsoft, Windows, SMB Client, ObjectStateDiagnostic
- On the “Actions” page on the right, “Disable Log”
Finally, review the events on the log in Event Viewer. You can filter the log to include only the SMB events that confirm that you have an SMB Direct connection or only error events.
The “Smb_MultiChannel” keyword will filter for connection, disconnection and error events related to SMB. You can also filter by event numbers 30700 to 30706.
- Click on the “ObjectStateDiagnostic” item on the tree on the left.
- On the “Actions” pane on the right, select “Filter Current Log…”
- Select the appropriate filters
You can also use a PowerShell window and run the following cmdlets to view the events. If there are any RDMA-related connection errors, you can use the following:
Get-WinEvent -LogName Microsoft-Windows-SMBClient/ObjectStateDiagnostic -Oldest |? Message -match "RDMA"
10) Conclusion
I hope this helps you with your testing of the Intel NE020 cards. I wanted to covered all different angles to make sure you don’t miss any relevant steps. I also wanted to have enough troubleshooting guidance here to get you covered for any known issues. Let us know how was your experience by posting a comment.