Deploying Windows Server 2012 Beta with SMB Direct (SMB over RDMA) and the Intel NetEffect NE020 card – Step by Step

  • Article

This blog post is obsolete. For the latest post on this topic, see https://blogs.technet.com/b/josebda/archive/2012/07/31/deploying-windows-server-2012-with-smb-direct-smb-over-rdma-and-the-intel-neteffect-ne020-card-using-iwarp-step-by-step.aspx

 

1) Introduction

We have covered the basics of SMB Direct and some of the use cases in previous blog posts and TechNet articles.

If you haven’t seen those, here are a few pointers:

However, I get a lot of questions about specifically which cards work with this new feature and how exactly you set those up. This is one of a few blog posts that cover specific instructions for RDMA NICs.

In this blog post, we’ll cover all the details to deploy the Intel NetEffect NE020 card, which use the iWARP “flavor” of RDMA.

2) Hardware and Software

To implement and test this technology, you will need:

  • Two or more computers running Windows Server 2012 beta
  • One or more Intel NetEffect Ethernet Adapter (NE020) card for each server
  • One or more 10GbE switches
  • Two or more cables required for the NE020 (typically using SFP+ connectors)

Intel states support for Windows Server 2012 SMB Direct and Kernel-mode RDMA capabilities on the following adapter models:

  • NetEffect™ Ethernet Server Cluster Adapter CX4 (Dover)
  • NetEffect™ Ethernet Server Cluster Adapter SFP+SR (Argus)
  • NetEffect™ Ethernet Server Cluster Adapter DA (Argus)

You can find more information about these adapters on Intel’s web site at https://www.intel.com/Products/Server/Adapters/Server-Cluster/Server-Cluster-overview.htm.

Important note: You should use NE020 cards with the chip version X2A1 (see picture below). If you have an old card with a X2A chip, it won't work with SMB Direct in Windows Server 2012.

clip_image001

3) Download and update the drivers

Windows Server 2012 Beta includes an inbox driver for the Intel NE020. However, Intel provides an updated driver for download. You should be able to use the inbox driver to access the Internet to download the updated driver.

The latest Intel NE020 driver can be downloaded from: https://www.intel.com/support/go/network/adapter/ne020/win8. The driver is provided to you as a single ZIP file that you should extract to a specific folder. It will contain a few files, including one or more SYS files with the driver itself, an INF text file with the required information to install the driver and a few supporting files.

To update to the new driver, follow these steps:

  • Open “Device Manager” and find the NE020 device, under “Network Adapters”
  • Right click the device and select “Update Driver Software”
  • Click on “Browse my computer for driver software”
  • Point to the folder where you extracted the ZIP file you downloaded
  • Follow the wizard to complete the installation

4) Configure IP Addresses

After you have the drivers in place, you should configure the IP address for your NIC. If you’re using DHCP, that should happen automatically, so just skip to the next step.

For those doing manual configuration, assign an IP address to your interface using either the GUI or something similar to the PowerShell below. This assumes that the interface is called RDMA1, that you’re assigning the IP address 192.168.1.10 to the interface and that your DNS server is at 192.168.1.1.

Set-NetIPInterface -InterfaceAlias RDMA1 -DHCP Disabled
Remove-NetIPAddress -InterfaceAlias RDMA1 -AddressFamily IPv4 -Confirm:$false
New-NetIPAddress -InterfaceAlias RDMA1 -AddressFamily IPv4 -IPv4Address 192.168.1.10 -PrefixLength 24 -Type Unicast
Set-DnsClientServerAddress -InterfaceAlias RDMA1 -ServerAddresses 192.168.1.1

Note: The Intel NE020 does not support iWARP with IPv6 addresses. You should disable IPv6 with this specific card if you’re planning to use the RDMA capability of the NIC. You can disable IPv6 for that interface using the following PowerShell cmdlet:

Disable-NetAdapterBinding -InterfaceAlias RDMA1 -ComponentID ms_tcpip6

5) Configure the firewall

iWARP uses TCP/IP for communications, so you need to configure the Firewall to allow that traffic. You essentially need to add a firewall rule to the SMB Server to allow incoming traffic from the SMB Direct clients. In Windows Server 2012, SMB Direct with iWARP uses TCP port 5445, in addition to the traditional 445 port used for SMB.

Here’s an example of how you would configure your firewall using a PowerShell cmdlet on the SMB server to allow access by the clients:

New-NetFirewallRule –Name SMBiWarp -DisplayName "SMB iWARP In" -Enabled True -Profile Any -Direction Inbound -Action Allow -Protocol TCP -LocalPort 5445

If you have multiple SMB servers, you will need to create and enable this firewall rule on every server that will use SMB Direct with iWARP.

Note: These instructions are for the Windows Server 2012 beta release only. We’re working to add a built-in firewall rule for iWARP before the final release. When that happens, you will be able to simply enable the pre-existing rule instead of creating a new one.

6) Allow cross-subnet access (optional)

One of the main advantages of iWARP RDMA technology is the ability to be routed across different subnets. While the most common setup is a single subnet (or maybe even single rack) deployment, you can use the Intel NE020 to connect computers across subnets. However, this capability is disabled by default on Windows Server  2012.

To enable Network Direct (and therefor SMB Direct) in this fashion, you do need to configure every system (SMB Servers and SMB Clients) to allow routing RDMA across subnets. This is done using the following PowerShell cmdlet:

Set-NetOffloadGlobalSetting -NetworkDirectAcrossIPSubnets Allow
Disable-NetAdapter -InterfaceAlias RDMA1
Enable-NetAdapter -InterfaceAlias RDMA1

Note: Disabling and re-enabling the interface makes the settings change effective without a reboot.

We recommend that you apply the configuration change above before creating any shares. If you do happen to apply it (or make any other major network configuration changes), the SMB client will re-evaluate its connections when new interfaces are detected or every 10 minutes. You can also tell SMB to update its connections immediately by using the following cmdlet on the SMB clients:

Update-SmbMultichannelConnection

7) Verify everything is working

Follow the steps below to confirm everything is working as expected:

7.1) Verify network adapter configuration

Use the following PowerShell cmdlets to verify Network Direct is globally enabled and that you have NICs with the RDMA capability. Run on both the SMB server and the SMB client.

Get-NetOffloadGlobalSetting | Select NetworkDirect
Get-NetAdapterRDMA

7.2) Verify SMB configuration

Use the following PowerShell cmdlets to make sure SMB Multichannel is enabled, confirm the NICs are being properly recognized by SMB and that their RDMA capability is being properly identified.

On the SMB client, run the following PowerShell cmdlets:

Get-SmbClientConfiguration | Select EnableMultichannel
Get-SmbClientNetworkInterface

On the SMB server, run the following PowerShell cmdlets:

Get-SmbServerConfiguration | Select EnableMultichannel
Get-SmbServerNetworkInterface
netstat.exe -xan | ? {$_ -match "445"}

Note: The NETSTAT command confirms if the File Server is listening on the RDMA interfaces.

7.3) Verify the SMB connection

On the SMB client, start a long-running file copy to create a lasting session with the SMB Server. While the copy is ongoing, open a PowerShell window and run the following cmdlets to verify the connection is using the right SMB dialect and that SMB Direct is working:

Get-SmbConnection
Get-SmbMultichannelConnection
netstat.exe -xan | ? {$_ -match "445"}

Note: If you have no activity while you run the commands above, it’s possible you get an empty list. This is likely because your session has expired and there are no current connections.

7.4) Verify the SMB events that confirm an RDMA connection

On the SMB client, open a PowerShell window and run the following cmdlets to view the SMB events that confirm that you have an SMB Direct connection. If there are any RDMA-related connection errors, you will also see them:

Get-WinEvent -LogName Microsoft-Windows-SMBClient/Operational | ? Message -match "RDMA"

8) Conclusion

I hope this helps you with your testing of the Intel NE020. I wanted to covered all different angles to make sure you don’t miss any relevant steps. I also wanted to have enough troubleshooting guidance here to get you covered for any known issues. Let us know how was your experience with the beta by posting a comment.