Q and A: Is it possible to run SMB Direct from within a VM?

Question received via blog mail:

Jose-

I picked up a couple ConnectX-2 adapters and a cable off of ebay for cheap (about $300 for everything) to test out SMB Direct.  I followed your blog "Deploying Windows Server 2012 with SMB Direct (SMB over RDMA) and the Mellanox ConnectX-2/ConnectX-3 using InfiniBand – Step by Step" and got it working.  It sure is fast and easy to setup! 

Another technology I was looking to explore was SR-IOV in Hyper-V.  When I created the virtual switch using the HCA and enabled single-root IO on it, SMB Direct no longer worked from the host.  Are these two technologies (SMB Direct and SR-IOV) mutually exclusive?  The Get-SmbServerNetworkInterface does not show an RDMA capable interface after enabling the virtual switch. 

I was hoping SMB Direct would work from within a virtual machine.  More specifically I was hoping I'd be able to utilize network direct/NDKPI from within the VM and I was using SMB Direct to verify if this was possible or not. 

Long story short, is it possible to run SMB Direct from within a VM?

Answer:

Those are known limitations of RDMA and SMB Direct. If you enable SR-IOV for the NIC, you lose the RDMA capabilities. If you team the RDMA NICs, you lose the RDMA capabilities. If you connect the RDMA NIC to the virtual switch, you lose the RDMA capabilities.

Essentially SMB needs to have a direct line of sight to the RDMA hardware to do its magic. You include any additional layers in between, we can no longer program the NIC for RDMA.

If you want to use RDMA in your Hyper-V over SMB configuration, you need to have a NIC (or two for fault tolerance) used for RDMA and a NIC (or two for fault tolerance) that you connect to the virtual switch.

More details at https://blogs.technet.com/b/josebda/archive/2013/02/04/hyper-v-over-smb-performance-considerations.aspx