15GB/s + 5 Days = 6 Petabytes of Streamed Data - SMB3.0 Multi-Channel + RDMA Demo Wins Best of TechEd

Article
07/19/2012

In this blog post, we would like to highlight one piece of the work we have done with Independent Hardware Vendors and OEM Storage partners who are adopting the new SMB 3.0 protocol available in Windows Server 2012. Don Cottam is a program manager in the Windows Server Partner and Customer Ecosystem Team who worked extensively with Independent Hardware Vendors, Independent Software Vendors and OEM Storage partners during the development of Windows Server 2012. He helped to facilitate engagements with many storage vendors throughout our development cycle; companies delivering on storage technologies including Offloaded Data Transfer, Thin Provisioning, Storage Spaces, Data Deduplication, SMB 3.0, and backup/recovery solutions, among other features. Dozens of companies in the storage industry have been working on new hardware and software to support Windows Server 2012 and Windows 8, and several of those companies are now ready to demonstrate their hardware and software. In his series of blogs, Don will highlight the many new features related to storage in this release of Windows Server 2012.

--Natalia Mackevicius,

Principle Group Program Manager, STB Partner and Customer Ecosystem Team

I’m sitting in the Philly airport on my way back from Toronto where I attended the Microsoft Worldwide Partner Conference 2012.I’m taking advantage of my four hour layover to review some photos I took at that conference as well as at the recent TechEd North America 2012 conference and have been reflecting on how awesome the new SMB 3.0 protocol is. I just have to blog about it!!

SMB 3.0: Not Your Dad’s Server Message Block

In September of 2011, just one week after Windows 8 was unveiled at the //BUILD/ conference, I attended the SNIA Storage Developer’s Conference in Santa Clara. Several new features in Windows Server 2012 were presented and demonstrated, and all week long there was an SMB Plugfest happening in the lower level of the hotel. That Plugfest, as well as several subsequent ones and recent industry events, showcase the close work that Microsoft has been doing with our storage partners. There have been some fantastic demos of transparent failover, Offloaded Data Transfer (ODX), Data Deduplication, amazing throughput speeds and other new storage technologies in the past few months related to Windows Server 2012. The partner collaboration for this release of Windows Server has been phenomenal. As we get ready to release Server 2012 many storage companies have announced support for SMB 3.0 as well as other new features, and many of them already have hardware ready to go now.

Prior to starting work on Server 2012 we had deep conversations with hundreds of customers and partners that generated thousands of very specific points of feedback. We then mapped the feedback into categories, and used those are the basis for the improvements that you can see today in the Release Candidate version of Server 2012. Two points of feedback that we heard consistently from customers was that they like the cost and the simplicity of management that file servers and file-based storage provide, but they wanted us to improve performance.

The Server Message Block protocol has been around for a long time, but with version 3.0 and Windows Server 2012 it has really matured and is now a viable competitor in many scenarios where more expensive Fibre Channel or iSCSI block storage arrays have previously been the only options for datacenter architects and administrators. Performance, Cost and Convenience are three decision drivers when selecting a storage solution. Block-based arrays have long held the edge when it comes to the Performance equation, but now SMB 3.0 will allow file-based storage solutions to match the performance of Fibre Channel, iSCSI and other block-based protocols, eliminating Performance as a primary driver when architecting storage solutions. As far as Cost (Capital Expenditure) as Convenience (Operational Expenditure) file-based storage generally beats block storage in both categories. File storage is less expensive to acquire, and is simpler to manage. File storage solutions also have the potential to provide an easier transition to cloud-based storage in the future.

Performance Improvements in SMB 3.0

An exciting new feature of SMB 3.0 and Server 2012 is the capability to use kernel-mode RDMA (Remote Direct Memory Access) with a new class of NICs that support networking protocols such as InfiniBand, iWARP, and RoCE. There are several vendors that already offer RDMA-capable NICs, including Intel (iWARP), Mellanox (InfiniBand, RoCE), and Chelsio (iWARP).

Jose Barreto (Microsoft) chats with Keith Hageman (X-IO) about the details of their demo.

SMB Direct – or SMB over RDMA – will automatically detect and leverage RDMA-capable NICs if they are available and properly configured. It will also fall back to TCP/IP if these NICs are no longer available for any reason. Even with non-RDMA NICs, there are significant performance improvements in SMB 3.0.

Another important improvement to SMB 3.0 is the addition of multichannel support. SMB Multichannel offers failover capabilities with both RDMA and non-RDMA NICs, and increased throughput when using multiple RDMA NICs. Fast and transparent failover is more easily achieved when multiple channels are proactively enabled. Upon failure of any one of the NICs, another NIC will already have an established channel and can be used without any delays associated with creating a channel, authenticating, etc. Also, if multiple RDMA NICs are available, SMB can leverage the additional speed and transfer more data by using multiple channels and spreading the load.

It’s this last point – leveraging the additional speed of multiple RDMA-capable NICs – that caught everyone’s eye and generated buzz at TechEd this year.

There are several exciting benefits to using SMB 3.0 and RDMA NICs, including:

· Low CPU cycles per byte transferred

· High bandwidth utilization with minimal client CPU utilization

· Low latency

· Zero-copy, zero-touch data movement

These four benefits of SMB Direct translate into a single word that everyone understands: speed!!

SMB Direct is fast. Microsoft has published some preliminary information on the speed potential of SMB3.0, but at TechEd X-IO blew everyone away with their real-world demo of this cool new technology.

X-IO Speed-Demon Demo

The X-IO booth at TechEd was dominated by a full-sized server rack filled with sexy X-IO ISE storage units, two HP DL380 G8 servers and two HP DL360 G7 servers interconnected with Mellanox Infiniband and QLogic fibre channel adapters and switches. X-IO showed how a storage system that they put together using this off-the-shelf equipment and running the Release Candidate of Windows Server 2012 could sustain 15 gigabytes-per-second throughput speeds and 125,000 IOPS in a demo built around the SMB 3.0 Multichannel and kernel-mode RDMA. This was especially impressive since they used only the 10,000rpm hard drives in their ISE-2 units (no SSD drives for caching!) and were able to achieve those speeds on spinning media.

What do all of those numbers mean? In the five days that the X-IO demo was running at TechEd 2012 at sustained speeds over 15 Gigabytes-per-second they transferred slightly more than 6 Petabytes of data. Those are huge numbers! Even the outspoken storage industry critic Jon Toigo was impressed by the X-IO demo and their ISE storage products – especially their IOPS per watt story – and raved about them a couple of day ago on his blog site DrunkenData.com.

The data sheet that X-IO provided at their TechEd booth had the following information:

This year’s demonstration simulates and stresses applications and user workloads over SMB 3.0 network file shares on Windows Server 2012. Multiple workloads were tested by using Iometer and SQLIO on the client that in one scenario mimics streaming large files from network shares. The file servers in this case are creating logical disks using X-IO’s X-Volume software to stripe data across multiple ISE volumes. All of the ISE Volumes were presented to the servers over a Fibre Channel storage area network comprised of QLogic 5800 Series, 8Gb switches and QLE2564 Fibre Channel Host Bus Adapters. The TCP/IP network between the servers and the client is comprised of a Mellanox SX6036 InfiniBand switch and Mellanox ConnectX-3 adapters.

Besides the SMB3.0 throughput demo, X-IO was also showing off their Hyper ISE product this year. Last year, at TechEd 2011, X-IO won both the Breakthrough Technology and Attendees’ Pick Awards for the new, innovative ISE drive enclosures. This year X-IO went into TechEd with a nomination as a finalist for the Best of TechEd 2012 award in the Hardware and Storage category for their new Hyper ISE product.

Not only did they win Best of TechEd 2012 for their category, but X-IO also took home the TechEd 2012 Attendees’ Pick Award. Great job, Team X-IO!! Many congratulations, and keep up to good work!

Taking Advantage of SMB 3.0

X-IO achieved these speeds using a storage system comprising several servers and 200 hard drives. While the results won’t be as spectacular, even a single server configured as a File Server, a SQL server, a Hyper-V server or similar role will be able to take advantage of the performance and failover benefits of SMB 3.0 if it is running Windows Server 2012 and has two or more certified NICs.

Taking advantage of SMB Direct or SMB Multichannel does not require changes to applications that use Win32 file APIs. Applications accessing remote file storage will work unmodified, since the negotiation of the available transports is done by SMB itself. This means that applications can leverage all of the benefits of SMB 3.0 – including kernel-mode RDMA and Multichannel capabilities – without the need to make any changes.

With the capability to establish RDMA channels using SMB 3.0, Windows Server 2012 is better equipped to offer file based storage to server applications such as SQL Server and Hyper-V, which are sensitive to network latency and can benefit from the reduced CPU cycles/byte. This will enable file-based storage in Windows Server to mimic the performance characteristics of Fibre Channel storage, while using lower cost network fabrics like InfiniBand, iWARP or RoCE.

Additional Resources

You can read the X-IO press release regarding this demo, or on their site you can provide basic registration information to access the data sheet with details about the equipment and configuration used. Also on the X-IO site you will find case studies translating the demo into real-world application.

More information related to SMB 3.0 Direct and SMB Multi-Channel can be found at the following resources:

· Taking Server Application Storage to Windows File Shares - Windows Server Blog

· Deploying Fast and Efficient File Servers for Server Applications - MSDN Content

· SMB Blogs by Jose Barreto:

o The basics of SMB Multichannel

o Demo at Interop shows SMB Direct at 5.8 Gbytes/sec

o Windows Server 2012 and SMB 3.0 demos at TechEd

o Step by Step: Deploying SMB Direct with Mellanox / InfiniBand

o Step by Step: Deploying SMB Direct with Intel NetEffect NE020

o Preliminary performance results with Windows Server 2012 Beta and SMB Direct

Don Cottam

Program Manager, Partner and Customer Ecosystem Team (PaCE)