Which Linux Integration Services should I use in my Linux VMs?

Overview
If you run Linux guest VMs on Hyper-V, you may wonder about how to get the “best” Linux Integration Services (LIS) for your Linux distribution and usage scenario.  Getting the “best” is a bit nuanced, so this blog post gives a detailed explanation to enable you to make the right choice for your situation.

Microsoft has two separate tracks for delivering LIS.  It’s important to understand that the tracks are separate, and don’t overlap with each other.  You have to decide which track works best for you.

“Built-in” LIS
One track is through the Linux distro vendors, such as Red Hat, SUSE, Oracle, Canonical, and the Debian community.  Developers from Microsoft and the Linux community at large submit LIS updates to the Linux Kernel Mailing List, and get code review feedback from the Linux community.  When the feedback process completes, the changes are incorporated into the upstream Linux kernel as maintained by Linus Torvalds and the Linux community “maintainers”.

After acceptance Microsoft works with the distro vendors to backport those changes into whatever Linux kernel the Linux distro vendors are shipping.  The distro vendors take the changes, then build, test, and ultimately ship LIS as part of their release.  Microsoft gets early versions of the releases, and we test as well and give feedback to the distro vendor.  Ultimately we converge at a point where we’re both happy with the release. We do this with Red Hat, SUSE, Canonical, Oracle, etc. and so this process covers RHEL, CentOS, SLES, Oracle Linux, and Ubuntu.  Microsoft also works with the Debian community to accomplish the same thing.

This track is what our documentation refers to as “built-in”.  You get LIS from the distro vendor as part of the distro release.  And if you upgrade from CentOS 7.0 to 7.1, you’ll get updated LIS with the 7.1 update, just like any other Linux kernel updates.  Same from 7.1 to 7.2. This track is the easiest track, because you don’t do anything special or extra for LIS – it’s just part of the distro release.  It’s important to note that we don’t assign a version number to the LIS that you get this way.  The specific set of LIS changes that you get depends on exactly when the distro vendor pulled the latest updates from the upstream Linux kernel, what they were able to include (they often don’t include every change due to the risk of destabilizing), and various other factors.  The tradeoff with the “built-in” approach is that you won’t always have the “latest and greatest” LIS code because each distro release is a snapshot in time.  You can upgrade to a later distro version, and, for example, CentOS 7.2 will be a later snapshot than CentOS 7.1.  But there are inherent delays in the process.  Distro vendors have freeze dates well in advance of a release so they can test and stabilize.  And, CentOS, in particular, depends on the equivalent RHEL release.

End customer support for “built-in” LIS is via your Linux distro vendor under the terms of the support agreement you have with that vendor.  Microsoft customer support will also engage under the terms of your support agreement for Hyper-V.   In either case, fixing an actual bug in the LIS code will likely be done jointly by Microsoft and the distro vendor.  Delivery of such updated code will come via your distro vendor’s normal update processes.

Microsoft LIS Package
The other track is the Microsoft-provided LIS package, which is available for RHEL, CentOS, and the Red Hat Compatible Kernel in Oracle Linux.  LIS is still undergoing a moderate rate of change as we make performance improvements, handle new things in Azure, and support the Windows Server 2016 release with a new version of Hyper-V.  As an alternative to the “built-in” LIS described above, Microsoft provides an LIS package that is the “latest and greatest” code changes.  We provide this package backported to a variety of older RHEL and CentOS distro versions so that customers who don’t stay up-to-date with the latest version from a distro vendor can still get LIS performance improvements, bug fixes, etc.   And without the need to work through the distro vendor, the Microsoft package has shorter process delays and can be more “up-to-date”.   But note that over time, everything in the Microsoft LIS package shows up in a distro release as part of the “built-in” LIS.  The Microsoft package exists only to reduce the time delay, and to provide LIS improvements to older distro versions without having to upgrade the distro version.

The Microsoft-provided LIS packages are assigned version numbers.  That’s the LIS 4.0, 4.1 (and the older 3.5) that you see in the version grids in the documentation, with a link to the place you can download it.  Make sure you get the latest version, and ensure that it is applicable to the version of RHEL/CentOS that you are running, per the grids.

The tradeoff with the Microsoft LIS package is that we have to build it for specific Linux kernel versions.  When you update a CentOS 7.0 to 7.1, or 7.1 to 7.2, you get changes to the kernel from CentOS update repos.  But you don’t get the Microsoft LIS package updates because they are separate.  You have to do a separate upgrade of the Microsoft LIS package.  If you do the CentOS update, but not the Microsoft LIS package update, you may get a binary mismatch in the Linux kernel, and in the worst case, you won’t be able to boot.  The result is that you have extra update steps if you use the Microsoft provided LIS package.  Also, if you are using a RHEL release with support through a Red Hat subscription, the Microsoft LIS package constitutes “uncertified drivers” from Red Hat’s standpoint.  Your support services under a Red Hat subscription are governed by Red Hat’s “uncertified drivers” statement here:  Red Hat Knowledgebase 1067.

Microsoft provides end customer support for the latest version of the Microsoft provided LIS package, under the terms of your support agreement for Hyper-V.  If you are running other than the latest version of the LIS package, we’ll probably ask you to upgrade to the latest and see if the problem still occurs.  Because LIS is mostly Linux drivers that run in the Linux kernel, any fixes the Microsoft provides will likely be as a new version of the Microsoft LIS package, rather than as a “hotfix” to an existing version.

Bottom-line
In most cases, using the built-in drivers that come with your Linux distro release is the best approach, particularly if you are staying up-to-date with the latest minor version releases.  You should use the Microsoft provided LIS package only if you need to run an older distro version that isn’t being updated by the distro vendor.  You can also run the Microsoft LIS package if you want to be running the latest-and-greatest LIS code to get the best performance, or if you need new functionality that hasn’t yet flowed into a released distro version.  Also, in some cases, when debugging an LIS problem, we might ask you to try the Microsoft LIS package in order to see if a problem is already fixed in code that is later than what is “built-in” to your distro version.

Here’s a tabular view of the two approaches, and the tradeoffs:

Feature/Aspect “Built-in” LIS Microsoft LIS package
Version Number No version number assigned.  Don’t try to compare with the “4.0”, “4.1”, etc. version numbers assigned to the Microsoft LIS package LIS 4.0, 4.1, etc.
How up to date? Snapshot as of the code deadline for the distro version Most up-to-date because released directly by Microsoft
Update process Automatically updated as part of the distro update process Requires a separate step to update the Microsoft LIS package.  Bad things can happen if you don’t do this extra step.
Can get latest LIS updates for older distro versions? No.  Only path forward is to upgrade to the latest minor version of the distro (6.8, or 7.2, for CentOS) Yes.  Available for a wide range of RHEL/CentOS versions back to RHEL/CentOS 5.2.  See this documentation for details on functionality and limitations for older RHEL/CentOS versions.
Meets distro vendor criteria for support? Yes No, for RHEL.  Considered “uncertified drivers” by Red Hat.  Not an issue for CentOS, which has community support.
End customer support process Via your distro vendor, or via Microsoft support.  LIS fixes delivered by distro vendor normal update processes. Via Microsoft support per your Hyper-V support agreement.  Fixes delivered as a new version of the Microsoft LIS package.