Thinking outside the Windows box – Bringing SQL Server to Linux

Bringing Microsoft’s enterprise database engine, SQL Server, to the Linux operating system was no simple task. The organisational and technical challenges of doing this must’ve been significant. So what has Microsoft done, why and how?

What has Microsoft done?

Microsoft has brought its enterprise data platform product, SQL Server, to Linux; it’s now possible to install it on the Red Hat, SUSE and Ubuntu variants of the Linux operating system. Microsoft’s focus so far has been on getting the database engine component of SQL Server working on Linux, but this could also mean bringing other SQL Server services to Linux in the future, such as Integration Services and Reporting Services.

SQL Server Deployment Options in 2017

When is Microsoft doing this?

Microsoft made its first announcement about SQL Server coming to Linux in early 2016. Back then, it spoke about a public release in mid-2017, with some sources going so far as to suggest June 2017. However, while we don’t yet know the exact release date, we now know Microsoft is nearer the end of the project than the beginning. It’s released several Community Technology Preview (CTP) builds of “SQL Server vNext”, the latest of which was in March 2017, with each of these showing just how close Microsoft is in bringing all of the database engine’s functionality to Linux. It’s clear Microsoft wants to make the SQL Server database engine on Linux identical to its Windows relation.

Why has Microsoft done this?

Recent technology trends have shown that IT leaders, developers and administrators increasingly want to choose the platforms they host their application solutions on. This might be SaaS or PaaS services which abstract the underlying technology entirely, or they want to standardise on Windows or Linux platforms hosted on IaaS virtual machines but not both. Whatever their decision, people now more than ever want to feel they had a say in making it. That’s one of the reasons why Microsoft has already brought its Visual Studio and .NET technologies to the Linux platform. It wants organisations who choose to use Linux rather than Windows to still be able to use its application and data platforms. That’s why its providing its database technology to audiences who want to use SQL Server but who don’t want to use Windows Server.

How has Microsoft done this?

Bringing SQL Server to Linux was no easy task. For over 20 years, the product was written to use readily available capabilities in the Windows operating system because it didn't need to run anywhere else. These capabilities provide some of its core functionality, such as executing CLR code and managing XML data, as well as give it its enterprise scale and performance by using CPU scheduling and memory management APIs. With even a conceptual understanding of this tight integration, it becomes clear that taking SQL Server to Linux means either porting a good chunk of Windows functionality with it, or writing a new version of SQL Server that has had every Windows feature replaced with a Linux feature. Both options which, until recently, were economically impossible and almost certainly technically impossible as well.

So what changed? Two things.

SQL Server on Linux containerisation model

The first was a major rewrite of the internal SQL Server database engine architecture in SQL Server 2016. SQL Server 2016 introduced a Platform Abstraction Layer, a PAL, within its core that removed its operating system specific code to external libraries that were then called through the PAL. This meant that the library for Windows could be quite lightweight, while the library for Linux could be packed with extra functionality yet the core SQL Server code didn’t need to know.

The second change, which was what made any of this possible, was the maturing of containerisation technologies and a Microsoft Research project called Drawbridge. These innovations allowed the SQL Server team to create an application container hosted by the Linux operating system that runs a very small version of the Windows operating system inside of it. Microsoft brought SQL Server to Linux – and the Windows functionality it depends on – by running an in-process “microkernel” version of Windows on Linux! While it’s invisible to anyone without a debugger, this microkernel version of Windows provides SQL Server with access to the 1200+ Windows APIs and features it needs to run. When SQL Server needs to perform an operating system task, such as mange CPU scheduling, then it’s the job of the microkernel to filter out which requests it needs to pass on itself to the underlying Linux operating system.

If you want to describe in a sentence how Microsoft makes SQL Server run on Linux, then SQL Server runs on a tiny version of Windows, which runs on Linux.

The reality

To bring SQL Server to Linux, Microsoft had to make some considerable changes to the SQL Server architecture, build on a Microsoft Research project and make sure it could still provide the same levels of functionality, security and performance on Linux as SQL Server provides on Windows. The overview above doesn’t do justice for the innovation that was needed. Instead, I recommend reading more about how Microsoft did it, written by the man who led the team doing it – Slava Oks. Slava wrote the core of the SQL Server 2005 database engine that’s still used today, and also led the team bringing it to Linux since 2015.

The first article I recommend reading is an MSDN article that describes the role of PAL, SQLPAL and Project Drawbridge in making SQL Server run on Linux.

The second is a 30 minute podcast by Slava in which he talks about how SQL Server on Linux works. There’s some good notes on the web page if you don’t want to listen to the podcast

Finally, for everything to do with the product itself, there’s the SQL Server home page on microsoft.com.

Gavin Payne is the Head of Digital Transformation for Coeo, a Microsoft Gold partner that provides consulting and managed services for Microsoft data management and analytics technologies.  He is a Microsoft Certified Architect and Microsoft Certified Master, and a regular speaker at community and industry events.