Microsoft Loves Linux Deep Dive #3: Linux Dynamic Memory and Live Backup

  • Article

This post was written by Michael Kelley, Principal PM Manager, Cloud + Enterprise team

Introduction

This blog post is #3 in a series of technical posts about running and managing Linux and FreeBSD in your on-premises datacenter.  Other posts in the series are here:

Overview

Running Linux and FreeBSD as a guest operating system on Hyper-V

Managing Linux and UNIX using System Center and PowerShell DSC

Linux Dynamic Memory

Dynamic memory is an optional feature in Hyper-V that lets Hyper-V automatically allocate and deallocate physical memory to guest virtual machines based on the virtual machines’ memory demand.   When dynamic memory is enabled, the aggregate memory size of the virtual machines running on Hyper-V can be more memory than is actually available.  This usage produces a memory overcommit, which is actually desirable for appropriate workloads because you get higher VM density, better utilization of the hardware, and lower per VM cost.  Dynamic memory works for both Windows and Linux guests, so it can even be used on Hyper-V hosts that are running a mixture of guest operating systems.

Dynamic memory is most appropriate for a mix of workloads where the memory usage varies over time.  The variability allows Hyper-V to remove memory from guests whose memory demand is low, and assign it to guests whose memory demand is high, and then reallocate back to the original VMs at a later time when memory demand changes.  Hyper-V removes and assigns memory automatically, without the administrator having the intervene or make memory allocation decisions.

If memory demand is fairly constant, there is little to be gained by Hyper-V reassigning memory from one VM to another, so dynamic memory is best left disabled.  Also, some workloads try to take all the memory in the VM and manage it themselves, bypassing the operating system.  Such workloads don’t work well with dynamic memory because the workload’s memory management conflicts with what Hyper-V tries to do.  Common examples are large scale databases and some Java application servers.

Dynamic memory is available in most Linux distros supported on Hyper-V, but it is not currently available for FreeBSD.  See the distro feature grids on TechNet for details of which distro versions have dynamic memory enabled.

How Does Dynamic Memory Work?

Dynamic memory uses two mechanisms that work independently of each other: “Hot-Add” and “Ballooning”.  To understand these two mechanisms, first recall that Hyper-V allows you to set three dynamic memory parameters on a VM:

  1. Startup memory – the amount of memory allocated to the VM when it starts. 
  2. Maximum memory – the maximum amount of memory that dynamic memory will assign to the VM
  3. Minimum memory – the minimum amount of memory that dynamic memory will assign to the VM.

Usually, Startup memory is between the Maximum and Minimum Memory.

The hot-add mechanism adds memory to the VM to grow the memory from the current size up to the Maximum Memory.  The newly added memory increases the total memory size as seen by the Linux guest OS – like adding another physical memory card to a physical computer.  There is no corresponding hot-remove mechanism; this mechanism can only add memory.

Memory is removed with the ballooning mechanism.  This is done by having the LIS dynamic memory driver claim the memory for itself, so that the Linux OS can’t use the memory.  Then the LIS dynamic memory driver gives the memory back to Hyper-V to allocate to some other VM.   This operation does not affect the total memory size as seen by the Linux OS.  Linux just thinks that the memory is in use as buffers somewhere in the operating system.  Ballooning never reduces the available memory below the Minimum Memory setting.

The ballooning mechanism can also add back memory to a VM.  When Hyper-V decides to give memory back to a VM, the ballooning mechanism first returns any memory it had previously claimed.  Once all of the ballooned memory is returned, then Hyper-V may use hot-add if even more memory is needed.

A Dynamic Memory Example

Let’s see how dynamic memory works, using an example that proceeds in three steps:

  1. A new virtual machine starts up, and its memory size is set to Startup Memory
  2. Memory demand increases in the virtual machine, so Hyper-V adds memory using Hot-Add
  3. Memory demand decreases, so Hyper-V removes memory using Ballooning

At each of the three steps, we will look at:

  • A schematic of how the memory allocation looks
  • The Hyper-V view of the memory using the “Memory” tab in Hyper-V Manager
  • The Linux OS view of the memory, using the free –m command from a Linux shell

The latter two items contrast the view from outside the VM with the view from inside the VM.

At Step 1, the virtual machine is running with no significant load.  The “Assigned Memory” for the VM is the “Startup Memory” specified when the VM was created, as shown above in the Memory Tab in Hyper-V Manager.  The Linux guest OS sees the same amount of memory, minus 32 Megabytes allocated by the BIOS for other purposes.

At Step 2, significant load has been put on the virtual machine, increasing the memory demand.  Hyper-V allocates additional memory to the VM using the hot-add mechanism.   The “Assigned Memory” as shown in Hyper-V Manager has increased, as has the memory size seen within the Linux guest OS (minus the 32 Megabytes previously mentioned).

At Step 3, the load has gone away, greatly reducing memory demand, and Hyper-V has removed memory from the VM using the ballooning mechanism.  The “Assigned Memory” as shown in Hyper-V Manager has gone back down again.  But the memory size seen within the Linux guest OS is unchanged, and is still at the previous high water mark.  The Linux guest OS doesn’t see a change in the overall memory size, but instead “holes” have been created in the memory extent (the light green in the schematic) that the Linux OS thinks are used buffers.  As expected, free –m shows that the used buffer count has grown significantly.

At any point after Step 3, if memory demand increased again, Hyper-V would first use the ballooning mechanism to fill the previously created “holes”, thereby reducing the used buffer count.  Once the holes are all filled, if memory demand continued to grow, Hyper-V would again allocate additional memory using hot-add as in Step 1.

Linux Live Backup
Now let’s look at Live Backup.  Live Backup lets you create a full backup of a Linux VM, including all of its virtual disks, without interrupting the running of the VM.  This feature is a major benefit in ensuring the greatest possible uptime for your important production workloads.

Backups are file system consistent.  When a backup is taken, the Linux guest file systems are very briefly frozen and the Linux file system buffers are flushed to (virtual) disk.  Flushing the buffers ensures that the file systems are consistent and that you won’t need to run ‘fsck’ and potentially lose data if the VM is restored from a backup that is inconsistent.

Let’s take a look at the internals of how Live Backup is accomplished, using the diagram above.  These steps are performed automatically once Step 1 is started.

  1. The process is started by the backup utility telling Hyper-V to perform a backup, and specifying one or more VMs.  A wide variety of backup utilities from Microsoft and 3rd parties will work with Hyper-V and do live backups of Linux VMs.
  2. Via the “VSS Driver” component of the Linux Integration Services, Hyper-V communicates to the Linux VM to start the backup procedures.  The driver is named “VSS” because performs a function somewhat similar to Volume Shadow Copy Services in Windows, but it doesn’t actually do the same thing, so don’t let the name confuse you.
  3. The VSS driver notifies the VSS daemon process in Linux user space.  The VSS daemon then makes Linux system calls to freeze the file systems and flush the file system buffers to the underlying virtual disks, which live in the storage you are using with your Hyper-V host.
  4. Hyper-V creates a VM checkpoint and tells LIS to unfreeze the file systems.  The Linux file systems are frozen only from step 3 to step 4 -- long enough to flush the buffers and create a VM checkpoint, which is usually less than 1 or 2 seconds.
  5. Windows VSS in Hyper-V creates a snapshot of the volume that hosts the virtual hard disk (VHD/VHDX).
  6. The backup utility copies the file system consistent VHD/VHDX from the snapshot to the backup media.

Summary

Dynamic memory and live backup are important features for your production workloads running on Linux on Hyper-V.  They are great examples of sophisticated functionality that requires cooperation between the Hyper-V host and guest OS, and of the high degree of technical integration that Microsoft has brought to Linux and Hyper-V.

Next week the topic is networking in Linux and FreeBSD guests on Hyper-V.  I’ll describe specific features for increasing availability and for network performance, and describe the level of performance Linux and FreeBSD guests can achieve on Hyper-V.