Huge thanks to Jaap Wesselius for this great Dynamic Memory write up that he kindly allowed me to pinch from its original home on the Simple-Talk online technical journal and community hub. More about Jaap below.
Jaap Wesselius is an independent consultant from The Netherlands focusing on (Microsoft) Business Productivity solutions. Prior to becoming an independent consultant, Jaap worked for 8 years for Microsoft Services in The Netherlands, specialising in Exchange Server. Jaap has a Bsc in Applied Physics & Computer Science, is an MCSE, MCITP and MCT, and has consistently been awarded the Microsoft MVP Award (Exchange Server, fifth year now) for his contributions to the Dutch Exchange community. For his Dutch blog posts, you can visit www.exchangelabs.nl. Besides Exchange Server, Jaap is also very active in virtualisation and is a founder of the Dutch Hyper-V Community. If you’d like to get in touch, you can reach Jaap via email at Simple-Talk@jaapwesselius.nl, or on Twitter as @jaapwess.
Dynamic Memory in Windows 2008 R2 SP1
Windows Server 2008 R2 SP1 has introduced what Microsoft calls ‘Dynamic Memory’.The start-up RAM VM memory assigned to guest virtual machines can be allowed to vary according to demand based on the workload of applications running inside. Fine, but it requires careful setting up for certain applications. Jaap explains.
In Windows Server 2008 and Windows Server 2008 R2 assigning memory to virtual machines is a static process. When you assign 4 GB to a virtual machine it uses these 4 GB, no more, no less. When not all of this memory is used than it’s bad luck. Once assigned it cannot be used for other purposes. So, when you have a Hyper-V host with 32GB of memory, you can create 15 virtual machines each configured with 2 GB of memory. The last 2 GB will be used by the parent partition itself.
New in Windows Server 2008 R2 SP1 is a feature called “Dynamic Memory”. This feature can assign more memory to a virtual machine (while running) when the virtual machine needs more memory. It can also remove memory from the virtual machine when this memory can be used for other virtual machines.
Some people think VMware’s and Microsoft’s approaches to dynamic memory is exactly the same. In this article I explain both the similarities and the differences.
Memory assignment basics
In earlier releases of Hyper-V memory assignment is static. Once memory is assigned to a virtual machine it will stay there as long as the virtual machine is running. If you need to reclaim memory the virtual machine has to be shut down and the configuration needs to be changed, either via the Hyper-V manager or via System Center Virtual Machine Manager (VMM) 2008 R2. Alternatively, if a virtual machine needs more memory it has to be shut down before memory can be assigned using the Hyper-V manager or via VMM (when physically available of course).
The challenge here is designing the Virtual Machines for the proper amount of memory. For There are plenty of guidelines, both from 3rd party software vendors as well as independent consultants. But all parties remain on the safe side; rather recommending too much memory than too little, because the latter could cause performance issues.
Windows Server 2008 R2 SP1 solves this issue by introducing dynamic memory. With dynamic memory a minimum amount of memory is assigned to a VM, but when the VM needs more memory the parent partition can assign more memory to the VM. Memory can grow while the VM is running, so it’s a fully dynamic solution. On the other hand, if the VM doesn’t need the amount of memory anymore, the parent partition can reclaim a portion of that memory from the VM and use it for other VM’s. So it really is a dynamic solution.
Note. When creating a VM the memory is set to ‘static’ by default, so you have to manually turn on dynamic memory, you cannot do this while the VM is running.
Dynamic memory is available on Hyper-V hosts when the host if (of course) running on Windows Server 2008 R2 SP1 or Hyper-V Server 2008 R2 SP1 and on the following operating systems as a guest:
- Windows Server 2008 R2;
- Windows Server 2008;
- Windows Server 2003;
- Windows 7 and Windows Vista;
An addition to this, the virtual machines need to be configured with the integration components of the parent partition, i.e. Windows Server 2008 R2 SP1. Unfortunately this is not true for all versions of Windows running in the VM. When running Windows Server 2008 Standard Edition in a VM the system needs a full installation of Service Pack 1.
Dynamic memory can be configured using the Hyper-V Management MMC snap-in. When the memory option is selected, memory management can be configured. By default memory management is set to static, just like previous versions of Hyper-V. When the virtual machine is not running, the setting can be changed from static to dynamic. The following settings are available:
- Startup RAM – this is the amount of memory the virtual machine is assigned when the virtual machine is booted. The amount of available memory will never be less than the amount of startup RAM;
- Maximum RAM – this is the maximum amount of RAM that can be allocated by Hyper-V for the virtual machine. It is set to 64GB of memory (it should never be set higher since Hyper-V is currently not capable of addressing more than 64GB of memory) but it will never grow beyond the amount of physical memory available in the parent partition;
- Memory buffer – a percentage of memory that is allocated and assigned to a virtual machine for small memory bursts.
Figure 1. Setting dynamic memory properties on a Virtual Machine
Memory weight is another setting used by Hyper-V for determining which virtual machine is more important than others when it comes to reassigning memory between virtual machines.
After running the virtual machine for some time Hyper-V will assign more memory to the virtual machine when it needs more memory. This is fully dynamic and no intervention is needed. When checking the Hyper-V Management snap-in while the virtual machine is running the amount of assigned memory can be monitored:
Figure 2. Monitoring dynamic memory using the Hyper-V Management MMC snap-in.
In this example the Hyper-V.nu Virtual Machine has 2048MB of memory assigned as “startup RAM” but only 1085MB is actually used. It should be possible to lower the amount of “startup RAM” on this particular VM.
The amount of memory that is requested by the virtual machine depends of the usage profile. We all know that Exchange Server is always in urgent need of memory and so is SQL Server. But running a webserver or a file server is a different story. I’ll get back to the (supported) scenario’s later in this article.
Dynamic memory is an interaction between the parent partition and the virtual machine. This is achieved through the use of the Integration Components. Integration Components in general consist of a server part, the Virtual Service Provider (VSP) and a client part, the Virtual Service Client (VSC). The latter is the one running in the virtual machine. The VSP and the VSC are connected via the VMBus structure. The VMBus is an in-memory bus structure, allowing the parent partition and virtual machine to communicate at very high speeds.
With SP1, new components are added to the VSP and the VSC to allow the use of dynamic memory. The VSC reports the memory requirements of the virtual machine to the VSP in the parent partition. The Memory Balancer coordinates all requests from all virtual machines and allocates memory to, or reclaims memory from the individual virtual machines.
When memory is added, the VSC report this to the kernel memory manager running inside the virtual machine which now can use the added memory. To reclaim memory from the virtual machine, a ballooning mechanism is used inside the virtual machine. The VSC interacts with the kernel memory manager inside the virtual machine, and the virtual machine can return unused memory pages to the VSC. The VSC in turn returns these memory pages to the VSP running inside the parent partition, allowing it to be used by other virtual machines. When these pages are not needed by the other virtual machines anymore, the memory pages are returned to the original virtual machine.
When the parent partition needs to reclaim memory, a mechanism called ballooning starts to reclaim these memory pages. When there are no free pages available inside the virtual machine, it starts paging to disk to reclaim memory to the parent partition. Basically this continues until the startup memory limit is reached. Setting this limit too low for virtual machines will result in paging and therefore in a performance decrease.
Please note that it is the virtual machine that starts paging, since the memory manager inside the virtual machine knows which memory pages to flush to disk and which pages not retain. Second level paging, i.e. the parent partition starts to flush virtual machine to disk does never occur with Hyper-V since the parent partition doesn’t know which parts of the virtual machine to flush to disk. This is a clear differentiator from VMware’s implementation of dynamic memory.
Figure 3. Dynamic memory architecture
Dynamic memory is often referred to as “Microsoft’s answer to VMware’s overcommit” but by now it should be clear that this is not the case. Memory can only be assigned once to virtual machines, and when the limit is reached additional virtual machines fail to start. Of course it is still possible to set the startup memory limit way too low, but you don’t have to be a guru to see that this approach doesn’t work.
Microsoft also doesn’t use memory page sharing between virtual machines like VMware does. Microsoft doesn’t do this from a security perspective, but also the added value using page sharing is very limited when using large memory blocks like Hyper-V does. Page sharing works absolutely fine as long as ‘normal’ 4KB memory pages are used, but when large memory blocks are used, the efficiency decreases rapidly since the parent partition has to find identical 2MB memory blocks and calculate its hash value. More information on this can be found on the VMware site: http://communities.vmware.com/message/1262016#1262016
For more information about large memory pages there’s also an interesting article on the AMD website: http://developer.amd.com/documentation/articles/pages/2142006111.aspx
Using dynamic memory with Windows Server 2008 R2 SP1 Hyper-V and running Windows Server virtual machines is 100% supported by Microsoft. But the trick is that the payloads of the virtual machines need to be taken into account.
For example, running Exchange Server 2010 in a virtual machine is 100% supported. But Exchange server is a memory hog and it allocates (and uses) as much memory as needed. When running Exchange Server inside a virtual machine with dynamic memory enabled, you’ll quickly see an increase in memory usage. But when the parent partition needs to reclaim memory from the virtual machine running Exchange Server, it will start paging straight away, resulting in a rapid performance decrease. Therefore using Exchange Server 2010 with dynamic memory is not a supported scenario. For more information please check the ”Hardware Virtualization” section on the Exchange 2010 System Requirements page: http://technet.microsoft.com/en-us/library/aa996719.aspx. When you check figure 2 for example you’ll see that the Exchange Server is not configured with the dynamic memory option.
Dynamic memory is a new feature of Windows Server 2008 R2 SP1 and permits a more efficient use of memory when running multiple virtual machines. It removes the barrier of statically assigning memory to virtual machine, and since assigning memory is always “better safe than sorry”, often too much memory was assigned.
Using dynamic memory gives you more flexibility, allowing you to run more virtual machines on the same hardware than before. Assigning dynamic memory to virtual machines is achieved using the Integration Components (or installing SP1 inside the virtual machine), and the Integration Components can interact with the kernel memory manager inside the virtual machine. This allows dynamic memory technology to work very efficient without nasty surprises. But, you have to be careful with the minimum amount of memory assigned to the virtual machine, and you have to check if the applications running inside the virtual machine are supported with dynamic memory.