One of my peers just purchased a very impressive server class machine. He’s really proud of it, and he should be! He has 2 quad core CPUs, 16GB RAM, and 2 TB of Disk space. He built this machine to be a virtualization monster, so once he got it up and running, he created an EBS configuration, all in virtual machines. I already have Virtual Machines of SBS 2008 and EBS 2008 running, so he pinged me to see if I had any suggestions on why his EBS installation was so sluggish. Let me say this loud and clear: just because you can create a configuration in virtual machines, does not mean that these configurations are “supportable” in production environments. We create a lot of virtual machines for demonstrations and education; not for production. While Microsoft supports most of our workloads created in Virtual Machines in production environments, I have not done the “due diligence” to ensure that every one of my virtual machine configurations is 100% supportable in a production environment. You must still do your “due diligence” to make the right decision for your situation.
Back to the performance issue: It turns out that you still need to tweak your host machine configuration to get the best performance possible out of your VMs. Five years ago when I started working with Virtual Machines, I learned early on (especially on slower machines) that you have to work hard to put your VMs in the best possible position to be successful. The rule of thumb is that you have to pay attention to Disk IO, CPU, and Memory (well what else is there??). While these are not always ranked in this order, more often than not, Disk IO will be your first and most significant bottleneck. While I was jealous of my friends 2 TB worth of disk space, it turns out that he only has two 1 TB drives. Having three VMs running on two spindles creates a lot of disk contention since every Windows installation has its’ own pagefile.sys and normal OS based disk IO. When you factor in the host OS, you now have two OS’ per spindle. Killer CPUs and memory can compensate for some of the Disk IO contention, but CPU and memory cannot overpower this much of a deficit. The lack of spindles can be the Achilles heel of this configuration.
Positioning your VMs on your “spindles” is crucial. The faster the disks, the lower your disk IO and the better performance you will achieve. I know it sounds simple, but when you mix in the complexity of virtual machines, you take on an additional layer of OS based IO. Each VM also has its own pagefile.sys as well its OS based IO. If you have a VM running on the same drive as your host OS, you now have two OS’s competing for the same “spindle”. Drive performance has not improved that much over the last five years, and spindles are usually the first culprit when it comes to poor VM performance. I actually have a Server Core VM that runs on my 8GB SD card. It performs well, and it’s kinda nice to have a second (small footprint) installation without adding an additional spindle. Granted, there isn’t a whole lot I can do with a small server core, but as I mentioned above, I create a lot of demo configurations, and showing off this additional VM does add some “pop” to my demonstration at a very little resource cost.
Windows Server 2008 now includes Resource Monitor, it’s accessible directly off of the Performance Tab in Task Manager. I really like this tool for its simplicity and “directness”. If the Disk graph stays “high”, disk IO is most likely the culprit, but just like conventional performance tuning, it is usually a disk IO issue, but it could also be a “masked” memory issue. Remember, if any machine does not have enough memory, the OS will spend all of its time paging virtual memory. Since Hyper-V and VPC do not allow memory over-commit, I doubt memory on the Host OS will be an issue. I usually suspect that disk IO is the true culprit. I mention memory though because this is not an absolute rule; but 95% of the time, Disk IO is the problem. There is a subtle memory problem that will look like a Disk IO problem, so lets take a look at it: Let’s say you create a VM with Windows Vista and you only allocate 512 MB of RAM. Vista will install and run in 512 MB of RAM, but it’s going to page like crazy, right? So this VM is now paging like crazy into a VHD on a spindle. You will see Disk IO as the bottleneck when in actuality, it’s the fact that you didn’t give your VM enough memory to adequately perform. Again, use performance tuning common sense…
As a rule, I try hard to run only one OS per spindle. I said OS, not VM, and I said spindle, not drive. While spindles and drives can be the same thing, I usually take my primary drive and only carve out 40-60 GB as C:. I then take the remainder of the primary drive and create a second partition with it. If you put a VM on this second partition, you’re in the same boat as if you had placed the VM on the C: drive itself. Another thing to check; where is your pagefile.sys? By default Windows puts pagefile.sys on the same partition as the OS, so unless you “tweaked” the pagefile.sys configuration, you should be OK, but it’s giving it a quick check. If pagefile.sys is on a different spindle, it will also create a lot of Disk IO. I’d shy away from putting a VM on my OS drive, or my drive that hosts pagefile.sys. Now if you put pagefile.sys on a separate spindle, your host OS will perform better, but you also loose the ability to generate some dumps in the event of a blue screen. If your focus is to “totally tweak” a machine for performance, consider giving pagefile.sys its own spindle. I’ve done a pretty often and I’m usually happy with the performance improvement. Make sense?
When I create a “big” VM, and my patience is low, I build the VM on my quad core, 8GB machine. I’ve taken two spindles on this machine and striped them. Striped drives (no parity) perform better than a single spindle since the Disk IO can be spread between the two drives. Of course, the risks with striped drives still prevail; if one of the two drives fail, you loose the contents of your striped set, so be sure you have good backups. This striped drive configuration has served me well when I’ve built VMs, or had VMs that I knew would create a larger than average amount of Disk IO. SATA 3.0 drives have been very good to me as well, but I still have a ton of PATA drives and they make good spindles for VMs too.
I’ve also found that while Hyper-V does not support memory “over-commit” is does support CPU over-commit. I usually have 3 or 4 VMs running on my quad core machine and I give each VM all 4 cores. This configuration gives Hyper-V the ability to move the CPU resources to the VM that has the immediate need. Since I, as a rule, give each VM as many cores as possible (4 is the current maximum per VM), I don’t have to necessarily worry about CPUs as a potential bottleneck, I can almost always strike CPU bottlenecks from my list of immediate concerns.
As far as memory itself, I try hard to reserve at least 1GB for the host OS. Hyper-V is very good at ensuring it has enough memory, so again, I don’t have to worry about memory that much, so that puts us right back to the spindle performance concerns. I have one 10K drive, and I’ve never seen that much of a performance increase with it so I finally just configured it as a boot drive for one of my Hyper-V machines and moved on. At this point, it’s just another drive that is still a lot more expensive than my larger 7200 RPM drives. I’d much rather purchase two 7200 RPM drives than one 10K drive. It just doesn’t make sense to me since I try to build my hardware “on the cheap”.
In summary, I try hard to allocate at least one spindle per VM, and if I know I’m going to have high performance expectations for a particular VM, I’ll stripe a few disks to increase performance. CPU’s are the least of my worry, that’s with the assumption that I’ve sized my CPU properly from the beginning. Memory can still create ‘wonky” behavior if you are too stingy with it, so be sure the scale your VM configurations just like you would any other configuration.
I hope my “informal” performance tuning thoughts have helped. Please let me know if you have any tips & tricks to improve the performance of your VMs.
Until next time!