I thought we were done with this topic yesterday. I really did. If you missed my last article and aren’t sure what I’m talking about, it’s here.
Scott Drummonds, VMware Product Marketing, apparently saw my response and attempted to respond. I say attempted because if you’re looking for facts Scott still doesn’t provide many.
I’ve taken the text below verbatim from his blog and will again provide the facts.
Scott’s Sort Of Blog Response
Since I posted the YouTube video showing Hyper-V blue screens last Friday I’ve received a lot of comments, questions, compliments and complaints. The video and descriptive text have raised more questions than answers, so here are a few details to help fill out the story.
- The workload was not technically VMmark. There are two reasons for this:
VMmark’s run rules specify that the VMs must be configured with a single virtual disk. Because this configuration can’t make use of Hyper-V’s paravirtualized SCSI driver, which requires a second virtual disk, the run rules were violated to make Hyper-V produce its best results.
Jeff Woolsey: I have no idea what he’s talking about. Hyper-V VMs support 4 virtual IDE disks, 4 virtual SCSI controllers and each virtual SCSI controller supports up to 64 virtual hard disks. That’s 4 times more than ESX. If you want to run a virtual machine with a single virtual disk just do it. My best guess is this: Hyper-V only boots from virtual IDE, not virtual SCSI, and whoever ran this test thinks that the test must be run from virtual SCSI for the best performance.
That’s not the case with Hyper-V.
Once you install the Integration Components in the guest OS it doesn’t matter. Our virtual IDE performs just as well as our virtual SCSI because we provide an optimized enlightened (para-virtualized) IDE path. This ensures our customers always get the best possible disk performance.
The only limitations here are:
- ESX needs virtual SCSI to get better performance because their virtual IDE doesn’t perform as well
- The person(s) running these test don’t understand what they’re testing
The vendors that provided requirements for VMmark included use of SMP Linux guests. Hyper-V’s lack of support for these configurations means that it is unable to run VMmark according to the rules. Those rules were ignored by the test team and the ESX and Hyper-V tests were run with uniprocessor Linux guests so that Hyper-V was able to produce some number.
Jeff Woolsey: I see no mention of the specific guest operating system version (Red Hat, SUSE, ???) or if the Linux Integration Components were installed. I’m going to assume by the stellar test methodology so far that’s a “No.”
- The server ran 15 tiles* when ESX was installed. So, the hardware is good.
Jeff Woolsey: As I pointed out in my last blog, of the 750,000 downloads of Hyper-V gold code, we’ve had 3 reports of crashes under stress and with the same error code as seen in the video bugcheck (0x00020001). The solution in all three cases was to upgrade the server BIOS which solved the problem. This can happen as hypervisors interact very closely with the hardware and BIOS updates generally include updated microcode for processors oftentimes to address errata.
As you still haven’t answered some of the most basic questions about this “test” (and I use the word test in the broadest sense as a test implies a methodology) you’ll have to excuse me if I disagree with the assertion that the hardware is properly configured and up-to-date.
Methodology, Heard Of It?
Even with this new information, Scott is still missing the most basic of information. What version of Hyper-V was this? Was this the version included with Windows Server 2008? If so, that’s the beta. Did you apply the RTM update? Did you use the final shipping Integration Components in the guests? Did you include the Linux Integration Components for the Linux guests?
The way this information is dribbling out indicates a lack of methodology and systematic testing. Apparently, VMware has no problem ignoring these precepts when it suits them.
Here’s what I mean.
The Hypocrisy of VMWare’s Benchmark Restrictions
Ever wonder why there are almost no public competitive benchmarks of VMware, Hyper-V and/or Xen?
Simple, VMware doesn’t permit it without their approval. No, I’m not kidding. Here’s the exact language: http://www.vmware.com/download/eula/esx_server.html
3.3 Restrictions. You may not (i) sell, lease, license, sublicense, distribute or otherwise transfer in whole or in part the Software or the Software License Key to another party; (ii) provide, disclose, divulge or make available to, or permit use of the Software in whole or in part by, any third party (except Designated Administrative Access) without VMware’s prior written consent; or (iii) modify or create derivative works based upon the Software. Except to the extent expressly permitted by applicable law, and to the extent that VMware is not permitted by that applicable law to exclude or limit the following rights, you may not decompile, disassemble, reverse engineer, or otherwise attempt to derive source code from the Software, in whole or in part. You may use the Software to conduct internal performance testing and benchmarking studies, the results of which you (and not unauthorized third parties) may publish or publicly disseminate; provided that VMware has reviewed and approved of the methodology, assumptions and other parameters of the study. Please contact VMware to request such review.
If you ask them why, their answer is:
“Benchmarking is a difficult process fraught with error and complexity at every turn. It’s important for those attempting to analyze performance of systems to understand what they’re doing to avoid drawing the wrong conclusions or allowing their readers to do so.” (from HERE)
So, what does that mean when VMware royally screws up?
Apparently, that’s OK as the video is still posted even after Scott is struggling to come up with the most basic of facts.
Scott Goes On To Say
I’m hoping to convince the people responsible for the test to shed their anonymity and come out with an official paper. I’ll provide those details as soon as I can get them.
Gosh, that’s awfully swell of you Scott.
In the meantime, I’m guessing you’re going to leave that defamatory video posted which has no merit or value except to a VMware salesman. This looks great for your credibility, VMware’s credibility and EMC’s.
Your management chain must be proud.
Principal Group Program Manager
Windows Server, Hyper-V