Expensive Hypervisors – a bad idea even if you can afford them

There’s a saying in politics People will fall more easily for a big lie than for a small one*. I think someone at VMware has picked this up because they’re trying a really big lie.
VMWare doesn’t need memory.

Or to quote the post more precisely: VMware Infrastructure’s exclusive ability to overcommit memory gives it an advantage in cost per VM the others can’t match.**

They were able to start 40 instances of Windows XP to achieve the 40 VMs, with 512MB of memory on a machine with only 4GB of RAM – a 5 times over commitment ratio.  Of course they didn’t actually run anything in them, because if you and I fired up Outlook, and IE (with our own mail boxes and choice of pages)  you open word and I open PowerPoint very few memory pages will be sharable (I’ve got 47 pages open in IE right now, and it’s using over 300MB of RAM, almost all for data). That means a lot of paging will have to happen in the virtualization stack. Brace yourself for really poor performance.

When I read they were using Windows 2000 in the server test I got suspicious, why use an OS which is out of support ? to get better results of course. (As I understand it they don’t support Windows Server 2008 yet). To get as much page overlap as possible they ran identical OS, and application on all the VMs, and gave them only a little memory – 512MB for a server is pretty meager. That’s important because it ensures memory is occupied by OS and programs  rather than data. Programs will have a lot of sharable pages of memory: data and caches won’t.  Suppose they had done the same over commitment with 1.5GB machines and got 4  on their system to 2 on Microsoft Virtualization.  Spare memory will get used as a cache and this will have almost no sharable pages. Data is in from disk to fill memory which isn’t there and has to be paged out in the Virtualization stack – that’s two sets of disk I/O for nothing.

The writer then cooks the figures: a $6000 for a  2-way server with 4GB of RAM ? I went onto Dell’s web site, and got a quote for 2xDual Core Processors, 16GB of RAM and 2x500GB drives in an SC1435 for under £2,000. Given that the UK prices for hardware are always higher than US prices I’m very dubious of that server price. But let’s allow that. According to the VMware post their software costs an extra $5,750 (note that’s without support). Again I’m working from the memory upgrade prices on Dells web site, an upgrade from 4 to  32 GB upgrade adds £2,720 to the price. At today’s Exchange rate, and ignoring the cheaper prices in the US, that’s $5,440. So according to VMware a $17,000 system based on their software would have 4GB of ram and run 14VMs. On 4GB the system would run 7 VMs on Microsoft software. But the same money using Microsoft Software would buy 32GB of RAM, the extra 28GB which allows you to run 56 more VMs. That’s 63 VMs on $17,000 of system with Microsoft ~ that $270 per VM for Microsoft, against $1268 for per VM for VMware. And with physical RAM for every byte the VMs see, the Microsoft system will perform better.
(To save smarter readers from pointing it out, a $6,000 server might run out of steam between the 7 VMs initially supported, and 63 VMs. The VMware work assumed there would enough CPU, DISK and network resource and the system would be RAM limited. If 14 VMs is where CPU becomes a limitation then we’d be looking at a 4GB upgrade costing £260 /$500, so the Microsoft system would come out about $5,250 cheaper for the same VMs – that’s about $375 per VM.)

That’s the argument ? Over $5000 of VMWare software is a good investment because it saves $500  on RAM ?

Further Cooking the figures. If you have a Microsoft support contract Microsoft Virtualization Software is covered. One of my colleagues tells me that VM platinum support for 3 years, would add $3666 to the total cost. I haven’t checked this numbers but divided over 14VMs that’s $261 per VM for VMware support… yes you read the numbers correctly support for VMware costs nearly as much per VM as the hardware, software and for the Microsoft solution. That’s before you allow for training, maintaining a separate management infrastructure for the virtual world, and so on.

If someone can show me a customer who is running, in production, a VMware VI3 Enterprise system with a 2:1 memory overcommit ratio on all the VMs, where spending the cost of VMware on RAM wouldn’t remove the need to use overcommitment then I’ll give… lets say $270 to their choice of charity. ***

Update. Someone asked me to remove a term which is OK in British usage, but tends to offend other English speakers.

* Foot note 1. Yes I know who wrote that, and I’m not going to suggest that VMware have been reading him. Like me, someone there might collect books of quotes and aphorisms.

**Footnote 2. As one of the commenters to the post points out VMware’s published recommendation is  "Make sure the host has more memory than the total amount of memory that will be used by ESX plus the sum of the working
set sizes that will be used by all the virtual machines."
I.e. Don’t use this feature, if you want decent performance.

*** One offer only. Void where prohibited. Over 18s only. My cause drowsiness; if affected to not operate virtual machinery. etc.

Comments (28)

  1. Anonymous says:

    In the comments an earlier post , "PMunro" raised an interesting point, and I mentioned it

  2. James ONeill says:

    Joe, you can just put $500 of RAM in the server and have 8GB instead of 4GB. Then free Microsoft software can run the same number of VMs as expensive VMware software.

    What you say only holds water *If* the server is limited by the amount of RAM that can be fitted. But a $6000 2 way server which only takes 4GB ? That’s daft. Dell can sell me a two-way server with 24GB for a shade over $2000

    (and see my later post for the trade of between cost of RAM, cost of VMware and OverCommit ratio for when VMware can be cheaper)

    Of course admin costs less on the Microsoft platform than the VMware one 🙂

  3. Anonymous says:

    Today’s post on Edge is an interview I did recently with Michael Niehaus and Jeremy Chapman regarding

  4. James ONeill says:

    @ Pmunro. We had it in the Alphas and if the market demands it we’ll put it back in. It’s a really, really bad idea to use it (VMwares own documents say as much).

    @ S Bevans. The Beta of Hyper-V is a beta, and not production ready. Actually the beta works pretty well until you put under stress like you would in production 🙂  Where did I say VMWare was unique in putting a marketing spin on things. When we say things that don’t stack up people take us to task on it. I think it’s perfectly reasonable hold other people to the same standard.

    @ "VM Ware customer". Drop me a mail with a customer name who’s doing this, where $5000 of RAM wouldn’t solve the problem, and the name of your chosen charity. VMWare has a very strong market position, and with VMotion they have a feature we won’t have until the next generation of hyper-V. There’s a difference between bitching about a competitor because they have a lead on us, and shooting down illogical statements that they make. I try not to do the former and will do the latter as and when required.

    @George. Do expand… for some customers VMWare is a good investment. But NOT because the ability to save a small about of money on RAM. The number of customers who think that it is a good investment will decline over time 🙂

  5. James ONeill says:


    Not sure which table you’re refering to.

    "You need half the number of servers to run a given number of VMs on ESX than on Hyper-V."

    No you don’t. You need half the amount of RAM. You don’t need half the amount of CPU, you don’t need half the disk I/Os (in fact you need more disk I/Os)

    If you were going to double the memory on the Microsoft system by doubling the number of servers the servers could

    (a) Run Windows Server Enterprise instead of Data-center

    (b) Use one Processor instead of 2

    (c) Use half the disk capacity, and slightly less than half the number of spindles (cheaper disks).

    but they would produce more than half the heat / power so the running cost for two severs would be more than one, but less than 2 consolidated servers.

    Once you to take that into account the rest of your numbers just unravel.

    @ Mike, I’ve replied on your blog. Assuming they’re in production I’ll pay and post something here.

  6. Anonymous says:

    It’s quite amusing to see the VMware blogger ( Citrix’s blog names him as Eric Horschmann) has come back

  7. Anonymous says:

    These past few days, we’ve seen some back and forth postings taking place on various corporate blogs around the topics of virtualization ROI and a unique feature found in VMware ESX Server called memory overcommitment. VMware’s Eric Horschman posted an

  8. James ONeill says:

    I’m only quoting the numbers that were on the blog post. But even if you use $1000 for the cost of VMware that’s still an 8GB upgrade for the Microsoft system, and the numbers in the post still don’t hold water.

    It’s always dangerous to try to match two vendors products too closely, but with Hyper-V at $28 and having a better management story to offset the lack of a "v-motion" alike, the number of good reasons to buy VMware will shrink. They won’t all vanish with the first version.

  9. Anonymous says:

    It has come to my attention that some of our competitors are making various claims about SCVMM 2008 and

  10. James ONeill says:

    @Murph182. $5000 isn’t the cost for the whole farm is it ? For every server where you add $5000 worth of VMWare add $5000 worth of RAM – that’s about 48GB, so we’re talking a Microsoft Model being on 64GB and the VMware on 16GB.

    @"me", ah… an intelligent question – what if you can’t add ram. Well read this one for how much memory you need to have the server for that to work.  


    The memory over head per VM I’ve been seeing varies (depending on what hardware is emulated and what is Native VM bus) but from memory was well inside that, and the overhead for server core is ~ 256MB

    @EuroVMUser, I meet typically a few thousand users per quarter. Desktop Virtualization lags server … and I suspect 15-20 Desktops per core would be optimistic – we’d tend to steer those solutions towards a thin client and Terminal services – in any event check the link above and do the sums for when VMWare is a better investment than RAM.

    That charity offer at the end of the main post is still open.

  11. James ONeill says:

    Your point would be what ?

    As far as I’m concerned when people develop any kind of intellectual property (from Music and Photos to software). They should at liberty to give it away, bundle it, or charge a kings ransom for it AS THEY SEE FIT. (Courts and Governments modify that in ways I don’t agree with ….)

    If other people want that intellectual property they take it on the terms offered.

    We went into Release candidate for Hyper-V today. VMware already has some features which we won’t have until (at least) the next version, and (rightly) can charge a premium for those features.  Customers decide if the features justify the premium – for example to one set of customers, Vmotion is of no value whatsoever and to another set its worth many times the cost of VMware. I’m not going to criticize anyone’s reading of the demand curve – so Let them "make their buck, their way"

    What I do object to, and will do my best to shoot down is spurious arguments being advanced for the premium priced features.

  12. Anonymous says:

    We’re two dates into our roadshow and I’ve twice been asked to do a comparison of VMware and Microsoft

  13. James ONeill says:

    @ "Me" Hyper-V handles running as many VMs as you’ve got memory (disk/CPU/Network for).

    That’s why this test is completely bogus, artificially constrain one resource [memory] and show that a piece of software gets round that constraint. However the cost of removing the constraint via software is 10 times that of removing it by hardware.

    I think the product group are aiming for better numbers than virtual server and we’ve shown that in public with *hundreds* of VMs on it.


    Well, VMware’s own perf guides say it’s a bad idea.

    Do you want to be in the support call with an app vendor who has taken steps to prevent critical bits of code being paged to disk by the OS only to have those self same bits paged out by the virtualization stack ?

    (actually that should be another post in its own right).

    Give some thought to what apps do when they see spare memory – they USE it. So you’re reading stuff from disk to cache it, only to have the virtualization stack page it back out again.

    There are exceptions – for example they only way to get a training lab going is to over commit a little.  

    If BobMu says we have to have it in the product is that because it’s expected or because it would be a good idea for customers to use it …

  14. pmunro says:

    So I take it that Microsoft won’t be adding this feature 🙂

  15. S Bevans says:

    With due respect. You might be missing the bigger picture here.  I have not had very good luck with my early access of 2008…  Production ready??? Insane thought at best.

    We have many production VM’s with high level of uptime and great performance  DRS, HA and Vmotion work as advertised.

    Live migration is better termed planned outage while VMotion is flawless.  

    and BTW my Vista migration is really going well, thanks for asking…

    can’t wait for 2008 migrations. What else can we freaking break.  

    Interesting that some people think that VMware is the ‘only’ vendor that puts marketing spin on things. Perhaps the only vendor I have seen that delivers on what they promise.  

  16. VMware customer says:

    Wake up James. Just ask some Windows customers and they will tell you that they are doing more than 2:1 overcommit using ESX. Don’t bitch about your competitor’s product just because you don’t have a shipping hypervisor yet (and it will have features when it ships, like "shipping is a feature too")

  17. George says:

    >Over $5000 of VMWare software is a good

    >investment because it saves $500  on RAM ?

    It actually is a good investment.

  18. Dave says:

    Are you running a virtual bullshit detector?  I had mine running on VMware but it was a real memory hog…

  19. Me says:

    Is it true that Hyper-V only handles six running VMs?

  20. pmunro says:


    Bob seems to think the feature is a good idea.

    We talked about Vmware ESX and its features like shared memory between VMs, "we definitely need to put that in our product" later he said it will be in the next release. Like hot add memory, disk and nic’s will be and Live migration of course, which didn’t make it in this release.

  21. VMware Customer says:

    I’ve got a wide variety of VM’s running in ESX.  They differ by OS, patch levels, applications running.  I still see the benefit of memory overcommitment, despite these differences.

    Also, with regards to having to purchase the Enterprise version to do all of this, if you aren’t using the advanced features (which Microsoft doesn’t have yet), then the cost does drop by $2000 to $3750.  And now that the 8GB limit has been removed from the Foundation Edition (formerly Starter Edition), the price drops again to $1000.  That being said, the $$$ that you say VMware will cost is very misleading.

    I look forward to when Microsoft’s offering is as capable.  

    That’s when it will really get interesting.

  22. Joe says:

    >Over $5000 of VMWare software is a good

    >investment because it saves $500  on RAM ?

    Your math is off.  If your virtual machines are bound by memory and not by CPU, than you are saving much more.  There is a good chance you can reduce the entire size of your farm based on memory over commitment.  Now you are saving the fully burdened cost of one or more servers which is significantly more than 5k.  Consider labor, real-estate, hardware, and hardware acquisitions costs.  Add in incremental costs such as power and administration.  

  23. murph182 says:

    jamesone, you’re missing joes point. Yeah, the cost for an extra 4GB of memory is less than the cost of the hypervisor…in a single server. But over an entire farm, you’re going to be able to reduce the number of servers required to do the job, along with the accompanying reduced costs in power, support, etc.

    Most virtualization projects I’ve been a part of have utilized servers with 16GB+, 32GB is seeming to be the usual…Now we’re talking about much larger discrepancies in the number of VM’s that can be hosted, as well as much larger cost savings in hardware and service/support costs.

  24. Me says:

    James: I cannot agree with VMware with a general over commitment ratio of 2, but the question here is not the 4GB RAM machine: take a server of whatever type and put in the maximum possible RAM (it is so cheap…).

    In that case you don’t have the option to put in cheap RAM to run more VMs: you have to buy a new server.

    And a technical question: how much overhead does a 1GB, 1VCPU, 32bit VM has in Hyper-V?

    (in case of ESX it is 97,35MB)

    How much RAM does Windows Server 2008 need if used for virtualization?

    (ESX Console needs ~272MB)

  25. euroVMWuser says:

    … "it’s a really, really bad idea to use it". James, tell us please, how many users of your software have you met in the past 6 months? Memory is the number 1 resource constraint in virtualised production environments! And when you virtualise desktops, memory overcommit is a godsend. We can put 15-20 desktops per core comfortably under ESX. What do your tests suggest you can do with Hyper-V?  

  26. Me says:

    Thanks for your reply.

    One more question: why have you omitted VI3 Foundation from the last comparison table…?  🙂

    Buy only 16GB RAM and VI3 Foundation for the same number of VMs as Hyper-V and you still have some free capacity in your server (e.g. if you need to run more VMs, you won’t have to buy a new server, you only have to buy some cheap RAM).

    Now let’s look at some more practical approaches/numbers.

    At first let’s stick at VI3 Enterprise and the standard "VM blogger prices" ($5998 Windows cost/server, $6000/server) and 2:1 ratio.

    You need half the number of servers to run a given number of VMs on ESX than on Hyper-V.

    Let’s say you need 2 ESX Servers, thus you need 4 Hyper-V servers:

    ESX cost: 2 *(5998 + 6000 + 5750) = $35,496

    Hyper-V cost: 4 * (5998 + 6000) = $47,992

    That is you save about $12,000 (25%) with ESX and you have:

    – more functionality

    – a proven technology

    Now you can say that 2:1 over commitment ratio is not real in general. Ok, I agree. But if that ratio is let’s say 1.2:1 compared to Hyper-V and you buy five $10,000 price ESX server:

    VI3 Enterprise cost: 5 * (5998 + 10000 + 5750) = $108,740

    VI3 Foundation cost: 5 * (5998 + 10000 + 995) = $84,965

    Hyper-V cost: 6 * (5998 + 10000) = $95,988

    Thus you pay

    – 10% less for a proven technology or

    – 10% more for more functionality and proven technology.

    You can choose.

    If you buy more expensive servers (or add related infrastructure costs such as UPS, networking, space, …) the difference between VI3 Enterprise and Hyper-V will be even less.

  27. Mike DiPetrillo says:


    You asked for a customer that was using memory overcommit in the real world. We just posted such an example. Go here for the details: http://blogs.vmware.com/virtualreality/2008/03/memory-overcomm.html.

    And thank you for the kind donation to the less fortunate in our world.

  28. mslicensing says:

    It would be great if Microsoft would adopt this “free” model to their CALs for all of their backoffice/server products.  Maybe if that happened they could have a valid platform upon which to stand and chastise VMware!  Until then, you (MS) are making your buck your way, and VMware is making their’s their way.