Windows Azure beats Amazon EC2, Google App Engine in cloud speed test
Microsoft’s Windows Azure has beaten all competitors in a year’s worth of cloud speed tests, coming out ahead of Amazon EC2, Google App Engine, Rackspace and a dozen others.
The independent tests were conducted by application performance management vendor Compuware using its own testing tool CloudSleuth which debuted last year. Anyone can get results from the past 30 days for free by going to the CloudSleuth website, but this is the first time Compuware has released results for an entire 12-month period.
Compuware uses 30 testing nodes spread around the globe to gauge performance of the cloud services once every 15 minutes. The company performed 515,000 tests overall for a year’s worth of data covering August 2010 to July 2011, which Compuware released today. Each test requires the loading of a simulated retail shopping site consisting of two pages, one page containing 40 item descriptions and small JPEG images, and the second page containing a single, larger image of 1.75MB.
The Windows Azure data center in Chicago completed the test in an average time of 6,072 milliseconds (a little over six seconds), compared to 6.45 seconds for second-place Google App Engine. Both improved steadily throughout the year, with Azure dipping to 5.52 seconds in July and Google to 5.97 seconds. Also scoring below 7 seconds for the whole year were the Virginia locations of OpSource and GoGrid along with BlueLock in Indiana. Rackspace in Texas posted an average time of 7.19 seconds, while Amazon EC2 in Virginia posted a nearly identical 7.20. Amazon’s California location scored 8.11 seconds on average.
Generally, Compuware says a 1.5- to 2-second spread between cloud providers represents a substantial performance difference.
Compuware admits the test is a limited one.
“The choice of a web site as the initial target application should be seen as a first step to understanding the availability, responsiveness and consistency of cloud service providers,” the company explains. “While admittedly monochromatic (especially in light of the richness of services provided by cloud providers), the choice reflects the observation that the majority of modern applications rely on the Internet protocols as their transport mechanism. It enables us to create a relatively small and simple application that still gives us great insight into the core performance of cloud service providers. Just as importantly, it can be easily implemented on both PaaS and IaaS cloud providers.”
Amazon EC2 has proven its worth in many real-world scenarios, including the building of a 30,000-core HPC cluster on EC2 and a separate EC2-based cluster that ranked in the world's Top 500 fastest supercomputers.
But within the limits of the Compuware test, Azure has improved in relation to its competitors. In July 2010, Network World conducted a test using the CloudSleuth tool and found that both Google App Engine and Amazon EC2’s Virginia location were faster than Windows Azure over the course of a month. (Google App Engine, perhaps because of its distributed nature, is tested by CloudSleuth as a whole rather than from specific locations.)
Although Compuware tries to make the tests expansive by spreading nodes throughout the world, the results are still highly affected by location. For example, both Azure and Amazon posted poor scores in their Singapore data centers (16.10 seconds for Azure and 20.96 seconds for Amazon, the worst time in the survey) but the discrepancies between North America and Asia are due in large part to limitations in the Compuware testing network. “Within Asia, the performance is generally abysmal by North American standards,” says CloudSleuth product manager Lloyd Bloom. But the measurements are skewed because “most of our measurement points are not in Asia.”
These times we've been discussing so far have been worldwide averages. North American-only times are generally about twice as fast. Azure continued to be the fastest in the past 30 days, both in North America only and in a worldwide average, according to results pulled from the CloudSleuth website this week. Azure also did well in availability, with its Chicago facility hitting 99.93 percent uptime over the past month, significantly better than the 97.69 percent score posted by Google App Engine, and among the highest overall. Rackspace in Texas achieved 99.96 percent uptime, while Amazon EC2 in California scored 99.75 percent and EC2 in Virginia was up 99.39 percent of the time.
While Compuware’s results may be a good starting point for customers trying to decide between various cloud services, they’re not perfect. For example, Salesforce’s Force.com cloud isn’t tested, even though it may be the most widely used platform-as-a-service cloud.
Compuware’s testing isn’t sponsored by any vendor, but several of the vendors in the testing are either Compuware partners or customers, including SoftLayer and OpSource. OpSource chief marketing officer Keao Caindec called the CloudSleuth test easy to use and among the most extensive in terms of the number of clouds tested, but the website loading test doesn’t include everything a potential customer might care about. “There are several ways of looking at load testing,” he noted. Another test is to simulate credit card transactions, which requires several steps. Compuware does provide that sort of testing, Caindec said, but you have to pay for it. Compuware is only willing to give away so much for free, but CloudSleuth does provide some interesting information.
“It’s been difficult in the past to actually come up with an objective way for our clients to compare us to other clouds,” Caindec said.
Compuware designed the webpage loading test to be fairly generic. “We wanted to make sure we didn’t play to the strengths of any one specific provider and their ability to accelerate pages,” said Compuware product marketing manager Ryan Bateman.
While Bateman noted that “some providers are going to fare much better because they’re geographically closer to the locations in which we’re testing from,” CloudSleuth provides enough granular data to help customers figure out the differences. For example, on CloudSleuth.net, you can click the measurement location in, say, New Jersey or Argentina and see a ranking of all the cloud providers from that location. Depending on the application or services you want to host with a cloud provider, the location-specific data may be more important than the global average.