Windows HPC Server 2008 RTM’s

We’re DONE with HPC Server 2008! We shipped our first beta last November, then a series of CTPs, then a second Beta in May, then a couple release candidates, and now we are FINISHED. 600MB of technical specifications, 500K lines of code, 250 customer based design changes, and 3000 beta downloads.

2 years and 3 months ago we shipped Compute Cluster Server 2003 (yes, a strange name for a product that shipped in 2006). Now we’re shippingWindows HPC Server 2008. First, we have a better name. The Clunk-Clunk of Compute Cluster’s cacophony was confusing. The “HPC” means we’re ready for the most demanding HPC workloads. For example, on the last Top 500 list we were number TWENTY-THREE, jumping 93 positions from our previous listing of #116.

Yes, there are a lot of skeptics. The HPC industry uses mostly Linux or UNIX servers. To even suggest Windows could be successful in HPC is blasphemy. To build our second release we went to customers, especially customers who didn’t use Windows. We conducted over 100 customer visits. We did internships, where we would work on site with HPC admins and developers. We created a customer advisory board with leading HPC experts from computational finance, engineering, government, academia and the life sciences and they were brutally honest with their feedback. We assisted several ISVs with their ports to Windows and conducted five separate week-long performance deep-dives with ISVs where we not only helped port, analyze, and tune their codes but we helped with improving concurrency in general. In the process we ate a lot of humble pie while learning how people really use their HPC servers: job schedulers, deployment tools, cluster administration tools, compilers, debuggers, and MPI stacks.

What did we learn from all these meetings? First, the traditional HPC die-hards loved beating up on a Microsoft person for a while. After a while, when they learned that we were earnest, they opened up a little. We learned Windows must perform as well as Linux. There isn’t a single feature we could build that would convince people to switch to Windows if we didn’t have the perf. Second, once we meet Linux on perf then people want solutions that let them focus on their day job instead of learning to be an HPC administrator. Many users just want to use their application, like Matlab, Mathematica, Fluent, BLAST, Excel (yep), or favorite open source app and ignore the fact that there is a supercomputer behind their application making things faster. What a bummer we spent so much time making the job submission UI so pretty. Heh.

With HPC Server 2008 we achieved our performance goals and our customers and partners are responding. One ISV’s CEO got up at his user group meeting and announced that in 5 years most of his users would switch to Windows. All jokes about the Finance Industry aside, one of our beta customers is running their financial models on our beta software. At the International Supercomputing Conference Mellanox was doing live demos of HPC Server 2008 achieving 2µsec latency and 2GB/sec (gigaBYTE not gigabit) throughput on their new ConnectX Infiniband cards. At this speed, the entire contents of the US Library of Congress – some 29 million books and 101 million other sound recordings, videos, etc. – can be transmitted in 2.5 hours!

Finally, Cray became our newestOEM partner. Cray! The Supercomputer Company! Even the skeptics are taking notice of this announcement. All of our customer focus and performance work means we can create affordable, easy-to-use supercomputing solutions. This in turn means HPC can go further into the mainstream. Cray is a big believer in this model and the Cray CX1 fits under your desk. Woah! Now, instead of waiting for hours to run your job on the big supercomputer you can run your models on the supercomputer in your office, saving the big jobs for the big cluster and running your regular jobs immediately. Oh, and the hardware is beautiful. Check it out.

What’s next? We’re working hand in hand with our customer advisory board and TAP customers to design the next release of HPC Server. We’ll have more to report next year as we continue to focus on performance while making HPC part of mainstream computing.