As part of our charter to be a very early adopter of emerging technologies, Microsoft.com is always pushing the envelope…we seem to live on the cutting edge and we venture out into the bleeding edge on a semi-regular basis. While the bleeding edge can be a bit hairy, (and messy…get out the band aides and sutures!!) at times, I can think of few better ways to provide better feedback into the core product teams.
Such is the case with the 64 bit architecture. We began testing the Windows Server x64 Edition of the O/S in early April 2004 with a single “white box” AMD64 reference machine and finally rolled it out across the board for www.microsoft.com at the end of April 2005.
It became evident early on that one of the compelling features of the x64 platform was the ability of running the x86 O/S on it with no performance hit. That fact alone simplified the hardware investment. We could continue to run pretty much “as is” in production while the x64 testing was proceeding.
The key takeaways from a hardware migration perspective were:
- Validate and test new platform in lab on a single server.
- Re-validate same scenarios with a single server taking live traffic.
- Phase in the new platform at our convenience since the x64 H/W can run the x86 O/S with no perf hit.
There were initial concerns around how we would port all of the legacy software applications to this new platform. The WoW64 layer virtually eliminated the need to touch or recompile any of the legacy code or 32bit ISAPIs. This was HUGE as I’m sure you can appreciate. The x64 bit processors have x86 registers so the perf hit was again negligible.
Nirvana’s new song may well be titled “Heaven Is Running Native x64”, but our software migrations did have some dependencies that required us to sing the song “Running 32bit IIS Worker Processes Under WoW64”. (Yikes was that a metaphorical stretch or what…)
Those dependencies were:
- ASP.Net 1.1 – there is no 64bit flavor for version 1.1 of the Framework – this new 64bit flavor is there in 2.0
- ISAPI extensions & filters – again leagacy code compiled years ago on x86
- Custom COM components – registered on the site that older ASP app use.
For details on how to set a metabase key that tells IIS to use 32bit worker processes instead of native 64bit go to this KB article “KB895976. How to run 32b it worker processes on 64-bit” http://support.microsoft.com/default.aspx?scid=kb;en-us;895976
The software migration gave us these key takeaways:
- Zero code changes required to move to x64 platform.
- Legacy ISAPI/COM components migrated without issue.
- ASP.Ney 1.1 apps migrated without issue.
- 32-bit processes running under WoW64 get 4Gb of addressable virtual memory.
- kernel memory isn’t sacrificed to squeeze out more VM for usermode processes. (Even with 32bit IIS worker processes, the kernel runs native 64bit.)
OK so aside from the fact that we really “like” to test drive new stuff…in production…(strap yourself in, hold on and take deep breath!). What are the compelling Performance reasons for going to x64? It turns out that they are substantial, primarily because of the larger virtual memory space available to IIS worker processes.
- App pools that recycling every 5 minutes on x86 leveled off and stopped recycling all together.
- Memory leaks or ASP.Net cache abuse from poorly behaving application were able to be identified. (2GB address space make its difficult to differentiate between an app leaking memory and one that was caching as part of its normal behavior.)
- 64bit addressing results in possibility of 2^64 unique address combinations (16 exabytes). X64 O/S uses 43 bits for addressing giving the kernel 8TB of addressable memory which leaves the other 8TB for user mode processes. Since the 32 bit processed no longer need to share ½ of the addressing space with the kernal, there is the full 4GB addressing space to use running under WoW64.
- Processor utilization improved dramatically, with average proc utilization of the x64 was 34% vs 64% for x86, primarily because apppools were no longer recycling or had settled down to recycling over weeks not minutes.
- x86 servers had noticeable CPU spikes to 100% for a sustained time period, due to one or more apppools recycling because they had run out of virtual memory. X64 server did not see these spikes because apppools are not running out of VM.
To give you a quick comparison:
X86 ASP req/sec 7.85, Response time 244ms
X86 ISAPI req/sec 110.85, Response time 248ms
X86 Static req/sec 41.9, Response time 135ms
X86 Static (cached) req/sec 47.11 Response time 1ms
X64 ASP req/sec 7.41, Response time 53ms
X64 ISAPI req/sec 125.43, Response time 18ms
X64 Static req/sec 31.01, Response time 3ms
X64 Static (cached) req/sec 54.51 Response time 1ms
(These figures generated using (SPA) Server Performance Advisor)
So to sum the x64 performance gains:
- Response time significantly improved for apps due to larger VM space
- Significant drop in CPU due to less frequent worker process recycling
- Server consolidation enhanced due to bette4r performance per server.
So I can hear some of you now saying, “All this is great stuff. But are you telling me there were NO problems?”
Actually, there were none. OK there were a few. We did have a couple of hiccups along the way and here are our Gotcha’s
1. Make sure to verify 3rd party dependencies, especially x64 compatible drivers for things like Antivirus Software; Backup Software, Imaging/Deployment Software, and common admin tools that use filter drivers like filemon and regmon.
2. Wow64 file system and registry redirection behaviors work a little differently in x64.
a. File system redirection – a 32bit process that attempts to access c:\windows\system32 get redirected by the Wow64 layer into c:\windows\syswow64. This holds all the 32 bit Windows binaries and prevents a 32bit process from trying to load a 64 bit binary…same goes for any scripts or tools running in a 32 bit process.
3. Registry Reirection
a. Works similar to the file system redirection. A 32 bit process trying to read or write to HKEY_LOCAL_MACHINE\Software gets redirected to HKEY_LOCAL_MACHINE\Software\WoW6432Node. This is more behind the scenes goodness that allow separate configurations to be maintained for 32/64 bit processes. Remember: any custom setting or keys set in this node may need to exist in both keys.
4. Script Host Bitness (cscript.exe/wscrpit.exe)
a. Scripts with dependencies on x86 COM objects need to be run with 32bit cscript or wscript. Specify either he 32bit or 64bit version of the scripting host.
X64 has had a huge beneficial impact on running www.microsoft.com. We welcome your questions about what we did, what we used, the methodology we used, our choice of musical metaphors or <your question goes here>.