A Few Sleepless Nights

  • Article

A while back Ruth did an interview with Cameron McKay about an upcoming infrastructure project he was in the planning stages on.  While the planning is done and the implementation also complete, I thought it was time to revisit with Cameron and talk about how things went.  Turns out Cam was one step ahead and posted an entry on his blog which we are cross posting here.

-------------------------

And here we are.... it's March 15th, I'm now 25 Years old, and the largest infrastructure project of my career to date is wrapping up. Now, I originally planned to have all work completed in a weekend... I may have been off by a week or two. Not a result of unknown factors, but delays in setting up systems and the occasional hour or two of sleep.

February 27th at 10PM my Team and I started by redirecting all web facing sites to the ubiquitous "planned downtime" graphic to alert clients that the geeks are hard at work somewhere... :)

First up on the task list was to modify our Cisco switches and firewalls and setup 10 new VLAN's. The tricky part here is that the Cisco VOIP servers also need to be re-IP addressed and this is where we had to take our time. While our contractors were busy checking everything with the telephony systems, our team was busy setting up the base AD Infrastructure (DNS, DHCP) and getting ready to deploy all our new servers.

What servers exactly?

  • Windows Server 2008 /w Hyper-V
  • Exchange Server 2007 SP1
  • SQL Server 2008
  • IIS7
  • Team Foundation Server 2008
  • Office Communications Server 2008 R2
  • Groove Server
  • ForeFront Client Security
  • SharePoint Office Server 2007

Not to mention:

  • WSUS
  • NAP
  • Certificate Services
  • Rights Management
  • File Screening

And if that wasn't enough... We rolled out Vista Enterprise and Office 2007 to 100 workstations and also deployed the latest Blackberry Enterprise Server. I also felt the need to deploy the latest version of GFI Faxmaker to handle our some 300 faxes a day at the office.

And of course, it's all managed by System Center.

So around 2am on Saturday morning all the network changes were completed and the base AD deployment completed. At this point, we have taken a flat network and diced it up into 10 VLAN's, a DMZ, and 4 Windows domains. I'm still pretty excited and pushing through the night.

Next up was taking all our existing physical and virtual servers and switching the IP's. This took us until around 6am...

We took a coffee break and then moved onto deploying Exchange Server 2007. About this time I realized that I haven't slept yet and shrugged it off... I'm an IT Pro. At 9am we brought our SQL Servers online and IIS 7 web farm. My network team was working on all the ACL's and settings for the DMZ to make sure that we were ultra secure, as this is a Fortune 18 we work for and information security is paramount.

Our developers came in around 10am to start migrating content from the old network and get the business up and running again. One challenge here is that there was no connectivity from the old LAN to the new network, so terabyte hard drives were used to move all the files.

This took the better part of the day to get all the files over and onto the new servers. Once this was done, the Dev's could start configuring all the new database servers and IIS sites. I had our network team leave the edge network accessible from the internal LAN to make it as easy as possible for the development team to get access and complete the migration.

Around 6pm on Saturday evening the Boss realized that I hadn't been to sleep yet and "suggested" that I go over to the hotel and get a few hours sleep. I slept from 7 until 11 and then showed back up at the office to continue working on the deployment.

Sunday morning we started our LT deployment of Vista and Office 2007 to all the desktops. I was left configuring Exchange 2007 and the Blackberry Server and a half dozen other servers. We were having some issues with migrating the data. As it turns out, moving everything onto SQL 2008 and IIS7 is a big deal. Who would have thought? At this point, the comment of "backing out" came up. I didn't like the sound of that... and the discussion became a pow-wow a couple hours later.

I'll spare everyone the discussion, but we decided to push forward and not roll back. Forgetting that all the workstations were already converted to Vista and on the new network, it would of been about 6 hours work to bring the old LAN back up.

Around 2am on Monday morning I had my first real challenge of the deployment. The Exchange Hub Transport and Edge Servers suddenly stopped working. The EdgeSync connectors were all there and all the settings were correct and by-the-book.... but still, I couldn't send or receive email. So, 6 hours from the start of business, email was down. This was my worse nightmare. After a couple hours of troubleshooting and rebuilding the EdgeSync, I had email working again.

Come Monday morning, we had core systems online and our CRM application was functional but none of the file shares or printers were up. Throughout the day we were busy assisting the developers, as their apps are all client facing and most of my Infrastructure changes were completed.

Again, sent over to the hotel late Monday afternoon for a few hours sleep. Back at the office come 2AM. At this period, I'm feeling like the biggest geek in the world... and loving it.

Tuesday I had the file shares and printers online. The task for the next couple days was to import all the old email from our 5.5 server and load it into each users new mailbox. This was a tedious task that took us until Friday to complete. Why? Lot's of old email...

Over the weekend we brought ForeFront Client Security online. This is probably my most favorite piece of the new Infrastructure. All workstations and servers were now protected.

The following week we were tweaking group policies and completing the configuration of all our web portals and databases.

Fast forward to today. It's Sunday March 15th... and we have the most modern network in the company. Built for High Availability and Security from the ground up using the latest Microsoft technologies. I am very proud to say that I was apart of this great project and it has definitely been a fun and exciting ride.

The best part is that 90% of the Datacenter is virtualized on Hyper-V. I'm a huge fan of the technology, and the benefits it provides our business in the DR / BC areas.

It was interesting to see the entire IT Team, all 9 of us, pull together and stand unified behind this project. Sure, it's difficult and I've only had 20 hours total sleep in two weeks, but this is what I live for. We now have a platform to work off of that will drive the business for the next 6 Years.

Was it as easy as I thought? NO. Would I do it again knowing what I know now? YES. Why? Because it had to be done. Our business is very competitive and we were working off technology that was 10 to 15 years old. More time was spent on maintenance and workarounds, then innovation.

And how do I feel? Great. I slept in this morning and looking forward to getting back into the office tomorrow to work on my new state-of-the-art Infrastructure. All 100% Microsoft software based.

------------------------------

You can read more about the project at Cameron’s blog!