AD In-Place Migration - Are YOU a candidate? The Series

I am passionate about Active Directory (is case you didn't know from talking with me before). It is the foundation for identity and authentication for Microsoft products. I've personally worked on over 36 Active Directory projects and migrations ranging from 200 users in a single site to 27000 spread nationally in 90+ sites.  I created the migration strategy and AD Practice at my previous Gold Certified Microsoft Partner who uses it for the bulk of its engagements. To put it simply - I love this stuff, I want YOU to get there and I will do almost anything to help.

One of the things that I found VERY frustrating over the last 5 or 6 years was the lack of clear documentation on In-place upgrades or single domain designs from the "white paper" engine that Microsoft has produced.  Let’s face it - are you a large multi-national organization with tons of physical sites? Probably not.  The proliferation of large white papers around AD design for these large organizations resulted in a lot of discussion and perception issues with my Canadian customers who didn't really need to undergo a more complex and lengthy multiple domain and Pristine domain migration which required 3rd party tools to streamline. That being said - I came across a nice little document that outlines an approach to in-place migrations - Way To Go!

If you still have one or more NT4 domains that you manage and haven’t gotten around to performing your upgrade because of various reasons, this article and subsequent posts are for you!

First off - my approach to AD design. This series of posts are about migration, not design.  That will be the topic of future posts.  But - because you have to have your "end point" AD design completed before you can start a migration - you have to have a vision on where you want to be before you get started, right?  Let me give you a couple of words of advice on AD design - they have served me well.  I was influenced by two main principles in my design projects. 1) KISS (Keep It Simple S{insert your term here}). The simpler the design you have with fewer domains, the easier it will be to implement and maintain on an ongoing basis. 2) Design for Nirvana (I stole that one from Stuart Kwan a while back from an interview he had about AD design). You want pie in the sky and are tired of working with your "old inherited and expand as you go" NT4 domain? Go Long - Go Deep!  Design for what your business needs with all the technology bells and whistles that will make it easier to manage and free your time.

The fewer domains you have (and obviously, the fewer forests) the easier your life as an IT pro will be. That being said - why would you design for multiple domains and un-needed complexity? This was another discussion I had a lot with customers who were going through a design project.  They'd present this multiple domain forest that looks a lot like their old NT4 environment and didn't really have any sound business reasons or technical reasons as to why they thought they needed it that way. I've done the single domain and domain consolidation thing many times and let me tell you - it's well worth it.  Even if you have one of those "complete trust" spider webs of NT 4 - one of your larger user account domains could be a candidate for a hybrid in-place migration.

Are you (like SO many other Canadian companies) a Candidate for an in-place Migration? Are you interested in streamlining and accelerating you migration while reducing and mitigating you risk? What makes a good candidate? Here's my perspective of a good candidate:

  • A domain with concentrated User accounts and computer accounts
  • the ability to "touch" all your BDCs in a short period of time in order to minimize the use of Mixed Mode
  • a migration team that has a clear vision of where they want to go and are willing to work at getting there
  • your target design will have a single forest and as few domains as possible (preferably 1 if you haven't caught on yet)

If you answered YES to these questions - you might want to consider reviewing my approach to in-place migrations.

But first you need a design: Minimal AD Design Concepts.
OK. I thought I wasn't going to get into AD design, but I feel I must - at least a bit - in order to make this series of posts flow.  I left the consulting side of things for my new post at Microsoft Canada because I was getting tired of all the deliverables and paper documents I created.  I've killed my fair share of trees in this line of work as an AD design and migration guy.  People ask me "How much of a design document is required?" - the answer - it depends on your audience. I would take great care in determining if the audience for EACH document was technical, executive or even end-user. Each document would be tailored to the audience to ensure that it could be digested and understood for what was important to them.  Obviously, the contract and the customer dictates the deliverables, format/medium and approximate brick thickness - but I would take the extra time to ensure the audience was kept in perspective at all times.  I've done AD design documents that are as short as 15-20 pages or as long as multiple hundreds.  One thing you WON'T find in my documents is ENDLESS pages of screenshots - carefully selected ones are quite useful.  Maybe it's just me, but do you need to show every screenshot for a migration process including the next button on a multi page wizard?  I had to review a number of competitors AD design documents on a number of occasions and I was astounded at the number of screenshots. But I digress...

How much of an AD design is required to start this migration process?  The classic consultant answer - it depends.  Most migration designs that I worked on included an AD design that had the following components completed.  The level of detail and the number of sections was directly tied to the audience.  Also - THIS IS NOT AN EXHAUSTIVE LIST - you might require more information then what is listed below. These are only "talking points" in what I would include in an AD design document.  Don't be scared of the list - some sections were as small as a paragraph, but others were quite long.

  • Review of existing environment (# domains, PDC/BDC placement, HW inventory, IP configuration, Trust relationships, User Rights, Administrative Group memberships, # global groups, Infrastructure services (DHCP, DNS, WINS), Login scrip review, System Policy review)
  • Logical Design (# forests, Domains, Password / audit policy, Domain controllers & Global Catalogues, FSMO placement, OU structure, Administrative delegation strategy, User/group placement in OUs, Computer/Server placement in OUs)
  • Physical Design (site definitions, subnets, DC and GC placement)
  • Server Design (DC hardware specs, GC hardware specs, DB placement, Sysvol placement, IP configuration)
  • infrastructure design (DNS strategy, DNS server placment, WINS, DHCP)

The Nice to have stuff - but not required for a minimal migration.
(This means that you can update these areas after the migration if you are pressed for time)

  • Group Policy design for Servers, Desktops / Laptops
  • Group Policy Strategy for administration
  • Updated Administrative delegation policy
  • DFS configurations
  • Workstation design (image deployment methods, application deployment strategy, application packaging)

I would find that people were not so interested in the review part, but after it got underway - they realized the impact of having all the information in hand when it came time to complete the design portion.

Creating the Lab environment for your AD

This section will deal with the mechanics of creating a representative test lab and getting the procedures documented just right. Please be forewarned - this process is not something that is cookie-cutter fitting to each migration project I worked on for the last 5 years. It is adaptable to each client I worked with in order to ensure all their needs and concerns were met.  You will have to take the basis of this post and apply it to your own environment to make sure you are well served by its overall use.

Most labs that I've created worked for two purposes.  One was created early on to document the design pieces that had to be in place to support the final end point of the AD design.  The other was to document the actual migration process.

The first Lab phase is one that was created based on the AD design Endpoint and is useful to share with management and non project IT staff who are not Migration or Design team members to allow them to kick the tires and provide feedback on how the design will work in the new environment (i.e. Buy-In!!!) They could see delegation working, sample GPOs tested out, inheritance flowing down OU structures - you name it.  It's a show and tell lab for what the new environment will be like.  It's an extremely valuable tool to get buy in from all the levels in an organization if you can have something they can see, feel and touch - rather then just having the design document to read.  It also is useful in order for you to document your post migration procedures for implementing your design.  This includes things like script creation to make the OU structure, administrative delegation strategies, sample GPO settings to implement right after migration, new helpdesk and administrative procedures - the list goes on and on.

The second lab phase is built to represent your current state and is then used to document the procedures required to get you migrated.  This includes as many safeguards for rollback as you comfortably require mitigating any risk you identify.  I can happily say that I have never had to roll back any migration I have designed or implemented in my 5+ years of doing this.  For this lab to work, you need a temporary BDC created in your production environment that you can pull into your lab environment (don't forget to delete this temporary BDC from your SAM before migration) This temporary BDC will be maintained in your isolated lab and promoted to become your lab PDC and hold a real copy of your live SAM database.  Whoa down there - Security concerns? No problem.  Change all the passwords in front of your security officer so that they can't be compromised.  It also facilitates user testing in future stages. I've used a script that resets EVERYONEs password (in the lab environment) to a complex password  and run it against on the database – just to show that we weren’t using the backup copy of the SAM as a cracking exercise. 

You then need to bring in or re-create your infrastructure (DNS, DHCP, WINS) and required testing services (maybe a LoB app or Web based application that is critical). This will require other IT staff members that might not be too forthcoming to help re-create something in your test lab.  How do you sell this?  It's their chance to ensure their application will continue to work post migration - rather simple, eh?  Go as big or as minimal as you feel is required.

So by now we're up to anywhere from a couple of servers to a datacenter number of servers in your test lab.  What? Don't have the cash for this?  You told me you had endless cash available for this project!  No worries - two things to help out here.  I usually am able to make the pitch that new domain controllers will be required for the new migration.  Secure them ahead of time in order to use them for your lab.  Still not enough? Have you considered Virtual Server 2005 as a solution?  It can take systems and patch them into multiple virtual networks or physical networks with great speed and ease,  The REAL benefit is the undo disks.  You can "reset" your lab back to the way you started by only choosing "shutdown and discard undo disks" option on the Virtual machines.  WOW - Virtual Server ROCKS! It used to take DAYS to do that before... Did I mention I am a BIG fan for Virtual Server 2005? 

Did you need to get Phase One lab back for some reason?  Just boot up the virtual machine files as required to have a second look.  When you're done, bring back the second phase lab to fine tune the migration again. Once the migration is finished - you have a representative test environment that you can use to test and implement new changes to your infrastructure BEFORE bringing them into production.

I can't stress the importance of getting a good representative lab in place for both "buy in - kicking the tires” of your new design AND for the actual migration step by step procedures.  You will actually find the night of the migration rather uneventful and anti-climactic since you would have rehearsed and planned almost all the steps and come up with contingency plans on what to do if something fails.

In-Place NT4 to AD migration recipe for success!

So what are the nuts and bolts of my modified In-place migration process I have used on almost all of my AD Design and migration projects?  Let's be brief and describe it here (I'll try my best) and expand each of the three sections in three more posts.

Pre-Upgrade Tasks

  • Install x number of new 2003 Member servers as part of your NT4 domain
  • backup all your domain controllers
  • Doublecheck for LMHOST files
  • Ensure WINS and DNS configuration
  • Introduce "offline BDC" into your production environment
  • Introduce "Temporary BDC: into your environment

Upgrade tasks

  • Swap PDC/BDC roles
  • Double check WINS entries for new PDC role
  • OS upgrade "temporary BDC"  to 2003
  • On reboot - complete the DCPROMO wizard to create your new AD forest and domain
  • Logon and review event logs on Temporary BDC
  • DCPROMO 1 or more of the X 2003 member servers to be new DCs
  • complete FSMO role transfer and other things from "temporary BDC" to final FSMO role holders

Post upgrade tasks

  • OS upgrade your NT4 BDC machines and cancel the DCPROMO process on first reboot
  • Shutdown old BDCs and remove their computer accounts from the new domain.
  • switch to native mode AD
  • Build OU structure
  • Implement Administrative delegation strategy
  • Implement group policy strategy
  • Move Users, groups and Computer accounts into proper OU structures

What was my worst case scenario that this method has gone through (and survived)?  Remember - I've never had to roll back yet!  I've implemented this on a client of 3500 users in 10 physical sites with the DCs centralized.  The process started on Saturday morning with the team getting a Processor failure on the PDC, hard drive issues on one of the BDCs, WINS replication errors, a fire alarm that required evacuation and top it all off with a winter blizzard that shut down the airports and prevented a Microsoft Consultant from showing up until the next day after the migration process was completed. It worked through all of those issues AND it was completed WITHOUT informing the user population that it was taking place (team choice that I now personally recommend). On Monday we found out that there were over 300 logons via remote access and "in-building" weekend workers and NOT A SINGLE helpdesk call or service outage issue reported. Not bad eh?

NT4 to AD Upgrade - Pre-Migration Tasks

Install x number of new 2003 Member servers as part of your NT4 domain
These servers will be your final production AD domain Controllers (DCs).  Notice they are introduced early in the game as fresh install and tweaked 2003 member servers working in your NT 4 domain?  They will house your new crisp and clean AD directory and will never have had any legacy NT 4 parts on them what so ever. They will have all new and signed windows 2003 Server certified drivers and all the latest patches installed to the operating system. Because they are member servers - they can be introduced any time in the pre-upgrade process. I've even implemented these with remote desktop enabled at remote sites in multiple provinces in order to minimize travel of having to be right in front of the servers for the actual DCPROMO process.

Backup all your domain controllers
This is obvious - it's one of your restore points that you can use if you so choose to backout. Make sure the backups worked and are verified. I've even gone through the steps of a single drive that was part of a mirror and replacing it with a new drive and allowing it to rebuild as a second layer of "protection"

Doublecheck for LMHOST files
I've gotten burned by this one early on - and it's now in all my pre-checks.  You might have implemented a custom LMHOST file that hard codes IP addresses and roles to specific machines (the 1B and 1C NetBios names that represent domain controllers and PDCs in an NT4 environment.  Just be sure you know where they are and how to remove or edit them according to future steps in this migration process.  This might seem trivial, but it’s tough to troubleshoot name resolution when a Backup Domain Controller can't locate its PDC anymore and therefore won't continue with an OS upgrade.

Ensure WINS and DNS configuration
Also a gotcha here as well.  In future steps, you will be swapping the role of PDB and BDC and you will need to know how to locate the 1B and 1C records to verify they are been registered correctly and they have replicated correctly.  You might not have known that your WINS replication was "hurting" in the first place... Get it identified and fixed now before migration day comes.

Introduce "offline BDC" into your production environment
This is another point of backup or rollback that you can do.  Bring up an NT 4 BDC and make sure it syncs with the PC. Take this guy offline and put a post it note on him identifying him as the back out machine and a big DO NOT TOUCH OR PLUG IN Warning on it.

Introduce "Temporary BDC: into your environment
My process is one where a temporary BDC is introduced and does the actual work of the migration.  Once it's complete, the new clean 2003 REAL production servers (which are currently member servers) are integrated, tested and configured.  Once they are driving the AD, this guy is expendable.  DON'T just turn him off - you have to DCPROMO him back down after the process is complete in a future step.  I’ve used both higher end new desktops and even a virtual machine for this guy in previous projects - use whatever class of system you can afford and are comfortable with.

Now - Pick your date - you are ready to migrate.

NT4 to AD Upgrade - Upgrade Tasks

Before proceeding it is important to note that I assume that you are not running any mission critical applications or file shares off your domain controllers.  As well - if you have infrastructure services running on these systems (like DNS, WINS or DHCP) you have a plan on how to move or update the services once the migration is complete or better yet - before you even begin.  You can always come up with a strategy to keep them on the same systems or move them to the new hardware ahead of the migration day.  I personally recommend that you move the infrastructure services OFF the old PDC/BDCs ahead of the migration and also swap IP addresses with the new machines ahead of time.  That way you DON'T have to revisit all your client systems (to update DNS or WINS server entries)  and you take as much work OFF the migration team and spread it out BEFORE the migration night. The less stuff you have to do on the migration night - the better.

Swap PDC/BDC roles
This part is quite simple. You go into server manager and select the "temporary BDC" and choose promote to primary. This will result in a brief period of server interruption where people will not be able to change their passwords.  It will not prevent them from login on and your users shouldn't notice that anything is going on at all.  Once this is complete - ensure the WINS server entry is updated (you can do a net stop netlogon and net start netlogon to help it along or even a nice reboot). Once this is done - this can be used as a backout / rollback process. How? Say the upgrade fails for some reason - you still have your old PDC and BDCs running NT4 and doing nothing in regards to the upgrade as of yet.  All you do is turn off the temporary PDC/BDC and promote the old guy back again - you are back to normal.

Double check WINS entries for new PDC role
Did I mention this was important? Without a proper 1B and 1C records registered for the domain, the backup domain controllers won't upgrade and your migration will stall. Check your wins database with the WINS manager. Also - in my previous post I warned about LMHOST files - are they present for some reason? If they are - update them to reflect the new PDC.

OS upgrade "temporary BDC" to 2003
Pop in the CD and off you go. Upgrade the OS (takes about 36 minutes or so based on your hardware of the Temporary BDC) and on the first reboot it will auto logon and start the DCPROMO process. You might get compatibility errors on drivers or whatever and might have to locate the appropriate drivers before continuing - no problems, you would have discovered these in your testing lab, right?

On reboot - complete the DCPROMO wizard to create your new AD forest and domain
You can either be a new domain in a new forest or you can join an existing forest as a new domain.  You can not become an additional Domain controller for an existing domain for obvious reasons.  Choose the default location for most of the things like database, logs and sysvol (all NTFS partitions) because this is only a temporary 2003 Domain Controller. The cool thing (ok, I think it is cool) is that you will see the number of object that have been created and migrated from your old environment as the migration wizard runs. It's just cool.  At the end of the upgrade process, it will reboot once more and come up for the first time as an AD Domain Controller (DC).  Congratulations, you have your first AD upgrade in  place (excuse the pun). From the time the NT4 OS requires the first reboot to continue until the time that the server comes up for the first time, your users will not be able to change their password.

Logon and review event logs on Temporary BDC
Now is when you go in and check out the event logs and ensure everything came up ok and things are humming along.  You will get warnings that the  server can't connect to various FSMO roles or issue RID pools - this is normal for the first time up and after a couple of minutes, you should see the next event log entry that states all is well.  I generally wait about 10 - 15 minutes or so after the first reboot to let things simmer nicely.

DCPROMO 1 or more of the X 2003 member servers to be new DCs
As soon as possible - get the member servers DC Promo'ed by either physically standing in front of them and running DCPROMO or Remote Desktop Connection to them and do the deed. This will prevent all your Windows 2000, Windows 2003 and Windows XP based machines from "piling on" to your single 2003 Domain Controller.

Complete FSMO role transfer and other things from "temporary BDC" to final FSMO role holders
So you have this temporary BDC now as your King Domain Controller (that's a Rick term) and you need to spread the FSMO roles from him to the real production domain controllers in order to safely remove him from the role of DC and remove him from the domain - he is "temporary" after all and has been upgraded from NT4 to 2003. The FSMO transfer process is accomplished by using the NTDS utility with the appropriate Enterprise administrator rights. This is a well documented procedure that can be found in the Microsoft Knowledge base (https://support.microsoft.com/kb/255504).  Your AD design document would outline where the FSMOs should be sitting and it's a simple process to do the actual move. Rather un-exciting, but required.

NT4 to AD Upgrade - Post Upgrade Tasks

What have we got now?  An NT4 domain that has been in place upgraded with a temporary BDC.  We have all new clean 2003 server installs now promoted to become DCs as well and we have distributed all the FSMO roles to the appropriate servers. We are running in mixed mode and things are just perfect.  There has been minimal interruption to your end users (they couldn’t' change their password for maybe an hour at most and no new accounts or groups could be made) but that was only temporary.  They could still log on and work as normal without any problems.   Let’s finish this up.

OS upgrade your NT4 BDC machines and cancel the DCPROMO process on first reboot
I want to get out of mixed mode as soon as possible.  Why?  Because without Native Mode, I can't do nested Global groups and I still have to worry about replication with old BDCs. Besides - those old BDCs would look great in my test lab or as a new foot rest under my desk. Go ahead with an OS upgrade on these NT4 BDCs - again about 36 minutes or so for each box - but they can be done in parallel to speed things up. Once the first reboot occurs, they will auto logon and try to run DCPROMO.  The first question it asks is if you want this machine to continue to be a domain controller - select NO and complete the wizard.  It goes ahead and deletes the old role and reboots the system.

Shutdown old BDCs and remove their computer accounts from the new domain.
Once they come up from their last reboot - they are now Member Servers and can be safely shut down and deleted from the AD.  If they are to be re-used, I suggest that you reformat and re-install Windows Server 2003 to get a clean install on the hardware and then add them back into the AD domain with new computer names.

Switch to native mode AD
The moment you have been waiting for - flip the switch (or rather change the drop down box) to Windows 2003 Native mode.  This can be done form Active Directory Users and Computers or from the Active Directory Domains and Trusts MMC console.  You won't have an impact on your users and your BDCs are no longer in existence.

Build OU structure
So you have your AD, you have all new Domain Controllers and you are now running in Native Mode.  Now is the time to implement the power of AD by creating your OU structure. This could be created by hand or an LDIF file can be implemented from a previous lab exercise - your choice.  It shouldn't be complex if you followed my KISS and NIRVANA principles for AD design.

Implement Administrative delegation strategy
Once you are in native mode, you can start to implement nested groups and cleanup that bloated DomainAdmins group that was migrated. You can create new global Groups and delegate out the rights as required for various users and then remove them from Domain Admins.  It's very satisfying to see this take place, but once again - you have designed this strategy in your AD design and tested it in the first phase of your lab environment.  Now is the time to clean up and regain control of your NT 4 administration nightmare.

Implement group policy strategy
Your group Policy testing took place in the first stage of the lab.  I hope that you were using the GPMC and were able to export your policies.  You can now import them back in to your production environment and have them applied to the OU structures you have just implemented.

Move Users, groups and Computer accounts into proper OU structures
Now that your delegation is in place and your GPOs are in place, it's time to move the users, Computers and Groups to your new structure and call it a day. Your existing systems won't notice they are part of an AD domain until they have to renew their computer security connection.  A reboot or patience will fix this. Soon enough you will have your GPOs humming and applied to your systems.

Welcome to the wonderful world of AD and all it entails. Any final comments before my final post on this In-Place migration series? Click on the Feedback/Comments link at the end of this post to make sure you are heard.

AD to NT4 In-Place Upgrade - Wrap up

This article was a lot of fun to revisit.

I wanted to share some final thoughts and wrap up stuff. I haven't mentioned testing at all yet - mainly because it is a highly personal thing that has to be determined by your migration team - more on this in a bit. Another thing I have learned throughout the refinement of this migration process is that your chances of success are directly related to the level of management AND technical staff “buy in" as well as a clearly defined communication plan that has been tailored to the appropriate audience.

Testing procedures need to be developed, documented and checked.  People have to "own" the tests and determine exactly what needs to be done for each test.  These test plans then need to be tried out BEFORE the migration to ensure they work in the first place (you'd be surprised if I told you how many WEREN’T tested before hand and didn't work after). These tests will be performed after the migration and post migration configurations are complete.  If they pass - you are a go.  If not - you have to continue to troubleshoot to get them working.

Management and Technical Staff "buy-in" is critical in being able to work though or around any political roadblocks that come up along the way.  You will need to be able to speak to the appropriate levels in the language they understand and satisfy their needs for this migration.  Sometimes needs are actually benefits and goals directly related to what is important to them and what drives the business.  You aren't doing this upgrade for the sake of technology - you are doing it to make your business more agile, reduce the amount of time and effort it takes to manage it which in turn will make your job easier by giving you more time to look after OTHER things that need your attention.  Without buy-in - you aren't going to get far and you will be shackled to political decisions that just don't make sense.

The communications plan is also critical.  Keeping the appropriate people in the loop along the process of the migration is very important.

Thanks for taking time to revisit this series of posts with me again. As always – please feel free to comment at the bottom of the post in order to improve this article for the online community and make it better for all of us.