Best practices around Active Directory Authoritative Restores in Windows Server 2003 and 2008
It’s your guest writer Herbert Mauerer again.
A very common AD disaster is an unexpected deletion or modification of objects. Unlike a bad football match or family meeting, you can prepare for that and make the crisis more bearable. In this blog, I will discuss best practices of Windows Server 2003 and 2008 forest level backup and restore. I will not discuss Windows 2000 as it’s at the end of its lifetime, and also not Windows 2008 R2 because we have a pretty good solution for object deletion without the need for backups: AD Recycle Bin.
AD Object deletions and unwanted modifications are often due to human error. Sometimes a bad script does that or a provisioning system, but then these are also created by humans. The common factor is the data loss. Now there is quite some complexity in the current KB article on AD object restores:
840001 How to restore deleted user accounts and their group memberships in Active Directory
https://support.microsoft.com/default.aspx?scid=kb;EN-US;840001
We basically follow method 1 in the article 840001, with a few tweaks and preparations:
- Windows Server 2003 Service Pack 2 or newer and Windows Server 2003 Forest Functional level. This allows the restoration of links using LDIF files written by NTDSUTIL.EXE.
- Preparing for the restore by converting the LEGACY group memberships to Link-Value Replication (LVR) group memberships.
Step 2 requires that you remove and re-add all group members. This is relatively easy using the Windows Server 2003 command line tools for querying and changing AD objects:
Dsget group CN=GroupX,OU=Groups,DC=contoso,DC=com /members | dsmod group CN=GroupX,OU=Groups,DC=contoso,DC=com /chmbr
This command gets the members of the group and replaces the members with the output, thus adding them as LVR members. This may mean some replication traffic if you apply this to many groups within a short time. However, since the removal and re-adding of members happened in a very short time, you should not see the member go away, as all changes should be replicated in the same replication job.
There are two problems with this approach:
- The version of DSGET.EXE in Windows Server 2003 gets a maximum of 1500 members, and the command above would discard the rest. In one of my test environments, DSGET does not show any members. I have not seen both problems with the Windows Server 2008 or later version of DSGET.
- If a group has lots of members, the execution of DSMOD.EXE can overflow the AD database version store, and adding the members would fail. So although the change of the membership type is a one-time action, it’s certainly a good idea to first export the member list to files first. I describe that below.
- Use updated operating system file versions, details also below.
How to Convert Groups for LVR
You can determine the state of linked attribute values like group memberships from the replication meta-data on the object. When you run "repadmin.exe /showobjmeta", the link "Type" says LEGACY:
Type Attribute Last Mod Time Originating DC Loc.USN Org.USN Ver Distinguished Name
======= ======== ============= =============== ========= ========= === ==========================================
LEGACY member CN=test-user1,OU=Test-OU,DC=contoso,DC=com
PRESENT member 2008-09-16 18:22:29 HQcontoso-DC1 175379684 175379684 1 CN=test-user2,OU=Test-OU,DC=contoso,DC=com
The line for LEGACY does not have data on the latest change all these values share that with the attribute meta-data listed in the first part of the output.
Hint: I use the "for" command in CMD for loops for direct execution on the command line. When you use CMD scripts, you need to change loop variables like "%f" to "%%f".
The steps are:
1. Freeze the group memberships, stop all group member management.
2. Export the groups in your domain:
Dsquery group dc=contoso,dc=com /limit 0 > grouplist.txt
Review the list and remove all built-in groups. You cannot remove all members there, and these groups usually don’t have many members to begin with. Also, there’s hope nobody deletes the memberships for these.
3. Get all group members on a Windows Vista member with RSAT or Windows Server 2008 computer:
For "delims=" %f in (grouplist.txt) do DSGET %f /members > groupmembers%f
Important: The DNs of the groups must not contain characters ":/*?". If you need to rename a group or an OU, it’s sufficient to change the DN.
4. Now we count the members:
In object counting, I use an older version of the MKS toolkit WC.EXE which counts lines, words and characters. You need a similar tool that counts lines in text files. You may have a favorite tool for that, or use a text editor that provides a line count, and use that to get the line count for the bigger output files.
For "delims=" %f in (grouplist.txt) do wc groupmembers %f >> membercount.txt
The file membercount.txt has the count of members in the first column. Groups with more than 5000 members require special treatment. I suggest moving their member list into a folder "big-groups1".
5. For all other groups, execute:
Dir /b groupmembers >grouplist-small.txtFor /f "delims=" %f in (grouplist-small.txt) do dsmod group "%f" /chmbr < " groupmembers%f"
Hint: "%f" is in double quotes here as "dir /b" does not print quotes, and we want to handle names with spaces properly.
So after this, the small groups are off our radar.
6. Now for the big groups:
My suggestion is to split the membership lists into multiple text files of approximately 5000 lines and put these sections into separate folders with the same file names (the DN of the group):
Dir /b big-groups1> big-groups1.txtFor /f "delims=" %f in (big-groups1.txt) do dsmod group "%f" /chmbr < " big-groups1%f"
Dir /b big-groups2> big-groups2.txtFor /f "delims=" %f in (big-groups2.txt) do dsmod group "%f" /chmbr < " big-groups2%f"
7. Verify the change by getting the group meta-data:
For "delims=" %f in (grouplist.txt) do repadmin /showobjmeta . %f > group-meta%f
The output files in “group-meta” should have "LEGACY" only for other attributes.
Well, this was quite some work. The good news is that we don’t have to worry about the primary group… :-)
I suggest you test-drive this with test groups, for these tests it does not matter that the members are in LVR mode already. Then expand the work to smaller OU trees until you tackle the rest of the domain. One idea would be to split out parts of the group-list.txt so you impact a few groups only.
Suggested Fixes
After Windows Server 2003 Service Pack 2 was released, we had two problems affecting replication after a (authoritative) restore:
937855 After you restore deleted objects by performing an authoritative restoration on a Windows Server 2003-based domain controller, the linked attributes of some objects are not replicated to the other domain controllers
https://support.microsoft.com/default.aspx?scid=kb;EN-US;937855943576 Active Directory objects may not be replicated from the restored server after an authoritative restore on a Windows Server 2003-based domain controller
https://support.microsoft.com/default.aspx?scid=kb;EN-US;943576
When you install Fix 943576 on the DCs you backup on a regular basis, you should not see any issues. In the long run, you should get this fix on all DCs in the enterprise. This problem is fixed in Windows 2008.
On top of that, there are also problem with the tool we use to restore the objects, NTDSUTIL.
Problems with object names containing extended characters:
886689 The Ntdsutil authoritative restore operation is not successful if the distinguished name path contains extended characters in Windows Server 2003 and in Windows 2000
https://support.microsoft.com/default.aspx?scid=kb;EN-US;886689910823 Error message when you try to import .ldf files on a computer that is running Windows Server 2003 with Service Pack 1: "Add error on line LineNumber: No such object"
https://support.microsoft.com/default.aspx?scid=kb;EN-US;910823Problems with excessive links in the LDIF files:
951320 The ntdsutil.exe utility in Windows Server 2003 writes out too many links to .ldf files during an authoritative restore process
https://support.microsoft.com/default.aspx?scid=kb;EN-US;951320
The good news is that having fix 951320 on the DCs you backup often also takes care of problem 910823. Fix 951320 is included in Windows Server 2008 Service Pack 2.
There is a new problem with authoritative restores where objects are not treated correctly in replication. We have a corrective fix in NTDSUTIL for that:
974803 The domain controller runs slower or stops responding when the garbage collection process runs
https://support.microsoft.com/default.aspx?scid=kb;EN-US;974803
This last problem is not fixed in Windows Server 2008 yet. The problem cannot happen on Windows Server 2008 R2.
How to Avoid Accidental Deletion
Most of the AD objects recovery cases we see in our support work are because of accidental deletions. Typically, there are whole OUs with lots of user, groups and computer accounts affected. These objects have attributes that can’t be recovered by re-creation as you can’t get the same Sid again.
For this problem, Microsoft has implemented an option in the Windows Server 2008 Admin Tools, also available for Windows Vista using RSAT (Remote Server Administration Toolkit). This version has a check box "Protect object from accidental deletion” when “advanced view” is enabled. This flag is set automatically on all OUs you create with this admin tool.
When this is used, the OU itself and its parent get ACEs (Access Control Entries) that deny "Everyone" the permission to delete the object itself and child objects. This works with Domains hosted on Windows Server 2003. You can add the ACEs this way using your own tools and scripts and you will get the same effect.
Thus such a mass deletion should not happen by accident anymore, as you need to step through clearing the check box. Yes, the deletion might still happen from a script, but then it’s written to remove these ACEs and brings the problem to the next level, the script programmer. At this point, you can’t call it accidental anymore.
A Solution
For the object deletion scenario, Windows Server 2008 R2 offers the AD Recycle Bin. The forest needs to run in the Windows Server 2008 R2 forest mode and the feature needs to be enabled separately.
All objects that are deleted will have no attributes removed anymore, and linked attributes will have a third state to signal they are inactive pointing to a deleted object. When the object is reactivated, the link state will be switched to active again.
The graphical user interface for this facility is not in this release, but the groundwork is done, PowerShell is available, and there are scripts to help you perform the recovery.
Until you can enable this feature, the task is to be prepared for object deletions. This article will still be useful for unwanted object attribute modifications.
Herbert “by way of Germany” Mauerer