The importance of following ALL the authoritative restore steps
Hello, David Everett here again. Recently a customer contacted Microsoft Product Support to determine why the Connect to Domain Controller option in Active Directory Users and Computers (aka: ADUC or dsa.msc) was generating an incomplete list of Domain Controllers (DCs) for one domain. Even though the list of available DCs was truncated we found we could manually enter the name of any DC not in the list and Active Directory Users and Computers would connect to the DC without issue.
Determining the scope of the issue:
Wanting to see if the truncated list of DCs was specific to Active Directory Users and Computers or if other tools also failed to locate all the DCs we ran nltest.exe /dclist:contoso.com. The output shown below revealed a complete list of domain controllers for contoso.com but many were missing their [DS] Site: information. We found that those DCs missing their [DS] Site: information happened to be the same DCs missing when Connect to Domain Controller was selected. One final observation was that the list of available DCs varied from one DC to the next when selecting Connect to Domain Controller.
Get list of DCs in domain 'contoso.com' from '\\dc01.contoso.com '.
MAYDC01.contoso.com [DS] Site: Mayberry
MAYDC02.contoso.com [DS] Site: Mayberry
DALDC01.contoso.com [DS] Site: Dallas
DALDC02.contoso.com
LADC01.contoso.com [DS] Site: LosAngeles
LADC02.contoso.com
SEADC01.contoso.com
SEADC02.contoso.com
The two DCs in the Los Angeles site saw themselves in the list of available DCs but not the other DC in the same site. Suspecting Active Directory (AD) replication might be at fault we ran Repadmin /showrepl * /csv > Showrepl.csv and found AD replication was free of errors forest wide.
Checking for Database Inconsistencies:
Since AD Replication was not at fault our focus switched to AD database inconsistencies. We focused on three primary objects which house all of the metadata needed for DC discovery:
- The Distinguished Name (DN) of the DC’s object in the domain partition
CN=LADC01,OU=Domain Controllers,DC=contoso,DC=com
- The DN of the DC’s NTDS Settings object and
CN=NTDS Settings,CN=LADC01,CN=Servers,CN=LosAngeles,CN=Sites,CN=Configuration,DC=contoso,DC=com
- The DN of the DC’s Server object which resides just above the DC’s NTDS Settings object in the Configuration partition
CN=LADC01,CN=Servers,CN=LosAngeles,CN=Sites,CN=Configuration,DC=contoso,DC=com
Using LDP.EXE we connected to both DCs in the Los Angeles site and gathered dumps of all three objects for both DCs and compared the output. For those who tend to avoid this tool, see MSKB 252335 on how to Bind and Connect but make certain to select CN=Configuration,DC=forestrootdomain from the Base DN: drop-down. Expand the configuration partition on the left until you locate the server object of the DC that was restored.
This LDP dump of LADC01’s Server object in the Configuration partition was taken while bound to LADC01 (notice the DC name in blue title bar indicating which DC we’re bound to). Looking at the third attribute from the bottom we find the serverReference forward link attribute in the list of attributes. This attribute contains the DN path of the corresponding DC object in the DC=contoso,DC=com partition. Below is an LDP dump of LADC01’s Server object while bound to LADC02. Notice the serverReference forward link attribute is missing which indicates it is not populated on this DC’s copy of the AD Database.
When we examined the LDP dumps of LADC02’s Server object we found the same was true. LADC02 had a DN for its own DC object but the LDP dump of LADC02’s Server object taken while bound to LADC01 had an empty serverReference attribute. Finally, those DCs which always appear in the list of domain controllers had a populated serverReference attribute on all DCs.
To determine how widespread this issue was we queried the serverReference attribute for both Los Angeles DCs from every DC in the forest using the repadmin /showattr command below. DCs that returned a serverReference attribute had the DC object DN and those DCs that had no serverReference attribute were empty:
Repadmin.exe /showattr * CN=LADC01,CN=Servers,CN=LosAngeles,CN=Sites,CN=Configuration,DC=contoso,DC=com /atts:serverReference
Fixing the problem:
We connected to the configuration partition on LADC01 using adsiedit.msc and manually added the “CN=LADC02,OU=Domain Controllers,DC=contoso,DC=com” DN to the serverReference attribute on LADC02. This change made LADC02 appear in the list of available DCs when the Connect to Domain Controller option was selected in Active Directory Users and Computers. Also, nltest.exe /dclist:contoso.com now showed [DS] Site: LosAngeles next to LADC02.contoso.com on all DCs. Not shown here, but once the DN of the DC’s object in the contoso.com domain was added to the serverReference attribute, the serverReferenceBL back-link attribute was automatically populated on the DC object in the contoso.com domain.
Determining how this occurred:
Now that we understood why the DC list was incomplete we started looking for how this occurred. To do this we gathered replication metadata from these three objects for both LADC01 and LADC02. The command used to gather the metadata from LADC01 for LADC02’s server object in the configuration partition is:
repadmin.exe /showobjmeta LADC01CN=LADC02,CN=Servers,CN=LosAngeles,CN=Sites,CN=Configuration,DC=contoso,DC=com
Comparing the objects dumped from LADC01 and LADC02 we found the Ver (version) numbers matched. It wasn’t until we looked at metadata of the DC object in the domain partition and compared it with the corresponding Server object in the configuration partition that we understood what occurred.
Here is a showobjmeta dump of CN=LADC02,CN=Servers,CN=LosAngeles,CN=Sites,CN=Configuration,DC=contoso,DC=com from LADC01:
repadmin.exe /showobjmeta ladc01 CN=LADC02,CN=Servers,CN=LosAngeles,CN=Sites,CN=Configuration,DC=contoso,DC=com
11 entries.
Loc.USN Originating DC Org.USN Org.Time/Date Ver Attribute
======= ============ === ====== ============= === =========
8363 b5c14a75-7f99-4f31-b84b-d755190a2c0d 213256008 2007-04-15 15:04:39 1 objectClass
85895 LosAngeles\LADC02 85895 2007-04-15 17:11:27 2 cn
8363 b5c14a75-7f99-4f31-b84b-d755190a2c0d 213256008 2007-04-15 15:04:39 1 instanceType
8363 b5c14a75-7f99-4f31-b84b-d755190a2c0d 213256008 2007-04-15 15:04:39 1 whenCreated
<snip>
Here is a truncated showobjmeta dump of CN=LADC02,OU=Domain Controllers,DC=contoso,DC=com from LADC01:
repadmin.exe /showobjmeta ladc01 CN=LADC02,OU=Domain Controllers,DC=contoso,DC=com
41 entries.
Loc.USN Originating DC Org.USN Org.Time/Date Ver Attribute
======= ============ === ======== ============= === =========
92248340 fb36d148-19fd-43f0-8876-91a027863f79 155898 2009-11-18 12:56:34 100001 objectClass
92248339 77dba4f6-3870-4eb5-b46a-4f1fb1ee0be6 92248339 2009-11-18 12:59:51 4 cn
92248340 fb36d148-19fd-43f0-8876-91a027863f79 155898 2009-11-18 12:56:34 100001 description
92248340 fb36d148-19fd-43f0-8876-91a027863f79 155898 2009-11-18 12:56:34 100001 instanceType
9027 4855f23c-744c-488d-852c-9c170dd3359c 108176481 2007-04-15 18:10:11 1 whenCreated
<snip>
Interpretation of the data:
The Version number of the attributes on LADC01’s DC object in the domain partition have a USN that is 100,000 higher than the DC’s corresponding Server object in the configuration partition. This strongly suggests the DC object in the Domain Controllers OU was authoritatively restored with the default version increase of 100,000 while the DC’s corresponding Server object in the configuration partition was not authoritatively restored. The customer then remembered accidentally deleting several of the DCs a while back and performing an authoritative restore on the entire Domain Controllers OU.
Understanding the inconsistencies:
Now that we knew an authoritative restore of the domain controllers OU was performed we needed to determine why the serverReference and serverReferenceBL attributes for restored DCs were missing and different across all DCs.
Anyone who has performed authoritative restores of users and groups will recall an issue where group membership is not correct on replica DCs after users and groups are authoritatively restored; this is discussed at length in KB280079. In the case of restored users and groups, when a user is deleted their membership from the remaining group is removed. If the user is then restored, but the group is not, the membership will not be restored on any DC except the DC where the restore took place. For those wondering what this has to do with DCs being restored, it is identical. DCs are security principals just like users, and the DC’s server object in the configuration partition behaves much like a group. If the DC object is deleted from the domain partition the serverReference attribute containing the forward link will be NULL’ed out on the server object in the configuration partition. If just the DC object in the domain partition is restored the serverReference attribute on the corresponding server object in the configuration partition will not be updated on replica DCs once the restored DC object inbound replicates to them.
Avoiding this issue:
Since the release of Windows Server 2003 Service Pack 1 ntdsutil.exe has automatically created LDF files for all partitions in the forest where restored objects have back-links. This is discussed further in MSKB 840001. In the case of user accounts you ensure all users have the correct group membership on all DCs by allowing the restored user accounts to replicate to all DCs/GCs. Once all DCs have the restored account you use ldifde -i -f <AR*.ldf> and import the user’s group membership against to the recovery DC. Doing this ensures the user’s DN is added to the member attribute on the group and the version of the member attribute is bumped higher causing it to replicate to all DCs. Since all DCs have a copy of the restored user account in their local database the DN on the member attribute is retained. As a rule of thumb, if you are authoritatively restoring users, computers or groups you should always import the LDF files created by ntdsutil.exe and avoid issues like this.
Or even better, deploy Windows Server 2008 R2 and enable the AD Recycle Bin – it automatically handles back links and forward links.
- Dave “metadata” Everett