Hello Everyone, my name is Zoheb Shaikh and I’m a Premier Field Engineer out of Malaysia. Today for my first post on AskPFEPlat, I wanted to share something interesting with you that I came across recently caused by a KRBTGT_RODC account deletion.
Before I talk more about the issue, I would like to share a bit of background about KRBTGT account and its use briefly. I could try to explain what the krbtgt account is, but here is a short article on the KDC and the krbtgt to take a look at:
“All instances of the KDC within a domain use the domain account for the security principal “krbtgt”. Clients address messages to a domain’s KDC by including both the service’s principal name, “krbtgt”, and the name of the domain. Both items of information are also used in tickets to identify the issuing authority. For information about name forms and addressing conventions, see RFC 4120.”
Likewise, a snip for the RODC krbtgt_##### account:
“The RODC is advertised as the Key Distribution Center (KDC) for the branch office. The RODC uses a different krbtgt account and password than the KDC on a writable domain controller uses when it signs or encrypts ticket-granting ticket (TGT) requests. This provides cryptographic isolation between KDCs in different branches, which prevents a compromised RODC from issuing service tickets to resources in other branches or a hub site.”
The krbtgt_##### account is unique to each RODC and minimizes impact if the RODC is compromised. The RODC does not have the krbtgt secret. It only has its own krbtgt_##### secret (and other accounts you have allowed). Thus, when removing a compromised RODC, the domain krbtgt account is not lost.
Getting back to the scenario, the customer had multiple DC’s running 2012 R2 and 3 Read Only Domain Controllers (RODC). We observed that the writable DC’s were flooded with the Event IDs 1168 stating “Internal error: An Active Directory Domain Services error has occurred”. They were not experiencing any functional loss because of this, but were worried about the h`ealth of the Domain Controllers.
Log Name: Directory Service
Date: 6/2/2017 3:18:01 AM
Event ID: 1168
Task Category: Internal Processing
Internal error: An Active Directory Domain Services error has occurred.
Error value (decimal):
Error value (hex):
So we asked, what changes have been made recently?
In this case, the customer was unsure about what exactly happened, and these events seem to have started out of nowhere. They reported no major changes done for AD in the past 2 months and suspected that this might be an underlying problem for a long time.
So, we investigated the events and when we looked at it granularly we found that the event 1168 was coming from a RODC:
Then we checked one of the RODC’s and could not see any reference to these. So, we turned up the Active Directory Diagnostics to 5 and saw an event Id Event 1084. (Refer blog for enabling Active Directory Diagnostic logging https://technet.microsoft.com/en-us/library/cc961809.aspx)
Event ID: 1084
Internal event: Active Directory Domain Services could not update the following object with changes received from the following source directory service. This is because an error occurred during the application of the changes to Active Directory Domain Services on the directory service.
Source directory service:
From this error, it was clear that this was caused by krbtgt_RODC account deletion and the customer said that they may have run a script to delete Disabled accounts.
So, we proposed below options to resolve this issue
- Restore the KRBTGT_RODC account from Active Directory Recycle Bin if it was enabled.
- Restore the KRBTGT_RODC account from a System State backup
- Demote and repromoted RODC as KRBTGT_RODC account is unique for each RODC
To reproduce this error in lab we followed the below steps: –
- Promoted a RODC in the environment
- Changed the attribute ms-ds-krbTGT-Link Removed the user (cn=krbtgt_37540,dc=contoso,dc=com) “key distribution center for service account > Put the value to <not Set>
- Once done checked events and it gave us the same event (1168)
- Added the ms-ds-krbTGT-Link value back to KRBTGT_RODC account and the event stopped coming.
If you have a RODC in your environment, do keep this in mind. Thanks for reading, and hope this helps!