IMPORTANT ANNOUNCEMENT FOR OUR READERS!
AskPFEPlat is in the process of a transformation to the new Core Infrastructure and Security TechCommunity, and will be moving by the end of March 2019 to our new home at https://aka.ms/CISTechComm (hosted at https://techcommunity.microsoft.com). Please bear with us while we are still under construction!
We will continue bringing you the same great content, from the same great contributors, on our new platform. Until then, you can access our new content on either https://aka.ms/askpfeplat as you do today, or at our new site https://aka.ms/CISTechComm. Please feel free to update your bookmarks accordingly!
Why are we doing this? Simple really; we are looking to expand our team internally in order to provide you even more great content, as well as take on a more proactive role in the future with our readers (more to come on that later)! Since our team encompasses many more roles than Premier Field Engineers these days, we felt it was also time we reflected that initial expansion.
If you have never visited the TechCommunity site, it can be found at https://techcommunity.microsoft.com. On the TechCommunity site, you will find numerous technical communities across many topics, which include discussion areas, along with blog content.
NOTE: In addition to the AskPFEPlat-to-Core Infrastructure and Security transformation, Premier Field Engineers from all technology areas will be working together to expand the TechCommunity site even further, joining together in the technology agnostic Premier Field Engineering TechCommunity (along with Core Infrastructure and Security), which can be found at https://aka.ms/PFETechComm!
As always, thank you for continuing to read the Core Infrastructure and Security (AskPFEPlat) blog, and we look forward to providing you more great content well into the future!
Disclaimer: For brevity and to get some key points across, quite a bit of detail about Kerberos has been purposely ommitted from this blog entry. I’m certain that those that are experiencing any of the problems below won’t mind 🙂
As a Premier Field Engineer, I visit new customers every week and every customer, and I mean every customer, has the KDC 11 events in their system event logs. Consequently, I have to explain to customers what this means and how to clean it up. But rather than just saying, “Look, these accounts have a duplicate SPN’s and use setspn or adsiedit to clean them up”, I like giving the back story about how duplicate SPN’s break authentication and what would happen if the KDC issued Kerberos tickets for resources with duplicate SPN’s. So, here’s the dialogue of my weekly explanation of Kerberos and duplicate SPN’s. By the end, most customers have that light bulb moment and “Get It”. At the end of this, you should be able to do the following:
1.) Fully understand what duplicate SPN’s are and the Kerberos Event ID 11.
2.) Understand why duplicate SPN’s break Kerberos authentication.
3.) How to find all duplicates across the entire forest with one simple command.
4.) Be able to determine which account the SPN should be set on.
5.) Understand the Kerberos Event ID 4: “KRB_AP_ERR_MODIFIED”.
What is this thing called a SPN?
It stands for service principal name and in most cases, it is the name of a resource that a client or application is connecting to. Every computer account has an attribute named ServicePrincipalName, which is usually populated. Users also have this attribute but it is normally blank unless some application is running under this user account.
Fileserver.contoso.com’s ServicePrincipalName attribute:
Let’s apply this to the real world and do some role playing. Imagine you are a phone operator and are responsible for issuing Kerberos tickets so I can talk to various people. But instead of Active Directory, let’s replace the directory with a phone book but it’s a super-secret phone book where the only two parties that know the phone numbers are the phone operator and the respective owner of each phone number. I realize no one uses phone books anymore but play along with me. 🙂
Ok, so there is this person I need to talk to named ‘John Smith’ and before I talk to John Smith, I need to acquire a Kerberos ticket to talk to him. I call you, the phone operator, and say that I need a Kerberos ticket for John Smith. In this instance, ‘John Smith’ would be the SPN. What’s the first thing you do with that SPN? Well, you’d look in your phone book for the name John Smith, right? Now, let’s say that when you find him, you take his super-secret phone number and encrypt a Kerberos ticket with it. Now when I talk to John, I provide him with this Kerberos ticket. Since he knows his own phone number, he can decrypt the Kerberos ticket, which proves it was issued by the phone operator, a trusted authority, and then we continue our conversation. But what would you do if you found multiple people in the phone book named ‘John Smith’. You’d probably ask for more information like address to further narrow it down, right?
Well, Kerberos and the domain controllers aren’t this smart. They take the name given and look for all accounts with an identical SPN attribute based on string match. If it finds two accounts with the same SPN attribute, it immediately throws a KDC 11 event, saying a duplicate SPN was found and replies to the client with KDC_ERR_C_PRINCIPAL_UNKNOWN. The client says “Cool, I’ll just go ahead and use NTLM instead”. But what would happen if you just randomly picked one of the John Smith’s, and encrypted a Kerberos ticket with his corresponding phone number?
Well, if you picked another John Smith than the one I needed to talk to, when I provided the Kerberos ticket over to my John Smith, he wouldn’t be able to decrypt it and Kerberos would fail. When this happens, my John Smith would throw a ‘KRB_AP_ERR_MODIFIED’ error because he can’t properly decrypt the Kerberos ticket. If you get the role play we just went through, you’re well on your way to understanding Kerberos and duplicate SPN’s. Congratulations! 🙂
Key Takeaway: The SPN attribute is the sole means by which the DC determines which account password to encrypt the Kerberos ticket with. Consequently, SPN attributes across all accounts must absolutely be 100% unique across the entire AD Forest, without fail. We’ll be discussing in Part 2, why this is, and how you can have duplicates across two different domains.
Now, If we apply this example to Active Directory and how we authenticate to resources, it would be almost identical:
Client connecting to file server
In this example, the user just connects to a share on the fileserver by going to \\fileserver.contoso.com. Upon doing so, the client will then go to the domain controller and request a Kerberos ticket for Host/fileserver.contoso.com. The domain controller uses this SPN provided by the client to determine which account password to use to encrypt the Kerberos ticket with. Domain controllers don’t issue SQL queries but for all you SQL geeks out there, the query to find the account would be similar to the following:
Select * from Active Directory where ServicePrincipalName = Host/fileserver.contoso.com
So in this scenario, can the user decrypt the Kerberos ticket intended for the fileserver?
A: Nope, because it’s encrypted with fileserver’s password.
Can the file server decrypt the Kerberos ticket?
A: Yes, because the ticket is encrypted with fileserver’s password.
Key Takeaway: This above scenario worked because the Host/Fileserver.contoso.com SPN was only set on the same AD account that we submitted the Kerberos ticket to. Hence, the fileserver could decrypt the ticket and then determine whether you have access to the file share. The above scenario is how it should work.
Now when it comes to Kerberos and SPN’s, always ask yourself the following questions:
1.) Who has to decrypt this Kerberos ticket?
2.) So that it can properly decrypt the ticket, what account must the SPN only be set on to make this work?
From now on, use these questions because the scenarios will start to get a little tricky. The following are the most common scenarios that cause duplicate SPN’s. By going through it this way, you should be able to directly apply this knowledge to most scenarios you encounter.
Scenario #1: Improper process for removing old accounts
Your file server crashed one day and had to be rebuilt. You renamed the old computer account in AD to fileserver_old, rebuilt the server, and then joined it back to the domain with the same name it used to have – fileserver.contoso.com. While their computer names may be different, their SPN attributes are still the same.
When the domain controller searches AD for the HOST/fileserver.contoso.com SPN provided by the client, it finds two computer accounts. At this point, the DC throws a KDC 11 event to the system log and replies to the client with KDC_ERR_C_PRINCIPAL_UNKNOWN. The client then attempts to use NTLM to the fileserver, which will probably work and they will gain access.
1.) Who has to decrypt this Kerberos ticket?
A: When connecting to this File server, the file server has to decrypt the Kerberos ticket.
2.) So that FileServer can properly decrypt the ticket, what account must the Host/Fileserver.contoso.com SPN attribute only be set on to make this work?
A: This SPN must only be set on File Server’s computer account in AD.
3.) Similar to the ‘John Smith’ scenario above, what would happen if the domain controller just randomly picked one of the two computer accounts to encrypt the Kerberos ticket with?
A: The user would submit this Kerberos ticket over to the fileserver and the fileserver may not be able to decrypt the ticket because it could be encrypted with fileserver_old’s password. Remember, the domain controller would never do this because instead it logs a KDC 11 event and replies to the client with KDC_ERR_C_PRINCIPAL_UNKNOWN at which time the client will attempt to use NTLM over the file server. But thinking about this way really highlights the importance of why SPN’s must only be unqiue to each account.
Iron out your process for removing old computer accounts in a timely manner. If you’re already having this issue, either delete the fileserver_old account if it is no longer valid or remove the HOST/fileserver.contoso.com SPN from the fileserver_old computer account using setspn or adsiedit. Once you think you fixed the problem by either of the above solutions, from a client, try to connect to \\fileserver.contoso.com again and then run ‘klist tickets’ from the command line. If Kerberos is now working, you’ll see that you acquired a Kerberos ticket for HOST/fileserver.contoso.com. Since computer objects have a SPN for both NetBIOS and FQDN, we would need to remove both from the fileserver_old computer account:
setspn -D Host/Fileserver fileserver_old
setspn -D Host/Fileserver.contoso.com fileserver_old
Scenario #2: Changing accounts that the SQL service runs under
The SQL Admin is asked to install a new SQL server. During the install, he hits Next, Next, Next. Accordingly, SQL is now running under the local system context, which is the computer account. Consequently, the MSSQLSVC/SQLServer.contoso.com SPN’s were added to the computer account. Many months later, company policy dictates that SQL must run under a service account. The same admin configures SQL to now run under the SQLAdmin service account. The SPN’s are then added to this SQLAdmin account. We now have duplicate SPN’s again.
Remember those questions I had you take note of earlier. Let’s go through them again for this scenario.
1.) Who has to decrypt this Kerberos ticket?
A: When connecting to this SQL server, the account that SQL is running under has to decrypt the Kerberos ticket, NOT the SQL computer account. So, in this case, it would be SQLAdmin.
2.) So that SQLAdmin can properly decrypt the ticket, what account must the MSSQLSVC/SQLServer.contoso.com SPN attribute only be set on to make this work?
A: This SQL SPN must only be set only on the SQLAdmin service account.
Use setspn or adsiedit and remove the MSSQLSVC/SQLServer.contoso.com SPN from the SQLServer.contoso.com computer account while leaving it on the SQLAdmin account.
setspn -D MSSQLSVC/SQLServer.contoso.com SQLServer
Scenario #3: Changing accounts that an IIS application runs under
This one is almost identical to the SQL one. The IIS admin is asked to install a new IIS application called http://intranet.contoso.com. During the install, he creates a new application pool within IIS and selects the local system account to run it under, which is the web server computer account. He then registers the HTTP/intranet.contoso.com SPN on the web server computer account. Later that year, he’s asked to change the web application to run under a service account. He switches the application pool to run under this new service account called WebAdmin. He also adds the HTTP/intranet.contoso.com SPN to this new service account.
Once again, remember those questions I had you take note of earlier. Let’s go through them for this scenario.
1.) Who has to decrypt this Kerberos ticket?
A: When connecting to this web application, the account that the web application is running under has to decrypt the Kerberos ticket. This would now be WebAdmin.
2.) So that WebAdmin can properly decrypt the ticket, what account must the HTTP/intranet.contoso.com SPN only be set on to make this work?
A: The SPN must only be set on the WebAdmin service account.
Use setspn or adsiedit and remove the HTTP/intranet.contoso.com SPN from the webserver.contoso.com computer account while leaving it on the WebAdmin service account.
setspn -D HTTP/intranet.contoso.com webserver
Scenario #4: The SPN is set on the wrong account
I left this scenario for last because while it doesn’t include any duplicate SPN’s, it can be a by-product of improper SPN cleanup and will cause authentication issues and once again highlights the importanace that the SPN must be configured on the right account. In any of the above scenarios, what if the admin was troubleshooting the Kerberos authentication issue and removed the SPN attribute from the wrong account, what would happen? Let’s go back to the original file server scenario and play it out. In this scenario, you discovered that you have duplicate SPN’s and remove the HOST/fileserver.contoso.com SPN from fileserver’s computer account while leaving this SPN on the JoeAdmin user account.
1.) Who has to decrypt this Kerberos ticket?
A: The file server computer.
2.) Will the file server be able to decrypt the Kerberos ticket?
A: No, when this happens, the file server will throw an Event ID 4: KDC_AP_ERR_MODIFIED to its system log because it is unable to decrypt the ticket because it’s encrypted with JoeAdmin’s password.
3.) In this scenario, will the client fail over to NTLM and successfully authenticate to the fileserver?
A: Actually, no. The client will only fail over to NTLM if it is unable to obtain a Kerberos ticket for the resource in question. Since the client was able to get a Kerberos ticket, although one that doesn’t work, the client will continually try Kerberos and it will fail until someone fixes the problem and the client purges and refreshes its Kerberos tickets. This is important to remember.
Use setspn or adsiedit and put the HOST/fileserver.contoso.com SPN back on the fileserver.contoso.com computer account and remove this SPN from the JoeAdmin user account.
Remove them from JoeAdmin
setspn -D HOST/fileserver JoeAdmin
setspn -D HOST/fileserver.contoso.com JoeAdmin
Add them back to the fileserver computer account:
setspn -A HOST/fileserver fileserver
setspn -A HOST/fileserver.contoso.com fileserver
Key Takeaway: This is not the only scenario that will cause this error but when the SPN is set on the wrong account other than the one we are submitting the Kerberos ticket to, authentication will continually fail, the user will not be able to authenticate to the resource, and KDC Event ID 4 will be logged to the system log on the destination file server.
How To Find Your Duplicate SPN’s:
Up until now, we’ve focused on troubleshooting duplicate SPN’s based on some given scenarios you might be having because you’ve seen KDC Event ID 4 or 11. What about any duplicate SPN’s you have in AD that just aren’t causing problems yet but may one day. How do we find those without spending many hours writing some PowerShell magic? Thankfully, the new version of setspn that is present on 2008 R2 will find all duplicates across the entire FOREST. Addtionally, this command only takes about a minute to run. Now, I’ve seen where a 2008 R2 server will have two versions of setspn installed and if you just go to the command prompt and run setspn, this version doesn’t support the necessary switches. Consequently, make sure to use the one located in %systemroot%\system32. The following is some example output:
C:\windows\system32\Setspn –x –f
The first line is the duplicate SPN and the next two lines are the two accounts that have that SPN set. If you run this command in your environment, most of the duplicates you have will probably match one of the given scenarios above. Now, if you have multiple domains and run this command, at the very bottom of the setspn output, you will see the reported SPN as having a duplicate:
Warning: These are by design. Do not mess with these SPN’s or these user accounts
Nonetheless, you now you have the knowledge to deal with these and clean them up.
This Kerberos blog will more than likely become a mini-series. In part two, I’ll talk about duplicate SPN’s between different domains/forests, and clustered applications and how to identify and resolve those. Part 3 will be inter-domain and inter-forest Kerberos authentication and Part 4 will be double-hop Kerberos authentication and constrained delegation. Please leave some feedback. Thanks.