Potential for Kerberos Issues When Using a Cisco VPN/ASA with Win2003 or later DC’s

Hey everyone, Rob Greene here back after a long hiatus from blogging. I had an interesting case come through that I thought many of you IT pros would be interested in.


The customer had an issue with using Cisco VPN and Cisco ASA concentrators and authenticating the user with Kerberos authentication. After they upgraded all their Windows 2000 domain controllers to Windows Server 2003 domain controllers, users could not authenticate through the VPN.

After doing this they gave Cisco a call and they were told to start DSA.MSC and check the box under Account Options to “Do not require Kerberos preauthentication” for all user accounts that need to connect through the appliances. When they did this as a test it did allow the user to start authenticating through the devices. However the customer did not like this option since:

1. This change lowers the security of Kerberos authentication by disabling password replay attack protection.

2. Everything had worked previously with Windows 2000.

If you want to learn more about preauthentication you can click here and look under the section:

Example AS Administration.

If you search Cisco’s website you will find that they have older documentation stating that you must disable this feature on the user’s account before the VPN will start working.

I don’t have real network traces (as I can’t show you a real customer’s network!), but I can tell you what we saw in the traces that led us to a potential workaround.

VPN Appliance –> UDP Port 88 / KRB5 AS-REQ –> Windows DC

VPN Appliance <– UDP Port 88 / KRB5KDC_ERR_REAUTH_REQUIRED <–Windows DC

VPN Appliance –> UDP Port 88 / AS-REQ (with Pre-Auth Data) –> Windows DC

VPN Appliance <– UDP Port 88 / KRB5KDC_ERR_RESPONSE_TOO_BIG <– Windows DC

A KRB5KDC_ERR_RESPONSE_TOO_BIG means that the requested Kerberos data was larger than the maximum default datagram size of 1465 bytes. The next step here usually would be for the client to start the Kerberos ticketing process over using TCP rather than UDP. If this was a Windows XP client instead of the Cisco VPN appliance we would have the customer implement the registry key change outlined in the below KB article.

244474 How to force Kerberos to use TCP instead of UDP in Windows

So basically the real issue is that the Cisco appliance is being asked by the KDC to use TCP instead of UDP to do the AS-REQ with pre-authentication data present. However, the Cisco appliance does not continue at this point. I spent the past few weeks working with Cisco directly (big shoutout – they have been great!), and they have published the following new articles:

Cisco VPN 3000: CSCed27444 – VPN3000 does not switch to TCP upon receiving Kerberos error code 52

Cisco ASA platform: CSCsi32224 – ASA does not switch to TCP upon receiving Kerberos error code 52

CSCtd92673 — “Kerberos authentication fails with pre-auth enabled”

Unfortunately, the Cisco VPN 3000 concentrator has been discontinued so there is no update available for the Cisco VPN 3000 to add this functionality. Also, you will need a support contract from Cisco to logon to those two articles and read them.

Please contact Cisco directly to find out more information about the issue, and with what IOS version it will be included in; we have provided this information to make it easier to find the content.

In the Meantime

So now the question becomes this: If that is what is going on, then how did it work when a Windows 2000 domain controller was used? And what can you do about it until an IOS update is released?

There was a change in the default datagram reply packet size used by the Kerberos Key Distribution Center (KDC) between Windows 2000 and Windows Server 2003. In Windows 2000 the default size is 2000 bytes, where it is 1465 bytes in Windows Server 2003. For those interested you can review the below KB article and find the registry key of MaxDatagramReplySize.

837361 Kerberos protocol registry entries and KDC configuration keys in Windows Server 2003

So that means this customer’s Kerberos ticket size for the user accounts is less than 2000 bytes, but more than 1465 bytes. Keep in mind that if you are reading this blog, your environment may still not work with the registry key setting at 2000. This is because the Kerberos ticket size is going to vary based on the number of groups that the user belongs to. You can review the KB below to learn more:

327825 New resolution for problems with Kerberos authentication when users belong to many groups

The only workaround that we can give a customer is to modify the MaxDatagramReplySize registry value to the size of your user’s largest Kerberos ticket size. It really depends on the group membership size of the user attempting to authenticate because of the PAC data that is included in the TGT. For those customers that have used the MaxTokenSize registry value should equal that value and restart the KDC service. If none of these acronyms are making sense you might want to check out my older blog post Kerberos for the Busy Admin.

Hey wow, this sounds like a pretty easy fix right? Well, you can actually have other problems because of this change. If your networking gear (switches, routers, firewalls) cannot handle large UDP packets without fragmenting them then all Kerberos authentication could start to fail in your environment. See KB244474 under more information to learn more about this.

Please keep in mind what you are doing when you implement the MaxDatagramReplySize registry value. You are in fact telling the KDC to never request the client send KRB5KDC_ERR_RESPONSE_TOO_BIG message back to a Kerberos client, and you should consider implementing the custom group policy setting included in KB244474 on all domain-joined machines to use TCP for Kerberos authentication always. If you are running purely Vista/2008 or later though, this is on by default.

Lastly, if you have to set this value very high, and you know that your networking gear will not support the packet sizes that had to be implemented, you may have to put a domain controller on the same IP subnet as the Cisco appliance to successfully get Kerberos tickets.

- Rob ‘I mean, you know’ Greene