Troubleshooting Authentication Issues in ISA Server Using Net Logon Logging

Background:

I recently had a case where web proxy clients were randomly being prompted for credentials when going to the Internet through ISA Server 2006. Even after entering the proper credentials they were not able to get to the Internet. This particular environment had Internet access restricted to a certain group of users so authentication was required. The problem seemed to plague random users at various times and would only last for about 15 minutes. Eventually the issue would go away on its’ own and Internet access would return for the affected users. Normally I might lean towards the domain controller being overloaded but since there were only about 1000 web proxy clients this didn’t seem to make sense.

Note: For a reference on how ISA Server 2006 handles Web Proxy Authentication for IE6 and IE7 (and later) read the article http://technet.microsoft.com/en-us/library/bb984870.aspx

To make the issue even a bit trickier to troubleshoot, the issue would often go away before their technical people could gather any diagnostic information. The ISA Best Practices Analyzer is an invaluable tool to help us capture and analyze data when reproducing the issue is easy to do. In this case, the IT Team was usually alerted to the issue after it was too late to gather the necessary data. When using this tool for reactive scenarios such as this you should capture data using the ISA Data Packager. Please see the article “Using ISABPA for Proactive and Reactive Work with ISA Server – Part 2 of 2” that has step by step instructions for this tool.

Analysis:

Since the issue involved authentication issues we decided to enable Net Logon logging on the Domain Controller which ISA Server had a secure channel established with. For this we need the tool NLTEST which is found in the Windows Support Tools on the Windows 2003 CD. An article to reference for NLTEST syntax can be found at http://technet.microsoft.com/en-us/library/cc786478.aspx

 

 

Note: To enable debug logging for Net Logon Service I followed instructions from this Knowledge Base article KB109626

I used the alternate method it mentions and ran nltest /dbflag:0x2080ffff in a command window. Next, I stopped and restarted Net Logon service. Note: You can actually run a slightly different flag if you want to eliminate some of the extraneous information: nltest /dbflag:0x20000004

 

After you enable debug logging and stop and restart the Net Logon Service a log will be created called netlogon.log in the Systemroot\Debug directory. Please note that performance can be degraded by this logging process and you will want to turn it off when you are done troubleshooting. To do this run nltest /dbflag:0x0 and then stop and restart the Net Logon Service.

 

Below is a sample of what the information in this log file that is created looks like.

 

To help you interpret the output and for additional details about Net Logon logging please see Maintaining and Monitoring Account Lockout

 

Using this article we see that the code 0x0 is a successful logon. Since this log file can get quite large depending on your environment it would help to be able to quickly find what you are looking for. Since we knew the account name of a user being affected (contoso\keith) it could be possible to filter this log file and find the information we needed. There is a nice command to help you do this at a command prompt using findstr :

 

findstr /I “contoso\keith” c:\windows\debug\netlogon.log >> c:windows\debug\failed.txt

 

Looking at the file we just created we now see information relevant to the account contoso\keith

 

When we looked up the code 0xC0000234 we see that the account was being locked out. I used the table found at http://technet.microsoft.com/en-us/library/cc776964.aspx for an explanation of the code.

 

We found out that the customer had a password policy which allowed for 3 bad password attempts set in Group Policy for the Domain. The Account Lockout Duration was set to 15 minutes. This explained why the issue would last only for a little while and then go away.

The next question was “Why were the accounts being locked out?” Debug logging was enabled on the other Domain Controllers and analysis was done using the articles previously mentioned. It turns out that a few machines in another site were infected by a virus that was using a dictionary attack to try to gain access to various accounts.

Note: A great article that explains how to troubleshoot account lockout can be found at the Microsoft Download Center .

Conclusion:

Authentication issues affecting ISA Server can often be tricky to troubleshoot. Using the tools and techniques outlined in this article, specifically Net Logon debug logging , can help us more quickly get to the source of the problem.

Author

Keith Abluton

Security Support Engineer – Forefront Edge Team

Microsoft – Charlotte

Technical Reviewer
Yuri Diogenes
Sr Security Support Escalation Engineer – Forefront Edge Team

Microsoft – Texas