Random authentication prompts while accessing internet through ISA Server followed by ISA Server becoming unresponsive

Introduction

Consider a scenario where users behind ISA Server (internal network) start to receive random prompts for authentication while trying to access internet using ISA Server as proxy. The authentication prompt persists even after entering the credentials. To resolve the issue it is necessary to restart Firewall Service.

Although you probably heard or read about this scenario many times, the goal of this post is to give you a compiled version of the action plan and what to look for while analyzing the data.

Data Collection

Start by following the plan from this post (basics section), along with that make sure that binding order is also correct i.e. internal NIC is higher in order then the external. Wrong binding order can cause issues such as the one mentioned here. In addition to the data gathering specified previously, also collect the following data:

1. Use ISA Data Packager while doing repro of the issue.
2. Enable netLogon logging on the ISA server nodes, using command nltest /dbflag:0x2080ffffin the command prompt as per KB109626.
2. Set the Performance counters as specified in this post.

Data analysis

When start reviewing the perfmon data you want to check the counter ISA Server Firewall Packet Engine\Backlogged Packets. You will notice a trend similar to the perfmon screenshot showed in this post. This can happen due name resolution issue as explained in this TechNet Article.

Next data to analyze is the netlogon.log, which also can be done using the same approach as the following post. In other words, look for the following pattern:

08/21 12:00:00 [DOMAIN] Contoso: Domain thread started 08/21 12:00:00 [DOMAIN] Contoso: Domain thread started doing API timeout 08/21 12:00:00 [SESSION] Contoso: Contoso: NlTimeoutApiClientSession: Unbind from server \\ab-cd.Contoso.local (TCP) 0.

From above data it appeared we can conclude that the Domain Controller to which ISA server had the secure channel established with, did not responded in time manner, which triggered the NlTimeoutApiClientSession in the netlogon logging. After that ISA Server resets the secure channel and tries to make secure channel with another DC.

Resolution for this Particular Case

In this particular case the clients were using WPAD (automatic detection), which by default returns the IP addressof the ISA Server rather than the name. This forced the client to use NTLM authentication rather than Kerberos (supported in IE7 or higher).

Note: The advantages to use Kerberos instead of NTLM are documented in this article.

In order to force WPAD to use FQDN instead of IP address we ran the script described in this post. After running the script, all the web proxy clients using WPAD started getting FQDN of the ISA server nodes and use Kerberos for authentication, which enhance the authentication traffic and decrease the number of authentication request.

Author
Suraj Singh
Security Support Engineer
Microsoft CSS Forefront Security Edge Team

Technical Reviewer
Yuri Diogenes
Sr Security Support Escalation Engineer
Microsoft CSS Forefront Security Edge Team