Troubleshooting Intermittent Pop-up Credentials in ISA Server 2004

Troubleshooting Intermittent Pop-up Credentials in ISA Server 2004

1. Summary

This article describes typical troubleshooting steps that you can use to determine the root cause for receiving the pop-up credential window when browsing Web sites when connected to an ISA Server 2004 computer.

2. Introduction

The following figure shows the topology that is used in this example.

 

Figure 1

  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

On the ISA Server computer, Integrated Windows authentication is used to access the Internet. The following figure shows the rule that defines the firewall policy.

Fig 2

 

 

 

 

 

 

When the ISA Server computer is a member of the domain and rules require that the domain user be authenticated, ISA Server must contact the domain controller to validate the credentials of the user. The following example emphasizes the main packets. Other packets will appear in the real environment:

1. The client tries to browse to the Web site www.msn.com. At this point, the ISA Server computer receives a packet, like this one:

192.168.0.180 192.168.0.3 HTTP GET http://www.msn.com/ HTTP/1.0

2. ISA Server accesses the firewall rules and checks if the rules require authentication. If the rules require authentication, the client receives an authentication request, like this one:

192.168.0.3 192.168.0.180 HTTP HTTP/1.1 407 Proxy Authentication Required ( The ISA Server requires authorization to fulfill the request. Access to the Web Proxy filter is denied. ) (text/html)

3. The client negotiates with ISA Server for authentication:

192.168.0.180 192.168.0.3 HTTP GET http://www.msn.com/ HTTP/1.0, NTLMSSP_NEGOTIATE

192.168.0.3 192.168.0.180 HTTP HTTP/1.1 407 Proxy Authentication Required ( Access is denied. ), NTLMSSP_CHALLENGE

192.168.0.180 192.168.0.3 HTTP GET http://www.msn.com/ HTTP/1.0, NTLMSSP_AUTH, User: CTEST\Administrator

4. After ISA Server negotiates the authentication method with the client and receives the credentials, the credentials are validated against the domain controller:

192.168.0.3 192.168.0.10 RPC_NETLOGON NetrLogonSamLogonEx request

192.168.0.10 192.168.0.3 RPC_NETLOGON NetrLogonSamLogonEx response

5. Assuming that the domain controller validates the credentials, the next step is for ISA Server to allow the client to connect to the Web site that was requested.

The next section explains what is needed to set up the environment to obtain the data to troubleshoot the issue.

3. Gathering Information

To address the issue, an understanding of the exact behavior is needed. To determine the issue, ask the following questions:

· What time does this problem usually occur?

· How long does the problem last?

· What can be done to fix the problem? Is there a workaround?

· Does an event appear in the ISA Server system log that shows when the issue occurs?

For this example, assume the following answers:

· The problem usually occurs between 07:30 (7:30 A.M.) and 08:30 (8:30 A.M.). But sometimes, it occurs between 13:00 (1:00 P.M.) and 14:00 (2:00 P.M.). This is the time that the users log on to the domain.

· The problem sometimes occurs for 10 minutes, sometimes for 5 minutes, and occasionally for a few seconds.

· Usually, nothing is done. Usually, users close the window and try again. In other situations, restarting the ISA Server computer solves the problem.

· The following two events appear in the ISA Server system log.

Event ID : 5719

Raw Event ID : 5719

Record Nr. : 1

Category : None

Source : NETLOGON

Type : Error

Generated : 8/23/2006 7:45:45 AM

Written : 8/23/2006 7:45:45 AM

Machine : ISANAME

Message : This computer was not able to set up a secure session with a domain

controller in domain DOMAINNAME..

Type: Error

Date: 08/23/2006

Time: 07:46:43

Event ID: 5783

Source: NETLOGON

User: N/A

Computer: ISANAME

Details:

The session setup to the Windows NT or Windows 2000 Domain Controller \\DCNAME for the domain DOMAINNAME is not responsive.

After gathering the information that you need to start troubleshooting, focus on the data collection in a synchronized way. The challenge is to collect enough data to determine if the issue is related to the ISA Server computer, to the domain controller, to the network infrastructure, or to performance on those servers.

3.1. Preparing the Domain Controller

The first server to prepare is the domain controller. Perform the following steps, in the order shown, to obtain the information that is needed:

1. Install Network Monitor on the server.

For more information about how to install Network Monitor, see How to Install Network Monitor in Windows 2000 at Microsoft Help and Support.

2. Enable the Netlogon debugging tool. Run the command nltest /dbflag:0x2080ffff.

For more information about debugging using Netlogon, see Enabling debug logging for the Net Logon service at Microsoft Help and Support.

3. Install the Windows Server Performance Advisor Tool. This tool analyzes the performance on the server.

Download this tool from the link at the Microsoft Download Center.

3.2. Preparing the ISA Server Computer

On the ISA Server computer, follow the same plan. In addition, make the process more automated by using the tool EventMon (Product Support Services tool) to keep monitoring the event ID 5783 on the system log. When this event occurs, you can start a batch file to begin collecting data. The following is an example of the batch file.

netcap /n:0 /b:200 /L:00:05:00 /c:c:\logs\ISANAME.cap

copy \\DCNAME\c$\windows\debug\netlogon.log c:\logs\DCNAME_netlogon.log

copy c:\windows\debug\netlogon.log c:\logs\ISANAME_netlogon.log

netdiag /v > C:\nediag.txt

In this example, Netcap will bind to the network ID 0, the buffer size will be 200 megabytes (MB), the process will run for 5 minutes, and the log will be saved in the C:\logs folder. The second part of the batch file copies the Netlogon.log file from the domain controller. This is an important step because the issue occurs rapidly, and sometimes the Netlogon.log is overwritten. When this happens, the needed information is not available. This also applies to the local Netlogon.log file.

To learn how to use Netcap and how to identify the network ID of the network adapter that you are using, see Description of the Network Monitor Capture Utility at Microsoft Help and Support.

4. Analyzing the Logs

The first action after the problem occurs is to make sure that you have all the data collected from the ISA Server computer and domain controller.

4.1. Starting from the Basics

Start to analyze the logs using a basic approach. Review the Netdiag.txt file to see if you have Domain Name System (DNS) errors, DNS bind order, and name resolution. Check the communication between the ISA Server computer and the domain controllers in this log. The following example shows one item that you can find in this file that can cause this kind of behavior.

Trust relationship test. . . . . . : Failed

    Test to ensure DomainSid of domain 'DOMAINNAME' is correct.

  Find DC in domain 'DOMAINNAME':

    Found this DC in domain ' DOMAINNAME ':

        DC. . . . . . . . . . . : \\DCNAME

        Address . . . . . . . . : \\192.168.0.10

        Domain Guid . . . . . . : {FD9BF05A-77CF-41FC-98DD-F91B5C2F67F4}

        Domain Name . . . . . . : DOMAINNAME

        Forest Name . . . . . . : DOMAINNAME

        DC Site Name. . . . . . : Default First Site

        Our Site Name . . . . . : Default First Site

        Flags . . . . . . . . . : PDC emulator GC DS KDC TIMESERV GTIMESERV WRITABLE DNS_DC DNS_DOMAIN DNS_FOREST CLOSE_SITE 0x8

    [FATAL] Secure channel to domain 'CTEST' is broken. [RPC_S_SERVER_UNAVAILABLE]

For this kind of situation, you can try to reset the secure channel using the NLTEST utility. For more information about NLTEST, see Domain Secure Channel Utility - Nltest.exe at Microsoft Help and Support.

For questions about DNS configuration on the ISA Server computer, review the recommendations in

Configuring DNS Servers for ISA Server 2004 at the Microsoft TechNet Web site.

4.2. Analyzing Netmon Results

Consider the following items from the Netmon file:

· In Network Monitor, create a filter for the domain controller's IP address and for the IP address of the internal interface on the ISA Server computer.

· When you analyze network captures, determine if the client request is actually getting to the server, or if the server is responding but the response is lost before the client receives it.

· If you have delays and packets are lost in the communication, to determine if the switch is using spanning tree protocol, see A Client Connected to an Ethernet Switch May Receive Several Logon-Related Error Messages during Startup at Microsoft Help and Support.

· To narrow the possible issues with the network infrastructure, see How to troubleshoot network connectivity problems at Microsoft Help and Support.

4.3. Analyzing Netlogon Results

Determine what time the issue occurred, based on the time that event 5783 occurred. Open the Netlogon.log file on the ISA Server computer using Notepad. Press CTRL+F to find the same hour. When you are on the same hour and minute, check for one of the following events.

Situation 1

<timestamp> [CRITICAL] WW001: NlSessionSetup: Session setup: cannot I_NetServerReqChallenge 0xc0020018

<timestamp> [MISC] Eventlog: 5719 (1) "<domain>" 0xc0020018 c0020018 .... <timestamp> [SESSION] <domain>: NlSetStatusClientSession: Set connection status to c000005e

Or

Situation 2

<timestamp> [CRITICAL] NlPrintRpcDebug: Couldn't get EEInfo for I_NetLogonSamLogonEx: 1761 (may be legitimate for 0xc0020018)

<timestamp> [CRITICAL] DOMAINNAME: NlpUserValidateHigher: denying access after status: 0xc0020018 1

Or

Situation 3

<timestamp> [SESSION] UAO: NlSessionSetup: Session setup Succeeded

<timestamp> 9 [SESSION] UAO: NlFinishApiClientSession: Unbind from server (null) (TCP) 2.

<timestamp> 9 [CRITICAL] UAO: NlFinishApiClientSession: timeout call to \\DCNAME Count: 1

<timestamp> 9 [CRITICAL] UAO: NlpUserValidateHigher: denying access after status: 0xc0020050 1

<timestamp> 9 [CRITICAL] NlPrintRpcDebug: Dumping extended error for I_NetLogonSamLogonEx with 0xc0020050

In the first situation, event 5719 occurs, which is the same event that appears in Event Viewer. This points to the communication between ISA Server and one specific domain controller.

In the second situation, the error 0xc0020018 occurs, which means "The RPC server is too busy to complete this operation". To learn what the error codes mean, use the Winerror tool. For more information, see Winerror.h at the MSDN Web site. To address issues with the Netlogon.dll file, see You may receive an "RPC server is too busy to complete this operation" error message when you try to log on to a computer that is running Windows Server 2003 with Service Pack 1 at Microsoft Help and Support.

In the third situation, error 0xc0020050 occurs, which means RPC_NT_CALL_CANCELLED (remote procedure call was cancelled). This also could be related to networking. The ISA Server computer might be losing connection to the domain controller during the negotiation time. If this is the case, go back to the Netmon capture and review it, to see if communication has been lost for a period of time. Another cause for this error can be issues with RPC communication. To troubleshoot, see Error message in a Windows Server 2003-based domain or in a Windows 2000 Server-based domain: "The remote procedure call failed and did not run" at Microsoft Help and Support.

4.4. Analyzing the Server Performance Analyzer Result

For performance, you need to verify items, such as: On the domain controller, what is the result for the Process Queue Length at the moment that ISA Server logs the events 5783 and 5719? To understand and detect if you are facing a bottleneck in one subsystem, see Windows 2000 Performance Tuning at the Microsoft TechNet Web site.

From a performance standpoint, after reviewing the Performance Monitor log, you should check for the page table entries (PTEs) available on the system. You might be facing a PTE exhaustion because the servers are using the /3GB parameter. To troubleshoot, see How to Configure the Paged Address Pool and System Page Table Entry Memory Areas at Microsoft Help and Support.

Although the following article is written for Microsoft Exchange Server, the idea behind this problem also applies to ISA Server. For details, see XADM: An Exchange 2000 Server with the "3/GB" Switch in the Boot.ini File May Lose Network Connectivity Under a Heavy Messaging Load at Microsoft Help and Support.

5. Conclusion

This article focuses on issues related to an intermittent pop-up window that asks for authentication on the client side while browsing the Internet through the ISA Server computer. Although many points are covered, there are other situations that can cause authentication problems. For information about authentication issues that occur while browsing some Web sites or issues with authentication errors that occur all the time, follow the guidelines in Troubleshooting Client Authentication on Access Rules in ISA Server 2004 at the Microsoft TechNet Web site.

Yuri Diogenes

Support Engineer – Latin America Team – Platforms

Microsoft