ISA Server 2006 form base authentication problem using UPN logon format on a multiple domain environment

1. Understanding the Environment

The scenario that I’m going to describe on this post was based on a case where the customer’s need was really specific and the configuration of the ISA Server was restricted by the company’s hardening policy. The topology was similar to the one below:

Figure 1 – Lab used to simulate this behavior.

As you can see above the ISA Server 2006 was a member of one of the child domains and it was using FBA (Forms Base Authentication) to publish an internal web server. The goal was to be able to use UPN format to logon to the Dallas.contoso.com child domain from outside. In this case customer was using multiple UPN suffixes for the same domain.

For more information about multiple UPN suffixes concept review the KB243280.

When the external client uses the SAM format (domain\user) it worked fine no matter what the domain was. However, using the UPN format the authentication was only working for users located on the same child domain as the ISA Server 2006. The error message that was showing up on the Internet Explorer right after the client logon was:

"The server requires authorization to fulfill the request. Access to the Web server is denied. Contact the server administrator.”

2. Reviewing the Data

To better understand the logon attempts we enabled the netlogon logging on the domain controller. To do this run the following command on the Domain Controller:

nltest /dbflag:0x2080ffff

2.1. Reviewing the Netlogon Log and the Security Logs

On the Domain Controller located on the Austin.contoso.com we could see the logon attempt:

[CRITICAL] NlPrintRpcDebug: Couldn't get EEInfo for I_NetLogonSamLogonEx: 1761 (may be legitimate for 0xc0000064)

11/30 08:45:50 [LOGON] SamLogon: Network logon of Austin.contoso.com\bob@dallas.contoso.com from ISASRV1 Returns 0xC0000064

The error code 0xC0000064 means that the user does not exist and we can confirm that on the security log of the ISA Server:

Event Type: Failure Audit

Event Source: Security

Event Category: Account Logon

Event ID: 680

Date: 04/10/2008

Time: 11:51:03 AP

User: NT AUTHORITY\SYSTEM

Computer: ISASRV1

Description:

Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0

Logon account: bob@dallas.contoso.com

Source Workstation: ISASRV1

Error Code: 0xC0000064

3. Hardening

On the first paragraph of this post I mentioned that the ISA Server was hardening due the company’s security policy. One of the changes that it was implemented on this case was the change for the LmCompatibilityLevel entry for 5. During the course of the troubleshooting for this case it was also identified that the issue didn’t happen using the default LmCompatibilityLevel value. But, we couldn’t implement this workaround due the compliance of this server to the company’s policy.

4. The Solution

This is a known issue that was recently fixed with the hotfix mentioned on the KB article below:

Authentication of trusted users fails on a Windows Server 2003-based server if the UPN format is used and if the value of the LmCompatibilityLevel entry is equal to or larger than 3

http://support.microsoft.com/?id=947861

Notice that even if you have Windows Server 2003 SP2 you still need to update the system using the fix above. Also, it is important to emphasize that although the article focus on FBA authentication, the same behavior can occur using normal HTTP authentication. In any of those instances, the SAM format logon is handled the same by ISA Server when calling LSALogonUser API. For more information on NTLM user authentication, review the KB102716.

Yuri Diogenes

Security Support Engineer – ISA Server Team

Microsoft Texas