1. Understanding the Environment
The scenario that I’m going to describe on this post was based on a case where the customer’s need was really specific and the configuration of the ISA Server was restricted by the company’s hardening policy. The topology was similar to the one below:
Figure 1 – Lab used to simulate this behavior.
As you can see above the ISA Server 2006 was a member of one of the child domains and it was using FBA (Forms Base Authentication) to publish an internal web server. The goal was to be able to use UPN format to logon to the Dallas.contoso.com child domain from outside. In this case customer was using multiple UPN suffixes for the same domain.
When the external client uses the SAM format (domain\user) it worked fine no matter what the domain was. However, using the UPN format the authentication was only working for users located on the same child domain as the ISA Server 2006. The error message that was showing up on the Internet Explorer right after the client logon was:
"The server requires authorization to fulfill the request. Access to the Web server is denied. Contact the server administrator.”
2. Reviewing the Data
To better understand the logon attempts we enabled the netlogon logging on the domain controller. To do this run the following command on the Domain Controller:
2.1. Reviewing the Netlogon Log and the Security Logs
On the Domain Controller located on the Austin.contoso.com we could see the logon attempt:
The error code 0xC0000064 means that the user does not exist and we can confirm that on the security log of the ISA Server:
Event Type: Failure Audit
Event Source: Security
Event Category: Account Logon
Event ID: 680
Time: 11:51:03 AP
User: NT AUTHORITY\SYSTEM
Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
Logon account: email@example.com
Source Workstation: ISASRV1
Error Code: 0xC0000064
On the first paragraph of this post I mentioned that the ISA Server was hardening due the company’s security policy. One of the changes that it was implemented on this case was the change for the LmCompatibilityLevel entry for 5. During the course of the troubleshooting for this case it was also identified that the issue didn’t happen using the default LmCompatibilityLevel value. But, we couldn’t implement this workaround due the compliance of this server to the company’s policy.
4. The Solution
This is a known issue that was recently fixed with the hotfix mentioned on the KB article below:
Authentication of trusted users fails on a Windows Server 2003-based server if the UPN format is used and if the value of the LmCompatibilityLevel entry is equal to or larger than 3
Notice that even if you have Windows Server 2003 SP2 you still need to update the system using the fix above. Also, it is important to emphasize that although the article focus on FBA authentication, the same behavior can occur using normal HTTP authentication. In any of those instances, the SAM format logon is handled the same by ISA Server when calling LSALogonUser API. For more information on NTLM user authentication, review the KB102716.
Security Support Engineer – ISA Server Team