Unable to authenticate using FBA with LDAP on ISA Server 2006

  • Article

Introduction

Consider a scenario where users are not able to authenticate using FBA with LDAP to access the websites published through ISA server. In this scenario ISA server was part of one domain (contoso.com) and users who would access the website are part of another domain (fabrikam.com). FBA with LDAP is used on the web listener of the web publishing rule to authenticate the users from fabrikam.com’s domain controller. But these users are not able to authenticate using this method.

Note: More about Ldap authentication please refer http://technet.microsoft.com/hi-in/library/bb794854(en-us).aspx#ldap

Data Collection

To troubleshoot the issue took network traces while creating LDAP user set on the ISA server (reference http://technet.microsoft.com/hi-in/library/bb794854(en-us).aspx#LDAPUser), which failed with error, “access to LDAP server is denied.”

Data Analysis

In the network captures we found:

1. LDAP Bind Request as below

clip_image002

2. LDAP Bind Response as below

clip_image004

Troubleshooting and Resolution

We tested the user credentials from a machine which is already part of the fabrikam domain and we were able to authenticate using same credentials. Then as per http://blogs.technet.com/b/isablog/archive/2008/04/17/isa-server-2006-form-base-authentication-problem-using-upn-logon-format-on-a-multiple-domain-environment.aspx we checked the HKLM\System\CCS\CONTROL\LSA\LMCompatibilityLevel on the ISA server and it was set to 0x2 (only allow LM and NTLM). Then checked same on the Domain Controller (which it was windows 2008 server) of the domain where users were located and it was set to 0x5(only allow NTLMv2 and block LM /NTLM).Since Domain controller only allows NTLM v2 it was not authenticating the request coming from the ISA server which was sending it with NTLM v1 as we can see in the NTLM challenge response in the network traces.

To resolve the problem we set LMCompatibilityLevel key on the ISA server to 0x3 (although we could also set it to 0x4) and restarted the ISA server. After that users were able to normally logon via FBA.

Author
Suraj Singh
Support Engineer
Microsoft CSS Forefront Security Edge Team

Technical Reviewer
Yuri Diogenes
Sr Support Escalation Engineer
Microsoft CSS Forefront Security Edge Team