Troubleshooting ERROR: Setup failed to install ADAM.rn (0x80074e46) and 0x80070643 while trying to install TMG 2010

Introduction

Consider a situation where the TMG Administrator was getting an error while trying to install TMG 2010 RTM Standard Edition Server on a domain joined Windows 2008 R2 Server. The error message was:

Setup failed to install ADAM.\r\n (0x80074e46)

The interesting part of this problem was that even after reinstalling the Operating System the same error message was happening again.

This post is about two different scenarios where TMG administrator was facing this issue while trying to install Forefront TMG 2010.

Scenario 1 – The Domain Policy Issue

The first step on each scenario is to understand what the issue, once this phase is done you can build an action plan for data gathering. To Troubleshoot Setup issues on TMG you will need to collect TMG Setup Installation logs, located at %windir%\temp and ADAM Setup log files located at %windir%\debug. Once this is collected you can proceed with the data analysis.

Data Analysis

As previously mentioned, TMG setup log files are by default located at %windir%\temp and the name will start with a pre-defined prefix, such as ISAADAM_INSTALL_XXX. For more information on the file name and description read the article Setup log files at Microsoft Technet. For this particular issue, when the error was happening on the GUI we notice the following error in the in the setup logs.

According to the error message above we can assume that ADAM is failing to install due to a trust relationship issue with the domain. By knowing that, the next troubleshooting step is

to verify if TMG has valid secure channel with the Domain Controller. To do that you can use the NLTEST command as shown below:

Unfortunately, even after perform this step the issue continued to happen. This would lead us to believe that the issue could be with a domain policy that might have been modified causing the restriction on the domain joined server. Browse to Start > Run à gpedit.msc > Computer Configuration > Windows Settings > Security Settings > Local Policies. Check the Security Options and User Right Assignment sections to verify if all the permissions are set to default.

Resolution

In our scenario, the issue seemed to have been due to a modified Domain Policy which restricted the installation of ADAM. We fixed the issue by performing the following steps:

1. Disjoin the Server you are about to install TMG on from the Domain.

2. Run TMG Setup again.

Once this is done, TMG should install fine while in a workgroup since the problem here is related to Domain Policy, at this point you can rejoin the server back to the Domain.

Scenario 2 – Another Domain Policy Issue

Using the same approach as was used before we collected setup log files from TMG and ADAM to start troubleshooting this issue.

Data Analysis

In this scenario the TMG setup failed in ldap_search_ext_s function with same error code as scenario 1

Setup Logs reveal the following:

13:58:07 ISA setup CA ERROR : pSecurityDesciptorValue == NULL

13:58:07 ISA setup CA ERROR : Adam_GetContainerSecurity(CN=Sites,CN=Configuration,CN={53A16AA0-C09E-4536-B55D-0FE4210F6D14}) failed, hr = 0x80070002

13:58:07 ISA setup CA ERROR : Failed to change DACL of Configuration objects (Apply failed) hr = 0x80070002

13:58:07 ISA setup CA ERROR : AdamSecurity.SetupEnterpriseSecurity failed, hr = 0x80070002

13:58:07 ISA setup CA ERROR : CreateStorage_Enterprise: Adam_SetupEnterpriseSecurity failed, hr=0x80070002

13:58:07 ISA setup CA ERROR : Setup failed while creating Forefront TMG storage.

13:58:07 ISA setup CA ERROR : (Error 28512) Setup failed while creating Forefront TMG storage.

13:58:07 ISA setup CA ERROR : EXIT: CreateStorage_Enterprise, Custom Action failed (0x643)

13:58:43 ERROR: Setup failed. Error returned: 0x643

13:58:43 ERROR: CBasicInstaller: Install failed, hr=0x80070643

13:58:43 ERROR: Installation failed. hr = 0x80070643

13:58:43 ERROR: Installation failed, hr=0x80070643

13:58:43 ERROR: InstallProducts: Install ISA (Core components) failed, hr=0x80070643

13:58:43 ERROR: Wrapper: Install failed, hr = 0x80070643

13:58:43 ERROR: Wrapper: DoSetup failed, hr = 0x80070643

13:58:43 ERROR: Wrapper: DoSetup failed, hr = 80070643

Once the issue was understood we could go to the next level of troubleshooting and use the ldapsd tool to perform the same ldap_search_ext_s function in order to retrieve the security descriptor of cn=sites,cn=configuration <guid>.

This tool queries ADAM the way TMG setup does, run it as follows:

Ldapsd /s <local Server name> /b cn=sites,cn=configuration,cn={guid}

In our case the guid 53A16AA0-C09E-4536-B55D-0FE4210F6D14 was obtained from the setuplog above. Open command prompt with elevated privileges and run the command below:

C:\ldapsd> .\ldapsd.exe /Servername /b "CN=Sites,CN=Configuration,CN={53A16AA0-C09E-4536-B55D-0FE4210F6D14 }" –t

The output of this command in this case:

ldap_init(Host- Servername, port- 2171t) succeeded, version- 3t

options: timelimit- 0t, sizelimit- 0t. hoplimit- 32t

ldap_bind_s() return 0h

ldap_search_ext_s(CN=Sites,CN=Configuration,CN={ 53A16AA0-C09E-4536-B55D-0FE4210F6D14 }, SeInfo- fh) return 0h

ldap_count_entries() return 1t

...processing entry no. 1t, CN=Sites,CN=Configuration,CN={ 53A16AA0-C09E-4536-B55D-0FE4210F6D14 }

Attribute nTSecurityDescriptor not found, err- 87t, 57h, LastLdap- 16t, 10h

ldap_unbind_s() return 0h

C:\ldapsd> .\ldapsd.exe /s Servername /b "CN=Sites,CN=Configuration,CN={ 53A16AA0-C09E-4536-B55D-0FE4210F6D14 }"

ldap_init(Host- Servername, port- 2171t) succeeded, version- 3t

options: timelimit- 0t, sizelimit- 0t. hoplimit- 32t

ldap_bind_s() return 0h

Not using LDAP server control

ldap_search_ext_s(CN=Sites,CN=Configuration,CN={ 53A16AA0-C09E-4536-B55D-0FE4210F6D14 }, SeInfo- fh) return 0h

ldap_count_entries() return 1t

...processing entry no. 1t, CN=Sites,CN=Configuration,CN={ 53A16AA0-C09E-4536-B55D-0FE4210F6D14 }

Attribute nTSecurityDescriptor not found, err- 87t, 57h, LastLdap- 16t, 10h

ldap_unbind_s() return 0h

This means  that ldap_search succeeded and it retrieved the distinguished name, but failed with the security descriptor.

In a good box this will return the following

nTSecurityDescriptor found, length- 648t

Resolution

This issue was found to be due to “Manage auditing and security log" user right on the Default Domain policy, normally this setting is Not Defined.

Normally the TMG server’s local security policies give this right to Builtin\Administrators. Customer had modified their domain policy and set it to right to the Domain\Exchange Enterprise Servers and Domain\Exchange Domain Servers groups ONLY. This change in policy overwrote the local policy and removed this right from the Administrator. To fix this issue the “Administrators” was added to the user right list for the default domain policy.

Authors 
Bala Natarajan 
Sr Support Escalation Engineer
Microsoft CSS Forefront Security Edge Team

Niladri Dasgupta
Support Engineer
Microsoft CSS Forefront Security Edge Team

Technical Reviewer
Yuri Diogenes
Sr Support Escalation Engineer
Microsoft CSS Forefront Security Edge Team

Doron Juster
Sr SDE
Forefront Endpoint Protection Team