Unable to Join a TMG server to the Stand Alone Array

Introduction:

In this scenario I am going to share a very unique issue which I came across while trying to join one of the TMG servers to the Stand alone array.

In this case we had two Enterprise Edition TMG servers installed on an Appliance.

Scenario:

We had two Enterprise Edition TMG servers and we were trying to join one of the servers to the Stand alone array pointing it to the other TMG server as the Array Manager.

But when we ran the ‘Join Array’ Wizard it failed with an error ‘KEYSET DOES NOT EXIST’ on the TMG server which we were trying to make the Array Member.

Troubleshooting:

Looking at the above error message it seems that TMG is trying to access some Folder/File which is either missing on the server or it does not have access to it.

So, we ran the Process Monitor on the TMG server while trying to join it to the Array. We filtered the Process Monitor file to show the results related to wspsrv.exe (Microsoft Forefront TMG Firewall Service) process.

And in the filtered trace we could see the following files being accessed by wspsrv.exe process:

2:02:36.6488085 AM wspsrv.exe 2756 CreateFile C:\ProgramData NAME COLLISION Desired Access: Read Data/List Directory, Synchronize, Disposition: Create, Options: Directory, Synchronous IO Non-Alert, Open Reparse Point, Attributes: N, ShareMode: Read, Write, AllocationSize: 0

2:02:36.6489655 AM wspsrv.exe 2756 CreateFile C:\ProgramData SUCCESS Desired Access: Read Attributes, Disposition: Open, Options: Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened

2:02:36.6489971 AM wspsrv.exe 2756 QueryBasicInformationFile C:\ProgramData SUCCESS CreationTime: 7/14/2009 8:50:08 AM, LastAccessTime: 5/5/2011 12:58:49 AM, LastWriteTime: 5/5/2011 12:58:49 AM, ChangeTime: 5/5/2011 12:58:49 AM, FileAttributes: HDNCI

2:02:36.6490214 AM wspsrv.exe 2756 CloseFile C:\ProgramData SUCCESS             

2:02:36.6491280 AM wspsrv.exe 2756 CreateFile C:\ProgramData\Microsoft NAME COLLISION Desired Access: Read Data/List Directory, Synchronize, Disposition: Create, Options: Directory, Synchronous IO Non-Alert, Attributes: S, ShareMode: Read, Write, AllocationSize: 0

2:02:36.6492088 AM wspsrv.exe 2756 CreateFile C:\ProgramData\Microsoft\Crypto NAME COLLISION Desired Access: Read Data/List Directory, Synchronize, Disposition: Create, Options: Directory, Synchronous IO Non-Alert, Attributes: S, ShareMode: Read, Write, AllocationSize: 0

2:02:36.6493243 AM wspsrv.exe 2756 CreateFile C:\ProgramData\Microsoft\Crypto\RSA NAME COLLISION Desired Access: Read Data/List Directory, Synchronize, Disposition: Create, Options: Directory, Synchronous IO Non-Alert, Attributes: S, ShareMode: Read, Write, AllocationSize: 0

2:02:36.6494460 AM wspsrv.exe 2756 CreateFile C:\ProgramData\Microsoft\Crypto\RSA NAME COLLISION Desired Access: Read Data/List Directory, Synchronize, Disposition: Create, Options: Directory, Synchronous IO Non-Alert, Attributes: S, ShareMode: Read, Write, AllocationSize: 0

2:02:36.6496038 AM wspsrv.exe 2756 CreateFile C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys NAME COLLISION Desired Access: Read Data/List Directory, Synchronize, Disposition: Create, Options: Directory, Synchronous IO Non-Alert, Attributes: S, ShareMode: Read, Write, AllocationSize: 0

2:02:36.6498710 AM wspsrv.exe 2756 CreateFile C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\a63f7ad5b2228889fc41ae79c417446b_c49c7513-8b86-4d0e-90bf-4805804a9318 SUCCESS Desired Access: Generic Write, Read Attributes, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: S, ShareMode: None, AllocationSize: n/a, OpenResult: Opened

2:02:36.6499060 AM wspsrv.exe 2756 QueryStandardInformationFile C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\a63f7ad5b2228889fc41ae79c417446b_c49c7513-8b86-4d0e-90bf-4805804a9318 SUCCESS AllocationSize: 4,096, EndOfFile: 1,467, NumberOfLinks: 1, DeletePending: False, Directory: False

2:02:36.6499310 AM wspsrv.exe 2756 WriteFile C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\a63f7ad5b2228889fc41ae79c417446b_c49c7513-8b86-4d0e-90bf-4805804a9318 SUCCESS Offset: 0, Length: 1,467, Priority: Normal

2:02:36.6499949 AM wspsrv.exe 2756 CloseFile C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\a63f7ad5b2228889fc41ae79c417446b_c49c7513-8b86-4d0e-90bf-4805804a9318 SUCCESS             

2:02:36.6501323 AM wspsrv.exe 2756 CreateFile C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\a63f7ad5b2228889fc41ae79c417446b_c49c7513-8b86-4d0e-90bf-4805804a9318 SUCCESS Desired Access: Read Attributes, Delete, Disposition: Open, Options: Non-Directory File, Open Reparse Point, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened

2:02:36.6501620 AM wspsrv.exe 2756 QueryAttributeTagFile C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\a63f7ad5b2228889fc41ae79c417446b_c49c7513-8b86-4d0e-90bf-4805804a9318 SUCCESS Attributes: SA, ReparseTag: 0x0

2:02:36.6501855 AM wspsrv.exe 2756 SetDispositionInformationFile C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\a63f7ad5b2228889fc41ae79c417446b_c49c7513-8b86-4d0e-90bf-4805804a9318 SUCCESS Delete: True

2:02:36.6502116 AM wspsrv.exe 2756 CloseFile C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\a63f7ad5b2228889fc41ae79c417446b_c49c7513-8b86-4d0e-90bf-4805804a9318 SUCCESS             

2:02:36.6506463 AM wspsrv.exe 2756 CreateFile C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys SUCCESS Desired Access: Read Data/List Directory, Synchronize, Disposition: Open, Options: Directory, Synchronous IO Non-Alert, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened

2:02:36.6506747 AM wspsrv.exe 2756 QueryDirectory C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\a63f7ad5b2228889fc41ae79c417446b_* NO SUCH FILE Filter: a63f7ad5b2228889fc41ae79c417446b_*

2:02:36.6507028 AM wspsrv.exe 2756 CloseFile C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys SUCCESS             

As you can see in the above logs Firewall service is trying to access a particular file in the C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys folder on that server and due to some reason it was not able to access it and hence was showing the message as ‘NO SUCH FILE’.

So now it was pretty clear that wspsrv.exe was looking for a ‘machine key’ file in C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys folder but it was not able to find it or it did not have access to it.

As the Microsoft Forefront TMG Firewall Service runs under the NetworkService account, we tried to give Permissions to the NetworkService account on the C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\a63f7ad5b2228889fc41ae79c417446b file. But while trying that we got an error ‘Access Denied’ .

This was most likely due to the fact that the SYSTEM service account is typically the ‘owner’ of the machine key file in question, and therefore we did not have permissions to give Read permission to the Network Service account.

So it looked to be some permissions issue with the server itself. We checked back with the Appliance vendor and came to know that the boxes were hardened. It looks as though the vendors hardening process may have changed the default permissions on the MachineKeys folder, it’s contents or both. They replaced that server with a new one and when we tried to join the TMG to the array on this new server, it got joined fine.

NOTE: The above troubleshooting was done on the TMG server which we were trying to join to the array and make it an Array Member.

CONCLUSION:

This was not a failure of the product. Problem was due to misconfiguration of server and overzealous hardening by the hardware vendor. TMG was working fine but not allowed to do what it should be able to in default installation.

Author:

Nitin Singh

Security Support Escalation Engineer

Microsoft CSS Forefront Security Edge Team

Technical Reviewers:

Billy Price

Security Sr. Support Escalation Engineer

Microsoft CSS Forefront Security Edge Team

Richard Barker

Security Sr. Support Escalation Engineer

Microsoft CSS Forefront Security Edge Team