Resolving TMG Array Join Failures - Reporting Server Leftovers

Introduction

Forefront TMG 2010 provides an enhanced array management feature known as array join / disjoin. This article describes a problem that may be encountered as a result of a failure during this process, the solution you can apply as well as a discussion of how and why this problem occurs.

The Problem

You attempt to join a TMG server to an empty EMS-managed array, but this action fails after some time because Intern Jamie unplugged the network cable from the EMS server by mistake (or so you choose to believe) during this process.

You have reconnected the EMS server to the network (and Jamie to his basic networking books) and attempt to join the server to the array once more, but now it fails with “The operation failed” message as shown below:

Although you have verified that Jamie is nowhere near the EMS server or any connecting devices between the EMS server and prospective array member, the array join failures continue with the same message; even with a completely different TMG server.

This problem does not occur when joining any array that contains functioning array members (which is why it can’t happen for Standalone arrays).

Note: when you encounter an array join failure, you must close and reopen the TMG manglement console before the array join wizard link is available.

The Solution

If you encounter this error during an array join process, perform the following steps on any computer that can establish connectivity to the EMS server on TCP port 2171.

1. Log on to the computer using an account with administrative rights to the TMG Enterprise
2. Download http://www.isatools.org/tools/cleartmgrptsrvrs.wsf to c:\tmgtools
3. Open a cmd window
   1. Hold down the Windows key and press the “R” key
   2. In the Run dialog, type cmd and hit <Enter>
4. In the cmd window, type
   1. cd c:\tmgtools
   2. cscript cleartmgrptsvrs.wsf /css:NameOfEmsServer

Note: the /css option may be omitted if you are running the script on the EMS server itself

The script performs the following actions:

1. connect to the specified EMS server (or “localhost” if none is provided)
2. check all arrays in the TMG Enterprise configuration
3. verify that the defined reporting server exists in the array servers set
4. if the server is not found in the array servers set, remove the server reference from the reporting configuration
5. log all actions to a text file

If the defined reporting server is not found in the array servers set, the reporting server definition is cleared and the following messages are displayed (and logged):

If the array is empty and has never experienced an array join failure of this type, the following messages are displayed (and logged):

If the array is properly configured, the following messages are displayed (and logged):

The script also creates an execution log file in the same folder from which it was run. If you followed the instructions, it will be found as c:\tmgtools\cleartmgrptsrvrs.log.

The Gory Details

Here’s what happens in this scenario…

When a server joins an array; whether this is the first server or the last, the array join wizard process performs several actions that make the new server known to the EMS and any array members that may already exist. One of these actions is to review the target array’s reporting configuration. If no server reference is found there, the array join wizard creates one, adds the new server’s UUID to it and moves along with the remainder of the array join actions.

If the array reporting configuration does include a server reference, the array join wizard tries to locate that server in the array’s server collection. If the defined reporting server cannot be found in the array servers collection, the “0x80070002” error is returned and the wizards halts; displaying the error results to the user.

When the array join wizard encounters a recoverable failure, it is designed to roll back (remove) all changes made in the target array up to the point of failure so as to leave the array in a clean state.

The core problem is caused because the array join wizard does not clean up the reporting server configuration during the rollback process.

Summary

This article describes only one cause and resolution for this sort of problem. It may actually be caused by any number of array management actions that (for any number of reasons) fail to complete and roll back successfully. In cases where the script fails to resolve array join failures to empty arrays that produce the same error message, it may actually be simpler to remove the empty array and replace it with a new array.

Author
Jim Harrison, Program Manager, AAG

Reviewers
Dima Datsenko, SDET, TUBT
Bala Natarajan. Sr Support Escalation Engineer