Exchange 2007 Synthetic Transactions against clustered Mailbox Servers may stop working

Are you using the new Exchange 2007 Management Pack?

Are you configuring the CAS and/or Mailflow synthetic transcactions?

Are any of your target Mailbox servers clustered?

If so, then you’ll want to verify that these transactions are actually running…and be sure to check it at least 12 hours after setting it up.

Here’s a very quick synopsis of the issue:

Symptom:

1. You create either CAS or Mailflow synthetic transactions where the target mailbox server is clustered
2. The synthetic transactions are discovered and appear to be running fine…objects are healthy and we see perf data
3. Within 12 hours, you notice that we are no longer seeing performance data from the transaction.

Cause:

There is a discovery called “RMS Target Relationship Discovery” which runs every 12 hours on the RMS.  The purpose of this discovery is to create a relationship between the Exchange Synthetic Transaction object and the target Mailbox server.  The purpose of the relationship is so that when the Mailbox server is put into Maintenance Mode, the Transaction object will also go into MM (otherwise, the transactions would continue to run and would fail and go into an unhealthy state).  The problem is that when the target mailbox server is clustered, some core OpsMgr rules relating to cluster monitoring end up getting applied to the CAS server and essentially disabling the workflows that run these transactions.  I know that’s not an extremely technical explanation, but it’ll do for now.  Regardless, if the target mailbox server is clustered, the transactions will completely stop running within 12 hours.

Resolution:

To resolve this, we need to disable the “RMS Target Relationship Discovery” discovery and use Powershell to remove the discovered relationship from OpsMgr.

Here’s a visual look at the problem:

Use the wizard to create the transactions:

EXCAS.OpsMgr.net is my CAS server:

I am going to enable the Active Sync and OWA tests:

EX07a is a standalone Exchange server, EX1V1 is a clustered mailbox server:

The discovery defaults to running every 24 hours, I changed it to 5 minutes for testing:

After giving the discoveries time to run (or just restarting the Health Service on the CAS server to force it), I go to the CAS Synthetic Transaction State view in the console and see that the objects are discovered.  I have an object for each Transaction to each Mailbox server (pay no attention to the unhealthy states in the screen shot, they are unrelated to this issue).

Looking in the performance view, I can see that I am receiving performance data from the transactions to both Mailbox servers (I configured the transactions to run every 1 minute, default is every 15 minutes):

After the RMS Target Relationship discovery runs, we are no longer seeing data from the transaction to the clustered mailbox server (the red line).  Normally you would see this within 12 hours of the transactions being created…I sped it up to run sooner to reproduce it easily:

Also, the state of the transaction object will remain the same as it was before the RMS Target Relationship discovery ran (until the problem is fixed)….if it was Healthy then, it will stay Healthy….if it was Unhealthy, it will stay Unhealthy.

To resolve this we must do the following two things (both MUST be done):

1. Create an Override to disable the “RMS Target Relationship Discovery” discovery

Go to Authoring\Object Discoveries

Select “View all targets"

Scope to “Exchange 2007 CAS Connectivity”

Select the “RMS Target Relationship Discovery” object – Overrides – Override the Object Discovery – For all objects of class: Root Management Server:

Set an Override for Enabled = False:

Now that the override is disabled, we will need to remove the relationships that were discovered by it.  To do this, we’ll use the Powershell command Remove-DisabledMonitoringObject.  Note that you MUST set the Enabled=False override mentioned above in order for this to work (if the discovery is not explicitly disabled, the Powershell command will not remove the objects discovered by it).  For more info on the Remove-DisabledMonitoringObject command, see Jonathan’s blog on it here.

Just open the OpsMgr Command Shell and run Remove-DisabledMonitoringObject:

Once the relationship is removed, the Transaction starts running again:

Bottom Line:

1. If you use the Exchange 2007 Synthetic Transactions for CAS or Mailflow tests against a Clustered Mailbox server, the transactions will stop running within 12 hours.
2. To fix this, you must disable the “RMS Target Relationship Discovery” discovery and run the Remove-DisabledMonitoringObject Powershell cmdlet.