Antigen 9.1 Hotfix Rollup 3 and Performance Monitor

While investigating an issue where mail was queuing in the Exchange Information Store, we discovered an issue that affects customers running Antigen 9.1 Hotfix Rollup 3 when there are performance monitoring tools such as Perfmon, Perfwiz, and the MOM client running.  This issue will manifest itself as mail queuing (and never un-queueing), particularly immediately after the store is restarted.  In this particular instance, we were seeing this happen when we failed from one cluster node to another.  This could also occur in non-cluster environments and it could occur if scanjobs are restarted for other reasons (such as scan timeouts).

Additionally, you may see entries in ProgamLog.txt similar to the following:

"ERROR: scanjobs.cpp::ConfigScanJobFile(): AddNewScanJob() Failed 0x80030021"
"ERROR: scanjobs.cpp::CheckScanJobs(): ConfigScanJobFile() failed. hr[0x80030021]"

"ERROR: Unexpected, RetrieveScanJobIdentifier could not find the index"
"ERROR: Problems retrieving ScanJob identifier from RegisterMonitor"
"ERROR: antigenvsapi.cpp::VSAPINavigatorThread(): RegisterMonitor() returned 8000ffff"

You may also see instances where you open the Antigen Administrator console and scanjobs are not visible.

The root cause of this is a regression in the Antigen performance counters DLL that results in Antigen services being unable to access the configuration information for scanjobs; thus, when the server is in this state, scanning processes cannot be started and the admin console cannot access scanjob configuration information.

These symptoms will not occur in all instances.

Recommendations:

If a server is having this issue, you should be able to resolve the immediate issue by stopping all applications that are performing performance monitoring and restarting Exchange services.

If you are running services/applications that gather performance data on your Exchange Server with Antigen 9.1 Hotfix Rollup 3, you can mitigate this in the short-term by disabling Antigen performance counters.  The following steps will disable those counters:

  1. At c:\program files\microsoft antigen for exchange\
  2. Enter command: antigenpmsetup -uninstall
  3. You will also have to restart any application that loads performance counters.  Rebooting the server will accomplish this; however, short of that, you can run 'tlist -m antigenpmdll.dll' to get a list.  (Tlist is part of the debuggers package.)

This will be resolved in Rollup 4 when it is released.  After Rollup 4 is available, we recommend re-enabling Antigen performance counters by running 'antigenpmsetup -install'.