Antigen 9.1 Hotfix Rollup 3 and Performance Monitor

Hello, this is Neil Carpenter. I’m an Escalation Engineer on the support side of our business and I work with Antigen and Forefront Security for Exchange Server and SharePoint.

We have been working on a hotfix rollup for Antigen 9.1 that will include a fix to help alleviate issues some of our customers have seen when using performance tools with Antigen. The hotfix will be ready soon, but we wanted to give our enthusiastic blog audience a heads up while we're still working on finalizing everything. When Hotfix Rollup 4 is available, this information will be cleaned up and included in a KB article.

Here are the details:

While investigating an issue where mail was queuing in the Exchange Information Store, we discovered an issue that affects customers running Antigen 9.1 Hotfix Rollup 3 when there are performance monitoring tools such as Perfmon, Perfwiz, and the MOM client running. This issue will manifest itself as mail queuing (and never un-queuing), particularly immediately after the store is restarted. In this particular instance, we were seeing this happen when we failed from one cluster node to another. This could also occur in non-cluster environments and it could occur if scanjobs are restarted for other reasons (such as scan timeouts).

Additionally, you may see entries in ProgamLog.txt similar to the following:

"ERROR: scanjobs.cpp::ConfigScanJobFile(): AddNewScanJob() Failed 0x80030021"
"ERROR: scanjobs.cpp::CheckScanJobs(): ConfigScanJobFile() failed. hr[0x80030021]"

"ERROR: Unexpected, RetrieveScanJobIdentifier could not find the index"
"ERROR: Problems retrieving ScanJob identifier from RegisterMonitor"
"ERROR: antigenvsapi.cpp::VSAPINavigatorThread(): RegisterMonitor() returned 8000ffff"

You may also see instances where you open the Antigen administration console and scanjobs are not visible.

The root cause of this is a regression in the Antigen performance counters DLL that results in Antigen services being unable to access the configuration information for scanjobs; thus, when the server is in this state, scanning processes cannot be started and the admin console cannot access scanjob configuration information.

These symptoms will not occur in all instances.

Recommendations:

If a server is having this issue, you should be able to resolve the immediate issue by stopping all applications that are performing performance monitoring and restarting Exchange services.

If you are running services/applications that gather performance data on your Exchange Server with Antigen 9.1 Hotfix Rollup 3, you can mitigate this in the short-term by disabling Antigen performance counters. The following steps will disable those counters:

1. At c:\program files\microsoft antigen for exchange\

2. Enter command: antigenpmsetup -uninstall

3. You will also have to restart any application that loads performance counters. Rebooting the server will accomplish this; however, short of that, you can run 'tlist -m antigenpmdll.dll' to get a list. (Tlist is part of the debuggers package.)

This will be resolved in Rollup 4 when it is released. After Rollup 4 is available, we recommend re-enabling Antigen performance counters by running 'antigenpmsetup -install'.