Solution: The ACS forwarder in Operations Manager 2007 may frequently log connection and disconnection events

Here's an issue I came across recently that I thought would be worth a mention here on our blog.  The issue is one where the ACS forwarder service shows frequent connections and disconnections on Windows XP POS computers but it could happen on any ACS Forwarder if the accounts being used or the permissions are configured incorrectly.  But first a little background:

The ACS Forwarder is a separate service (AdtAgent.exe) called the Operations Manager Audit Forwarding Service.  It is deployed automatically with the Operations Manager 2007 agent but must be explicitly enabled to initiate security log collection. The Operations Manager Audit Forwarding Service listens to the local Windows Event Log service and processes security events, in near real-time, then forwards the events to a central collector. During failover and connectivity outages the local Security log acts as the Forwarding Service queue.

After you install the ACS Collector and database you can then remotely enable this ACS Forwarding Service on agents through the Operations Manager 2007 console by running the Enable Audit Collection task.  By default the ACS Forwarding service runs using the Network Service Account, but since in Windows XP POS it does not have read permission on the security event log you may see frequent connection and disconnection events logged in the Operations Manager event log on the ACS Collector server:

Log Name: Operations Manager
Source: AdtServer
Date:
Event ID: 4628
Task Category: None
Level: Information
Keywords: Classic
User: NETWORK SERVICE
Computer:
Description: An Audit Forwarder connected.
Name: <>
Address: <>
Port: 266
DbId: 2
Value: 1

Log Name: Operations Manager
Source: AdtServer
Date:
Event ID: 4629
Task Category: None
Level: Warning
Keywords: Classic
User: NETWORK SERVICE
Computer: <>
Description: An Audit Forwarder disconnected.
Name: <>
DbId: 2
Value: 1
Reason: Forwarder initiated disconnect or connection broken.

Log Name: Operations Manager
Source: AdtServer
Date:
Event ID: 4628
Task Category: None
Level: Information
Keywords: Classic
User: NETWORK SERVICE
Computer: <>
Description: An Audit Forwarder connected.
Name: <>
Address: <>
Port: 2314
DbId: 2
Value: 1

For detailed troubleshooting you can enable debugging for the ACS Forwarder and gather more information. To enable verbose logging on a local Forwarder, create a new registry value and restart the AdtAgent service as shown below.  Just remember to turn off debugging when completed.

1. Browse to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\AdtAgent\Parameters

2. Create DWORD value = TraceFlags and set it to a decimal value of 524420.

3. Restart AdtAgent service.

4. Review log, C:\Windows\Temp\AdtAgent.log.  You'll probably see something similar to this:

[20100420 143003,721][Error ]EventLogReader::Open(0x1520): 0x00000522
[20100420 143003,721][Error ]OpenReaders(): EventReader::Open(Security) returned 0x00000522.
[20100420 143003,721][Warning]AgentRun(): Transmit() returned 0x00000522
[20100420 143003,721][Info ]AgentRun(): Disconnecting after Transmit() returned 0x00000522.

Error Number 0x00000522 resolves to "A required privilege is not held by the client"

5. Turn off debugging when done by removing above created DWORD and restarting the AdtAgent service.

Resolution

There are two resolutions to fix this issue.

1. Grant Read permission on the security event log for the Network Service account.  For information on this see the following Knowledge Base article:

KB323076 - How to set event log security locally or by using Group Policy in Windows Server 2003

or

2. Change the “Log on As” value for the ACS Forwarding service from “Network Service” to “Local System”.

Hope this helps,

Dipesh Kumar Rathod | System Center Support Engineer