When using connectors to forward alerts in Operations Manager 2007, in certain situations such as an alert storm (defined as a large number of alerts being generated in a very short period of time) there may be alerts that are not forwarded via a connector. When this occurs, these alerts will never be forwarded and will remain in a "New" state.
This can occur if there is an alert storm and the connector processes alerts in batches (e.g. 100 at a time). If there are too many alerts to be processed within 5 seconds (the default value) of when they were inserted into the database, they will be skipped by connector processing and never be reprocessed.
To resolve this issue you can increase the amount of time allowed for the alerts to be processed. There is a registry value called AlertSubscriptionWatermarkLatencyMilliseconds that can be set to increase the latency value such that when the query executes to collect the alerts for forwarding, the alerts that have a timestamp less than or equal to the current time minus the latency value (the current default is 5000 milliseconds).
This registry value does not exist by default but it can be created and will be read by the Root Health Service.
The registry key is: SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\AlertSubscriptionWatermarkLatencyMilliseconds
The default is 5000.
Note: When this key is added, the Health Service on the Root Management Server must be restarted to implement the change.
The value should only be increased, never decreased. It should be incremented (for example, adding 5000 milliseconds at a time) and tested to make sure the value is sufficient to not skip alerts. Setting it too high can cause performance issues by causing the connector to look at too many alerts at the same time unnecessarily.
For the latest information regarding this issue please see the following Knowledge Base article:
KB2277825 – Alerts may not get forwarded as expected via a connector in System Center Operations Manager 2007
J.C. Hornbeck | System Center Knowledge Engineer