Alerting on Events – waiting for a specific amount of time to pass


 

Consider the scenario -  you want to monitor the event logs for a specific event, however, this event has a tendency to “storm” or log hundreds of events in a short time window.  Not a good condition for a monitoring system, as you can quickly overwhelm the system, nor do you want hundreds or thousands of alerts for a single condition.

 

The traditional approach to this would be to enable “Alert Suppression” which will increment a repeat counter on the alert.  This has a few negative effects:

1.  You still overwhelm the monitoring system, as you have to write this incremented counter to both the OpsDB and the DW.  Although this is not as expensive and creating multiple individual alerts, it still has significant impact.

2.  You will only get a notification on your FIRST alert.  All subsequent alerts will increment the counter, but you will never get another email/ticket on this again, as long as the original alert is still open.

 

Another approach – is to use a consolidator condition detection.  This is similar to the solution I provided here:  https://blogs.technet.microsoft.com/kevinholman/2014/12/18/creating-a-repeated-event-detection-rule/

The different, however, is instead of waiting for a specific “count” of events to fire in a specific time window, this example will do the following:

  1. Wait for the event to exist in the event log.
  2. Start a timer upon the first event, then wait for the timer to expire
  3. Create an alert for the event(s), no matter if there was a single event or thousands of events in the timed window.

 

The XML is fairly simple for this.  We will have the following components:

  1. Event datasource  (Microsoft.Windows.EventProvider)
  2. Consolidation Condition Detection  (System.ConsolidatorCondition)
  3. Alert Write Action   (System.Health.GenerateAlert)

 

Here is the datasource:   we simply look for event ID “123”

 

<Rule ID="Demo.AlertOnConsolidatedEvent.Event123.Alert.Rule" Enabled="true" Target="Windows!Microsoft.Windows.Server.OperatingSystem" ConfirmDelivery="true" Remotable="true" Priority="Normal" DiscardLevel="100"> <Category>Alert</Category> <DataSources> <DataSource ID="DS" TypeID="Windows!Microsoft.Windows.EventProvider"> <ComputerName>$Target/Host/Property[Type="Windows!Microsoft.Windows.Computer"]/NetworkName$</ComputerName> <LogName>Application</LogName> <Expression> <SimpleExpression> <ValueExpression> <XPathQuery Type="UnsignedInteger">EventDisplayNumber</XPathQuery> </ValueExpression> <Operator>Equal</Operator> <ValueExpression> <Value Type="UnsignedInteger">123</Value> </ValueExpression> </SimpleExpression> </Expression> </DataSource> </DataSources>

 

Here is the condition detection.  Notice there is no counting condition, simply the timer window, where my example uses 30 seconds.

 

<ConditionDetection ID="CD" TypeID="System!System.ConsolidatorCondition"> <Consolidator> <ConsolidationProperties> </ConsolidationProperties> <TimeControl> <WithinTimeSchedule> <Interval>30</Interval> <!-- seconds --> </WithinTimeSchedule> </TimeControl> <CountingCondition> <CountMode>OnNewItemNOP_OnTimerOutputRestart</CountMode> </CountingCondition> </Consolidator> </ConditionDetection>

 

And finally – a simple write action to generate the alert:

 

<WriteActions> <WriteAction ID="WA" TypeID="Health!System.Health.GenerateAlert"> <Priority>1</Priority> <Severity>1</Severity> <AlertMessageId>$MPElement[Name="Demo.AlertOnConsolidatedEvent.Event123.Alert.Rule.AlertMessage"]$</AlertMessageId> <AlertParameters> <AlertParameter1>$Data/Count$</AlertParameter1> <AlertParameter2>$Data/TimeWindowStart$</AlertParameter2> <AlertParameter3>$Data/TimeWindowEnd$</AlertParameter3> <AlertParameter4>$Data/Context/DataItem/EventDescription$</AlertParameter4> </AlertParameters> </WriteAction>

 

When I fire off a LOT of Event ID 123 events:

eventcreate /T ERROR /ID 123 /L APPLICATION /SO TEST /D "This is a Test event 123"

image

 

I only get a single, consolidated Alert, after the 30 second time window expires:

 

image

 

I will attach the entire MP example here:

 

https://gallery.technet.microsoft.com/SCOM-Alerting-on-Events-930464cc


Comments (0)

Skip to main content