How to monitor events logged by another computer or cluster


Or – How to use the <AllowProxying> XML item.

 

When you monitor the event log in OpsMgr, there is some built in security that the agent modules perform.  Normally this wont affect you, except for unique situations where events are logged from/by another computer, or in the case of MS Clusters, where events get logged by the virtual cluster computer name.

For instance – look at this Application event log on a SQL 2-node cluster:

 

image

 

The nodes are SQL1CLN1 and SQ1CLN2.  The virtual clustered SQL instance is SQL1V1.  As you can see – I have events logged by the node, and sometimes the cluster virtual.

 

When we target event based monitoring to something generic, like the “Windows Server Operating System” class, this will require that the principal name of the instance of the class, matches the Logging Computer column of the event in the log.  If it does not – this is considered a security risk as a spoof, and we will not alert or match on the event datasource criteria.

Sometimes this is good, other times it is not desired, and we want our event rules to match on any computer.

 

Another common scenario is when we are targeting a cluster aware class, like “SQL 2005 DB engine” where the principal name of the instance will be the cluster virtual name.  We could have the same problem – if the event is logged by the node, and not the cluster virtual computer name.  In R2 (and in a hotfix for SP1) – a fundamental change was made, to always allow matching on the event, if the workflow targets a clustered instance, and the event is logged by the virtual name, OR either node.  This is good and fixed a lot of issues with monitoring clustered applications like SQL.

 

The challenge you might face today – is when you are writing a custom event based workflow, but targeting a non-cluster aware class (like Operating System)

Let’s walk through some scenarios:

 

1.  If I target “Windows Server Operating System” and the event is logged by the Computer\Node name, I will match and get an alert.

2.  If I target “Windows Server Operating System” and the event is logged by the Cluster Virtual Computer name, or some other computer logging the event remotely, I will NOT match and will not alert.

3.  If I target a cluster aware class, like “SQL DB Engine” and the event is logged by the Cluster Virtual Computer name, I will match and get an alert.

4.  If I target a cluster aware class, like “SQL DB Engine” and the event is logged by the Computer\Node name, I will match and get an alert PROVIDED I am running R2, or SP1 with http://support.microsoft.com/kb/959865

 

The remainder of this article will cover scenario #2 above.

 

There is an XML tag you can add to your rules which will basically ignore the Logging Computer field in an event based workflow… called “<AllowProxying>”.  Using this tag will ignore the computername and match on the event criteria.

Lets have a look at a Rule example:

Below is a very simple rule which targets Windows Server Operating System, and looks for an event ID 101 in the application log:

 

<Rule ID="kevinsmp.BadEvent101" Enabled="true" Target="Windows!Microsoft.Windows.Server.OperatingSystem" ConfirmDelivery="true" Remotable="true" Priority="Normal" DiscardLevel="100">
  <Category>Alert</Category>
  <DataSources>
    <DataSource ID="DS" TypeID="Windows!Microsoft.Windows.EventProvider">
      <ComputerName>$Target/Host/Property[Type="Windows!Microsoft.Windows.Computer"]/NetworkName$</ComputerName>
      <LogName>Application</LogName>
      <Expression>
        <SimpleExpression>
          <ValueExpression>
            <XPathQuery Type="UnsignedInteger">EventDisplayNumber</XPathQuery>
          </ValueExpression>
          <Operator>Equal</Operator>
          <ValueExpression>
            <Value Type="UnsignedInteger">101</Value>
          </ValueExpression>
        </SimpleExpression>
      </Expression>
    </DataSource>
  </DataSources>

 

Now – to add the ability to “ignore” the computer name – we just need to add the <AllowProxying> tag:

 

<Rule ID="kevinsmp.BadEvent101" Enabled="true" Target="Windows!Microsoft.Windows.Server.OperatingSystem" ConfirmDelivery="true" Remotable="true" Priority="Normal" DiscardLevel="100">
  <Category>Alert</Category>
  <DataSources>
    <DataSource ID="DS" TypeID="Windows!Microsoft.Windows.EventProvider">
      <ComputerName>$Target/Host/Property[Type="Windows!Microsoft.Windows.Computer"]/NetworkName$</ComputerName>
      <LogName>Application</LogName>
      <AllowProxying>true</AllowProxying>
      <Expression>
        <SimpleExpression>
          <ValueExpression>
            <XPathQuery Type="UnsignedInteger">EventDisplayNumber</XPathQuery>
          </ValueExpression>
          <Operator>Equal</Operator>
          <ValueExpression>
            <Value Type="UnsignedInteger">101</Value>
          </ValueExpression>
        </SimpleExpression>
      </Expression>
    </DataSource>
  </DataSources>

 

Voila!

In the authoring console – here is how it will appear, and you can gain access to the XML by clicking the “Edit” butting circled in red.  Or you can edit the XML directly of the MP.

 

image

 

The reason this works – is because rules by default will use the “Microsoft.Windows.EventProvider” data source, from the Microsoft.Windows.Library management pack.  If we examine that data source, we can see that “AllowProxying” is an optional parameter that we are allowed to pass to this data source:

 

image

 

So – essentially – for any workflow which leverages the “Microsoft.Windows.EventProvider” data source, we can pass this “Allow Proxying” as a parameter, and this solution will work.

 

That covers most simple rules.  What about monitors?

 

Now – if you are interested in making this work for monitors – a much deeper understanding of what is going on under the covers is required.  The last part of this article will discuss the modules that make this work, and how to use them to make this work for a monitor.

 

To understand the Microsoft.Windows.EventProvider datasource – this is a composite datasource of two modules:  The Microsoft.Windows.BaseEventProvider DS, and a condition detection of System.ExressionFilter (which is the expression you will pass in the rule for the event criteria).  You can see this in the Authoring console if you load the Microsoft.Windows.Library:

image

 

The Microsoft.Windows.BaseEventProvider is a Native Datasource module, the fundament core of event monitoring that we are using.  Here we can see THIS is the module that defines which parameters are available:

 

image

 

 

When we create a unit monitor using the UI, we are actually running a wizard, which based on the type of monitor we choose, will pick an existing “MonitorType” that is appropriate for what we are trying to do with the monitor.  A unit monitor is used to monitor a particular aspect of an object such as an application, device or service.  A unit monitor is defined to be of a particular unit monitor type.  We can create new MonitorTypes, or choose from existing ones.

 

In this example – I will create a new monitor, which will look for an Event 102 in the application log.  It will be a timer reset monitor.  What this means, is that as I create this monitor using the Authoring console or the Operations Console UI, is because I chose to make it a Timer Reset monitor – it automatically chose a good existing MonitorType… which in this case is “Timer Reset” which is the display name for Microsoft.Windows.SingleEventLogTimer2StateMonitorType from the Microsoft.Windows.Library MP.

 

If I try to use the same XML tag we used for rules above – we find it will not work.  Here is a screenshot from the Authoring console – after I added <AllowProxying>true</AllowProxying> to the XML manually:

 

image

 

Why does this work for rules and not for monitors?

It actually will work fine for both.  The issue is – that the existing MonitorType of Microsoft.Windows.SingleEventLogTimer2StateMonitorType did not expose ALL of the optional parameters allowed for the event module.  Whoever created this MonitorType likely didn't think anyone would need this very often. 

Let me explain.  If we look at that MonitorType of Microsoft.Windows.SingleEventLogTimer2StateMonitorType in the authoring console, by loading the Microsoft.Windows.Library, this is MonitorType is composed of the following Member Modules:

 

image

 

The DataSource is the BaseEvent Provider we discussed above.

The FilterCondition is the expression which contains the event criteria we want to match on.

The TimerCondition is the timer module to reset the state of the Monitor.

The OnDemandReset is a probe action added to support on demand recalculation when the timer rest occurs.

 

The problem with THIS MonitorType, if we look at the data source configuration, is that the author only allows for ComputerName and LogName to be passed as variables from the Monitor to the MonitorType.  Therefore – when we try and add <AllowProxying>true</AllowProxying> to the XML manually, we get that nasty popup!

 

image

 

Hopefully now – this makes more sense… AllowProxying is not a valid parameter… only <ComputerName> and <LogName> will be accepted by this MonitorType, even though the datasource module we are using would accept it.

Bummer.

So what can we do?  Simple – create a NEW MonitorType, just like this one, and add the optional parameter that we want for <AllowProxying> to be accepted by the MT, and passed to the datasource.  We can copy a lot of the basic stuff we see in the Microsoft Windows Library.

 

***Note:  I recommend creating your custom MonitorTypes into a custom Library MP that you create… and then sealing it.  This will be your Library of MonitorTypes that you can reuse in lots of different Management Packs you are authoring.  Just like ALL MP’s reference the Microsoft Windows Library by default, your custom MP’s will also reference your “Custom.Authored.Library.mp” (or whatever you want to name it)

 

So let’s begin. 

Create a new MP named “Custom.Authored.Library” in the Authoring Console.

Browse to the Type Library screen, and create a new Composite Monitor Type.

I will give mine the ID of Custom.Authored.Library.SingleEventLogTimer2StateMonitorType

For the Display Name – I will call mine “Timer Reset with Proxy”

image

 

For the States – I will use the same states as seen in the built-in monitor type:

image 

 

For the Member Modules – I will add in the exact modules and configuration that the built-in module used:

Add the Data Source of Microsoft.Windows.BaseEventProvider, and give it the name “DataSource”

image

 

Click “Edit” on the configuration page for the data source, and here we need to input the variables we will allow to be passed, directly to the XML.  We will add in:

<ComputerName>$Config/ComputerName$</ComputerName>
<LogName>$Config/LogName$</LogName>
<AllowProxying>true</AllowProxying>

Here is how it will look when done:

image

 

This is the KEY difference in our custom MonitorType.

 

Next – add the filter condition for System.ExpressionFilter, and name it “FilterCondition”

 

image

 

At this point we should edit the expression to accept a variable the same way the original MonitorType does.  Unfortunately the authoring console does not allow this because it does error checking and does not understand the variable… so we will skip this step and do that manually in XML later.  Just click OK to accept the default expression configuration for now.

 

Next, add the TimerCondition,

 

image

 

At this point we should edit the expression to accept a variable the same way the original MonitorType does.  Click Edit, and change the default of <TimerWaitInSeconds>1</TimerWaitInSeconds> to the variable <TimerWaitInSeconds>$Config/TimerWaitInSeconds$</TimerWaitInSeconds>

You will get a popup error that the Authoring Console does not like accepting the variable when is expects an integer.  Just click Ignore and then OK.

 

Next, add the OnDemandReset System.PassThroughProbe:

image

 

No configuration is necessary for this module, so just click OK.

 

When you are done we should have 4 modules added, and we will only have to come back and revisit the System.ExpressionFilter as we were not able to configure it in the Authoring console.

image

 

Move on to the “Regular” tab, and configure the two states exactly as the built-in MonitorType:

 

image

image 

 

Skip the OnDemand configuration for now – as we will also handle this part in XML.

 

image

 

Move on to the “Configuration Schema” tab, and add configure it exactly as the original MonitorType.

This is all case sensitive so make sure you add it EXACTLY as seen below:

 

image

 

Lastly – go to the Options tab, and change the MonitorType to Public.

At this point, you should be able to save your Monitor Type.

Now – we will go into XML to configure the last couple items that can only be handled in XML due to shortcomings in the Authoring console.

Save your MP, and then edit it with Notepad, Notepad++, or your XML editor of choice.

 

Change the Filter condition expression from:

<ConditionDetection ID="FilterCondition" TypeID="System!System.ExpressionFilter">
  <Expression>
    <SimpleExpression>
      <ValueExpression>
        <XPathQuery>XPathQuery</XPathQuery>
      </ValueExpression>
      <Operator>Equal</Operator>
      <ValueExpression>
        <XPathQuery>XPathQuery</XPathQuery>
      </ValueExpression>
    </SimpleExpression>
  </Expression>
</ConditionDetection>

TO:

<ConditionDetection ID="FilterCondition" TypeID="System!System.ExpressionFilter">
  <Expression>$Config/Expression$</Expression>
</ConditionDetection>

 

ADD the following XML:  (this should be placed after </RegularDetections> and BEFORE </MonitorImplementation>

<OnDemandDetections>
  <OnDemandDetection MonitorTypeStateID="TimerEventRaised">
    <Node ID="OnDemandReset" />
  </OnDemandDetection>
</OnDemandDetections>

 

SAVE the MP.

SEAL the MP with your own custom key, and keep that key safe.  You can now reference this MonitorType in your own custom Monitors, using the Authoring console.  If you need other MonitorTypes that also use an event datasource, go through this same process, add these

 

Now – when you create a new monitor, you must use the Authoring Console to use your custom monitortype.  In your custom monitor, add a reference to your custom library.  Next, create a New > Custom Unit Monitor.  On the configuration tab – you will be able to select your custom MonitorType.

The only thing that is a bit more painful about using these custom monitortypes – is that you will not be able to use a nice GUI for creating the event expression criteria.  You will have to write in the expression in XML using the Edit button in the Authoring console.

 

The default expression just gives a little example:

<Expression>
  <SimpleExpression>
    <ValueExpression>
      <XPathQuery>XPathQuery</XPathQuery>
    </ValueExpression>
    <Operator>Equal</Operator>
    <ValueExpression>
      <XPathQuery>XPathQuery</XPathQuery>
    </ValueExpression>
  </SimpleExpression>
</Expression>

Here is a good sample expression using Event ID and Event Source:

<Expression>
  <And>
    <Expression>
      <SimpleExpression>
        <ValueExpression>
          <XPathQuery Type="UnsignedInteger">EventDisplayNumber</XPathQuery>
        </ValueExpression>
        <Operator>Equal</Operator>
        <ValueExpression>
          <Value Type="UnsignedInteger">101</Value>
        </ValueExpression>
      </SimpleExpression>
    </Expression>
    <Expression>
      <SimpleExpression>
        <ValueExpression>
          <XPathQuery Type="String">PublisherName</XPathQuery>
        </ValueExpression>
        <Operator>Equal</Operator>
        <ValueExpression>
          <Value Type="String">TEST</Value>
        </ValueExpression>
      </SimpleExpression>
    </Expression>
  </And>
</Expression>

If you have trouble nailing the right expression – you can create a dummy monitor using the Authoring console and built-in monitor type, and then grab the expression from there.

Comments (2)

  1. Anonymous says:

    Outstanding information, thank you very much for this!

  2. Damir says:

    regarding …… Or – How to use the <AllowProxying> XML item.

    Thanks for this Kevin …really helpful… how on earth did you figure it out 🙂

    just a thougth … wouldnt it also work if I change

    $Target/Host/Property[Type="MicrosoftWindowsLibrary6172210!Microsoft.Windows.Computer"]/NetworkName$

    to

    $.$

    ?

Skip to main content