SNMP Trap monitoring with SCOM 2012 R2


 

There are several really good blogs out there which document some of the struggles and workarounds with getting SNMP TRAP monitoring to work.  I’ll reference some of them at the bottom, but this will be mine to document what worked, what didn’t, and some MP authoring examples that don’t exist anywhere else on the web.

In order for SCOM 2012 to accept traps from network devices, there is a laundry list of configuration prerequisites.  Lets just jump in.

First clarification (and update to this blog post) – There is NO NEED to install or configure the SNMP service on the management server.  We don’t use the SNMP stack.  In fact – if the SNMP TRAP service is running (it does not by default) then only thing I need to do is make sure it is disabled! 

There are LOTS of blogs out there saying the SNMP service is required, and quite frankly – it isn't…. the SCOM SNMP trap listener uses a MonitoringHost.exe process and does not interact with the Windows SNMP stack.  I initially posted about configuring this because ALL of the blog resources pointed to the need for the SNMP service, but in my testing this is totally not required.  Thanks to Mihai for setting me straight again.  Smile

 

Open Services.  The only thing we need to make sure is that the SNMP Trap service should be disabled.  There were lots of incorrect postings about the trap service early on.  Notice – the SNMP service is not even installed here:

image

 

 

Next, I create a network monitoring resource pool in SCOM.  I want to lock down which management server I will use to receive traps, and to lock this to a single management server for any troubleshooting. 

image

 

 

I’ll assign a single MS to this pool for the purposes of trap reception:

image

 

 

Next – in order to accept traps from any network device – that device MUST be discovered and use SNMP as an access mode.  See:  http://blogs.technet.com/b/kevinholman/archive/2011/07/21/opsmgr-2012-discovering-a-network-device.aspx

In this example, I will discover a Linux System running Ubuntu, because that is easy to generate SNMP traps.  Remember – SCOM 2012 filters out any SNMP traps and will not discover a Windows Computer as a SNMP device, therefore using a Windows Computer and trapgen is not suitable for testing. 

Here is a good walkthrough on setting up the Ubuntu Server:  http://www.it-slav.net/blogs/2009/02/05/install-and-configure-snmp-on-ubuntu/

Once you install and config SNMPD on Ubuntu, you can send test traps from Ubuntu to the SCOM 2012 server from the command line.  First – we need to discover the Ubuntu server:

Under SCOM Administration, Run As accounts, create a new account, of type Community String.  I will be using “public” for mine but you can use anything configured for your network, as long as you used this on the Ubuntu server.   Remember, this is a password, and it is case sensitive.

imageimage

image

 

 

Distribute the account to your Management server and/or resource pool for network monitoring.

Next, create a discovery for the Ubuntu server:

image

 

 

Make sure this device discovers correctly:

image

 

 

Next up – we need a “catch all” rule in SCOM to collect all received traps, and collect them as events.  This will be helpful for troubleshooting.  If you are going to receive a large number of traps, you will probably want to turn this off later.

New SNMP Trap event collection rule:

image

 

Give your rule a name, and target class MUST BE “Node”

image

 

For the configuration of the SNMP trap provider – you can leave this blank – it will then match on all:

image

 

Click Create.

Now – create an event view in the Console in our management pack for SNMP monitoring, and configure it to show data related to Node:

image

 

 

Nice – now we need to export our MP to XML, and do a very specific edit.  This is outlined at:   http://blogs.msdn.com/b/wei_out_there_with_system_center/archive/2014/02/15/opsmgr-customizing-the-snmp-trap-collection-rule-for-all-snmp-version-traps.aspx

When we receive traps by default – we ONLY accept taps in the same SNMP version as we discovered the device in.  This is not really optimal, because some devices send SNMP v1 traps but are discovered as an SNMP v2 device.  We can strip this filter from the XML by finding our rule, and removing the lin defining “Version”:

image

 

 

Delete that entire line containing Version to remove this as a filter for the rule:

image

 

 

Now increment the XML version of the MP in the Manifest section, and re-import the MP.  This will limit confusion and SNMP version issues down the road.

 

Lets’ take a recap:

1.  We ensured SNMP Trap service is not running on the SCOM server.

2.  We created a resource pool and community string run as account for network monitoring.

3.  We discovered our network device that will send us SNMP Traps.

4.  We created a “catch all traps” event rule to collect all SNMP traps received as events.

 

 

Next up – we should send some SNMP traps from the Ubuntu server to the SCOM server:

The command line to send a SNMP V2 trap would be something like this:

sudo snmptrap -v 2c -c public 10.10.10.11 "" .0.1.2.3.4 0.1.2.3.5 s "string one" 0.1.2.3.6 int 12345

That is basically stating to run the snmptrap command, with a SNMP V2 version, public community string, ip address of the remote SCOM server, two double quotes to encapsulate “uptime” value (a require parameter), and then a trap OID, which I just made up as .0.1.2.3.4.  I then added a string and integer varbind.

The command to send an SNMP V1 trap would look something like this:

sudo snmptrap -v 1 -c public 10.10.10.13 '1.2.3.4.5.6' 10.10.10.44 6 99 '' 0.1.2.3.4 s "string one" 0.1.2.3.5 s "string two"

10.10.10.13 is destination, 10.10.10.44 is the source (Ubuntu server)  All the rest is just incremental varbind strings.

If we did everything right, we should be able to see this TRAP on a network trace.

If you have problems – DON’T COPY and PASTE the lines above.  Copying often pastes in the wrong “” and – characters, so if you Linux session throws errors with you SNMPTRAP syntax – type it out manually.

I will use Wireshark to see this.  Install wireshark on the SCOM server, and start it up.  You can create a filter to ONLY see traffic to and from your Ubuntu server by using this filter:

ip.addr==10.10.10.44     (or whatever your Ubuntu server IP is)

image

 

Send a trap from your Ubuntu server or network device, and you should see it register in Wireshark:

image

 

If everything went perfectly – you will also see this collected as an event in SCOM:

image

 

 

Click the “View Event Data” to see how SCOM breaks down each dataitem:

I will format this out a little better to make it more understandable:

<EventData> <DataItem type="System.SnmpData" time="2015-02-03T11:44:01.6480756-06:00" sourceHealthServiceId="BA5D0090-EDAA-EFF4-10BB-3882E6B231E0"> <Source>10.10.10.44</Source> <Destination>127.0.0.1</Destination> <ErrorCode>1</ErrorCode> <Version>2</Version> <SnmpVarBinds> <SnmpVarBind> <OID>.1.3.6.1.2.1.1.3.0</OID> <Syntax>67</Syntax> <Value VariantType="19">489831453</Value> </SnmpVarBind> <SnmpVarBind> <OID>.1.3.6.1.6.3.1.1.4.1.0</OID> <Syntax>6</Syntax> <Value VariantType="8">.1.2.3.4.0</Value> </SnmpVarBind> </SnmpVarBinds> </DataItem> </EventData>

 

This is important, if we want to manipulate the data, or create further filters/condition detections.  Note the SNMPVarBinds – these are essentially event parameters in an SNMP Trap event. (more on this later)

Next up – lets create a generic Alert rule for SNMP traps, which will help us in testing and troubleshooting future more specific alert rules.

Create an Alert generating SNMP trap rule:

image

 

Leave OID blank:

image

 

Configure your alert page like this:

image

 

Notice in the Alert description – you can gather each data item associated with a specific OID like an event parameter.  This will help us create better, filtered alerts later. 

Save the new Alert Rule.  Don’t forget to export the MP and delete the line with the <Version> filter in it for V1 vs V2 SNMP traps.

Also – create a new view in our MP, for Alerts.  Scope it to “Node” class so we will see open alerts for SNMP traps.

 

Now – when we create a trap on the Ubuntu server, we should collect it as an event, AND alert on it.  This time, on the trap command line, lets add two more OID’s to the trap to simulate a more realistic trap:

snmptrap –v 2c –c public 10.10.10.13 “” .1.2.3.4.0 .1.2.3.4.0.1 int 12345 .1.2.3.4.0.2 int 67890

image

 

Event Collected:

image

 

Alert Generated:

image

 

Notice in the alert – the alert description variables we added previously help us interpret which SNMP Varbind (parameter) is which.  In the collected event, it breaks down like this:

image

 

 

You can also express this based on the OID in an Xpath query, such as:

SnmpVarBinds/SnmpVarBind[OID=".1.2.3.4.0.1"]/Value

See more on this at:  https://technet.microsoft.com/en-us/library/hh563870.aspx

 

Ok, next up – lets build an alert rule based on a specific OID.

Create a new alert rule, and this time lets input the specific OID of our test trap, “.1.2.3.4.0”

image

image

 

And customize the alert description as before:

image

 

Now – when we send a trap based on OID .1.2.3.4.0 this workflow will alert, but a trap based on .1.2.3.4.1 will not:

image

 

However, what if we want to get even more specific?  What if the OID of a trap is generic in nature, and there is data inside a trap that we want to alert ONLY when that data inside a trap matches specific criteria?

In this case, we need to add a condition detection to a rule.  I could not find any examples of how to do this on the web, and for some reason we don’t have a built in datasource which allow for SNMP trap data and a simple expression filter.  We could create an advanced composite datasource for this, and reuse it, but I want to show something much simpler, which still allows you to author the rule in the UI and just make a simple tweak.

So, in this case, we will want to make an alert when the OID is .1.2.3.4.0, and when SNMPVarbind3 (the 3rd parameter down) Equals 12345.

Start by creating the EXACT same rule we did before with a new name:

image

image

 

 

But on the alert description – let’s get a little sexier:

You can use the flyout on the right to create these:

image

image

 

 

Now – we need to export this MP to XML and do a manual edit.

Increment the version in the manifest.

Find the rule with all the sexy alert description stuff we just wrote (hint – look in the write action section of the rule)

We need to insert the following code in between the <Datasource> and the <WriteAction> sections.  Here is the code:

<ConditionDetection ID="FilterSpecificVarbind" TypeID="System!System.ExpressionFilter"> <Expression> <SimpleExpression> <ValueExpression> <XPathQuery Type="String">EventData/DataItem/SnmpVarBinds/SnmpVarBind[3]/Value</XPathQuery> </ValueExpression> <Operator>Equal</Operator> <ValueExpression> <Value Type="String">12345</Value> </ValueExpression> </SimpleExpression> </Expression> </ConditionDetection>

Rule before:

<Rule ID="Demo.SNMP.monitoring.AlertOnTrapOidAndVarbind" Enabled="true" Target="SystemNetworkManagementLibrary71102260!System.NetworkManagement.Node" ConfirmDelivery="false" Remotable="true" Priority="Normal" DiscardLevel="100"> <Category>Alert</Category> <DataSources> <DataSource ID="DS" TypeID="SystemNetworkManagementLibrary71102260!System.NetworkManagement.SnmpTrapEventProvider"> <IP>$Target/Property[Type="SystemNetworkManagementLibrary71102260!System.NetworkManagement.Node"]/SNMPAddress$</IP> <Version>$Target/Property[Type="SystemNetworkManagementLibrary71102260!System.NetworkManagement.Node"]/SNMPVersion$</Version> <OIDProps> <OIDProp>.1.2.3.4.0</OIDProp> </OIDProps> <EventOriginId>$Target/Id$</EventOriginId> <PublisherId>$Target/Id$</PublisherId> <PublisherName>Snmp Event</PublisherName> <Channel>SnmpEvent</Channel> <LoggingComputer /> <EventNumber>1501</EventNumber> <EventCategory>5</EventCategory> <EventLevel>10</EventLevel> <UserName /> <Params /> </DataSource> </DataSources> <WriteActions> <WriteAction ID="Alert" TypeID="Health!System.Health.GenerateAlert"> <Priority>1</Priority> <Severity>0</Severity> <AlertName>SNMP Alert Rule - Alert on Specific OID and Specific Varbind</AlertName> <AlertDescription>We received an SNMP Trap from {0} with a critical event with ID {1} The source IP address is {2} The primary owner is {3} The vendor is {4} The location is {5}</AlertDescription> <AlertOwner /> <AlertMessageId>$MPElement[Name="Demo.SNMP.monitoring.AlertOnTrapOidAndVarbind.AlertMessage"]$</AlertMessageId> <AlertParameters> <AlertParameter1>$Target/Property[Type="SystemNetworkManagementLibrary71102260!System.NetworkManagement.Node"]/sysName$</AlertParameter1> <AlertParameter2>$Data/EventData/DataItem/SnmpVarBinds/SnmpVarBind[3]/Value$</AlertParameter2> <AlertParameter3>$Target/Property[Type="SystemNetworkManagementLibrary71102260!System.NetworkManagement.Node"]/SNMPAddress$</AlertParameter3> <AlertParameter4>$Target/Property[Type="SystemNetworkManagementLibrary71102260!System.NetworkManagement.Node"]/PrimaryOwnerContact$</AlertParameter4> <AlertParameter5>$Target/Property[Type="SystemNetworkManagementLibrary71102260!System.NetworkManagement.Node"]/Vendor$</AlertParameter5> <AlertParameter6>$Target/Property[Type="SystemNetworkManagementLibrary71102260!System.NetworkManagement.Node"]/Location$</AlertParameter6> </AlertParameters> <Suppression /> <Custom1 /> <Custom2 /> <Custom3 /> <Custom4 /> <Custom5 /> <Custom6 /> <Custom7 /> <Custom8 /> <Custom9 /> <Custom10 /> </WriteAction> </WriteActions> </Rule>

Rule after:

 

<Rule ID="Demo.SNMP.monitoring.AlertOnTrapOidAndVarbind" Enabled="true" Target="SystemNetworkManagementLibrary71102260!System.NetworkManagement.Node" ConfirmDelivery="false" Remotable="true" Priority="Normal" DiscardLevel="100"> <Category>Alert</Category> <DataSources> <DataSource ID="DS" TypeID="SystemNetworkManagementLibrary71102260!System.NetworkManagement.SnmpTrapEventProvider"> <IP>$Target/Property[Type="SystemNetworkManagementLibrary71102260!System.NetworkManagement.Node"]/SNMPAddress$</IP> <Version>$Target/Property[Type="SystemNetworkManagementLibrary71102260!System.NetworkManagement.Node"]/SNMPVersion$</Version> <OIDProps> <OIDProp>.1.2.3.4.0</OIDProp> </OIDProps> <EventOriginId>$Target/Id$</EventOriginId> <PublisherId>$Target/Id$</PublisherId> <PublisherName>Snmp Event</PublisherName> <Channel>SnmpEvent</Channel> <LoggingComputer /> <EventNumber>1501</EventNumber> <EventCategory>5</EventCategory> <EventLevel>10</EventLevel> <UserName /> <Params /> </DataSource> </DataSources> <ConditionDetection ID="FilterSpecificVarbind" TypeID="System!System.ExpressionFilter"> <Expression> <SimpleExpression> <ValueExpression> <XPathQuery Type="String">EventData/DataItem/SnmpVarBinds/SnmpVarBind[3]/Value</XPathQuery> </ValueExpression> <Operator>Equal</Operator> <ValueExpression> <Value Type="String">12345</Value> </ValueExpression> </SimpleExpression> </Expression> </ConditionDetection> <WriteActions> <WriteAction ID="Alert" TypeID="Health!System.Health.GenerateAlert"> <Priority>1</Priority> <Severity>0</Severity> <AlertName>SNMP Alert Rule - Alert on Specific OID and Specific Varbind</AlertName> <AlertDescription>We received an SNMP Trap from {0} with a critical event with ID {1} The source IP address is {2} The primary owner is {3} The vendor is {4} The location is {5}</AlertDescription> <AlertOwner /> <AlertMessageId>$MPElement[Name="Demo.SNMP.monitoring.AlertOnTrapOidAndVarbind.AlertMessage"]$</AlertMessageId> <AlertParameters> <AlertParameter1>$Target/Property[Type="SystemNetworkManagementLibrary71102260!System.NetworkManagement.Node"]/sysName$</AlertParameter1> <AlertParameter2>$Data/EventData/DataItem/SnmpVarBinds/SnmpVarBind[3]/Value$</AlertParameter2> <AlertParameter3>$Target/Property[Type="SystemNetworkManagementLibrary71102260!System.NetworkManagement.Node"]/SNMPAddress$</AlertParameter3> <AlertParameter4>$Target/Property[Type="SystemNetworkManagementLibrary71102260!System.NetworkManagement.Node"]/PrimaryOwnerContact$</AlertParameter4> <AlertParameter5>$Target/Property[Type="SystemNetworkManagementLibrary71102260!System.NetworkManagement.Node"]/Vendor$</AlertParameter5> <AlertParameter6>$Target/Property[Type="SystemNetworkManagementLibrary71102260!System.NetworkManagement.Node"]/Location$</AlertParameter6> </AlertParameters> <Suppression /> <Custom1 /> <Custom2 /> <Custom3 /> <Custom4 /> <Custom5 /> <Custom6 /> <Custom7 /> <Custom8 /> <Custom9 /> <Custom10 /> </WriteAction> </WriteActions> </Rule>

 

Now import this MP back in, and test your traps.

When we send a trap that contains both the OID and the data in Varbind3, we should get a very specific alert, with a nice Alert Description pulling from data within the trap:

image

 

Summary:

1.  We ensured the SNMP Trap service is not running on the SCOM server

2.  We created a resource pool and community string run as account for network monitoring.

3.  We created an Ubuntu server to send test SNMP traps from.

4.  We discovered our network device that will send us SNMP Traps.

5.  We created a “catch all traps” event rule to collect all SNMP traps received as events.

6.  We modified our rules to strip the SNMP <Version> filter from them so we can receive traps of any version.

7.  We demonstrated using Wireshark to validate that SNMP traps are received by the network interface.

8.  We create an alert rule to alert on all traps, and modified the alert description to show the SNMP Varbinds and how they related to data collected in a SNMP trap event.

9.  We created an alert rule for specific OID’s in a SNMP Trap.

10.  We created an alert rule that matches on OID and specific dataitems within the SNMP Trap data, with a rich alert description

 

I will attach my example MP to this article.

Demo.SNMP.monitoring.xml.zip


Comments (94)

  1. MWeterings says:

    Please remove/ignore the previous post, i hope this is more readable;

    Hi Kevin,

    Great article! I do have a question though, despite your very clear description.

    Let's say i want to create an alert based on multiple snmpVarBind conditions. I was hoping to achieve this by separating my expressions by inserting AND in between, but that's probably not how it should work. SCOM also refuses to import the modified MP. Any
    ideas?

    This is what i tried;

    [code]

    EventData/DataItem/SnmpVarBinds/SnmpVarBind[7]/Value

    Equal

    99

    AND

    EventData/DataItem/SnmpVarBinds/SnmpVarBind[8]/Value

    Equal

    99

    AND

    EventData/DataItem/SnmpVarBinds/SnmpVarBind[9]/Value

    Equal

    99

    [/code]

  2. Kevin Holman says:

    Are you receiving V1 or V2 traps from the device? What do the traps look like? Does agent-addr contain the same IP address as the discovered device? Have you tested with traps from a Linux box like the article shows?

  3. Kevin Holman says:

    SNMP on the OS should have ZERO bearing on any of this - SCOM does not use the SNMP service on the OS. To accept V1 traps, the rule taking action needs to have the tag removed from the XML, and you need to ensure the community string is the same as the
    discovered community string. If they are different, add that community string in your list of available community strings, or hard code the string for that rule.

  4. Kapil Dham says:

    Astonishing but true, I see the value under Simple Network Management Protocol in Wireshark for the trap that were received called snmp.community: SNMP_trap but the device has the Read community string totally different. Maybe that seems to be the issue.
    I added the comunity string within SCOM and distributed it to the resource pool but still no cigar. Suggestions?

  5. MWeterings says:

    Ah, of course, that makes sense. Better yet, SCOM imports the management pack now without any errors, Thanks Kevin!

  6. Kapil Dham says:

    So finally made it work!!! took an out of the box approach and changed the community string of the device in question to match the SNMP community string returned by traps as per Wireshark and it piped straight into SCOM console. I had to rediscover the
    device with new community string, make change to the Alert rule by removing SNMP version dependency as per Kevin's suggestion by editing the xml file for the MP and worked just fine. Now the device is discovered as a SNMP v2 device, SCOM is able to receive
    traps as V1 and right into SCOM console.

    Thanks Kevin for a great write-up.

    Regards,

    Kapil Dham

  7. Kevin Holman says:

    Did you make sure you disabled/aren't running the SNMP TRAP service?
    Is there a chance that you discovered the device using a specific community string - but the trap is sent using a different community string? Look at the community in wireshark.

  8. Kevin Holman says:

    @Martijn -

    Yes, you just need to use the correct expression syntax for an AND statement in XML. There are many examples of this, you could make one just by making an event rule with two event ID's to see an example of the XML. Here is a sample.

    https://msdn.microsoft.com/en-us/library/ee692979.aspx
     
     
     
    <Expression>
      <And>
        <Expression>
          <SimpleExpression>
             <ValueExpression>
               <XPathQuery Type="String">EventData/DataItem/SnmpVarBinds/SnmpVarBind[3]/Value</XPathQuery>
             </ValueExpression>
             <Operator>Equal</Operator>
             <ValueExpression>
               <Value Type="String">12345</Value>
             </ValueExpression>
          </SimpleExpression>
        </Expression>
        <Expression>
          <SimpleExpression>
             <ValueExpression>
               <XPathQuery Type="String">EventData/DataItem/SnmpVarBinds/SnmpVarBind[4]/Value</XPathQuery>
             </ValueExpression>
             <Operator>Equal</Operator>
             <ValueExpression>
               <Value Type="String">foo</Value>
             </ValueExpression>
          </SimpleExpression>
        </Expression>
      </And>
    </Expression>
     
     

  9. Kevin Holman says:

    @SajMo -

    That's good - with wireshark we know we are receiving the trap. Couple things to check:

    1. If multiple management servers - did you lock down the network monitoring pool to a single MS and that is where traps are going?
    2. Are you receiving traps from a discovered device, that shows up as ICMPSNMP access mode?
    3. Did you strip out the line of code for in the rule to make sure that if we are getting a V1 trap we will still log it?

  10. Kevin Holman says:

    Should work fine. It has to be some data in the trap that is root cause.... most of the time it came down to version (fixed by removing this from the rule) or IP address (agent-addr in trap must match discovered device IP)

  11. Tero Ilenius says:

    Great article! Thanks!

    -Tero

  12. Yeah, thanks Mihai! I was more than confused reading your initial post in my feed reader and was asking myself "man, how did I make that work so often?" 🙂

  13. SajMo says:

    Excellent article. Not worked for me. Wirehshark shows SNMP traps but SCOM 2012 still not picking them up even though catch all trap rule set.

  14. SajoMo says:

    Only got one MS, but went ahead and created the resource pool anyway. Yes device discovered as ICMPSNMP. Yes to Q.3.

  15. SajMo says:

    SNMP Service and SNMP Trap are both disabled. Community name is the same as per discovery and what appears in Wireshark.

  16. SajMo says:

    V1 traps are being received in Wireshark. Agent-addr ip has now been added as network device, still no traps showing in SCOM. I am trying to get SCOM to monitor EMC RPA's which use linux Net-SNMP agent 5.1.

  17. Kapil Dham says:

    Hello Kevin, I have a similar case like SajMo. The traps are getting received by SCOM 2012 R2 instance we have for few devices that are discovered as network device. Alerting is working fine too. The issue is that we discovered a new device that got discovered
    as SNMP v2. It sends trap using SNMP v1. I forced the discovery to be as a v1 device and it got discovered successfully.

    The issue is that the device is sending traps to one MS in our NMPool and I confirmed the same using WireShark. But not getting into SCOM. Created empty UID alert rule, collection rule and they capture all other traps from other devices but not from this one.
    One interesting thing you wrote is that to ensure the community string that the device is using should be same as one used to discover device. I can see the trap in wireshark but no reference to the community string within it. Any idea where can I find that
    info in a sample trap? will be a big help.
    Thanks. Just to let you know, you are considered a rockstar within my support team!!! Keep up giving back to the community.
    Regards,
    Kapil Dham

  18. SajMo says:

    SCOM discovers my EMC RPA cluster as snmp v2. I know the traps come in as v.1 only as set like that by storage guy. No option for v.2 only v1 and v.3. Is there a way I can configure SNMP on OS to look for v.1 only traps ? SNMP service now running.

  19. Frank says:

    Hi Kevin,

    several month ago I've published how to receive snmp traps from a MS windows based vCenter server on SCOM 2012 R2. Maybe this is worth a try:

    http://www.fricnet.de/scom2012r2-trapreceiver/scom2012r2-trapreceiver.html

    regards,

    Frank

  20. Roy Beard says:

    Thanks so much for this post Kevin! Great job as usual.

  21. Sean Tompkins says:

    Troubleshooting steps -
    1. Is the trap making it to the SCOM server?
    2. Is the originating IP in the "network" discovered section?
    3. Is your rule/monitor targeting "Node"?
    4. Is the SNMP trap received a different version than the device you discovered? (V1, V2) -- see editing out the version filter above
    5. Is SCOM listening on the standard SNMP trap port? (If the SNMP service or Trap service is running, likely THEY are grabbing the trap)
    6. Is the originating IP a Windows machine? SNMP will be dropped from any Windows machine.
    7. This one needs tested... but my experience was that once I had installed the SNMP service, even if it was disabled, I still needed to update the Traps and Security sections of the SNMP service properties - I *think* the security section there was blocking
    some SNMP traffic.

  22. Martijn Weterings says:

    Hi Kevin,

    Great article! I do have a question though, despite your very clear description.

    Let's say i want to create an alert based on multiple snmpVarBind conditions. I was hoping to achieve this by separating my expressions by inserting AND in between, but that's probably not how it should work. SCOM also refuses to import the modified MP. Any
    ideas?

    This is what i tried;

    EventData/DataItem/SnmpVarBinds/SnmpVarBind[7]/Value

    Equal

    2

    AND

    EventData/DataItem/SnmpVarBinds/SnmpVarBind[8]/Value

    Equal

    99

    AND

    EventData/DataItem/SnmpVarBinds/SnmpVarBind[9]/Value

    Equal

    1

    Thanks,
    Martijn

  23. Ben Lambert says:

    THANK YOU! I can't tell you how many hours I spent trying to get this to work using the various methods found on the internet. I stumbled across this today and had it working in less than an hour.

    If you feel like revising, if you could put in something about HOW to export and import the management pack, that threw me for a little bit (but I found it finally, duh).

  24. Any Linux agent for SNMP to work says:

    Hi Kevin, May we know if there is a need to install any SCOM agent to a linux server for SNMP to work and be discovered in SCOM? The network devices are found in the network devices in SCOM already.

  25. Niki4 says:

    any third party tool to convert MIBs into a MP?

  26. Niki4 says:

    Perfect i will look into that product.

  27. SV says:

    Kelvin, I have 300 Servers to be monitored from SCOM 2012 R2 all are Win2012 R2 and pls can you tell me will it be possible to install the SNMP Services on all the Server and generate the alerts using SNMP, the main reason for this is that the client want
    to integrate this will BMC Remedy and they want to configure this using SNMP Trap only..

  28. Kevin Holman says:

    @SV -

    My name is Kevin, not Kelvin. This is important in a dialogue. 🙂

    For some reason - the product group decided not to allow SNMP monitoring or traps from another Windows Computer object. These are filtered out. I have heard this is hackable - and you could change this, but I don't have the info handy.

  29. Andrew says:

    I need to raise SNMP Trap alerts on 100+ OIDs all with ".1.3.6.1.4.1.3167.99.1.1.xxxx"

    Is it possible to create only one rule to capture all the traps to raise alerts?

  30. SV says:

    Dear Kevin, Apologize for the typo error in your name.. Extremely Sorry for that.. 🙂 🙂 Thanks a lot for your immediate response on my post... 🙂 🙂

  31. Kevin Holman says:

    @Andrew - yes - see the "catch all traps" rule - see if the OID is sent as a varbind - then create an expression.

    @SV - no worries. 🙂

  32. Vinayak_Giri says:

    Hi Kevin,

    Another great article. Your blogs always helps me. Thank you.

    I have a situation. On one my gateway server SNMP service is used by another application. So I have to keep it running.

    I have noticed the event id 12300 –
    Log Name: Operations Manager
    Source: Health Service Modules
    Date: 8/19/2015 7:43:56 PM
    Event ID: 12300
    Task Category: Health Service Module
    Level: Error
    Keywords: Classic
    User: N/A
    Computer: *********
    Description:
    Error: The SNMP Trap port is already in use by another program. Please uninstall or disable other SNMP services.

    One or more workflows were affected by this.

    And then custom rules are failing...
    does this SNMP services used by another application causing this?

  33. Pacman says:

    Hi Kevin, do you know if these steps can be used to receive SNMP traps from another/different monitoring system running on Windows server since we won't be able to discover it as a network device?

    Regards,
    MA

  34. Vinayak_Giri says:

    @Pacman,

    Kevin already answered it -- "For some reason - the product group decided not to allow SNMP monitoring or traps from another Windows Computer object. These are filtered out."

  35. Marlon says:

    Hi All -

    I am going to monitor a UPS device (Eaton 93E UPS), there's already available management pack and successfully imported in SCOM. The UPS device was discovered with SNMP v1. Windows SNMP service was uninstalled and SNMP Trap service was already disabled. We
    have setup the device to send trap and verified that is was also sending SNMP Trap under v1 but unfortunately SCOM did not received any traps. We are using network monitor to trace the trap and found that it was received by the MS but did not see on SCOM.

    Do i miss anything here?

    Thanks!

    Marlon

  36. Please ignore my post above because I have found the root cause of the problem. The protocol that the device used to communicate back to the management server is port 162 and we have manually assigned it with port 161.

    Thanks Kevin for this very helpful blog! More power!

  37. CyrAz says:

    Hi Kevin,

    I've add two new MS to my management group, added them to my "network monitoring" resource pool but they don't seem to enable their "snmp trap receiver" feature...
    I can see using wireshark that they receive snmp traps, but their answer is "port unreachable".
    "Netstat -na | findstr 162" shows that they are not listening on port 162.

    Am I missing something here? For now, the traps are still received by the old MS but I'll have to take it offline sooner or later, so I need my new MS to be able to receive traps as well.

    Any idea?

    Thanks!

  38. Kevin Holman says:

    @Cyraz -

    Try changing your network device discovery to be run by a new MS. I believe only the MS that discovers the SNMP device will listen for traps.

  39. CyrAz says:

    Done that in the first place as it was part of my "migration process", and re-ran the discovery rule.
    I just did it again I do have events showing that this has been taken into account such as 12121/12127/12003/12004 (topology cleared/proceeding to discover/probing/probing completed), but it's still not listening on udp 162...

  40. CyrAz says:

    (thanks for this incredebly fast answer, though!)

  41. Dan_IT says:

    Hi Kevin,
    Nice work with this custom SNMP Alerting, exactly what I needed 🙂

    I followed to the letter what you did and it worked like a charm, i'd like to push this further, and I am wondering if you could help.
    We are trying to monitor Tripp Lite batteries, and if work well.. i.e:
    Trap filtered to ti OID of the batteries, limiting the "spam" on the trap, which is perfect.
    Made and alert like you explained and when we unplug the battery we get and alert
    $Data/EventData/DataItem/SnmpVarBinds/SnmpVarBind[5]/Value$ >> it Reports "On Battery", meaning the power is off, which is great

    Now I have noticed there is and Alert Suppression button on the Alerting page, so my question is this.
    How can I have this alert resolve automatically when I plug back the power... It reports "On Utility power" with the same varbind.
    I dunno much about xml programming unfortunately, so I don't know how to capture in a variable and pass it in the expression to suppress the alert, or the steps required, if any, to modify the MP for it to work.

    Thanks

  42. justin says:

    What about the HP Storage , Proliant and Blade MP?
    In my environment they all throw out SNmp and require snmp config of the trap service. How do you set those up now?

  43. sam says:

    Excellent Article. Its worked for me. One doubt, when the alert triggered, with in 1 minute it is moving to closed state. What is the reason for that.

  44. Anonymous says:

    I previously wrote about using the network device monitoring in SCOM here:
    http://blogs.technet.com/b

  45. Harun Akboga says:

    If the Device is not discoverable (like DELL TPAM), it does not allow incomming PING or SNMP, how to manually add the device anyway? even if they are not discoverable.
    Harun Akboga

  46. Md Nur Hossain says:

    Thanks. Nice Article.

  47. Cloud-Ras says:

    Kevin, i'm a fan of your blogs 🙂

  48. Bob Compono says:

    This is a really old entry and my question is a little off topic, but what if you have SNMP trap messages being sent from an application running on another server, in my case AIX and Linux? Is there no way to simply set up SCOM to listen for traps from
    a particular address? The server doesn't run SNMP as it's not needed just to send trap messages, so it can't be discovered by SCOM.

  49. Kevin Holman says:

    @ Bob Compono -

    SCOM unfortunately must discover an object in order to receive traps from it. I dislike this requirement, and if I ever find time, I think we can add network devices on our own via script based discovery. I was planning on showing an example of that, using
    scripts to read a CSV file, and discover network objects for just this very purpose, bypassing the interrogation method that is built into SCOM.

  50. Hi,

    I want to send Snmp trap from SCOM 2012 R2 to Nagios, how can I do that .I think the above code is for receiving trap to SCOM.
    Can anybody say me about how to send snmp trap from SCOM to Nagios, Please.
    Thank You.

  51. Ash says:

    Great article but I have a few questions if anyone can help answer these that would be great.

    1. Will this work with systems using SNMP V3 ?

    2. How is this setup where the devices to be monitored by behind a firewall and the management servers are located in different DMZ ? Can gateway be used ?

    1. Kevin Holman says:

      @ Ash -

      Great article but I have a few questions if anyone can help answer these that would be great.

      1. Will this work with systems using SNMP V3 ?

      SNMP V3 devices are supported.

      2. How is this setup where the devices to be monitored by behind a firewall and the management servers are located in different DMZ ? Can gateway be used ?

      Yes, in a firewall scenario where a management server does not have SNMP access to a device, we support using Gateways in network monitoring resource pools to manage firewalled devices.

      1. Henrik Andersen says:

        When you say V3 devices are supported. Does that apply to V3 traps? Can only find articles that says it does not. Will V1/V2 traps work with devices discovered with snmp V3

        1. Mace says:

          Hi Henrik,
          SCOM 2012 does not and even SCOM 2016 does not (MS Request No.117021415314872). You can discover and monitor v3 devices, but you cannot catch their (v1/v2/v3) traps with SCOM 201x. Maybe we should vote that up to make it happen one day:
          https://systemcenterom.uservoice.com/forums/293064-general-operations-manager-feedback/suggestions/12332553-support-for-snmp-v3-traps

  52. Woodall77 says:

    To potentially save some poor fellow an hour or two, if you can't reimport your MP after making the Condition changes, check your references to ensure that the "System" Alias is present against the System Library. I was saving my test Rules to general use, full of crap, Management Pack, which had that MP in already as a reference, but with a different Alias

  53. Sumi says:

    Hi Kevin,
    Thanks for this article,
    i couldn't find the class "node" when creating the rules for the SNMP event trap.
    Instead i could see the below in the rule target when search for Node.

    Dell Sled Server Node
    Dell Sled Server Node with Operatinf system
    Dell sled Server Node without Operating system
    Dell Windows Sled Server Node

    1. Sumi says:

      Please ignore my previous post. I could find it.
      I searched in view common target instead of view all targets.
      Thanks.

  54. lchua says:

    is there any link that show how to insert expression for SCOM 2007R2 same as SCOM 2012 in the SNMP rule?

  55. lchua says:

    i have force discover windows based server as v1 to receive traps from vcenter.. i receive traps from both vcenter but only 1 of them is published into SCOM. both are using the same rule and the traffics are capture via wireshark. anywhere i need to check??

  56. Mark Derouen says:

    Ever needed to create a report on all the SNMP events you collect? I need to and cant seem to display any data. Trying to run a report on all traps received for a specific node. Tried from inside the console running a report on events where I target the object node and limit it to the specific device I want to report on.. "No go", so then I search on the trap receiver server for an event log named SNMPEvent, and that doesn't exist, so I search all the common logs for event ID 1501, and you guessed it, nada. Kind of stumped here.

  57. SergIT says:

    Hello, Kevin!

    I don't understand this step:
    - Now increment the XML version of the MP in the Manifest section, and re-import the MP. This will limit confusion and SNMP version issues down the road.

    Could you explain?

    1) Export MP to XML
    2) Delete /
    3) Re-import MP - on this step error, that MP already present in system.

    1. SergIT says:

      Fuf it works! After few days and install test ubuntu 🙂

      In my questions:
      1) go administration-> management pack -> export created MP
      2) delete "version" as in article
      3) change "version" in manifest 1.0.0.0 -> to 1.0.0.1
      4) Import xml to SCOM

      Thanks Kevin! Good job!

  58. Dexter says:

    Hi Kevin, Is it possible for using SNMP trap to calculate HP storage capacity? If yes, could you please help with the procedure?

  59. Vance says:

    Fantastic article 🙂

    One issue I did run into was the listener didn't start quickly on port 162. I left it over night and it started, also found a reboot will start it if you cannot wait.

  60. Vance says:

    Hi Kevin, I've been trying to setup a trap based monitor to capture if an appliances services are down or not (has 3 services that run). I have it working as a rule without any issues but when I try and recreate it via a monitor I have no success at all.

    Below is a trap captured in SCOM using your method in this post.

    Object Identifier Syntax Value
    .1.3.6.1.2.1.1.3.0 Timeticks 1167094645
    .1.3.6.1.6.3.1.1.4.1.0 Oid .1.3.6.1.4.1.23365.10000.0.1051
    .1.3.6.1.4.1.23365.10000.7.1.1.1 Integer 1

    I'm trying to set up the monitor using the using the expression below while having the "First SnmpTrapProvider" blank.

    SnmpVarBinds/SnmpVarBind[OID=”.1.3.6.1.4.1.23365.10000.7.1.1.1”]/Value

    Any insight as to why this isn't working?

    1. Kevin Holman says:

      If you are using the UI - to create the monitor - there is a bug:

      https://social.technet.microsoft.com/Forums/systemcenter/en-US/282b61e6-69d9-4bd4-ba14-a9d43a40d093/snmp-integer-value-convert?forum=operationsmanagergeneral

      It defaults to string - and you cannot use an integer based expression with string - so you have to do some XML edits (see link above)

      1. Vance says:

        Thanks for your reply Kevin.

        I did try changing it but it still doesn't trigger with the OID.

        SnmpVarBinds/SnmpVarBind[OID=”.1.3.6.1.4.1.23365.10000.7.1.1.1”]/Value

        Equal

        1

        SnmpVarBinds/SnmpVarBind[OID=”.1.3.6.1.4.1.23365.10000.7.1.1.1”]/Value

        Equal

        0

        I did get it working by using "SnmpVarBinds/SnmpVarBind[3]/Value" but this doesn't allow me to monitor all 3 services individually.

  61. Mark Derouen says:

    I have SCOM as a trap receiver for a while now. We introduced some re-provisioned switches which were monitored from another system, and that system was trying to connect to these switches after we changed the community name on the switches. Over 1 million failed auth traps. We have fixed the other system to quit trying to authenticate to the switches and now I would like to clear out all the 1 million + traps. Any idea on how to do this since the health service is the trap receiver? I cant find anywhere these traps are stored. Clearing the health service state folder didn't work, Any ideas?

    1. Kevin Holman says:

      Not sure I understand.

      Where is the "problem"? What do you want to delete?

      1. Mark Derouen says:

        I had over 1 million traps sitting in my view for All SNMP Traps. I corrected this by tuning the days to keep events in Administration\Settings\Database Grooming.

      2. Jesty says:

        Hi Kevin,

        We have SNMP monitoring configured and we would like to exclude few alert triggered with few keywords. Is this achievable?

  62. Jesty says:

    We have created a SNMP rule to monitor autosys jobs based on OID's. We want to exclude few jobs from alerting from these rules. Is there an option to exclude few job failures from alerting?

    1. Kevin Holman says:

      Yes - you would simply add criteria where varbind (n) doesnt contain or doesnt equal "foo" in your rule.

      1. Jesty says:

        Thanks Kevin 🙂

  63. Erik says:

    Hi Kevin,

    Thank you for your clear step-by-step guide. I followed every step in detail, including testing from an ubuntu box. Small recap:
    - Ubuntu box is discovered as an snmp device, using snmp version 2c.
    - Wireshark shows incoming version 2c traps on the management server
    - Even though both traps and discovery are using the same version I modified filtered out the version requirement using the suggested export/edit/import method
    - I verified the snmp trap service is disabled on the management server
    - 'netstat -ano' shows UDP port 162 and a process ID, 'get-process' shows the corresponding process is monitoringhost
    However, the traps don't appear in the console. To me it seems like the rule doesn't process the incoming traps.

    What to do now?

    1. alexander says:

      Exactly the same problem. The port listens to the process, but there are no events.
      What to do?

      1. hnnycs says:

        Same problem here, any idea Guys?

  64. Adrien says:

    Thanks for the article very helpful.
    I succeeded to receive SNMP traps on my SCOM consol. Those traps are generated by a third party application on a Linux systems. The application is configured to send SNMP traps to 1 of my management server. But what will happens if this management server fall down? I've got two others in the same resource pool but they don't received the SNMP traps...

    Any idea how I can manage the failover?

    Thanks in advance for your help.

    Adrien

    1. Kevin Holman says:

      In order to have high availability - you must send traps to all IP addresses of all servers in the resource pool.

      1. Adrien says:

        Hi Kevin,

        Thanks for your quick reply. Yes it's what I thought, we will try this.

        Adrien

      2. Adrien says:

        Hi Kevin,

        After some tests it seems that SCOM only catch SNMP traps from the MS which discovered the device. I've got tree management servers in the same "Network ressouce pool" with the community string distributed on each one. But only one received the traps...

        Any idea ?

        Thanks in advance !

        1. Kevin Holman says:

          I already commented on this above.

          It is NOT the MS that discovers the device. It is the MS that HOSTS the device. If you have three MS in a pool hosting network devices, you must send the traps to ALL THREE Management servers, because of load balancing the network device object could be hosted on any of the management servers at any given time.

  65. Jonathan DeLong says:

    Hi Kevin, wondering if you can give me some help on an SNMP trap issue.

    I've done these steps I believe, but I also followed your other guide on setting up a Windows Server as an SNMP device.

    I did those steps, and I have my VEEAMONE server setup as an SNMP node in SCOM 2012 R2. I have several other SNMP devices as well. Confirming through Wireshark, I am getting the traps sent to my SCOM management server, but I am not getting the alerts showing up in my event view, but I am with the other SNMP traps I send.

    Help!! SCOM is not in production, but we are probably migrating towards it soon and need to be able to get these alerts sent from VEEAM ONE

  66. Hsanchez says:

    Hi! need a example that SCOM2012 Sends Traps to another system.... Is it Possible?

  67. John Sandman says:

    Hello,

    will this work with SCOM 2016 on Windows server 2016?

    Thanks

  68. hnnycs says:

    Hello,

    Will This Work on Windows Server 2016 (SCOM2016 aswell)?

    Thanks

  69. Thomas Frimo says:

    I have tried this, but it does not work. I even tried your MP. I can see in NETMONITOR that the traps go with the same community string, but the rule does not show any of them. 🙁

    1. Kevin Holman says:

      Did you place ONLY a single management server in the resource pool used for network monitoring?

      1. Thomas Frimo says:

        Yes, I did, but doesnt work. I have two MGMT, but only one in resource pool. 🙁

      2. hnnycs says:

        I got it, problem was with local firewall. Now its working perfectly!!
        Thanks

    2. Thomas Frimo says:

      Yes

Skip to main content