OpsMgr: Public release of the Alert Update Connector


 

The Alert Update Connector for SCOM 2012 is now public: 

http://www.microsoft.com/en-us/download/details.aspx?id=34783

 

The AUC was a tool that many large enterprises used in OpsMgr 2007, where they integrated OpsMgr alerts with an upstream alert system, ticketing, or incident management system.  It helped (tremendously) to offer the following solutions:

  • Create the ability to insert specialized data into custom fields on alerts from rules or monitors, after they are generated, for a ticketing system to interpret and use to assign to the correct queues or take special actions on.
  • Create a simple to use *filter* with a user interface – to quickly and simply pick and choose which alerts would be sent to a ticketing system, rather than relying on “All Critical” or “All Critical with High priority”.  Often times customers would have to generate WAY too many overrides to use standard alert as the filter for which alerts would go to a connected system.

The only way to do this without the AUC, was to write powershell scripts, or code and develop your own “pre-connector” service.

    ***Note: I don’t recommend that customers deploy the AUC unless you have a good business need for it.  It becomes a link in the chain of events to get in your incident management system, and could be one more thing to break.  However, for customers who are already using it, or for customers who didn’t realize this was available – it is an awesome tool.  It is being updated for OpsMgr 2012 to help those customers who depended on it transition into their 2007R2 > 2012 upgrade a little easier.

     

    Long term, System Center Orchestrator is going to be the solution for these tools, where Orchestrator runbooks can handle the alert modification, enrichment, and forwarding to an upstream system.

     

    Let’s have a deeper look.

     

    When you extract the MSI download – you have the following files:

    image

     

    AlertUpdateConnector.exe is the file used for the service that will be installed.

    AlertUpdateConnector.exe.config is an ACSII configuration file you edit for making changes.

    AlertUpdateConnector.rtf is a readme.

    AlertUpdateConnector.tmf is used for troubleshooting when needing to perform an ETL trace.

    AlertUpdateConnector.xml is a management pack for monitoring the AUC service.  (note – I don’t recommend you use this – better to create a service monitor for the service and add a recovery to restart it on your own – this MP could create too many state changes in some situations).

    ConnectorConfiguration.exe is the User Interface used to pick which rules and monitors to make changes to.

    TracingGuidsAlertUpdateConnector.txt is used for ETL tracing/troubleshooting.

     

     

    Installing the connector

     

    A good location to install the connector is any management server.  The connector is USUALLY pretty lightweight, so it isn’t a big concern.  It just needs to be installed on a server that also has the console components installed as we need to have access to the SDK.  We can point the connector at ANY management server after install, since all management servers run the SDK service.  This does not need to be pointed at the RMSe.

    We need to create a folder on this server, and copy all the files to it.  I like C:\Program Files\AlertUpdateConnector

    Follow the instructions VERY carefully when installing the connector.  Be methodical – ensure you are using the 64bit install, and path to InstallUtil.exe, and the correct .NET folder.  Here is my full command line:

    C:\Program Files\AlertUpdateConnector>c:\Windows\Microsoft.NET\Framework64\v2.0.50727\InstallUtil.exe AlertUpdateConnector.exe

    This step above installed the Service.  Open the Services.msc applet and find the service:

    image

    Change the logon credentials of the service to the SDK account, or any account that has SCOM admin rights.  DO NOT start the service yet.

    image

     

     

    The next step is to install the connector into SCOM.  Here is my command line for that:

    C:\Program Files\AlertUpdateConnector>AlertUpdateConnector.exe -InstallConnector

    You will now see a new connector show up in the UI:

    image

     

    Next up – we need to configure the connector config file.  Look in C:\Program Files\AlertUpdateConnector and open the AlertUpdateConnector.exe.config file in notepad.

    Under <appSettings> modify the following:

    <add key="RootManagementServerName" value="localhost" />

    You should change localhost to your management server name – ONLY IF you did not install the connector service on a management server.  It just needs to point to any server running the SDK (DAS) service.

    <add key="PollingIntervalInSeconds" value="10" />

    You shouldn’t change this in most cases.  We want the alert subscription module to grab these alerts quickly.

    <add key="ConfigurationFilePath" value="C:\Connector_Configuration.xml" />

    Modify the path to where you will store the Alert configuration file.  Mine looks like:

    <add key="ConfigurationFilePath" value="C:\Program Files\AlertUpdateConnector\Connector_Configuration.xml" />

     

    Next – we need to open the Alert Update Connector UI – and create an Alert Configuration file.  Run the ConnectorConfiguration.exe program located at C:\Program Files\AlertUpdateConnector.  For the Root Management Server Name – give it the name of any Management Server that runs the SDK, and hit connect.  This will take some time to fully populate all the rules and monitors in your management group.

    Choose File > Save Configuration > browse to the correct location and supply the name we used for the alert config file above:

    image

     

    Open this Connector_Configuration.xml file for editing in notepad.  Add in the ExcludeResolutionState="255" like you see in my example below.  This will keep our connector from re-opening alerts that auto-close quickly.

     

    <ConnectorConfig GlobalResolutionState="251" ExcludeResolutionState="255">
      < AlertSources />
    < /ConnectorConfig>

     

    Save the file.

    You can now start the Alert Update Connector service.  You will see events logged in the OpsMgr event log for any issues, or see a normal startup and connection logged.

    Once this is working – we only have a few more steps.

    We need to create a couple new resolution states.  One for all alerts that we process, but do not want to send across the connector.  Another – for all alerts that will be sent across the connector.  The AUC defaults to 251 for “processed” alerts, and we can use any resolution state ID for what gets used to send to the connector.  I like using 251, and 252.

    In the console – browse Administrator > Settings > Alerts.  See my graphic below for my examples – you can name these whatever you like:

    image

     

    Next – we need a subscription to subscribe to the alerts that we want the Alert Update Connector to inspect.  My preference here is to create a subscription to inspect ALL NEW alerts, regardless of their criteria.  This ensures that all alerts will either be set to a “Processed” or a “SendToConnector” resolution state.  This is important as this will impact you you subscribe to email notifications from within SCOM.  No longer will we include “New” resolution state for emails – because alerts will be changed to one of these custom resolution states very quickly after being created.

    In the console, Administration > Product Connectors > Internal Connectors.

    Open the properties of the Alert Update Connector.  Click Add – to add a new subscription.  Give it a name, like “Process All NEW Alerts”.  Scope it to all groups, all targets, and check ALL severities, ALL priorities, and UNCHECK “Closed” alerts.  Only NEW resolution state should be inspected for these.

    image

    Save the subscription.

     

    From this point forward – you will see all NEW alerts getting assigned to the Alert Update Connector, with a forward pending status:

     

    image

     

    Then the AUC will take action – and either modify the alert, or will place it in the “Processed” resolution state if it takes no action:

    image

     

     

    Configuring the AUC:

     

     

    Speaking of taking action – that’s what this connector business is all about – lets configure that now!

    Open your ConnectorConfiguration.exe UI.

    Connect to the management server first, THEN – open your previously saved Connector_Configuration.xml file.  File > Load Configuration > Connector_Configuration.xml

    Explore to a class where you have a targeted workflow that you want to send to a connector.  (hint – you can uncheck to View > Group By Management Pack to make it easier sometimes). 

    I am going to modify my “Test Alert” rule.  Right click it – and choose “Specify Properties to Modify”

     

    image

     

    A new window pops up.  Right click and choose to “Add Property”

    In Custom Field 1 – I am going to add the text “Test Team”.  Optionally I can scope it to only modify this property if the alert is raised about an instance contained in a specific group.

     

    image

     

    I can keep making changes, to the following properties:

     

    image

    Once I am done – save the configuration.  Overwrite your existing file.  AUC will automatically save a backup copy of your previous version in C:\Program Files\ directory. 

    The Alert Update Connector will also notice that the file has changed – and reload the in memory config from the file.

    Now – when this alert in generated in the future – it will follow this path:

    1. Generated as NEW
    2. Assigned to the AUC
    3. AUC inspects the alert ID based on what is loaded in the config file.
    4. If it matches, it will make the modifications specified in the config file.

    I am going to make the following changes to this Alert:

    Custom Field 1:  Core Infra Team  (the upstream system can use this to identify which team to notify)

    Custom Field 2:  Page  (the upstream system can take additional special actions, like send to a paging system)

    Custom Field 3:  PRODUCTION  (only if in special server group)

    Custom Field 3:  NONPROD  (on if in special infra group)

    Owner:  Infrastructure Queue  (this could tell the upstream system which ticketing queue the ticket belongs in or who to assign it to)

    Resolution State:  252  (this places the alert in a special resolution state, to which a product connector can subscribe to)

    image

     

    Now – when my alert comes in as new, it gets inspected by the AUC, and then modified as we see here:

     

     image

     

    image

     

    image

     

    ***Note – it is best not to use Custom Fields 5 – 10.  The Exchange 2010 MP used custom fields as part of its process.  While this is a bad practice for Microsoft MP’s to make use of custom fields because of the disruption it could cause to customer alert lifecycle processes, the Exchange 2010 MP is hard coded on these for now, so best to avoid using those or over-writing them.

     

     

    The main benefit of using the AUC over scripts to make these types of modifications are that the AUC is a commonly used tool, it has an easy to use UI to select alerts, and it is far less resource intensive than executing powershell scripts on a timed basis as it leverages the alert subscription module and an in-memory configured service.

    The main benefit of using the AUC over Orchestrator, is the simple UI to help select which alerts need to be modified.  A superb solution would be something like using the UI in AUC to create the config file, then using Orchestrator run-books to make the modifications based on the config file.  That might be something I try and work on down the road.

     

    Summary:

     

    Again – this isn’t a catch-all tool that every organization should deploy.  However, if you find that you’d benefit from using custom fields in an upstream incident management system, or you’d like a more granular filter for “what gets ticketed” than using tons of overrides on all workflows, this tool can be very handy.  This tool is similar to a Microsoft Resource Kit tool – it is not directly supported by Microsoft.  However, it is making simple SDK calls, which those SDK functions are supported.


    Comments (46)

    1. Anonymous says:

      @Bill C –

      Ugh.  I was told we would allow customers to edit these back once we broke them.  I really dislike the fact that we added this dev-ops capability in SP1, but we hard-coded these resolution states, breaking existing customer solutions.  I have provided this feedback to the product team.

      The best solution is to switch your process to leverage a different resolution state (the PG recommend using low numbers for customer resolution states apparently)

      I am checking if there is a supportable way to modify these for customers who don't care about Dev-Ops/TFS integration, and why we saw fit to lock out editing them in the UI.

       

      Bill – my friend Daniele referred me to this link for information how to unprotect these for editing:  http://www.muscetta.com/2008/09/13/protecting-custom-resolution-state-in-opsmgr-2007/

    2. Kevin Holman says:

      @Brian –

      I don’t know of any specific issues, but this one theoretical: When the subscription assigns alerts to a connector, it does this based on a configured polling cycle (default is 60 seconds). Then – the connector modifies alerts on its own polling cycle, which
      is every 10 seconds. It is possible having two connector services might try and modify the same alert at the same time or near time. This might produce occasional unexpected results. For instance, the AUC was updated a long while back for a specific condition,
      where an alert auto-closed after it was assigned to a subscription, but just before the connector modified the resolution state. This re-opened a closed alert. Therefore – the code for the connector was changed to actually ignore the alerts that were in the
      closed resolution state just before modifying them. Having two connector services might cause something similar to this, where they are both trying to act on an alert at the same time. In theory, I cannot think of any settings that might conflict though, because
      they would both share the same configuration, and just attempt the same action on the same alert at the same time. So even if one connector completes first, the second one would simply be modifying the same properties over again. The only negatives offhand,
      would be that you might see some evidence of this in the alert history. Worst case scenario, would be some sort of exception that crashed the connector service if they ran at the same time.

    3. Anonymous says:

      This version is only available for 2012.  I don't know that it would work on OpsMgr 2007R2.  However, there is a version that was leveraged for 2007R2, but never published.  You might be able to get it via your TAM.

    4. Anonymous says:

      Bill – I have observed the same behavior and reported it early on… but it was difficult to reproduce apparently.  The connector is what it is… I found that if I removed the $ServerName$ variables – the problem went away.

      Wish I had better news….

    5. Kevin Holman says:

      @Anonymous –

      The AUC is not designed for that purpose. It is designed to add specific information to specific alerts. If you want more flexibility, then use SC Orchestrator.

    6. Anonymous says:

      Hello Kevin,

      Thanks for sharing with a detailed write up.  I followed all the steps outlined in this article, but the workflows are not showing up while trying to configure AUC (last section in your article).  What could be wrong? There are no events generated.

      The generated connection_configuration.xml file had only two lines in it as shown below. I added ExcludeResolutionState as described in your article.  Is that correct?

      <ConnectorConfig GlobalResolutionState="251" ExcludeResolutionState="255">

       <AlertSources />

      </ConnectorConfig>

      TIA

      Rajemma

    7. Anonymous says:

      Yes – and I believe that is covered in the instruction document.  No restart of the service is necessary.

    8. Anonymous says:

      @ M.Mathew –

      Great question.  The answer is not really.  You can only have one of these running at any given time.  However – you do have a couple options:

      1.  Install the service on multiple management servers, then disable it on one.  If you have a failure, turn it on, on a different management server.  Replicate the config file, or place the config file on a highly available file share and access it via \UNC path on either machine.

      2.  Install the AUC service on a windows cluster, or highly available VM.

    9. Anonymous says:

      Bill – do you any of your alert modifications contain the $ServerName$ variable?

    10. Anonymous says:

      Teresa – did you execute from an elevated cmd prompt?

    11. Anonymous says:

      Can we use this connector in SCOM 2007 R2 environment?

    12. Kevin Holman says:

      @Brian –
      The connector service just needs to connect to a SDK service. It doesn’t matter if that SDK service is direct, or part of an NLB cluster, so yes, NLB is supported. However, we don’t support multiple services running at the same time. This will cause issues.
      You can install it on two servers, and have one disabled and the other enabled, and use this as part of a DR plan, but the only highly available solution for the connector service itself would be using a Windows Failover cluster.

    13. Anonymous says:

      Thx for the detailed writeup. Is there an option to configure the Alert Update Connector for High Availability?ie Having it run on multiple Managment Servers  & also  for an option for the Connector to automatically failover its SDK Connection  to a secondary Managment Server in case the Primary MS fails.

      Thx,

      M.Mathew

    14. Anonymous says:

      Before using Custom Fields in your enviroment check not to be used by other Management Packs out there! If you have Exchange and Nworks… you are almost full with the custom fields.

    15. Anonymous says:

      Hi Kevin,

      Sorry, I was busy with other things and couldn't come back on this.  I am using SP1.  Actually after a while, I could see the events generated and alerts status changed as expected.  Thanks again for quick reply.

      I have a requirement to add  Monitor Threshold values as part of Custom Fields so that field persons can see the Thresholds in the Notification email itself.  Is there a way to configure Alert Update connector to run a script which would add these values to Custome Fields.

      TIA

    16. Rob says:

      Hi Kevin,

      Is there a 64 bit version of the connector?  My install defaults to c:Program Files (x86)…

    17. quick question: Does the service handle updates to the config file (added rules) without needing to restart it?

    18. Teresa says:

      When trying to install the AlertUpdateConnector I've having the following issues:

      1. Tried to run the command line: C:Program FilesAlertUpdateConnector>c:WindowsMicrosoft.NETFramework64v2.0.50727InstallUtil.exe AlertUpdateConnector.exe

      and get "access is denied".  I'm logged in remotelly as SCOM Action Account, SCOMDA account and also local adm account.

      2. Tried to run the InstallUtil.exe from v2.0.50727 using windows and DOS and the file will not run.

      Help Help very frustrated.

    19. Teresa says:

      HI Kevin thanks for quick response.  Yes CMD – run as adminstrator

    20. Ron Williams says:

      I lurve the Alert Update Connector.  Thanks for the public release!

    21. Bill says:

      @Rob, when you run the .msi file, it just extracts the files to an x86 path. You then need to read the AlerUpdateConnector.rtf document for the rest of the instructions as Kevin outlined above to install the service, etc.  See the very next line above under "Let's have a deeper look"

    22. Cat says:

      SolarWinds has created Alert Central for incident reporting that routes emails only to the parties relevant to the issue at hand.  It's completely free and very intuitive.  You can sign up for the beta at http://swalertcentral.com

    23. Bill C says:

      Hi Kevin, I've found that with my sandbox 2012 RTM environment that New and Closed resolution states are the default, and as stated cannot be modified which is just fine. However in SP1 there are 5 additional states provided by default and of course one of them conflicts with our setup, Acknowledged (249). The dialog box still states that New and Closed cannot be changed/deleted, but these 5 news ones can't be either. We're using SCOrch to connect to Remedy and can certainly have separate runbooks for both environments, but what a pain. Any idea if they will be 'unlocked' in a future UR?

    24. Clark K says:

      I am working on getting attributes from AD and populating into custom fields.  What would you recommend for this?

    25. Bill C says:

      So in testing with a few alerts defined in the config file, the AUC works fine and modifes the fields flawlessly. After sanitizing our config file of about 500 items from our 2007R2 environment, updating non-matching GUIDs for Alerts and Groups, the AUC is not processing at all. No errors in the event log other than it was successful in connecting to the SDK.

      With a few items, the service starts and stops manually just fine. With our large config file, it takes quite a long time to stop and generates this in the event log.

      "Could not find any resources appropriate for the specified culture or the neutral culture. Make sure "Microsoft.EnterpriseManagement.Mom.AlertUpdateConnector.Resource.resources" was correctly embedded or linked into assembly "AlertUpdateConnector" at compile time, or that all the satellite assemblies required are loadable and fully signed."

      And yes it states 'with CSS guidance' but from the guide I implemented the tracing items for AUC, started trace, started AUC service, waited 5 min, stopped service, stopped trace,  and the log has a long delay starting at this point: (then continues minutes later when I stopped the service)

      [AlertUpdateConnector] [Verbose] [TRACE_LEVEL_VERBOSE] :AlertPropertiesModificationProcessingAction.ResolveServerName{alertpropertiesmodificationprocessingaction_cs154}( 00000000036BF1CF )Enter.

      What server name it be trying to resolve? I've tried both the default localhost, and the FQDN in the service config file with the same results.

      As always, thanks for your great insight to all things SCOM Kevin.

    26. Bill C says:

      Yeah, just about every one of them. We use CF7 to pass through the computer name.

    27. John Cawte says:

      I can't get this to install. I'm using windows 2008 r2  and SCOM 2012 SP1. If i install it on a management server, the service fails to start and times out. I get the following error in the application event log:

      Windows detected your registry file is still in use by other applications or services. The file will be unloaded now. The applications or services that hold your registry file may not function properly afterwards.  

      DETAIL –

      15 user registry handles leaked from RegistryUserS-1-5-21-1066489894-1993119028-2280170125-2767 etc etc

      I tried installing it on another server which had the SCOM console installed, but the connector is not installing when i run the command AlertUpdateConnector.exe –InstallConnector

      i get the following error: Cannot start the service from a command line or a debugger. A windows service must first be installed (using installutil.exe) and then started with ServerExplorer, Windows Service Administrative Tool or the NET START command.

      i get the following error in the application event log when i try to start the service without installing the connector:

      Service cannot be started. The service process could not connect to the service controller

      So far I've tried this on 2 management servers and the sql server hosting the databases. Any help or pointers would be appreciated. The source code would be great if possible!

      thanks

    28. Hi Rajemma says:

      Are you using SCOM 2012 without the SP or with SP1?

    29. Zeb_1026 says:

      Hi Kevin…

      We are getting ready to migrate existing 2007 sp1 and r2 environments into a new 2012 SP1 infrastructure.

      We have some customers that want the Alert's "Product Knowledge" used as 1st troubleshooting steps.

      Is there a way to tag that data and pass it to a CustomField in AUC so that that data flows into our mgmt console with the event?  

    30. Arrhur silvany says:

      Hi kevin,

      I created two dynamic groups containing  production and nonproduction windows servers and cathegorized them with custom fields to some monitors and rules. How could I cathegorize all new alerts without doing this one by one? Is it possible to change the alert id in the config xml file to a value that specify all alerts?

    31. DSloyer says:

      In the configuration for custom fields, is there a way to specify a registry value, or value in the SCOM database for the AUC to process?

    32. Binoy Das says:

      Hello Kevin,
      Thank you so much for such a lovely post. I am using SCOM 2012 R2 and applied the changes as documented and alerts getting changed to resolution state Processed. Hoever, I am seeing the state change alerts are not closing automatically.
      Thanks for your help.

    33. Brian Nelson says:

      Kevin, the AUC 2012 documentation says that you can point the configuration as and NLB cluster name for a pool of management servers. You mentioned in response to an earlier questions that you can’t run this in a highly available failover mode. I just
      want to verify if you can create an NLB pool and have the service running on more than one MS?

    34. Brian Nelson says:

      Thank you Kevin, do you know what problems it may cause? We tested it running on two management servers and then we saw this blog entry. We had not come across any issues yet but we obviously would not want to implement something that is problematic and/or
      not supported. Thank you again.

    35. Jim DeVries says:

      One thing the documentation doesn’t provide is an authoritative list of replacement variables. I’ve tried using the list Mark Manty provided in his his May 2012 "Alert Updater Service" blog, but I’m not having much success. In fact the only two that I’ve
      gotten to work are ServerName & WebConsoleURL. ServerName isn’t even an alert property, so I’m a bit perplexed.

    36. Daya Ram says:

      Hi Kevin,

      When I try to install the connector in SCOM by running C:Program FilesAlertUpdateConnector>AlertUpdateConnector.exe -InstallConnector I get this error: Windows service start failure. Can’t start service from the command line or a debugger. A windows service
      must first be installed and then started.
      Please help.

    37. Ashman says:

      Hi Kevin,
      Great blog on AUC. We’ve been using this for SCOM 2007 with no great problem.
      I’m now testing this on SCOM 2012 and I’m finding strange behaviour. I have a lab with Management servers installed. I install AUC on one of the management servers and set the config file to use . Everything is setup pretty much the way you have described the
      configuration in your blog.
      When the Management server that does not have AUC installed is down, then the Management server that is up and has AUC installed does not process NEW alerts. As soon as the other management server is powered up and running then the alerts are processed. I cannot
      see why this behaviour is occuring.
      I tried running a trace and the logs don’t show any problems. My original trace showed that the Management servers required powershell V3 installed and I’ve done this on the server with AUC installed but this doesn’t appear to be the fix. I can also see that
      the Management server with AUC installed is constantly looking for the Management server that is down, and I would expect this to be normal behaviour.
      I have not configured any servers to be n RMS Emulator.
      Any help or pointers would be much appreciated.

    38. Ashman says:

      Looks like I’ve figured this out now. The Alert Update Connector has a dependency on the RMS, so in SCOM 2012 the RMS Emulator must always be up and running. I’ve now tested this and when I flip the RMS Emulator role from one MS to the other where the
      AUC is running then the problem is solved.

    39. Dhiraj J says:

      Did anyone test this on Windows server 2012 and was able to work?

    40. Simon D. says:

      Hi Kevin, Can we just put string in customfields with AUC ??

    41. Simon D. says:

      Hi Kevin, Can we just put string in customfields with AUC ??

    42. Simon D. says:

      Hi Dhiraj J, Yes i have installed AUC in my Server 2k12r2 with OpsMgr 2012 r2 UR8.

      It’s work fine.

    43. Habeeb says:

      Is Orchestrator a better solution than that of AUC as of now, We have been using AUC in our previous MG, while starting a new management for a different client, evaluating if Orchecstrator is mature enough to be set up easily and manageable.

    44. J Pritchard says:

      @Habeeb: Also wondering if anyone has set up an Orchestrator Runbook to essentially do what the Alert Update Connector does? Surprised we don’t have more community resources for the Runbooks as opposed to the IPs, am I missing something?

      For an AUC-style Runbook, at least for changing Resolution State should be simple. Assigning Custom Fields values based on certain group membership would be a bit tougher in Orchestrator, from my limited knowledge.

    45. Anonymous says:

      When running the connector installation, with the command line “AlertUpdateConnector.exe -installconnector”, I receive the error: —————————
      Windows Service Start Failure
      —————————
      Cannot start service from the command line or a debugger. A Windows Service must first be installed (using installutil.exe) and then started with the ServerExplorer, Windows Services Administrative tool or the NET START command.

      I have found that the command line switch is case sensitive. The connector install must be run exactly as stated in the article including the case of the switch.

      Running the command exactly as follows works: AlertUpdateConnector.exe -InstallConnector

    46. Zoran says:

      Would it be okay to setup the service on one SCOM server to process all new alerts and on another SCOM server to process all closed alerts, since I need to modify both new and closed alerts that were generated by specific MPs and use the values in the custom field for forwarding notifications? Or is it possible to do it with a single instance?