The problem: I recently ran into a problem at one of my customer sites, with System Center Operations Manager (SCOM) 2007 & 2012, where numerous custom overrides had been saved to the Default Management Pack over the course of quite some time. Obviously there was an issue with administrators going in and quickly creating overrides without following procedures or properly understanding what they were doing. Best practice is to create new custom management packs associated with each management pack imported to the system as a placeholder for any customizations and overrides. To identify and resolve this issue, and prevent it from happening again we followed steps listed below.
Identifying the Overrides: The built-in Overrides report in the Reporting Workspace is useful in identifying. In the ‘Microsoft Generic Report Library’ the ‘Overrides’ report can be run. By default all Management Packs are selected, it is necessary to clear the selections for all but the ‘Default Management Pack’. The default criteria of ‘Select overrides DEFINED in the following Management Packs’ is correct. When the report is generated it will list the Management Pack affected, the rules or monitors. The affected element/value, and the target. These are useful both in identifying who may have created the override and identifying where to go to delete the current override and create in its correct location. After identifying the overrides in question it is necessary to manually delete each one and re-create it in the correct custom Management Pack.
Monitoring for future occurrences. Since we have two separate environments; one running 2007 and the other 2012 I decided to experiment with two different approaches.
- In the 2007 environment I used steps similar to what are defined here: http://blogs.technet.com/b/jonathanalmquist/archive/2008/11/12/monitor-default-management-pack.aspx. I am not going to go into detail on this approach since it is covered so well in the linked blog post. Using the steps set forth in the link above; I created two custom SCOM Rules. The first rule ran once per day and queried the AllOverrideView and ManagementPackView in the OperationsManager Database for any members where an override existed in the Default Management Pack. If more than 0 rules were found an event was created in the Operations Manager Event log. The second rule simply watched for the occurrence of the custom event and created an alert in SCOM if it was found. This process worked well and I was able to immediately start monitoring for any new occurrences of the problem. I set up an alert subscription so that I received an email anytime the alert was raised.
- In the 2012 environment I decided to use a different tactic. My customer recently implemented System Center Orchestrator 2012 and I decided to try using it to accomplish the same goal using this tool in what would possibly be a cleaner and more controllable manner. I created a Runbook with 4 simple steps. (note: the SCOM Integration Pack needs to be installed for these steps to work).
- Monitor Date/Time – This step is simply set to run every 1 day. The connecting Link to the following step continues if the step returns success.
- Check Schedule – On this step I simply scheduled a pre-created schedule to run daily between 9pm & 3am. The connecting link continues if the Check Schedule returns success.
- Query Database - This third step is the key step in this Runbook. The query I used is “Select count(*) as Overrides from AllOverrideView ao inner join ManagementPackView mp on ao.ManagementPackID = mp.ID where mp.name = ‘Microsoft.SystemCenter.OperationsManager.DefaultUser’”. I specified a SQL Server Database type for the connection and entered the server name and Initial Catalog (default name for the SCOM Database is OperationsManager). The connecting link after this step does require a few customizations. On the include step I selected ‘Numeric return value of the query’ from the Query Database ‘is greater than or equal to’ 1. I chose 1 because there should not be any overrides found here.
- SCOM – Alert – Here I simply put in the Management Server name for the connection, set the Priority to High, the Severity to Critical, selected Description and Name as additional fields to include in the alert. I set the name to Default MP Overrides and description toe Overrides have been saved in the Default Management Pack.