UR7 for SCOM 2012 R2 – Step by Step

 

 

KB Article for OpsMgr:  https://support.microsoft.com/kb/3064919

KB Article for all System Center components:  https://support.microsoft.com/en-us/kb/3069110

Download catalog site:  https://catalog.update.microsoft.com/v7/site/Search.aspx?q=3064919

IMPORTANT Be aware that the SCOM Web Console package for UR7 includes an important security update (https://technet.microsoft.com/en-us/library/security/MS15-086)

*** Take notice – this update is HUGE with LOTS of fixes, and also has an important security update for a vulnerability in the web console. I’d recommend giving serious consideration to getting this through lab testing and in production to your environments.

 

Key fixes:

  • Security Issue:   The home page link in the Web Console Noscript.aspx file is vulnerable to cross-site scripting (XSS)
    A security vulnerability exists in the Web Console for System Center 2012 R2 Operations Manager that could allow elevation of privilege if a user goes to an affected website by using a specially crafted URL. This fix resolves that vulnerability. For more information, see Microsoft Security Bulletin MS15-086.

  • Report Fix: Agents by Health State" report shows duplicate entries and inconsistent data
    Sometimes a single agent has multiple entries displayed in the "Agents by Health State" report. The fix for this issue makes sure the most recent state of the agent is displayed in the report.

  • Grooming fixes:   Dependent tables are not groomed (Event.EventParameter_GUID table)
    The following issues are fixed:

    • In a database, the grooming of certain MT$X$Y tables were missed because of the filtering logic. Therefore, the tables were never groomed. There were scenarios in which lots of unwanted data was stored in these tables. This issue is now fixed, and data is groomed data from these table. This results in performance gains because there is less data from which to query.
    • In Data Warehouse, the grooming of certain tables was missed occasionally because current logic expects the rows to be returned in a certain order. This issue is now fixed, and the grooming of these tables will not be missed. In some scenarios, millions of rows were stored in these tables. This issue is now fixed. Data is now groomed from these tables. This results in performance gains because there is less data from which to query.
  • Management Packs do not synchronize between management servers
    In a SQL AlwaysOn setup, the SQL in the times of failover does not acknowledge the notifications that are registered by SCOM. This leads to inconsistent data in the environment, and the changes in management pack are not reflected in the whole environment. This update resolves this issue.

  • Grooming fixes:   Leaked transaction causes over 100 SPIDs in SCOM database to be permanently blocked by the "p_DataPurging" stored procedure
    Sometimes, because of a leaked transaction in the p_DataPurging stored procedure, the SPID becomes stuck. This causes other SPIDs to be blocked, and SCOM is brought to a standstill. This issue is fixed in this update. The fix prevents other SPIDs from being blocked.

  • Operations Manager SDK crashes because of SQL errors when QueryResultsReader.Dispose is called
    The Operations Manager SDK could potentially crash when it disposes of a database connection in some scenarios. Additionally, you receive an error message that resembles the following:

    Exception object: 00000004058197a0
    Exception type: System.Data.SqlClient.SqlException
    Message: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.)
    InnerException: System.ComponentModel.Win32Exception, Use !PrintException 0000000405819050 to see more.

    This update handles these scenarios gracefully.

  • You can't view dashboard performance counters that are created by using the TCP Port Monitoring template
    When you use the TCP Port Monitoring template to monitor network connectivity and the availability of local and remote assets, the template is missing a write action to the data warehouse. With the update, you can present this information on dashboards.

  • Dynamic inclusion rule is added to a group definition unexpectedly if an explicit member instance of the group disappears
    If all the explicit member instances of the group disappear, a dynamic rule is added to the group unexpectedly. With the update, no dynamic rule is generated in these situations.

  • You can't create group by using the SQL Server 20XX Installation Seed
    On the Dynamic Members tab of the Group Creation wizard, if you have a host class of your desired class, and if you try to select the inherited property of the host class, group creation fails. For example, if you select the Display Name(Object) property of Host=Windows Computer of the Management Server class on the Dynamic Members tab, group creation fails, and you receive the following exception:

    Processing the template failed. See inner exception for details.
    Verification failed with 1 errors:
    -------------------------------------------------------
    Error 1:
    Found error in 1|StressCollectPerformancecounterMP|1.0.0.0|UINameSpaceb2240e1340254758bc3a0f1bd0082f4d.Group.DiscoveryRule/GroupPopulationDataSource|| with message:
    The configuration specified for Module GroupPopulationDataSource is not valid.
    : Cannot find specified MPSubElement DisplayName, on MPElement= Windows!Microsoft.Windows.Computer, in expression: $MPElement[Name="Windows!Microsoft.Windows.Computer"]/DisplayName$

    This update resolves this issue.

  • Add MPB support to the SCOM online catalog
    The Management Pack catalog supports only those management packs that have the .mp extension and not the .mpb file-name extension. When this feature is implemented, the Management Pack catalog now supports MPB files.
    This update helps include management packs such as Azure Management Pack and SQL Management Pack on the Management Pack catalog that were not featured because of the .mpb file-name extension.

  • Active Directory Integration in Perimeter Network fails when there is only an RODC present
    When Active Directory Integration is enabled, the SCOM agent cannot talk to an RODC to obtain SCP information and instead looks for a RW domain controller. With this update, the Agent obtains SCP information from the RODC if information is available.

  • Subscriptions that use the filter to search for specific text in the description (of the message) do not work
    When you create a message subscription by using a criterion that contains specific text in the message description, no alerts were received through notifications. With this update, you receive notifications when the message description has specific text.

  • CLR load order change
    The current behavior for agents is to choose a CLR version based on the operating system version. For Windows Server 2012 and newer, the .NET Framework 4.0 is loaded. For operating systems older than Windows Server 2012, the .NET Framework 2.0 family is loaded. On management servers, the .NET Framework 2.0 family is loaded. This essentially maps the .NET Framework version used to the version available out-of-box on the server. The problem with the current behavior is that even if the Management Pack author knows that .NET Framework 4.0 is present on the system, it cannot be used.
    In the new behavior, the agent loads the .NET Framework 4.0 if it is available else it falls back to the .NET Framework 2.0.

  • Problems in obtaining monitoring objects by using "managementGroup.EntityObjects.GetObjectReader"
    In large System Center Operations Manager installations, when entity objects under a management group are retrieved by the object reader by using buffered mode, the object reader sometimes encounters System.Collections.Generic.KeyNotFoundException messages. With this update, the object reader ignores the invalid objects if they are not available.

  • The "Threshold Comparison" setting in the consecutive-samples-over-threshold monitor cannot be configured
    Although you configured the Threshold Comparison setting in the consecutive-samples-over-threshold monitor, the conversion of the Threshold float value from the management pack was incorrect for the German locale and caused monitor configuration failures. This issue is now fixed in this update for every supported locale.

  • Agentless Exception Monitoring (AEM) causes the Health Service to crash because the maximum path length of 248 character is exceeded
    When AEM client monitoring is turned on, sometimes the Windows error reporting file is created in a large file hierarchy. In scenarios in which the path is longer than 248 characters, AEM monitoring was causing the Health Service to crash. This issue is fixed.

  • After you update management packs, the newly added default (visible) columns to view are not visible automatically
    The first time that a view is opened on the console, registry keys are written in the HKEY_Current_User hive. The customization settings for the user are written in the registry. If the default view changes, the customization settings in the registry are not updated to reflect the new defaults. This update adds the newly added default column in the view.

  • Branding update
    Updates the "Operational Insights" name to "Operations Management Suite" in the System Center Operations Management console.

 

Xplat updates:
  • In some cases, Unix and Linux agents report incorrect CPU Processor Time
    The Unix and Linux agents use Percent IO Wait Time when calculating the Percent Processor Time for a CPU object. The agents no longer include the IO Wait Time in the calculation when they return Percent Processor Time.

  • Updates that are included in this update rollup

  • The following files are updated to support the following manageable operating systems.

Debian 8 (x64)  Microsoft.Linux.UniversalD.1.mpb

Debian 8 (x86)  Microsoft.Linux.UniversalD.1.mpb

 

 

Lets get started.

From reading the KB article – the order of operations is:

  1. Install the update rollup package on the following server infrastructure:
    • Management servers
    • Gateway servers
    • Web console server role computers
    • Operations console role computers
  2. Apply SQL scripts.
  3. Manually import the management packs.
  4. Update Agents

Now, we need to add another step – if we are using Xplat monitoring – need to update the Linux/Unix MP’s and agents.

       5.  Update Unix/Linux MP’s and Agents.

1. Management Servers

image

Since there is no RMS anymore, it doesn’t matter which management server I start with.  There is no need to begin with whomever holds the RMSe role.  I simply make sure I only patch one management server at a time to allow for agent failover without overloading any single management server.

I can apply this update manually via the MSP files, or I can use Windows Update.  I have 3 management servers, so I will demonstrate both.  I will do the first management server manually.  This management server holds 3 roles, and each must be patched:  Management Server, Web Console, and Console.

The first thing I do when I download the updates from the catalog, is copy the cab files for my language to a single location:

image

Then extract the contents:

image

Once I have the MSP files, I am ready to start applying the update to each server by role.

***Note: You MUST log on to each server role as a Local Administrator, SCOM Admin, AND your account must also have System Administrator (SA) role to the database instances that host your OpsMgr databases.

My first server is a management server, and the web console, and has the OpsMgr console installed, so I copy those update files locally, and execute them per the KB, from an elevated command prompt:

image

This launches a quick UI which applies the update.  It will bounce the SCOM services as well.  The update does not provide any feedback that it had success or failure.  You can check the application log for the MsiInstaller events for that:

Log Name: Application
Source: MsiInstaller
Date: 8/17/2015 1:17:39 PM
Event ID: 1036
Task Category: None
Level: Information
Keywords: Classic
User: OPSMGR\kevinhol
Computer: SCOM01.opsmgr.net
Description:
Windows Installer installed an update. Product Name: System Center Operations Manager 2012 Server. Product Version: 7.1.10226.0. Product Language: 1033. Manufacturer: Microsoft Corporation. Update Name: System Center 2012 R2 Operations Manager UR7 Update Patch. Installation success or error status: 0.

You can also spot check a couple DLL files for the file version attribute. 

image

Next up – run the Web Console update:

image

This runs much faster.   A quick file spot check:

image

Lastly – install the console update (make sure your console is closed):

image

 

A quick file spot check:

image

 

 

Secondary Management Servers:

image

I now move on to my secondary management servers, applying the server update, then the console update. 

On this next management server, I will use the example of Windows Update as opposed to manually installing the MSP files.  I check online, and make sure that I have configured Windows Update to give me updates for additional products:

image29

This shows me three applicable updates for this server:

NOTE:  Because the web console fix is a security vulnerability – it will show up in “Important” as opposed to “optional”

 

image

image

 

I apply these updates (along with some additional Windows Server Updates I was missing, and reboot each management server, until all management servers are updated.

 

 

Updating Gateways:

image

I can use Windows Update or manual installation.

image

The update launches a UI and quickly finishes.

Then I will spot check the DLL’s:

image

I can also spot-check the \AgentManagement folder, and make sure my agent update files are dropped here correctly:

image

2. Apply the SQL Scripts

In the path on your management servers, where you installed/extracted the update, there are two SQL script files: 

%SystemDrive%\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\SQL Script for Update Rollups

(note – your path may vary slightly depending on if you have an upgraded environment of clean install)

image

First – let’s run the script to update the OperationsManager database.  Open a SQL management studio query window, connect it to your Operations Manager database, and then open the script file.  Make sure it is pointing to your OperationsManager database, then execute the script.

You should run this script with each UR, even if you ran this on a previous UR.  The script body can change so as a best practice always re-run this.

image

Click the “Execute” button in SQL mgmt. studio.  The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation.  I have had customers state this takes from a few minutes to as long as an hour.  In MOST cases – you will need to shut down the SDK, Config, and Monitoring Agent (healthservice) on ALL your management servers in order for this to be able to run with success.

You will see the following (or similar) output:

image47

or

image

IF YOU GET AN ERROR – STOP!   Do not continue.  Try re-running the script several times until it completes without errors.  In a production environment, you almost certainly have to shut down the services (sdk, config, and healthservice) on your management servers, to break their connection to the databases, to get a successful run.

Technical tidbit:   Even if you previously ran this script in UR1, UR2, UR3, UR4, UR5, or UR6, you should run this again for UR7, as the script body can change with updated UR’s.

image

Next, we have a script to run against the warehouse DB.  Do not skip this step under any circumstances.    From:

%SystemDrive%\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\SQL Script for Update Rollups

(note – your path may vary slightly depending on if you have an upgraded environment of clean install)

Open a SQL management studio query window, connect it to your OperationsManagerDW database, and then open the script file UR_Datawarehouse.sql.  Make sure it is pointing to your OperationsManagerDW database, then execute the script.

If you see a warning about line endings, choose Yes to continue.

image

Click the “Execute” button in SQL mgmt. studio.  The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation.

You will see the following (or similar) output:

image

3. Manually import the management packs?

image

There are 26 management packs in this update!

The path for these is on your management server, after you have installed the “Server” update:

\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\Management Packs for Update Rollups

However, the majority of them are Advisor/OMS, and language specific.  Only import the ones you need, and that are correct for your language.  I will remove all the Advisor MP’s for other languages, and I am left with the following:

image

The TFS MP bundles are only used for specific scenarios, such as DevOps scenarios where you have integrated APM with TFS, etc.  If you are not currently using these MP’s, there is no need to import or update them.  I’d skip this MP import unless you already have these MP’s present in your environment.

The Advisor MP’s are only needed if you are using Microsoft Operations Management Suite cloud service, (Previously known as Advisor, and Operation Insights).

However, the Image and Visualization libraries deal with Dashboard updates, and these always need to be updated.

I import all of these shown without issue.

 

 

 

4. Update Agents

image43_thumb

Agents should be placed into pending actions by this update (mine worked great) for any agent that was not manually installed (remotely manageable = yes):   One the Management servers where I used Windows Update to patch them, their agents did not show up in this list.  Only agents where I manually patched their management server showed up in this list.  FYI.

image46_thumb

If your agents are not placed into pending management – this is generally caused by not running the update from an elevated command prompt, or having manually installed agents which will not be placed into pending.

In this case – my agents that were reporting to a management server that was updated using Windows Update – did NOT place agents into pending.  Only the agents reporting to the management server for which I manually executed the patch worked.

You can approve these – which will result in a success message once complete:

image

Soon you should start to see PatchList getting filled in from the Agents By Version view under Operations Manager monitoring folder in the console:

image

 

 

 

5. Update Unix/Linux MPs and Agents

image

Next up – I download and extract the updated Linux MP’s for SCOM 2012

https://www.microsoft.com/en-us/download/details.aspx?id=29696

7.5.1045.0 is current at this time for SCOM 2012 R2 UR6. 

****Note – take GREAT care when downloading – that you select the correct download for R2. You must scroll down in the list and select the MSI for 2012 R2:

image

Download the MSI and run it.  It will extract the MP’s to C:\Program Files (x86)\System Center Management Packs\System Center 2012 R2 Management Packs for Unix and Linux\

Update any MP’s you are already using.   These are mine for RHEL, SUSE, and the Universal Linux libraries. 

image

 

You will likely observe VERY high CPU utilization of your management servers and database server during and immediately following these MP imports.  Give it plenty of time to complete the process of the import and MPB deployments.

Next up – you would upgrade your agents on the Unix/Linux monitored agents.  You can now do this straight from the console:

image

image

You can input credentials or use existing RunAs accounts if those have enough rights to perform this action.

Mine FAILED, with an SSH exception about copying the new agent.  It turns out my files were not updated on the management server – see pic:

image65

I had to restart the Healthservice on the management server, and within a few minutes all the new files were there.

Finally:

image

 

6. Update the remaining deployed consoles

image

This is an important step.  I have consoles deployed around my infrastructure – on my Orchestrator server, SCVMM server, on my personal workstation, on all the other SCOM admins on my team, on a Terminal Server we use as a tools machine, etc.  These should all get the matching update version.

Review:

Now at this point, we would check the OpsMgr event logs on our management servers, check for any new or strange alerts coming in, and ensure that there are no issues after the update.

image

Known issues:

See the existing list of known issues documented in the KB article.

1.  Many people are reporting that the SQL script is failing to complete when executed.  You should attempt to run this multiple times until it completes without error.  You might need to stop the Exchange correlation engine, stop all the SCOM services on the management servers, and/or bounce the SQL server services in order to get a successful completion in a busy management group.  The errors reported appear as below:

------------------------------------------------------
(1 row(s) affected)
(1 row(s) affected)
Msg 1205, Level 13, State 56, Line 1
Transaction (Process ID 152) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
Msg 3727, Level 16, State 0, Line 1
Could not drop constraint. See previous errors.
--------------------------------------------------------