UR12 for SCOM 2012 R2 – Step by Step

 

image

KB Article for OpsMgr:  https://support.microsoft.com/en-us/help/3209587/system-center-2012-r2-om-ur12

Download catalog site:  https://www.catalog.update.microsoft.com/Search.aspx?q=3209587

Additional MP Update download required:  https://www.microsoft.com/en-us/download/details.aspx?id=55033

 

 

NOTE:   I get this question every time we release an update rollup:   ALL SCOM Update Rollups are CUMULATIVE.   This means you do not need to apply them in order, you can always just apply the latest update.  If you have deployed SCOM 2012R2 and never applied an update rollup – you can go straight to the latest one available.  If you applied an older one (such as UR3) you can always go straight to the latest one!

Key Fixes:

  • SCOM 2016 Upgrade failure:  When you try to upgrade System Center 2012 R2 Operations Manager Reporting Server to System Center 2016 Operations Manager reporting server, the upgrade fails for the following configuration:

    • Server A is configured as System Center 2012 R2 Operations Manager including Management Server.
    • Server B is configured as System Center 2012 R2 Operations Manager, including Operations Manager Database (OpsMgrDB), Operations Manager Data Warehouse (OpsMgrDW) and Operations Manager Reporting Server.
    • ( X ) Management Server Upgraded Check - The management server to which this component reports has not been upgraded.
  • Recovery tasks on "Computer Not Reachable" messages in the System Center Operations Manager Monitor generate failed logons for System Center Operations Manager Agents that are not part of the same domain as the Management Groups.

  • Resource pools:  When a Management Server is removed from the All Management Servers Resource Pool, the monitoring host process do not update the Type Space Cache.

  • SHA2 support for certificates:   SHA1 is deprecated for the System Center 2012 R2 Operations Manager Agent and SHA2 is now supported.

  • Override fixes:   Because of incorrect computations of configuration and overrides, some managed entities go into an unmonitored state. This behavior is accompanied by event 1215 errors that are logged in the Operations Manager log.

  • IntelliTrace Profiling workflows fail on certain Windows operating system versions. The workflow cannot resolve Shell32 interface issues correctly.

  • Notifications:   There is a character limitation of 50 characters on the custom fields in the notification subscription criteria. This update increases the size of the limitation to 255 characters.

  • OMS:   You cannot add Windows Client computers for Operational Insights (OMS) monitoring. This update fixes the OMS Managed Computers wizard in the System Center Operations Manager Administration pane to let you search or add Windows Client computers.

  • When you use the Unix Process Monitoring Template wizard to add a new template to the monitor processes on UNIX servers, the monitored data is not inserted into the database. This issue occurs until the Monitoring Host is restarted. Additionally, the following is logged in the Operations Manager log file:

    Log Name:      Operations Manager
    Source:        Health Service Modules
    Event ID:      10801
    Level:         Error
    Description:    Discovery data couldn't be inserted to the database. This could have happened because of one of the following reasons:
          - Discovery data is stale. The discovery data is generated by an MP recently deleted.
          - Database connectivity problems or database running out of space.
          - Discovery data received is not valid.

    Additionally, you may receive the following exception, which causes this issue to occur:

    Exception:

    Exception type:   Microsoft.EnterpriseManagement.Common.DataItemDoesNotExistException
    Message:          ManagedTypeId = ccf81b2f-4b92-bbaf-f53e-d42cd9591c1c
    InnerException:   <none>
    StackTrace (generated):   SP IP Function   000000000EE4EF10 00007FF8789773D5 Microsoft_EnterpriseManagement_DataAccessLayer!Microsoft.EnterpriseManagement.DataAccessLayer.TypeSpaceData.IsDerivedFrom(System.Guid, System.Guid)+0x385

     

New Linux operating systems supported
  • RHEL 7 on Power8 is now supported in System Center2012 R2 Operations Manager

     
Issues that are fixed in the UNIX and Linux management packs
  • SHA2 support:   SHA1 is deprecated and SHA2 is now supported on the management server that's used to sign the UNIX/Linux Operations Manager i (OMi) certificate.
  • OMI and FIPS support:   OMi start attempts fail on all FIPS-enabled UNIX/Linux systems. This fix updates the agents to support FIPS-enabled systems.
  • HPUX fix:   The Average Physical disk sec/transfer performance counters are not displayed for Hewlett Packard systems.
  • Solaris 10 fix:   OMi displays incorrect memory information for Solaris 10 systems.
  • SLES fix:   The Network Adapter Performance counter is not displayed for SLES 12 x64 platforms in the console.
  • HPUX fix:   You can't discover file systems on HPUX 11.31 IA-64 computers that have more than 128 disks. Previously, only 128 VGs were supported. This update extends support to 256 VGs.
  • JBOSS fix:   Deep monitoring can't be started successfully for some Jboss applications because the discovery of the Jboss application server sets the DiskPath for the Jboss server incorrect. Deep monitoring is not started in JBoss stand-alone mode when a nondefault configuration is used. This update provides additional support for JBoss stand-alone mode.

 

 

 

Lets get started.

From reading the KB article – the order of operations is:

  1. Install the update rollup package on the following server infrastructure:
    • Management servers
    • Audit Collection servers 
    • Gateway servers
    • Web console server role computers
    • Operations console role computers
    • Reporting
  2. Apply SQL scripts.
  3. Manually import the management packs.
  4. Update Agents
  5. Unix/Linux management packs and agent updates.

 

 

1. Management Servers

image

It doesn’t matter which management server I start with.  There is no need to begin with whomever holds the “RMSe” role.  I simply make sure I only patch one management server at a time to allow for agent failover without overloading any single management server.

I can apply this update manually via the MSP files, or I can use Windows Update.  I have 2 management servers, so I will demonstrate both.  I will do the first management server manually.  This management server holds 3 roles, and each must be patched:  Management Server, Web Console, and Console.

The first thing I do when I download the updates from the catalog, is copy the cab files for my language to a single location:

image

 

Then extract the contents:

image

 

Once I have the MSP files, I am ready to start applying the update to each server by role.

***Note: You MUST log on to each server role as a Local Administrator, SCOM Admin, AND your account must also have System Administrator role to the SQL database instances that host your OpsMgr databases.

 

My first server is a Management Server Role, and the Web Console Role, and has the OpsMgr Console installed, so I copy those update files locally, and execute them per the KB, from an elevated command prompt:

image

 

This launches a quick UI which applies the update.  It will bounce the SCOM services as well.  The update usually does not provide any feedback that it had success or failure. 

 

You can check the application log for the MsiInstaller events to show completion:

Log Name:      Application
Source:        MsiInstaller
Date:          8/31/2016 9:01:13 AM
Event ID:      1036
Description:
Windows Installer installed an update. Product Name: System Center Operations Manager 2012 Server. Product Version: 7.1.10226.0. Product Language: 1033. Manufacturer: Microsoft Corporation. Update Name: System Center 2012 R2 Operations Manager UR11 Update Patch. Installation success or error status: 0.

 

You can also spot check a couple DLL files for the file version attribute. 

image

 

Next up – run the Web Console update:

image

 

This runs much faster.   A quick file spot check:

image

 

Lastly – install the console update (make sure your console is closed):

image

 

A quick file spot check:

image

 

 

Additional Management Servers:

image

I now move on to my additional management servers, applying the server update, then the console update and web console update where applicable.

On this next management server, I will use the example of Windows Update as opposed to manually installing the MSP files.  I check online, and make sure that I have configured Windows Update to give me updates for additional products: 

image

 

The applicable updates show up under optional – so I tick the boxes and apply these updates.

image

 

After a reboot – go back and verify the update was a success by spot checking some file versions like we did above.

 

 

Updating ACS (Audit Collection Services)

image

 

 

You would only need to update ACS if you had installed this optional role.

On any Audit Collection Collector servers, you should run the update included:

image

 

image

 

A spot check of the files:

image

 

 

 

Updating Gateways:

image

I can use Windows Update or manual installation.

image

The update launches a UI and quickly finishes.

I was prompted for a reboot.

image

 

Then I will spot check the DLL’s:

image

 

I can also spot-check the \AgentManagement folder, and make sure my agent update files are dropped here correctly:

image

***NOTE: You can delete any older UR update files from the \AgentManagement directories. The UR’s do not clean these up and they provide no purpose for being present any longer.

 

I can also apply the GW update via Windows Update:

image

 

 

Reporting Server Role Update

image

I kick off the MSP from en elevated command prompt:

image

 

This runs VERY fast and does not provide any feedback on success or failure.

image

 

 

 

 

 

 

2. Apply the SQL Scripts

In the path on your management servers, where you installed/extracted the update, there are two SQL script files: 

%SystemDrive%\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\SQL Script for Update Rollups

(note – your path may vary slightly depending on if you have an upgraded environment or clean install)

image

First – let’s run the script to update the OperationsManagerDW (Data Warehouse) database.  Open a SQL management studio query window, connect it to your Operations Manager DataWarehouse database, and then open the script file (UR_Datawarehouse.sql).  Make sure it is pointing to your OperationsManagerDW database, then execute the script.

You should run this script with each UR, even if you ran this on a previous UR.  The script body can change so as a best practice always re-run this.

If you see a warning about line endings, choose Yes to continue.

image

Click the “Execute” button in SQL mgmt. studio.  The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation.

You will see the following (or similar) output:   “Command(s) completes successfully”

 

 

image

Next – let’s run the script to update the OperationsManager (Operations) database.  Open a SQL management studio query window, connect it to your Operations Manager database, and then open the script file (update_rollup_mom_db.sql).  Make sure it is pointing to your OperationsManager database, then execute the script.

You should run this script with each UR, even if you ran this on a previous UR.  The script body can change so as a best practice always re-run this.

 

image

Click the “Execute” button in SQL mgmt. studio.  The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation.  

 

I have had customers state this takes from a few minutes to as long as an hour. In MOST cases – you will need to shut down the SDK, Config, and Monitoring Agent (healthservice) on ALL your management servers in order for this to be able to run with success.

You will see the following (or similar) output: 

image

or

image

 

IF YOU GET AN ERROR – STOP!   Do not continue.  Try re-running the script several times until it completes without errors.  In a production environment with lots of activity, you will almost certainly have to shut down the services (sdk, config, and healthservice) on your management servers, to break their connection to the databases, to get a successful run.

Technical tidbit:   Even if you previously ran this script in any previous UR deployment, you should run this again in this update, as the script body can change with updated UR’s.

 

 

 

3. Manually import the management packs

image

There are 58 management packs in this update!   Most of these we don’t need – so read carefully.

The path for these is on your management server, after you have installed the “Server” update:

\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\Management Packs for Update Rollups

However, the majority of them are Advisor/OMS, and language specific.  Only import the ones you need, and that are correct for your language.  I will remove all the MP’s for other languages (keeping only ENU), and I am left with the following:

image

 

What NOT to import:

The Advisor MP’s are only needed if you are using Microsoft Operations Management Suite cloud service (OMS), (Previously known as Advisor, and Operations Insights).

The APM MP’s are only needed if you are using the APM feature in SCOM.

The Alert Attachment and TFS MP bundle is only used for specific scenarios, such as DevOps scenarios where you have integrated APM with TFS, etc.  If you are not currently using these MP’s, there is no need to import or update them.  I’d skip this MP import unless you already have these MP’s present in your environment.

However, the Image and Visualization libraries deal with Dashboard updates, and these always need to be updated.

I import all of these shown without issue.

 

Update:   There is a bug in the “System Center Core Monitoring Agent Management MP (Microsoft.SystemCenter.OperationsManager.AM.DR.2007.mp) version 7.1.10226.1304 that shipped as an updated MP with SCOM 2012 UR12.  The bug is that when an agent issues a heartbeat failure, it will also issue a “Computer Unreachable” alert even if that computer is not down and responds to ping.  An updated MP has been released for this and is available at the link below.  This fix will be included in UR13.  In the meantime, if you are running SCOM 2012 UR12, you should be using 7.1.10226.1339 of the “System Center Core Monitoring Agent Management MP” (Microsoft.SystemCenter.OperationsManager.AM.DR.2007.mp)

https://www.microsoft.com/en-us/download/details.aspx?id=55033

 

 

 

 

4. Update Agents

image

Agents should be placed into pending actions by this update for any agent that was not manually installed (remotely manageable = yes):  

One the Management servers where I used Windows Update to patch them, their agents did not show up in this list.  Only agents where I manually patched their management server showed up in this list.  FYI.   The experience is NOT the same when using Windows Update vs manual.  If yours don’t show up – you can try running the update for that management server again – manually.

image

If your agents are not placed into pending management – this is generally caused by not running the update from an elevated command prompt, or having manually installed agents which will not be placed into pending.

In this case – my agents that were reporting to a management server that was updated using Windows Update – did NOT place agents into pending.  Only the agents reporting to the management server for which I manually executed the patch worked.

I manually re-ran the server MSP file manually on these management servers, from an elevated command prompt, and they all showed up.

You can approve these – which will result in a success message once complete:

image

 

Soon you should start to see PatchList getting filled in from the Agents By Version view under Operations Manager monitoring folder in the console:

image

 

 

 

5. Update Unix/Linux MPs and Agents

image

The current Linux MP’s can be downloaded from:

https://www.microsoft.com/en-us/download/details.aspx?id=29696

7.5.1068.0 is the SCOM 2012 R2 UR12 release version.  

****Note – take GREAT care when downloading – that you select the correct download for SCOM 2012 R2. You must scroll down in the list and select the MSI for 2012 R2:

image

 

Download the MSI and run it.  It will extract the MP’s to C:\Program Files (x86)\System Center Management Packs\System Center 2012 R2 Management Packs for Unix and Linux\

Update any MP’s you are already using.   These are mine for RHEL, SUSE, and the Universal Linux libraries. 

image 

 

You will likely observe VERY high CPU utilization of your management servers and database server during and immediately following these MP imports.  Give it plenty of time to complete the process of the import and MPB deployments.

Next – you need to restart the “Microsoft Monitoring Agent” service on any management servers which manage Linux systems.  I don’t know why – but my MP’s never drop/update the UNIX/Linux agent files in the \Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\AgentManagement\UnixAgents\DownloadedKits folder until this service is restarted.

Next up – you would upgrade your agents on the Unix/Linux monitored agents.  You can now do this straight from the console:

image

 

You can input credentials or use existing RunAs accounts if those have enough rights to perform this action.

Finally:

image

 

 

 

6. Update the remaining deployed consoles

image

This is an important step.  I have consoles deployed around my infrastructure – on my Orchestrator server, SCVMM server, on my personal workstation, on all the other SCOM admins on my team, on a Terminal Server we use as a tools machine, etc.  These should all get the matching update version.

You can use Help > About to being up a dialog box to check your console version:

image

 

 

 

Review:

image

Now at this point, we would check the OpsMgr event logs on our management servers, check for any new or strange alerts coming in, and ensure that there are no issues after the update.

 

 

Known issues:

See the existing list of known issues documented in the KB article.

1.  Many people are reporting that the SQL script is failing to complete when executed.  You should attempt to run this multiple times until it completes without error.  You might need to stop the Exchange correlation engine, stop all the SCOM services on the management servers, and/or bounce the SQL server services in order to get a successful completion in a busy management group.  The errors reported appear as below:

——————————————————
(1 row(s) affected)
(1 row(s) affected)
Msg 1205, Level 13, State 56, Line 1
Transaction (Process ID 152) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
Msg 3727, Level 16, State 0, Line 1
Could not drop constraint. See previous errors.
——————————————————–

2.  The Computer Unreachable alert will be generated anytime a Heartbeat failure alert is generated, because of a bug in the ping diagnostic of the MP included in UR12.  You should update the MP included at https://www.microsoft.com/en-us/download/details.aspx?id=55033