UR2 for SCOM 2016 – Step by Step


 

image

 

KB Article for OpsMgr:  https://support.microsoft.com/en-us/help/3209591/update-rollup-2-for-system-center-2016-operations-manager

Download catalog site:  http://www.catalog.update.microsoft.com/Search.aspx?q=3209591

Recommended hotfix page:  https://blogs.technet.microsoft.com/kevinholman/2009/01/27/which-hotfixes-should-i-apply/ 

Additional MP update download needed:  https://www.microsoft.com/en-us/download/details.aspx?id=55025

 

NOTE:  I get this question every time we release an update rollup:   ALL SCOM Update Rollups are CUMULATIVE.  This means you do not need to apply them in order, you can always just apply the latest update.  If you have deployed SCOM 2016 and never applied an update rollup – you can go straight to the latest one available. 

 
 
Key fixes:
  • When you use the Unix Process Monitoring Template wizard (adding a new template) to monitor processes on UNIX servers, the monitored data is not inserted into the database because of the following failure:

    Source: Health Service Modules
    Event ID: 10801
    Description: Discovery data couldn’t be inserted to the database. This could have happened because of one of the following reasons:
    – Discovery data is stale. The discovery data is generated by an MP recently deleted.
    – Database connectivity problems or database running out of space.
    – Discovery data received is not valid.

    The following is the underlying exception that causes this issue:

    Exception type: Microsoft.EnterpriseManagement.Common.DataItemDoesNotExistException Message: ManagedTypeId = ccf81b2f-4b92-bbaf-f53e-d42cd9591c1c InnerException: StackTrace (generated): SP IP Function 000000000EE4EF10 00007FF8789773D5 Microsoft_EnterpriseManagement_DataAccessLayer!Microsoft.EnterpriseManagement.DataAccessLayer.TypeSpaceData.IsDerivedFrom(System.Guid, System.Guid)+0x385

  • When a management server is removed from the All Management Servers Resource Pool, the monitoring host process does not update the TypeSpaceCache.
  • When alerts are closed from the Alerts view after you run a Search, the closed Alerts still appear in the View when the Search is cleared.
  • When you press Ctrl+C to copy an alert in Operations Manager Alert view and then press Ctrl+V to paste it to Notepad, the Created time is in UTC time, not local time.
  • Groups disappear from Group view after they are added to a Distributed Application.
  • IM notifications from Operating Manager to Skype fail when an incorrect exception causes NullReferenceException in the SipNotificationTransport.Send method.
  • When the maintenance mode option for the dependency monitor is set to “Ignore,” and the group (consisting of the server to which this dependency monitor is targeted) is put in Maintenance mode, the state of the monitor changes to critical and does not ignore maintenance mode.
  • Because of a rare scenario of incorrect computation of configuration and overrides, some managed entities may go into an unmonitored state. This behavior is accompanied by 1215 events that are written to the Operations Manager log.
  • Recovery tasks on “Computer Not Reachable” Operations Manager Monitor generate failed logons on SCOM Agents that are not part of the same domain as the management groups.
  • The ManagementGroupCollectionAlertsCountRule workflow fails and generates a “Power Shell Script failed to run” alert.
  • Get-SCOMGroup cmdlet fails when thousands of groups are created in Operations Manager.
  • Organizational unit properties for computers that are running Windows are not discovered or populated. This discovery is part of the System Center Internal Library MP. After this update, organizational unit properties will be discovered for all computers that are running Windows.
  • When the Operations Manager Health Service agent starts, and the agent is configured for AD integration, if the agent cannot contact Active Directory at all, it immediately goes dormant and stops trying to connect and obtain the policy from Active Directory.

 

Issues that are fixed in the UNIX and Linux management packs
  • SHA1 is deprecated, and SHA256 certificates are now supported on the management server that’s used to sign the Unix/Linux OMI certificate.
  • OMI does not work on Linux servers configured for FIPS compliance.
  • Avg. Physical disk sec/transfer performance counters are not displayed for Hewlett Packard computers.
  • OMI displays incorrect Memory information on Solaris 10 computers.
  • Network adapter performance is not displayed for SLES 12 x64 platform in the Operations Manager Console.
  • Cannot discover file systems on HPUX 11.31 IA-64 computers with more than 128 disks. Previously it supported only 128 VGs. Now support is extended to 256 VGs.
  • Deep monitoring cannot be started successfully on some JBoss applications because the discovery of the JBoss application server sets the Disk Path for the JBoss server incorrectly. Deep monitoring was not being started in JBoss stand-alone mode when a nondefault configuration was used.
 
 
 


Lets get started.

From reading the KB article – the order of operations is:

  1. Install the update rollup package on the following server infrastructure:
    • Management server or servers
    • Web console server role computers
    • Gateway
    • Operations console role computers
  2. Apply SQL scripts.
  3. Manually import the management packs.
  4. Apply Agent Updates
  5. Update Nano Agents
  6. Update Unix/Linux MP’s and Agents

 

Perfect!

 

 

 

1.  Management Servers

image_thumb3

Since there is no RMS anymore, it doesn’t matter which management server I start with.  There is no need to begin with whomever holds the “RMSe” role.  I simply make sure I only patch one management server at a time to allow for agent failover without overloading any single management server.

I can apply this update manually via the MSP files, or I can use Windows Update.  I have 2 management servers, so I will demonstrate both.  I will do the first management server manually.  This management server holds 3 roles, and each must be patched:  Management Server, Web Console, and Console.

The first thing I do when I download the updates from the catalog, is copy the cab files for my language to a single location, and then extract the contents:

 

 

 

Once I have the MSP files, I am ready to start applying the update to each server by role.

***Note:  You MUST log on to each server role as a Local Administrator, SCOM Admin, AND your account must also have System Administrator role to the SQL database instances that host your OpsMgr databases.

My first server is a management server, and the web console, and has the OpsMgr console installed, so I copy those update files locally, and execute them per the KB, from an elevated command prompt:

image

 

This launches a quick UI which applies the update.  It will bounce the SCOM services as well.  The update usually does not provide any feedback that it had success or failure….  but I did get:

image

 

You can check the application log for the MsiInstaller events to show completion:

Log Name:      Application
Source:        MsiInstaller
Event ID:      1036
Computer:      SCOM1.opsmgr.net
Description:
Windows Installer installed an update. Product Name: System Center Operations Manager 2016 Server. Product Version: 7.2.11719.0. Product Language: 1033. Manufacturer: Microsoft Corporation. Update Name: System Center 2016 Operations Manager UR2 Update Patch. Installation success or error status: 0.

 

You can also spot check a couple DLL files for the file version attribute. 

image

 

Next up – run the Web Console update:

image

 

This runs much faster.   A quick file spot check:

image

 

Lastly – install the console update (make sure your console is closed):

image

 

A quick file spot check:

image

 

Or help/about in the console:

image

 

 

 

 

 

Additional Management Servers:

image75

Windows Update contains the UR2 patches for SCOM 2016.   For my second Management Server – I will demonstrate that:

image

 

Then:

image

 

 
 

 

Updating Gateways:

image

Generally I can use Windows Update or manual installation.  I will proceed with manual:

image

 

The update launches a UI and quickly finishes.

 

Then I will spot check the DLL’s:

image

 

I can also spot-check the \AgentManagement folder, and make sure my agent update files are dropped here correctly:

image

 

***NOTE:  You can delete any older UR update files from the \AgentManagement directories.  The UR’s do not clean these up and they provide no purpose for being present any longer.

 

I can also apply the GW update via Windows Update:

image

 

 

 

 

 
 
 
 
2. Apply the SQL Scripts

image99

 

In the path on your management servers, where you installed/extracted the update, there is ONE SQL script file: 

%SystemDrive%\Program Files\Microsoft System Center 2016\Operations Manager\Server\SQL Script for Update Rollups

(note – your path may vary slightly depending on if you have an upgraded environment or clean install)

 

Next – let’s run the script to update the OperationsManager (Operations) database.  Open a SQL management studio query window, connect it to your Operations Manager database, and then open the script file (update_rollup_mom_db.sql).  Make sure it is pointing to your OperationsManager database, then execute the script.

You should run this script with each UR, even if you ran this on a previous UR.  The script body can change so as a best practice always re-run this.

image

 

Click the “Execute” button in SQL mgmt. studio.  The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation.  

I have had customers state this takes from a few minutes to as long as an hour. In MOST cases – you will need to shut down the SDK, Config, and Monitoring Agent (healthservice) on ALL your management servers in order for this to be able to run with success.

You will see the following (or similar) output: 

image10

 

IF YOU GET AN ERROR – STOP!  Do not continue.  Try re-running the script several times until it completes without errors.  In a production environment with lots of activity, you will almost certainly have to shut down the services (sdk, config, and healthservice) on your management servers, to break their connection to the databases, to get a successful run.

Technical tidbit:   Even if you previously ran this script in any previous UR deployment, you should run this again in this update, as the script body can change with updated UR’s.

 
 
 
 
3. Manually import the management packs

image18

 

There are 30 management packs in this update!   Most of these we don’t need – so read carefully.

 

The path for these is on your management server, after you have installed the “Server” update:

\Program Files\Microsoft System Center 2016\Operations Manager\Server\Management Packs for Update Rollups

However, the majority of them are Advisor/OMS, and language specific.  Only import the ones you need, and that are correct for your language.  

This is the initial import list: 

image

image

 

What NOT to import:

The Advisor MP’s are only needed if you are using Microsoft Operations Management Suite cloud service, (Previously known as Advisor, and Operations Insights).

DON’T import ALL the languages – ONLY ENU, or any other languages you might require.

The Alert Attachment MP update is only needed if you are already using that MP for very specific other MP’s that depend on it (rare)

The IntelliTrace Profiling MP requires IIS MP’s and is only used if you want this feature in conjunction with APM.

 

So I remove what I don’t want or need – and I have this:

image

 

These import without issue.

 

Update:  There is a bug in the “System Center Core Monitoring Agent Management MP (Microsoft.SystemCenter.OperationsManager.AM.DR.2007.mp) version 7.2.11822.0 that shipped as an updated MP with SCOM 2016 UR2.  The bug is that when an agent issues a heartbeat failure, it will also issue a “Computer Unreachable” alert even if that computer is not down and responds to ping.  An updated MP has been released for this and is available at the link below.  This fix will be included in UR3.  In the meantime, if you are running SCOM 2016 UR2, you should be using 7.2.11846.0 of the “System Center Core Monitoring Agent Management MP (Microsoft.SystemCenter.OperationsManager.AM.DR.2007.mp)

https://www.microsoft.com/en-us/download/details.aspx?id=55025

 

 

4.  Update Agents

image24

Agents should be placed into pending actions by this update for any agent that was not manually installed (remotely manageable = yes):  

image

 

If your agents are not placed into pending management – this is generally caused by not running the update from an elevated command prompt, or having manually installed agents which will not be placed into pending by design.

You can approve these – which will result in a success message once complete:

image30

 
 
 
5.  Update Unix/Linux MPs and Agents

image36

 

The “UR2” Linux MP’s and agents have been updated to align with UR2 for SCOM 2016.  You can get them here:

https://www.microsoft.com/en-us/download/details.aspx?id=29696

The current version of these MP’s for SCOM 2016 UR2 is 7.6.1072.0 – and includes agents with version 1.6.2-338

 

Make sure you download the correct version for your SCOM deployment:

image

 

Download, extract, and import ONLY the Linux/UNIX MP’s that are relevant to the OS versions that you want to monitor:

image

 

This will take a considerable amount of time to import, and consume a lot of CPU on the management servers and SQL server until complete.

Once it has completed, you will need to restart the Healthservice (Microsoft Monitoring Agent) on each management server, in order to get them to update their agent files at \Program Files\Microsoft System Center 2016\Operations Manager\Server\AgentManagement\UnixAgents

You should see the new files dropped with new timestamps:

image

 

 

Now you can deploy the agent updates:

image

 

image

 

Next – you decide if you want to input credentials for the SSH connection and upgrade, or if you have existing RunAs accounts that are set up to do the job (Agent Maintenance/SSH Account)

image

 

image

 

image

 

If you have any issues, make sure your SUDOERS file has the correct information pertaining to agent upgrade:

https://blogs.technet.microsoft.com/kevinholman/2016/11/11/monitoring-unixlinux-with-opsmgr-2016/

 

 

6.  Update the remaining deployed consoles

image

 

This is an important step.  I have consoles deployed around my infrastructure – on my Orchestrator server, SCVMM server, on my personal workstation, on all the other SCOM admins on my team, on a Terminal Server we use as a tools machine, etc.  These should all get the matching update version.

 

Review:

image60

Now at this point, we would check the OpsMgr event logs on our management servers, check for any new or strange alerts coming in, and ensure that there are no issues after the update.

 

Known Issues:

 

1.  The OU property on Windows Computers only works if the computer object is in an actual OU and not the built in “Computers” container.  The problem is in a script in “Microsoft.SystemCenter.WindowsComputerPropertyDiscovery” datasource.  This works fine if the domain computer accounts actually exist in a custom OU.

2.  The “PatchList” property on the HealthService doesn’t work.  This was re-written in PowerShell for SCOM 2016, and it looks like we still have a bug in the PatchList script.  I’d recommend taking a look at https://blogs.technet.microsoft.com/kevinholman/2017/02/26/scom-agent-version-addendum-management-pack/ because I offer a better solution than PatchList, actually changing the discovered HealthService version property, which shows up in the Agent Managed screen.

3.  The Computer Unreachable alert will be generated anytime a Heartbeat failure alert is generated, because of a bug in the ping diagnostic of the MP included in UR2.  You should update the MP included at https://www.microsoft.com/en-us/download/details.aspx?id=55025


Comments (19)

  1. SCOM-PHL says:

    Known issues # 4: The updates kept running in a loop and it never stops. After this my SCOM console doesn’t start again and the systems center data access service does not restart. I tried manually upgrading and that doesn’t start the data center service either.

    An exception was thrown while initializing the service container.
    Exception message: Initialize
    Full exception: Feature of type ‘Microsoft.EnterpriseManagement.ServiceDataLayer.IPerformanceCountersFeature, Microsoft.EnterpriseManagement.DataAccessService.Core, Version=7.0.5000.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35’ cannot be added to the container.

    1. Kevin Holman says:

      The windows update loop was fixed – it was an issue on the Windows Update backend.

  2. I was able to resolve the issue with the install loop and the service not starting by increasing the size of the OperationsManager database… The Primary filegroup was full.

  3. John Austin says:

    No love for the Reporting update?

    1. Kevin Holman says:

      There is no reporting update. When the download was first made available, there was an empty reporting and ACS update, and those have been removed.

      1. John Austin says:

        Wow. Ok, thanks!

  4. Christer Fredriksson says:

    After uppgrade to CU2 I get “Web Console Configuration Required” if i run Configure it says success, but if i run refresh i got the promt back to configure.
    If I select Skip it shows the dahsboard until next refresh, and i also says “Your configuration i being saved” in the left corner.
    Win7 clients and IE11

    1. Vyacheslav says:

      We have the same situation with Web-console configuration in two SCOM 2016 UR2 installations. Also there are some threads on MS forum about it, but no solution.
      There is an assumption that the problem in the expired certificate “Microsoft code signing PCA 2011”. It was added in process of console configuration and i see that it expired at 28.01.2017.
      Kevin, did you encounter this problem?

      1. Martijn says:

        I’m also seeing Silverlight problems in the webconsole displaying dashboards. Running the silverlight.configuation says succesful, but upon reloading the page in pops up again suggesting it’s missing something. Most likely the proper certificate with which the UR2 code is signed with. Where is the certificate ?

    2. Martijn says:

      I accidentally found the answer on this page:

      https://technet.microsoft.com/en-us/system-center-docs/get-started/release-notes

      The PCA 2011 certificate being installed by the silverlightconfiguration.exe is an old one. The new one can be found and installed by following the procedure below:

      Quite silly! Because this is not how it should be.

      Launching the Operations Manager Web Console may result in a blank screen

      Description: When you first open the Operations Manager web console you may encounter a blank screen.

      Work around: To resolve the issue:
      1.Click on “Configure” button.
      2.When prompted to Run or Save SilverlightClientConfiguration.exe, click Save.
      3.Run SilverlightClientConfiguration.exe. (Not needed).
      4.Open the file properties of exe (right-click) and open the Digital Signatures tab.
      5.Select the certificate with Digest Algorithm as sha256 and click on Details.
      6.In the Digital Signature Details dialog box, Click on View Certificate.
      7.In the dialog box which appears next, click on Install Certificate.
      8.In the Certificate Import Wizard, Set store location as – Local Machine. Click Next.
      9.Select the option – “Place all certificates in the following store” Browse to Trusted Publishers.
      10.Click Next and then Finish.
      11.Refresh the Browser

      This will install a valid PCA 2011 certificate which make the website work again.

      1. Anthony Ashmead says:

        Thanks Martijn, awesome find.

  5. John Austin says:

    Note that the “Failed to Connect to Computer” alert (“Computer Not Reachable” monitor) is also broken in SCOM 2016 UR2 (confirmed with MS support). When heartbeats fail to come in and the ping test is attempted, it will ALWAYS fail and generate the “Failed to Connect” alert. I’m told this will be fixed in UR3.

    1. Kevin Holman says:

      Yep – or sooner.

      If you are impacted – simply revert the specific MP you updated in UR2 – back to the one from the SCOM media. Microsoft.SystemCenter.OperationsManager.AM.DR.2007.mp

      1. John Austin says:

        Great; thanks for that!

  6. Jamie Lowe says:

    Hi Kevin – Do you have a step by step guide available for upgrading from scom 2012 ur11 to scom 2016. Your Note at the start of these guides says they are cumulative “if you have deployed 2016 and never applied an update rollup.” Wondering if you have a guide going from one release up to the other. We normally use your guide for the UR rollups with great success.

    1. Kevin Holman says:

      I don’t. I should do something like that. I just haven’t because large enterprise customers tend not to do in-place upgrades. Major versions generally are done in a side-by-side fashion because of the requirement for new OS versions, new SQL versions, and an opportunity to only bring in newer MP’s, not carry over all the old stuff, etc. So I have actually never supported a customer doing an in place upgrade. For smaller environments that meet the prerequisites, they make perfect sense.

    2. p@man says:

      With no disrespect to Kevin, there are other MVPs who have posted some quite good guides; I used Marnix Wolf’s ones here:
      http://thoughtsonopsmgr.blogspot.co.uk/2016/10/upgrading-to-scom-2016-rtm-01.html
      http://thoughtsonopsmgr.blogspot.co.uk/2016/10/upgrading-to-scom-2016-rtm-02-lets.html

  7. Parez Faizulla says:

    Hello Kevin,
    Is SCOM 2016 Reporting Server support SQL 2012 SP3? I have installed SQL 2012 SP3 native reporting on my SCOM 2016 management server, but when I try to install SCOM Reporting Server I get below error :

    The installed version of SQL Server could not be verified or is not supported. Verify that the computer and the installed version of SQL Server meet the minimum requirements for installation, and that the firewall settings are correct. See the Supported Configurations document for further information.

    Thanks,
    Parez

  8. Birdal says:

    Hi Kevin,
    I deployed UR2 on a fresh deployed SCOM 2016 infrastructure. I did not import any MPs from UR1 before.
    Must I now first import “old” MPs from UR1 and then Update them to UR2, or can I directly import MPs from UR2?
    Thank you in advance.
    Best Regards
    Birdal

Skip to main content