UR4 for SCOM 2016 – Step by Step


 

image

 

KB Article for OpsMgr:  https://support.microsoft.com/en-us/help/4024941/update-rollup-4-for-system-center-2016-operations-manager

Download catalog site:  http://www.catalog.update.microsoft.com/Search.aspx?q=4024941

Updated UNIX/Linux Management Packs:  https://www.microsoft.com/en-us/download/details.aspx?id=29696

Recommended hotfix page:  https://blogs.technet.microsoft.com/kevinholman/2009/01/27/which-hotfixes-should-i-apply/

 

 

NOTE:  I get this question every time we release an update rollup:   ALL SCOM Update Rollups are CUMULATIVE.  This means you do not need to apply them in order, you can always just apply the latest update.  If you have deployed SCOM 2016 and never applied an update rollup – you can go straight to the latest one available. 

There is some conflicting documentation with the above statement in the UR4 KB article, so I will try and explain.  The KB article states that you must apply UR2 before UR4.  This is misleading.  UR2 fixed a bug in the update process to allow any subsequent UR (post UR2) to have the ability to be uninstalled.  This bug was first fixed in UR2.  So if you want the ABILITY to uninstall UR4, you should apply UR2 or UR3 first.  However, once you apply UR4, you have the fix, and any subsequent UR’s will be able to be uninstalled.  In my opinion, there is very little to no value of applying UR2 first.  Simply apply UR4 to whatever version you are at (since it IS cumulative), and any subsequent Update Rollups will have the ability to be uninstalled.

 

Ok, now that’s clear as mud, lets get rolling:

 

Key fixes:
  • Adds support for TLS 1.2.

    • For more information about how to set up, configure, and run your environment with TLS 1.2, see the following article in the Microsoft Knowledge Base:  4051111 TLS 1.2 Protocol Support Deployment Guide for System Center 2016

  • APM Crash fix:  This update resolves an issue that causes a crash of IIS application pools that are running under CLR 2.0 when the APM feature is installed on the server as part of SCOM Agent. The code now uses appropriate memory instructions, based on the CLR version.

  • Fix:  Addresses an issue in which the APM AppDiagnostics console fails to create a Problem Management rule due to a FormatException. The appropriate string is now used for formatting, and the Problem Management wizard is able to run without issues.

  • Fix:  When a log file is being monitored by SCOM, Monagent locks the file and won't allow it to be renamed.

  • Fix:  Failure of GetOpsMgrDBWatcherDiscovery.ps1 script causes the Monitoring Host to crash.

  • Fix:  WMI Health monitor doesn't work if WINRM is configured to use https only.  (servers configured with HTTP SPN’s)

  • WMI Health monitor doesn't work if SPN http://servername is set to a user account.  (servers configured with HTTP SPN’s)

  • Fix:  SCOMpercentageCPUTimeCounter.ps1 script generates WMI errors that are caused by Service Principle Name (SPN) configuration issues.  (servers configured with HTTP SPN’s)

  • Product knowledge of "Windows Cluster Service Discovery" includes an incorrect reference to "Windows NT."

  • After a network outage, the management server does not reconnect to the gateway server if the gateway server was installed with the /ManagementServerInitiatesConnection=True option.

  • A configuration change to the network device triggers a rediscover of the device, and this process changes the SNMP agent address.

  • The UseMIAPI registry subkey prevents collection of custom performance rules data for all Linux servers.

 

 


Lets get started.

 

 

From reading the KB article – the order of operations is:

  1. Install the update rollup package on the following server infrastructure:
    • Management server or servers
    • Audit Collection Server (ACS) 
    • Web Console server role computers
    • Gateway
    • Operations Console role computers
  2. Apply SQL scripts.
  3. Manually import the management packs.
  4. Apply Agent Updates
  5. Update Nano Agents
  6. Update Unix/Linux MP’s and Agents

 


 
1.  Management Servers

image

It doesn’t matter which management server I start with.  I simply make sure I only patch one management server at a time to allow for agent failover without overloading any single management server.

I can apply this update manually via the MSP files, or I can use Windows Update.  I have 2 management servers, so I will demonstrate both.  I will do the first management server manually.  This management server holds 3 roles, and each must be patched:  Management Server, Web Console, and Console.

The first thing I do when I download the updates from the catalog, is copy the cab files for my language to a single location, and then extract the contents

 

image

 

Once I have the MSP files, I am ready to start applying the update to each server by role.

***Note:  You MUST log on to each server role as a Local Administrator, SCOM Admin, AND your account must also have System Administrator role to the SQL database instances that host your OpsMgr databases.

 

My first server is a Management Server, and the Web Console, and has the OpsMgr console installed, so I copy those update files locally, and execute them per the KB, from an elevated command prompt:

image

 

This launches a quick UI which applies the update.  It will bounce the SCOM services as well.  The update usually does not provide any feedback that it had success or failure….  but I did get a reboot prompt.  You can choose “No” and then reboot after applying all the SCOM role updates.

 

image

 

You can check the application log for the MsiInstaller events to show completion:

Log Name:      Application
Source:        MsiInstaller
Event ID:      1036
Description:
Windows Installer installed an update. Product Name: System Center Operations Manager 2016 Server. Product Version: 7.2.11719.0. Product Language: 1033. Manufacturer: Microsoft Corporation. Update Name: System Center 2016 Operations Manager Update Rollup 4 Patch. Installation success or error status: 0.

You can also spot check a couple DLL files for the file version attribute.

 

image

 

Next up – run the Web Console update:

 

image

 

 

This runs much faster.   A quick file spot check:

 

image

 

Lastly – install the Console Update (make sure your console is closed):

 

image

 

A quick file spot check:

 

image

 

Or help/about in the console:

 

image

 

 

Additional Management Servers:

image

Apply the UR updates for Server, Web Console, and Console roles as needed for all additional management servers.  You may also use Windows Update.

 

Updating ACS (Audit Collection Services)

image

One of my management servers is also my ACS Audit Collection Server role.  I will apply the update for that.

 

From an elevated command prompt:

image

 

image

Note the above image states “Operations Manager 2012”.  This is a known issue documented in the KB article. 

 

Updated files:

image

 

 

Updating Gateways:

image

Generally I can use Windows Update or manual installation.  I will proceed with manual:

 

image

 

The update launches a UI and quickly finishes.

Then I will spot check the DLL’s:

image

 

 

I can also spot-check the \AgentManagement folder, and make sure my agent update files are dropped here correctly:

image

***NOTE:  You can delete any older UR update files from the \AgentManagement directories.  The UR’s do not clean these up and they provide no purpose for being present any longer.

I could also apply the GW update via Windows Update.

 

 

 

2. Apply the SQL Scripts

image

***Note – this update SQL script did NOT change from UR3 to UR4.  If you had previously applied this file in UR3, you may skip this step.  However, if you are unsure, you may always re-apply it.  Reapplication will never hurt.

 

In the path on your management servers, where you installed/extracted the update, there is ONE SQL script file: 

%SystemDrive%\Program Files\Microsoft System Center 2016\Operations Manager\Server\SQL Script for Update Rollups

(note – your path may vary slightly depending on if you have an upgraded environment or clean install)

Next – let’s run the script to update the OperationsManager (Operations) database.  Open a SQL management studio query window, connect it to your Operations Manager database, and then open the script file (update_rollup_mom_db.sql).  Make sure it is pointing to your OperationsManager database, then execute the script.

You should run this script with each UR, even if you ran this on a previous UR.  The script body can change so as a best practice always re-run this.

image

 

Click the “Execute” button in SQL mgmt. studio.  The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation.  

I have had customers state this takes from a few minutes to as long as an hour. In MOST cases – you will need to shut down the SDK, Config, and Monitoring Agent (healthservice) on ALL your management servers in order for this to be able to run with success.

You will see the following (or similar) output: 

image

 

IF YOU GET AN ERROR – STOP!  Do not continue.  Try re-running the script several times until it completes without errors.  In a production environment with lots of activity, you will almost certainly have to shut down the services (sdk, config, and healthservice) on your management servers, to break their connection to the databases, to get a successful run.

 

 

3. Manually import the management packs

image

 

There are 36 management packs in this update!  Most of these we don’t needso read carefully.

 

The path for these is on your management server, after you have installed the “Server” update:

\Program Files\Microsoft System Center 2016\Operations Manager\Server\Management Packs for Update Rollups

However, the majority of them are Advisor/OMS, and language specific.  Only import the ones you need, and that are correct for your language.  

This is the initial import list: 

image

image

 

What NOT to import:

The Advisor MP’s are only needed if you are using Microsoft Operations Management Suite cloud service, (Previously known as Advisor, and Operations Insights) and have your on premise SCOM environment connected to the cloud service.

DON’T import ALL the languages – ONLY ENU, or any other languages you might require.

The Alert Attachment MP update is only needed if you are already using that MP for very specific other MP’s that depend on it (rare)

The IntelliTrace Profiling MP requires IIS MP’s and is only used if you want this feature in conjunction with APM.

 

So I remove what I don’t want or need – and I have this:

image

#Note:  If the “Install” button is greyed out – this means you might already have one or move of these MP’s with the same version installed.  In my case, I had already applied an out of band MP hotfix for “System Center Internal Library” version 7.0.8437.10, so I had to remove that to continue.  Only do this if it is blocking you from continuing.

 

 

 

4.  Update Agents

image24

Agents should be placed into pending actions by this update for any agent that was not manually installed (remotely manageable = yes):

image

 

If your agents are not placed into pending management – this is generally caused by not running the update from an elevated command prompt, or having manually installed agents which will not be placed into pending by design, OR if you use Windows Update to apply the update rollup for the Server role patch.

You can approve these – which will result in a success message once complete:

image

 

You normally could verify the PatchLevel by going into the console and opening the view at:  Monitoring > Operations Manager > Agent Details > Agents by Version

image

HOWEVER – due to a bug in UR4 agent patch – it is still reporting as UR3.  Sad smile

 

I *strongly* recommend you take a look at this community MP, which helps see the “REAL” agent number in the “Agent Managed” view console:

https://blogs.technet.microsoft.com/kevinholman/2017/02/26/scom-agent-version-addendum-management-pack/

image

 

And my SCOM Management Group Management mp (updated for UR4), which will help show you REAL UR levels based on a better discovery.  This has long been a pain point in SCOM:

https://blogs.technet.microsoft.com/kevinholman/2017/05/09/agent-management-pack-making-a-scom-admins-life-a-little-easier/

image

image

 

 
 
5.  Update UNIX/Linux MPs and Agents

image

 

The UNIX/Linux MP’s and agents at the time of this article publishing have not changed since SCOM 2016UR3 was released.

You can get the current Unix/Linux MP updates here:  https://www.microsoft.com/en-us/download/details.aspx?id=29696

The current version of these MP’s for SCOM 2016 UR4 is 7.6.1076.0 – and includes agents with version 1.6.2-339

 

Make sure you download the correct version for your SCOM deployment:

image

 

Download, extract, and import ONLY the updated Linux/UNIX MP’s that are relevant to the OS versions that you want to monitor:

image

 

This will take a considerable amount of time to import, and consume a lot of CPU on the management servers and SQL server until complete.

Once it has completed, you will need to restart the Healthservice (Microsoft Monitoring Agent) on each management server, in order to get them to update their agent files at \Program Files\Microsoft System Center 2016\Operations Manager\Server\AgentManagement\UnixAgents

You should see the new files dropped with new timestamps:

image

 

Now you can deploy the agent updates:

image

image

 

Next – you decide if you want to input credentials for the SSH connection and upgrade, or if you have existing RunAs accounts that are set up to do the job (Agent Maintenance/SSH Account)

image

image

 

If you have any issues, make sure your SUDOERS file has the correct information pertaining to agent upgrade:

https://blogs.technet.microsoft.com/kevinholman/2016/11/11/monitoring-unixlinux-with-opsmgr-2016/

 

 

6.  Update the remaining deployed consoles

image

 

This is an important step.  I have consoles deployed around my infrastructure – on my Orchestrator server, SCVMM server, on my personal workstation, on all the other SCOM admins on my team, on a Terminal Server we use as a tools machine, etc.  These should all get the matching update version.

 

 

Review:

image60

Now at this point, we would check the OpsMgr event logs on our management servers, check for any new or strange alerts coming in, and ensure that there are no issues after the update.

 

Known Issues:

1.  The ACS update shows “Operations Manager 2012” in the UI but is actually for SCOM 2016.

2.  The Patchlist for agents mistakenly shows UR3, and was not updated for UR4.  This can be worked around by importing my “Agent Version” and “SCOM Management” mp’s.

3.  The Web Console is broken for any Silverlight dashboards.  When you view a Silverlight Dashboard, you are prompted to “Configure” but even when you configure it, you are prompted over and over again.  This is because the certificate being published in the utility is not correct for the version hosted on the Web Console server.  You can follow the steps in the UR4 KB article to resolve the issue.


Comments (30)

  1. Raoul says:

    Hi Kevin, We experienced the IIS Application Pool crash on Application Pools that run .NET CLR Version 4.0
    Does this UR also solve that?

    In our test environment we solved this using the noapm switch.

    1. Kevin Holman says:

      This UR does not resolve ALL of the IIS application pool scenarios. There were some other scenarios (like on Sharepoint servers even using .NET4 app pools) which were not included in this UR, but will be included in the next UR, or possibly released as a standalone update in the near future. For this reason, I am still recommending to install agents with -NOAPM switch, and remove the APM mp’s from management groups that do not leverage the APM functionality.

      1. Raoul says:

        Hi Kevin, Thanks for your reply.
        The servers we have this problem on, are no Sharepoint servers. Would remove the APM mp’s be enough, or do I still need to set the noapm switch then?

        1. John Bradshaw says:

          Thx Kevin.
          This IIS Application Pool issue has hit us badly after WSUS updates this month. We only knew about it when users complained. Hope you might be able to help push along a fix for this.
          Is there a way we can work out what servers are affected before the users complain?

          Cheers
          JB

          1. Kevin Holman says:

            Honestly – my recommendation is that all agents with IIS (or just all agents) be switched to -NOAPM until we know this is 100% resolved. I also advocate removing all the APM MP’s from the management group – unless you are using the APM feature. The risk is too high, and the monitoring system should never create an outage. https://blogs.technet.microsoft.com/kevinholman/2017/08/05/reinstalling-your-scom-agents-with-the-noapm-switch/

            1. Hung Le says:

              What a disappointment!
              What a inability!
              Microsoft promissed to fix the APM issue since UR3 in May. 5 months later the issue is not fix with the new UR,

              1. Kyle says:

                Unfortunately if you have servers within azure, you’re hooped as with the workspaces in azure the workspaces will automatically push out new versions of the operations manager agent. You are also not told about the new version being pushed out. To work around this i made a monitor to tell me when a version of the agent was installed that did not match the current version of the agent installed, thus letting me know when microsoft pushes out a new patch to my azure servers. At least this lets me know i have to go reinstall the azure operations manager agent with NO APM so my servers don’t bork.

  2. Dmitry says:

    Hi Kevin!
    Thanks for info!
    Yesterday I did upgrade using Windows Update services, but then I couldn’t see agents required update in Pending Management. This morning I applied manually only SCOM server update, now I can see this list. This is strange.

    Best regards,
    Dmitry

    1. cyaz says:

      That may be “strange”, but it’s also detailed in the article 😉
      “If your agents are not placed into pending management – this is generally caused by not running the update from an elevated command prompt, or having manually installed agents which will not be placed into pending by design, OR if you use Windows Update to apply the update rollup for the Server role patch.”

  3. Hung Le says:

    What a disappointment! Microsoft promissed to fix the APM issue since UR3 in May. 5 months later the issue is not fix with the new UR,

    1. JB says:

      Why cannot MS just wise up and make the default agent install without the APM service. It should not have taken 5 months to figure that out. If you want APM then run a task to install and enable the service.

      It is best to be a minimalist but i know that is against the Microsoft way.

      1. Kevin Holman says:

        I completely agree and we have asked for that…..

        1. JB says:

          BTW you are awesome Kevin. The culmination of lack of QA with all things MS lately is just getting frustrating. 🙂

      2. rob1974 says:

        And while they are at it, do the same for ACS. That’s even less used then APM

  4. Tonka Pushchaira says:

    Mister Holeman,we have found a bug.. minor to some… which create issue. Discovery of Microsoft.SystemCenter.WindowsComputerPropertyDiscovery NETBIOS DOMAIN look at WMI query returned “Domain” property. In previous version it give different value then previous version.. brakes our many dynamic group discovery. This bug appears again in UR4. Please see more here: https://blogs.technet.microsoft.com/omx/2017/08/03/netbios-domain-name-incorrect-in-scom-2016/

    thanks you kindly in advance

    1. Kevin Holman says:

      You are correct. I thought this was already fixed an in interim release, and totally assumed this would be addressed in UR4. I will follow up with the product group and try to get the status. The interim fix posted at that blog is likely a good idea until this gets addressed.

      1. cyraz says:

        On a side note, I don’t get why the netbios domain name is purposedly discovered as this for workgroup servers having a FQDN :
        var arrNameSplit = strDNSComputerName.split(“.”);
        strNetBIOSDomain = arrNameSplit[1];

        What that effectively does is that a server in workgroup named “server01.contoso.com” will have a netbios domain name property of “contoso” and a dns domain property of “workgroup” (returned from WMI query), which doesn’t sound right to me.
        Unless I misunderstand how the discovery script works and I have an issue somewhere…

        Also looks like the ps1 discovery in SCOM 2016 works the same way.

        Not a big issue anyway, just wanted to rant about it 😀

  5. michael says:

    Hi Kevin, not sure what I’ve done wrong but when I upgrade the agents that require update. they are successful but are still on the version 8.0.10918.0.

    1. Kevin Holman says:

      That’s totally normal. 8.0.10918.0 is the base level agent build version of SCOM 2016 agents. Out of the box – we don’t update this “version” in the “Agent Managed” view under Administration pane.

      This is why I recommended the two management pack add-ons that I wrote, in the above UR4 article. Have you implemented those?

      1. michael says:

        I’ve done that now and I can see the version correctly now thanks

  6. Raed says:

    Hi Kevin,
    I have SCOM 2016 single deployment , one server with all the roles ( Management Server , Operations Console ,Web Console , Reporting server ).

    So I should install the below rollup packages in order :-
    – Management server.
    – Web Console server.
    – Operations Console.

    I did not see the operations console package installation in this article.

    Thank you.

  7. Jason Aranha says:

    Hi Kevin,

    I just pushed UR4 from console and also did an manual install, Still the agent show in the console & installed Update on Agent as Update Rollup Patch 3 , Is this normal or am I missing something.

    1. Kevin Holman says:

      Yes. I call this out in the article, and in the known issues, and provided workarounds.

  8. john says:

    Thanks for the great work, again, Kevin.

    Question: Am I remembering incorrectly, or did previous updates disable enabled adtagent service state? I’m seeing agents that I’ve updated with the adtagent state as running. Was this something that happened with the 2012R2 to 2016 update?

    Also, the MPs you noted above are great. I especially like being able to see all the details under Scom Agents. One note, however. The ability to right click an agent in that view and enable maintenance mode seems to be broken. You can click it and do normal MM things, but the agent stays out of MM.

    1. cyaz says:

      That’s because the view is scoped to “scom agents management class” (a class that only exists on this MP, used to add and display all the properties) and not to the actual SCOM agent or Windows computer class,
      Nothing goes to maintenance mode because the “scom agents management class” doesn’t host anything…

  9. Igor Puhalo says:

    If you are bored with installing MPs on update rollup installation you can use my script. I tested with UR4. Everything its fine. there is no new MPs 🙂 only new version of the same . You can download script at https://gallery.technet.microsoft.com/Importing-SCOM-management-f9ae78c0

  10. RichardRuben says:

    Please be aware that if you activate the “Automatic provisioning” on the “Security Center” blade in Azure that this will automatically install the MicrosoftMonitoringAgent extension on your VM and update or install OMS/SCOM agents on all servers on that Subscription . (See https://docs.microsoft.com/en-us/azure/security-center/security-center-get-started).

    My security Team activated this and update rolled out to all servers even to servers where I had installed de SCOM agent with NOAPM.

  11. George says:

    Has anyone experienced any issues with UR4? On 2 MS, the installation never seems to complete, I left it run for hours without success. Rebooted the server and tried again and it got installed in less then 2 minutes?

    1. Kevin Holman says:

      I had this on a customer environment. It seemed it was waiting on the services to start back up. We simply re-installed the update like you, and it went fine the second time.

Skip to main content