UR3 for SCOM 2016 – Step by Step


 

image

 

KB Article for OpsMgr:  https://support.microsoft.com/en-us/help/4016126/update-rollup-3-for-system-center-2016-operations-manager

Download catalog site:  http://www.catalog.update.microsoft.com/Search.aspx?q=4016126

Updated UNIX/Linux Management Packs:  https://www.microsoft.com/en-us/download/details.aspx?id=29696

Recommended hotfix page:  https://blogs.technet.microsoft.com/kevinholman/2009/01/27/which-hotfixes-should-i-apply/

 

NOTE:  I get this question every time we release an update rollup:   ALL SCOM Update Rollups are CUMULATIVE.  This means you do not need to apply them in order, you can always just apply the latest update.  If you have deployed SCOM 2016 and never applied an update rollup – you can go straight to the latest one available. 

 
 
Key fixes:
  • The Application Performance Monitoring (APM) feature in System Center 2016 Operations Manager Agent causes a crash for the IIS Application Pool that's running under the .NET Framework 2.0 runtime. Microsoft Monitoring Agent should be updated on all servers that use .NET 2.0 application pools for APM binaries update to take effect. Restart of the server might be required if APM libraries were being used at the time of the update.
  • Organizational Unit (OU) properties for Active Directory systems are not being discovered or populated.
  • The PatchLevel discovery script was fixed to properly discover patch level.
  • SQL Agent jobs for maintenance schedule use the default database. If the database name is not the default, the job fails.
  • When the heartbeat failure monitor is triggered, a "Computer Not Reachable" message is displayed even when the computer is not down.
  • An execution policy has been added as unrestricted to PowerShell scripts in Inbox management packs.
  • The Microsoft.SystemCenter.Agent.RestartHealthService.HealthServicePerfCounterThreshold recovery task fails to restart the agent, and you receive the following error message:  (LaunchRestartHealthService.ps1 cannot be loaded because the execution of scripts is disabled on this system.)   This issue has been resolved to make the recovery task work whenever the agent is consuming too much resources.
  • The Get-SCOMOverrideResult PowerShell cmdlet doesn't return the correct list of effective overrides.
  • The Event ID: 26373 event, which happens when there are large amounts of rows returned from an SDK query, has been changed from a “Critical” event to an “Informational” event (because there is nothing you can do about it).
  • When you run System Center 2016 Operations Manager in an all-French locale (FRA) environment, the Date column in the Custom Event report appears blank.
  • The Enable deep monitoring using HTTP task in the System Center Operations Manager console doesn't enable WebSphere deep monitoring on Linux systems.
  • When overriding multiple properties on rules that are created by the Azure Management Pack, duplicate override names are created. This causes overrides to be lost.
  • When creating a management pack (MP) on a client that contains a Service Level (SLA) dashboard and Service Level Objects (SLO), the localized names of objects aren't displayed properly if the client's CurrentCulture settings don't match the CurrentUICulture settings. In cases where the localized settings are English English, ENG, or Australian English, ENA, there's an issue when the objects are renamed.
  • The UseMIAPI registry subkey prevents collection of processor performance data for RedHat Linux system. Also, custom performance collection rules are also impacted by the UseMIAPI setting.
  • This update adds support for OpenSSL1.0.x on AIX computers. With this change, System Center Operations Manager uses OpenSSL 1.0.x as the default minimum version supported on AIX, and OpenSSL 0.9.x is no longer supported.

 

 


    Lets get started.

     

    From reading the KB article – the order of operations is:

    1. Install the update rollup package on the following server infrastructure:
      • Management server or servers
      • Web console server role computers
      • Gateway
      • Operations console role computers
    2. Apply SQL scripts.
    3. Manually import the management packs.
    4. Apply Agent Updates
    5. Update Nano Agents
    6. Update Unix/Linux MP’s and Agents

     

     

    1.  Management Servers

    image_thumb3

    It doesn’t matter which management server I start with.  I simply make sure I only patch one management server at a time to allow for agent failover without overloading any single management server.

    I can apply this update manually via the MSP files, or I can use Windows Update.  I have 2 management servers, so I will demonstrate both.  I will do the first management server manually.  This management server holds 3 roles, and each must be patched:  Management Server, Web Console, and Console.

    The first thing I do when I download the updates from the catalog, is copy the cab files for my language to a single location, and then extract the contents:

    image

     

     

    Once I have the MSP files, I am ready to start applying the update to each server by role.

    ***Note:  You MUST log on to each server role as a Local Administrator, SCOM Admin, AND your account must also have System Administrator role to the SQL database instances that host your OpsMgr databases.

     

    My first server is a management server, and the web console, and has the OpsMgr console installed, so I copy those update files locally, and execute them per the KB, from an elevated command prompt:

    image

     

    This launches a quick UI which applies the update.  It will bounce the SCOM services as well.  The update usually does not provide any feedback that it had success or failure….  but I did get a reboot prompt.  You can choose “No” and then reboot after applying all the SCOM role updates.

    image

     

    You can check the application log for the MsiInstaller events to show completion:

    Log Name:      Application
    Source:        MsiInstaller
    Event ID:      1036
    Computer:      SCOM1.opsmgr.net
    Description:  Windows Installer installed an update. Product Name: System Center Operations Manager 2016 Server. Product Version: 7.2.11719.0. Product Language: 1033. Manufacturer: Microsoft Corporation. Update Name: System Center 2016 Operations Manager Update Rollup 3 Patch. Installation success or error status: 0.

     

    You can also spot check a couple DLL files for the file version attribute. 

    image

     

    Next up – run the Web Console update:

    image

     

    This runs much faster.   A quick file spot check:

    image

     

    Lastly – install the console update (make sure your console is closed):

    image

    A quick file spot check:

    image

     

    Or help/about in the console:

    image

     

     

    Additional Management Servers:

    image75

    Windows Update contains the UR3 patches for SCOM 2016.   For my second Management Server – I will demonstrate that:

    image

     

     

    Updating Gateways:

    image

    Generally I can use Windows Update or manual installation.  I will proceed with manual:

    image

     

    The update launches a UI and quickly finishes.

    Then I will spot check the DLL’s:

    image

     

    I can also spot-check the \AgentManagement folder, and make sure my agent update files are dropped here correctly:

    image

    ***NOTE:  You can delete any older UR update files from the \AgentManagement directories.  The UR’s do not clean these up and they provide no purpose for being present any longer.

     

    I could also apply the GW update via Windows Update:

    image

     
     
     
     
     
    2. Apply the SQL Scripts

    image99

    In the path on your management servers, where you installed/extracted the update, there is ONE SQL script file: 

    %SystemDrive%\Program Files\Microsoft System Center 2016\Operations Manager\Server\SQL Script for Update Rollups

    (note – your path may vary slightly depending on if you have an upgraded environment or clean install)

    Next – let’s run the script to update the OperationsManager (Operations) database.  Open a SQL management studio query window, connect it to your Operations Manager database, and then open the script file (update_rollup_mom_db.sql).  Make sure it is pointing to your OperationsManager database, then execute the script.

    You should run this script with each UR, even if you ran this on a previous UR.  The script body can change so as a best practice always re-run this.

    image

     

    Click the “Execute” button in SQL mgmt. studio.  The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation.  

    I have had customers state this takes from a few minutes to as long as an hour. In MOST cases – you will need to shut down the SDK, Config, and Monitoring Agent (healthservice) on ALL your management servers in order for this to be able to run with success.

    You will see the following (or similar) output: 

    image10

     

    IF YOU GET AN ERROR – STOP!  Do not continue.  Try re-running the script several times until it completes without errors.  In a production environment with lots of activity, you will almost certainly have to shut down the services (sdk, config, and healthservice) on your management servers, to break their connection to the databases, to get a successful run.

    Technical tidbit:   Even if you previously ran this script in any previous UR deployment, you should run this again in this update, as the script body can change with updated UR’s.

     

     

     
    3. Manually import the management packs

    image18

    There are 33 management packs in this update!   Most of these we don’t need – so read carefully.

    The path for these is on your management server, after you have installed the “Server” update:

    \Program Files\Microsoft System Center 2016\Operations Manager\Server\Management Packs for Update Rollups

    However, the majority of them are Advisor/OMS, and language specific.  Only import the ones you need, and that are correct for your language.  

    This is the initial import list: 

    image

    image

     

    What NOT to import:

    The Advisor MP’s are only needed if you are using Microsoft Operations Management Suite cloud service, (Previously known as Advisor, and Operations Insights).

    DON’T import ALL the languages – ONLY ENU, or any other languages you might require.

    The Alert Attachment MP update is only needed if you are already using that MP for very specific other MP’s that depend on it (rare)

    The IntelliTrace Profiling MP requires IIS MP’s and is only used if you want this feature in conjunction with APM.

     

    So I remove what I don’t want or need – and I have this:

    image

     

    These import without issue.  If the “Install” button is greyed out – this means you have an MP in your import list that is already imported and not updated.  The “Microsoft System Center Advisor Resources (ENU)” MP was causing this for me – since it hasn’t been updated, I simply remove it from the list so I can install.

     

     
     
    4.  Update Agents

    image24

    Agents should be placed into pending actions by this update for any agent that was not manually installed (remotely manageable = yes):  

    image

     

    If your agents are not placed into pending management – this is generally caused by not running the update from an elevated command prompt, or having manually installed agents which will not be placed into pending by design, OR if you use Windows Update to apply the update rollup for the Server role patch.

    You can approve these – which will result in a success message once complete:

    image

     

    You can verify the PatchLevel by going into the console and opening the view at:  Monitoring > Operations Manager > Agent Details > Agents by Version

    image

     

    I also recommend you take a look at this community MP, which helps see the “REAL” agent number in the “Agent Managed” view console:

    https://blogs.technet.microsoft.com/kevinholman/2017/02/26/scom-agent-version-addendum-management-pack/

     

     
    5.  Update UNIX/Linux MPs and Agents

    image36

     

    The UNIX/Linux MP’s and agents have been updated to align with UR3 for SCOM 2016.  You can get them here:

    https://www.microsoft.com/en-us/download/details.aspx?id=29696

    The current version of these MP’s for SCOM 2016 UR2 is 7.6.1076.0 – and includes agents with version 1.6.2-339

     

    Make sure you download the correct version for your SCOM deployment:

    image

     

    Download, extract, and import ONLY the updated Linux/UNIX MP’s that are relevant to the OS versions that you want to monitor:

    image

     

    This will take a considerable amount of time to import, and consume a lot of CPU on the management servers and SQL server until complete.

    Once it has completed, you will need to restart the Healthservice (Microsoft Monitoring Agent) on each management server, in order to get them to update their agent files at \Program Files\Microsoft System Center 2016\Operations Manager\Server\AgentManagement\UnixAgents

     

    You should see the new files dropped with new timestamps:

    image

     

    Now you can deploy the agent updates:

    image

    image

     

    Next – you decide if you want to input credentials for the SSH connection and upgrade, or if you have existing RunAs accounts that are set up to do the job (Agent Maintenance/SSH Account)

    image

    image

     

    If you have any issues, make sure your SUDOERS file has the correct information pertaining to agent upgrade:

    https://blogs.technet.microsoft.com/kevinholman/2016/11/11/monitoring-unixlinux-with-opsmgr-2016/

     
     
     
     
    6.  Update the remaining deployed consoles

    image

    This is an important step.  I have consoles deployed around my infrastructure – on my Orchestrator server, SCVMM server, on my personal workstation, on all the other SCOM admins on my team, on a Terminal Server we use as a tools machine, etc.  These should all get the matching update version.

     

     

     

     

    Review:

    image60

    Now at this point, we would check the OpsMgr event logs on our management servers, check for any new or strange alerts coming in, and ensure that there are no issues after the update.

     

    Known Issues:

    None!


    Comments (29)

    1. Morten says:

      I ran into some issues with some of the new powershell scripts in the internal MPs.
      As part of a refactoring pass I changed the problematic SCOMpercentageCPUTimeCounter to use SpreadInitializationOverInterval instead of SyncTime.

      Can be found here:
      https://gallery.technet.microsoft.com/SCOM-2016-System-Center-1611d6c8?redir=0
      and here for the other fixes I did:
      https://gallery.technet.microsoft.com/SCOM-2016-System-Center-d9fc238b?redir=0

      1. Kevin Holman says:

        Good stuff! You are spot on. Those monitoring scripts are buggy. I know they are being worked on but you did some great work there. I was going to recommend just disabling them as they have always been buggy and I don’t feel offer that much value anyway. I’ll check yours out.

        1. Morten says:

          Cool, it’s the same core code as before. Though I have added more error handling and logging. One can now set the level of logging to the eventlogs as an override. Also run the script directly in PS commandline and logging will be done directly to the window instead. This will make it much easier to identify environment specific problems with running the code.

          I also did some cleanup of the code for populating information about the monitored object, like also setting IsVirtualMachine for VmWare VMs also. I also fixed the logic that populates the OU information and getting it to work even for objects in the built in Computer OU.

          It all started with the change to use WinRM to get information from the local machine. I removed the dependence on WinRM, so one will not have problems if the HTTP SPN for the machine is in use by another application.

          1. Kevin Holman says:

            Nice man! One comment – the logic was fixed for the OU properties in UR3 – that was one thing that did get fixed.

            1. Morten says:

              Yeah I discovered that they had done the fix after I implemented mine. My fix just uses the same AD fetch logic, though expands on how the OU information gets extracted from the LDAP string. Now they are still using this logic, but added extra complexity by doing another fetch of information. Which is more robust I do not know. But one of my goals was to remove as much unnecessary code as I could manage.

      2. Nice work Morten, thanks.

    2. Joerg says:

      Thanks, Kevin – just one more thing:
      I disabled APM on some machines as of the UR2 bug – updating the agents after installing UR3 _does not!_ bring back APM feature to those machines (I checked the services after agent was updated to UR3, no APM showing).

      So I had to do a repair after those agents after upgrading to UR3 from the console and there you, APM feature is back on those machines.

      1. Joerg says:

        UR3 _does not!_ fix the APM bug for me! 🙁

        Did a repair as described, rebooted SharePoint server in my lab and BOOM – blown up, just the same way it was with UR2.
        At least the workaround by re-installing the agent with NOAPM=1 also works in UR3. After doing so and rebooting the server once again, it’s up and running.

        Not very nice… 🙁

        1. Georg says:

          Same shit with Sharepoint Server 2016…

      2. Dev Ramdin says:

        APM is still causing issues for me after applying UR3. As soon as I disabled it IIS stopped crashing.

        It’s my reporting server with SquaredUp in a .NET v4.0 application pool which is puzzling.

      3. Artūras says:

        Same here. Had to reinstall with NOAPM=1

      4. Installed this UR for one customer so far, still seems to be a problem with APM and IIS.

    3. Nader Fattah says:

      Hey Kevin,

      Have you had any issues using PowerShell cmdlets on servers that are non-management servers since upgrading to SCOM 2016 or apply Update Rollups? We’re currently having an issue where importing the ‘OperationsManager’ module on servers that are not Management Servers and/or Gateways works just fine, however every single OpsMgr cmdlet does not work.

    4. Joerg says:

      Now things are getting really weird – UR3 agent + repair, Classic .NET2.0 AppPools => no problem anymore (so in fact, MS did not lie, that bug seems to be fixed).

      Therefore we now have a problem with UR3 agent and APM that _ALL!_ .NET4.x AppPools keep on crashing when APM feature is installed. Killed my OWA on Ex2016 last night. Re-Installing agent with NOAPM=1 seems to fix it.

    5. Tim says:

      Dear all,

      since ur13 we have an issue an windows 10. if i righ click an alert it takes up to 13-15 sec until it opens the context menue.

      any suggestions?

      KR

      Tim

    6. Parez Faizulla says:

      How can we prevent the server restart mentioned below? Is there a way to stop the APM libraries being used at the time of the update to prevent the restart?

      “Restart of the server might be required if APM libraries were being used at the time of the update.”

      Also, what is the status with UR3 fixing the IIS crash of NET2.0 AppPools? I have SCOM 2016 upgraded to UR2 currently and all the agents were pushed from SCOM console to the servers.

      Please advise.

      Thanks,
      Parez

    7. Gene Lindsay says:

      I see that UR3 fixes some stuff for UNIX/Linux that I need in my environments. I wanted to know if I can import the new Linux 2016 Pack even though I am still staying at UR2. I just want to make sure that the UNIX/Linux script/workflows don’t rely on being at UR3.
      It seems as if the UniversalR MP is missing all kinds of collection rules and I am hoping that this newest Linux PAK has them back in.

      1. Kevin Holman says:

        I don’t know if there are any dependencies/requirements in the UNIX/Linux MP’s for the agent binaries, I doubt it. However, we generally only test and evaluate MP’s and binary packages at the same time.

    8. julien says:

      Thanks.

    9. Birdal says:

      Hi Kevin,
      I deployed a brand new SCOM 2016 infrastructure.
      You wrote: “the path for these is on your management server, after you have installed the “Server” update: \Program Files\Microsoft System Center 2016\Operations Manager\Server\Management Packs for Update Rollups”
      But there is no product related MPs under this folder, such as Windows Server 201 R2, etc…
      Must I import Windows MPs nd also Unix/Linux MPs manually?
      Best Regards
      Birdal

    10. Sivakumar says:

      Not sure if anyone noticed it. In the blog “https://support.microsoft.com/en-us/help/4016126/update-rollup-3-for-system-center-2016-operations-manager”, under installation notes, it is specifically mentioned to install UR2 before installing UR3.
      @ Kevin, any suggestions on that!

      1. Kevin Holman says:

        I have refrained from commenting on this….. partially because of the message that sends to you guys, and partially because I think the reasoning they are pushing that is largely irrelevant, in my opinion.

        Here is the deal:

        UR’s are cumulative. They always have been. You are going to be seeing some changes very soon in how updates are delivered that will be changing this, but that is beside this point.

        Ok, with regard to why the PG put in that blurb and recommends applying UR2 before UR3:

        Technically, we have historically supported the ability to *uninstall* an Update Rollup. However, it was found that the ability to *uninstall* was broken in UR1. It was fixed in UR2 forward. However, UR2 cannot be uninstalled, it contained the fix so that SUBSEQUENT UR’s can be uninstalled. So if you apply UR2, UR3 can technically be uninstalled from a server via the MSP process. If you go straight to UR3, you cannot uninstall UR3. However, you WOULD be able to uninstall any later subsequent UR’s you chose to apply.

        So the reasoning behind recommending applying UR2 then UR3 is ONLY for the purpose of being able to uninstall the update.

        In my opinion, this is pretty meaningless. We cannot really uninstall a UR because of Management Pack changes and SQL script changes made to the management group and databases. So IF you did decide to roll back an Update Rollup, you have to also restore your databases from backup. Most people would choose to simply snapshot their VM’s and just restore a snapshot or backup of a VM as part of a “backout plan”.

        In my experience in the enterprise, needing to backout a UR is almost unheard of. It is great that we support uninstall of a UR so people using physical hardware don’t have to do a bare metal restore as part of a “backout plan” for their change management, and to provide additional options for customers using VM’s. HOWEVER, this is not critical as there are other options, and will fix itself after the next update.

        For this reason, I am not recommending to my customers that they apply UR2 then UR3 on a new deployment, because there is ZERO need to uninstall UR3 in that scenario, and any subsequent UR’s will be uninstallable.

        I hope this brings some clarity to that statement, let me know if you have any questions.

        1. Sivakumar says:

          Great! Thanks for the Clarification Kevin. Regarding installing UR3, many of our colleagues here complains that they are facing issues with APM , .Net and IIS. I have installed UR1 for SCOM 2016 today and looking out for some suggestions on UR2 and UR3 installation if they are good to install! Thanks.

          1. Kevin Holman says:

            The latest is here:

            https://blogs.technet.microsoft.com/momteam/2017/06/06/update-on-apm-fix-for-agent-crashing-issue-shipped-in-ur3/

            Basically – we did fix one of the .NET/APM issues that was impacting servers with IIS and .net2.0 based app pools. However, what was discovered is that there were other issues that were not reported because people assumed they were related to this issue that was being addressed. Those are being actively worked on, and the potential workarounds and recommendations are on that URL.

        2. “”For this reason, I am not recommending to my customers that they apply UR2 then UR3 on a new deployment, “”

          Kevin, unfortunately this time it is not this way.
          UR2 has also updates for ACS and Reporting.
          1. UR3 does not include these
          2. today i’d installed UR3 on UR1 webconsole and got error 500. after console reinstallation and UR2+UR3 installation, my webconsole works fine

    11. Stephen says:

      This is not a bug with UR3 per se, but I noticed that when upgrading the Universal Linux Agent from 1.6.2.338 to 1.6.2.339, you can load the Apache module, but it won’t discover any properties – you either get a “SUCCESS_NO_DATA” or “DISCOVERY_NOT_FOUND.” Rolling back the agent to 1.6.2.338 seems to work.

      1. Rafael Monteiro says:

        How can I rollback to 1.6.2.338 ?

    12. Stephen says:

      Another issue/bug I’ve encountered (SCOM 2016, Windows Server 2016, SQL 2014 in AAG), but not sure it is UR3 related as I’m fairly new to SCOM. However, I’ve created a group under Authoring with Explict Members. Later, I edited the group and removed a couple of explicit members. I don’t see the members in View Group Members or when checking the Properties of the group. However, a custom process monitor I created using the Unix/Linux Process Monitoring and specifying my custom group as the Target group still sends me alerts for the removed explicit members. I’d really rather not have to recreate the a few groups where this is occurring.

      On another note, does anyone know of any Community Management Pack for Unix/Linux automatic services similar to following for Windows? https://blogs.technet.microsoft.com/meamcs/2016/02/01/windows-automatic-services-monitoring-using-scom/

    Skip to main content