UR3 for SCOM 2012 R2 – Step by Step


 

 

KB Article for OpsMgr:  http://support.microsoft.com/kb/2965445

KB Article for all System Center components:  http://support.microsoft.com/kb/2965090

Download catalog site:  http://catalog.update.microsoft.com/v7/site/Search.aspx?q=2965445

 

Key fixes:

 

  • Reliability fix:  A deadlock condition occurs when a database is connected after an outage. You may experience this issue may when one or more HealthServices services in the environment are listed as Unavailable after a database goes offline and then comes back online.  Management servers cannot reconnect to SQL after a SQL outage because of thread exhaustion. 
  • The Desktop console crashes after exception TargetInvocationException occurs when the TilesContainer is updated. You may experience this issue after you leave the console open on a Dashboard view for a long time.
  • The Password expiration monitor is fixed for logged events. To make troubleshooting easier, this fix adds more detail to Event IDs 7019 and 7020 when they occur.
  • The Health service bounces because of high memory usage in the instance MonitoringHost: leak MOMModules!CMOMClusterResource::InitializeInstance. This issue may be seen as high memory usage if you examine monitoringhost.exe in Performance Monitor. Or, the Health service may restart every couple of days , depending on the load on the server.
  • The Health service crashes in Windows HTTP Services (WinHTTP) if the RunAs account is not read correctly.
  • Windows PowerShell stops working with System.Management.Automation.PSSnapInReader.ReadEnginePSSnapIns. You may see this issue as Event ID 22400 together with a description of "Failed to run the Powershell script."
  • The PropertyValue column in the contextual details widget is unreadable in smaller widget sizes because the PropertyName column uses too much space.
  • The update threshold for monitor "Health Service Handle Count Threshold" is reset to 30,000. You can see this issue in the environment, and the Health Service Handle Count Threshold monitor is listed in the critical state.
  • An acknowledgement (ACK) is delayed by write collisions in MS queue when lots of data is sent from 1,000 agents.
  • The execution of the Export-SCOMEffectiveMonitoringConfiguration cmdlet fails with the error "Subquery returned more than 1 value.”
  • The MOMScriptAPI.ReturnItems method can be slow because a process race condition may occur when many items are returned, and the method may take two seconds between items. Scripts may run slowly in the System Center Operations Manager environment.
  • When you are in the console and click Authoring, click Management Pack, click Objects, and then click Attributes to perform a Find operation, the Find operations seems unexpectedly slow. Additionally, the Momcache.mdb file grows very large.
  • A delta synchronization times out on SQL operations with Event ID 29181.
  • Operations Manager grooms out the alert history before an alert is closed.
  • The time-zone settings are not added to a subscription when non-English display languages are set. Additionally, time stamps on alert notifications are inaccurate for the time zone.
  • Web Browser widget requires the protocol (http or https) to be included in the URL.
  • You cannot access MonitoringHost's TemporaryStoragePath within the PowerShell Module.
  • The TopNEntitiesByPerfGet stored procedure may cause an Operations Manager dashboard performance issue. This issue may occur when a dashboard is run together with multiple widgets. Additionally, you may receive the following error message after a time-out occurs:

[Error] :DataProviderCommandMethod.Invoke{dataprovidercommandmethod_cs370}( 000000000371AA78 )
An unknown exception was caught during invocation and will be re-wrapped in a DataAccessException. System.TimeoutException: The operation has timed out.  at Microsoft.EnterpriseManagement.Monitoring.DataProviders.RetryCommandExecutionStrategy.Invoke(IDataProviderCommandMethodInvoker invoker) at Microsoft.EnterpriseManagement.Presentation.DataAccess.DataProviderCommandMethod.Invoke(CoreDataGateway gateWay, DataCommand command)

 
Xplat updates:
  • Slow results are returned when you run the Get-SCXAgent cmdlet or view UNIX/Linux computers in the administration pane for lots of managed UNIX/Linux computers.
    Note To apply this hotfix, you must have version 7.5.1025.0 or later of the UNIX/Linux Process Monitoring, UNIX/Linux Log File Monitoring, and UNIX/Linux Shell Command Template management pack bundles.
  • Accessing the UNIX/Linux computers view in the administration pane can sometimes trigger the following exception message:

    Microsoft.SystemCenter.CrossPlatform.ClientLibrary.Common.SDKAbstraction.ManagedObjectNotFoundException

 

Lets get started.

From reading the KB article – the order of operations is:

  1. Install the update rollup package on the following server infrastructure:
    • Management servers
    • Gateway servers
    • Web console server role computers
    • Operations console role computers
  2. Apply SQL scripts.
  3. Manually import the management packs.
  4. Update Agents

Now, we need to add another step – if we are using Xplat monitoring – need to update the Linux/Unix MP’s and agents.

       5.  Update Unix/Linux MP’s and Agents.

 

 

1.  Management Servers

image

Since there is no RMS anymore, it doesn’t matter which management server I start with.  There is no need to begin with whomever holds the RMSe role.  I simply make sure I only patch one management server at a time to allow for agent failover without overloading any single management server.

I can apply this update manually via the MSP files, or I can use Windows Update.  I have 3 management servers, so I will demonstrate both.  I will do the first management server manually.  This management server holds 3 roles, and each must be patched:  Management Server, Web Console, and Console.

The first thing I do when I download the updates from the catalog, is copy the cab files for my language to a single location:

image

Then extract the contents:

image

Once I have the MSP files, I am ready to start applying the update to each server by role.

***Note:  You MUST log on to each server role as a Local Administrator, SCOM Admin, AND your account must also have System Administrator (SA) role to the database instances that host your OpsMgr databases.

My first server is a management server, and the web console, and has the OpsMgr console installed, so I copy those update files locally, and execute them per the KB, from an elevated command prompt:

image

This launches a quick UI which applies the update.  It will bounce the SCOM services as well.  The update does not provide any feedback that it had success or failure.  You can check the application log for the MsiInstaller events for that:

Log Name:      Application
Source:        MsiInstaller
Date:          8/6/2014 3:00:46 PM
Event ID:      1022
Task Category: None
Level:         Information
Keywords:      Classic
User:          OPSMGR\kevinhol
Computer:      SCOM01.opsmgr.net
Description:
Product: System Center Operations Manager 2012 Server – Update 'System Center 2012 R2 Operations Manager UR3 Update Patch' installed successfully.

You can also spot check a couple DLL files for the file version attribute. 

image

Next up – run the Web Console update:

image

This runs much faster.   A quick file spot check:

image

Lastly – install the console update (make sure your console is closed):

image

A quick file spot check:

image

 

 

Secondary Management Servers:

image

I now move on to my secondary management servers, applying the server update, then the console update. 

On this next management server, I will use the example of Windows Update as opposed to manually installing the MSP files.  I check online, and make sure that I have configured Windows Update to give me updates for additional products:

image29

This shows me two applicable updates for this server:

image

I apply these updates (along with some additional Windows Server Updates I was missing, and reboot each management server, until all management servers are updated.

 

Updating Gateways:

image

I can use Windows Update or manual installation.

image

The update launches a UI and quickly finishes.

Then I will spot check the DLL’s:

image

I can also spot-check the \AgentManagement folder, and make sure my agent update files are dropped here correctly:

image

 

 

2. Apply the SQL Scripts

In the path on your management servers, where you installed/extracted the update, there are two SQL script files: 

%SystemDrive%\Program Files\System Center 2012\Operations Manager\Server\SQL Script for Update Rollups

image

First – let’s run the script to update the OperationsManager database.  Open a SQL management studio query window, connect it to your Operations Manager database, and then open the script file.  Make sure it is pointing to your OperationsManager database, then execute the script.

image

Click the “Execute” button in SQL mgmt. studio.  The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation.

You will see the following (or similar) output:

image47

or

image

IF YOU GET AN ERROR – STOP!  Do not continue.  Try re-running the script several times until it completes without errors.  In a large environment, you might have to run this several times, or even potentially shut down the services on your management servers, to break their connection to the databases, to get a successful run.

Technical tidbit:  This script has been updated in UR3.  Even if you previously ran this script in UR1 or UR2, you must run this again.

 

image

Next, we have a script in UR3 to run against the warehouse DB.  Do not skip this step under any circumstances.    From:

%SystemDrive%\Program Files\System Center 2012\Operations Manager\Server\SQL Script for Update Rollups

Open a SQL management studio query window, connect it to your OperationsManagerDW database, and then open the script file UR_Datawarehouse.sql.  Make sure it is pointing to your OperationsManagerDW database, then execute the script.

If you see a warning about line endings, choose Yes to continue.

image

Click the “Execute” button in SQL mgmt. studio.  The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation.

You will see the following (or similar) output:

image

 

 

3. Manually import the management packs?

image

We have 6 updated MP’s to import  (MAYBE!).

image

The TFS MP bundles are only used for specific scenarios, such as DevOps scenarios where you have integrated APM with TFS, etc.  If you are not currently using these MP’s, there is no need to import or update them.  I’d skip this MP import unless you already have these MP’s present in your environment.

The Advisor MP’s are only needed if you are using System Center Advisor services.

However, the Image and Visualization libraries deal with Dashboard updates, and these need to be updated.

I import all of these without issue.

image

 

 

4.  Update Agents

image

Agents should be placed into pending actions by this update (mine worked great) for any agent that was not manually installed (remotely manageable = yes):

 image

If your agents are not placed into pending management – this is generally caused by not running the update from an elevated command prompt, or having manually installed agents which will not be placed into pending

You can approve these – which will result in a success message once complete:

image

 

Soon you should start to see PatchList getting filled in from the Agents By Version view under Operations Manager monitoring folder in the console:

image

 

 

5.  Update Unix/Linux MPs and Agents

image

Next up – I download and extract the updated Linux MP’s for SCOM 2012 SP1 UR3

http://www.microsoft.com/en-us/download/details.aspx?id=29696

7.5.1025.0 is current at this time for SCOM 2012 R2 UR2. 

****Note – take GREAT care when downloading – that you select the correct download for R2.  You must scroll down in the list and select the MSI for 2012 R2:

image

Download the MSI and run it.  It will extract the MP’s to C:\Program Files (x86)\System Center Management Packs\System Center 2012 R2 Management Packs for Unix and Linux\

Update any MP’s you are already using.   These are mine for RHEL, SUSE, and the universal Linux libraries:

image

You will likely observe VERY high CPU utilization of your management servers and database server during and immediately following these MP imports.  Give it plenty of time to complete the process of the import and MPB deployments.

Next up – you would upgrade your agents on the Unix/Linux monitored agents.  You can now do this straight from the console:

image

image

You can input credentials or use existing RunAs accounts if those have enough rights to perform this action.

image

 

 

6.  Update the remaining deployed consoles

image

This is an important step.  I have consoles deployed around my infrastructure – on my Orchestrator server, SCVMM server, on my personal workstation, on all the other SCOM admins on my team, on a Terminal Server we use as a tools machine, etc.  These should all get the UR3 update.

 

 

 

Review:

Now at this point, we would check the OpsMgr event logs on our management servers, check for any new or strange alerts coming in, and ensure that there are no issues after the update.

image

 

 

Known issues:

See the existing list of known issues documented in the KB article.

1.  Many people are reporting that the SQL script is failing to complete when executed.  You should attempt to run this multiple times until it completes without error.  You might need to stop the Exchange correlation engine, stop the services on the management servers, or bounce the SQL server services in order to get a successful completion in a busy management group.  The errors reported appear as below:

——————————————————
(1 row(s) affected)
(1 row(s) affected)
Msg 1205, Level 13, State 56, Line 1
Transaction (Process ID 152) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
Msg 3727, Level 16, State 0, Line 1
Could not drop constraint. See previous errors.
——————————————————–


Comments (33)

  1. Tom_Floor says:

    Xahmed: As far as I know, these updates are cumulative, so you don’t need to deploy UR1 and UR2, you can just deploy UR3.

    Kevin: Thanks for this comprehensive guide on updating. Love the part about checking DLL versions to see if the update actually was applied.

  2. Sn0w_MOnkEY says:

    Thank You.

  3. Ed (DareDevil57) says:

    Thank you

  4. Kevin Holman says:

    @Richard – no the SQL scripts can only be applied manually. WU doesn’t cut it.

    @Amjad – if you add another management server later, you simply need to update the management server itself, for any roles on the MS, such as the MS role, the Console, and the Web Console, where applicable. You don’t need to go back and do anything with the
    database.

  5. Kevin Holman says:

    @steven – No. Update Rollups are cumulative.

  6. Kevin Holman says:

    @Steve –

    The KB article is in error. I will notify those who own that to get it updated. Those were issues in UR2, in my testing they do not exist as issues in UR3. Thanks for pointing that out.

  7. Anonymous says:

    If you guys are seeing errors when running the DW update such as:

    The module ‘PerformanceAggregate’ depends on the missing object ‘DebugMessageInsert’. The module will still be created; however, it cannot run successfully until the object exists.

    Or

    Msg 208, Level 16, State 1, Procedure GetStateDaily, Line 63
    Invalid object name ‘vManagedEntityMonitor’.
    Msg 208, Level 16, State 1, Procedure GetStateHourly, Line 63
    Invalid object name ‘vManagedEntityMonitor’.

    The root cause is that some datasets never got deployed – this is due to not enough rights. You should ensure your DW read and DW write accounts are SCOM admins, and local admins on your management servers, after doing this and waiting a bit you should see
    this problem go away.

  8. Steve says:

    The Known Issues section of the KB article indicates that the agents need to be updated via the Repair Install process, not by simply approving them from the Pending Management view. Is this not correct? Your article indicates that they can be updated
    normally from the Pending Management view.

  9. Howard says:

    Thank you Kevin for a detailed step-by-step! Really appreciate it.

  10. Anonymous says:

    This is updated as of 8-11-2014 In general – you should evaluate all hotfixes available, and only apply

  11. andyinsdca says:

    Good to see you back! However, this webpage appears to render images with the word "image" and not a real pic in ff and in IE I get the red X.

  12. andyinsdca says:

    We did the update here and even in our tiny environ (87 agents, 2 MS, 2 GW), we had to stop the SCOM services on our MSs to do the SQL scripts. Otherwise, it went flawlessly, we had a couple of agents that didn’t update, but that had nothing to do with
    SCOM (full C: drives, anyone?)

  13. Jon M Honhart says:

    Glad you are back. Thanks for the details!

  14. Brian Hansen says:

    We update our agents manually (via SCCM). So far I have installed the URS agent update manually to about 10 servers. Every server wanted to reboot. I thought Microsoft committed to no reboots for agent updates. We can suppress the reboot in SCCM but will
    this cause any issues with the agent?

  15. Dudu Sakharovich says:

    Hi Kevin,

    A few days ago I’ve started receiving alerts in SCOM regarding "Failed to store data in the Data Warehouse. The operation will be retried.
    Exception ‘InvalidOperationException’: The given value of type String from the data source cannot be converted to type nvarchar of the specified target column".

    I’ve searched through the net and didn’t find any proper solution regarding this issue.
    In addition since I get these errors I’m experiencing problems with my RMS. Every time and now it turns to gray along with the Gateway server and only after I restart the SCOM services it returns to green.

    1. Do you think that these issues are related ?
    2. Is this fix mentioned in the post here can fix this issue ?

    Thx in advance.

    Dudu Sakharovich

  16. microgod says:

    Hi Kevin,

    I installed the Update Rollups 3 on each system / role. Now when I run the first SQL script (UR_Datawarehouse.sql) against the OperationsManagerDW Database I get the following errors:I’ve installed all components successfully and I followed the right order
    by updating the components.
    Msg 208, Level 16, State 1, Procedure GetStateDaily, Line 63
    Invalid object name ‘vManagedEntityMonitor’.
    Msg 208, Level 16, State 1, Procedure GetStateHourly, Line 63
    Invalid object name ‘vManagedEntityMonitor’.

  17. Xahmed says:

    Hi actually i’m planning to install new SCOM 2012 R2 so shall i install only UR3 or start from UR1 until UR3.
    Thanks

  18. andreas says:

    cool and useful docu, thanks a lot!

  19. Mpho says:

    Here is a simple work around.

    Known issues: Transaction (Process ID 152) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

    Stop all management server services (Monitoring, Data Access, Management system) on all the Management servers and rerun the SQL update.

    Results Command completed Successfully.

  20. David says:

    Hi all,
    during installation of our agents, I noticed an error event about MsiInstaller:

    Product: Microsoft Monitoring Agent — Error 1923.Service ‘@C:Windowssystem32AdtAgent.exe,-500’ (AdtAgent) could not be installed. Verify that you have sufficient privileges to install system services.

    My account is Domain Admin and machine administrator, also the UR3 update is correctly installed:

    Product: Microsoft Monitoring Agent – Update ‘System Center 2012 R2 Operations Manager UR3 Update Patch’ installed successfully.

    Can I safely ignore this event?

    Thanks,
    David

  21. Richard Schwartz says:

    Hi Kevin,

    Question in regards to Step #2: Apply the SQL Scripts. If you are using the WU method of updating the SCOM infrastructure, do you still have to run the SQL scripts or is this automgically completed for you. I would think that because these scripts require SA
    rights to on SQL the answer is "no". That said, how do you extract the UR3 Server MSP to gain access to the scripts? Just a point of contention that I wanted to call out for those going the full Windows Update method for these UR’s.

    Many thanks in advance,

    Richard Schwartz, Sr. Engineer, SO-MM, CDW

  22. Qazi says:

    you are simply the best in SCOM

  23. Amjad says:

    Hi, i want to ask that i have two Management server now after upgrading my servers with above RU if later i add another management server or OpsMgr server role in current hierarchy then what would be case, mean i have to update that new server and SQL
    datbases?

  24. Diego Rios says:

    Kevin good afternoon I already manually updated a fresh SCOM 2012 R2 install to the UR3 as it is pretty well explained in this post, but after that I noticed that the new widgets view does not show the new one, I am pretty sure that I followed this guide
    step by step, and the SQL scripts run OK. I appreciate any coment. Thanks in advance.

  25. Hi Kevin,
    We have updated from SCOM 2012 SP1 to R2 CUR3 and after the upgrade we tried to update the Unix SCOM agent and found that when we clicked update agent it caused the SCOM console to hang. Have tried SP1 and R2 consoles and both have same issue.
    Anyone experience this?

  26. Hi Kevin,
    Worked it out, turns out that the SQL broker was not enabled, once enabled this looks to have fixed the issue.

  27. Marco says:

    Why did you extract KB2965445-i386-Agent and KB2965445-AMD64-Agent ?
    You didn’t use or run these files anywhere in this guide…
    Please explain what needs to be done with these files

  28. Steven says:

    do i need to update the previous update UR1 & UR2 before the UR3?

  29. John says:

    Has anyone seen an agent in the "Agents by Version" view not show the updated patch level even though the Application log on the agent machine shows it was updated properly? Out of my ~10 test agents one is having an issue. I’ve cleared the cache, restarted
    the agent, checked logs, everything looks good except it’s not reporting properly in the OM console.

  30. Lukas says:

    Thanks Kevin.

  31. Sanjay Dubey says:

    i am getting this error while running the sql script. please help.

    The module ‘p_AlertGrooming’ depends on the missing object ‘dbo.p_ConvertLocalTimeToUTC’. The module will still be created; however, it cannot run successfully until the object exists.
    Msg 208, Level 16, State 1, Line 22
    Invalid object name ‘__MOMManagementGroupInfo__’.
    The module ‘p_GroomTypeSpecificLogTables’ depends on the missing object ‘dbo.p_InternalJobHistoryInsert’. The module will still be created; however, it cannot run successfully until the object exists.
    The module ‘p_GroomTypeSpecificLogTables’ depends on the missing object ‘dbo.p_InternalJobHistoryUpdate’. The module will still be created; however, it cannot run successfully until the object exists.
    The module ‘p_GroomTypeSpecificLogTables’ depends on the missing object ‘dbo.p_InternalJobHistoryUpdate’. The module will still be created; however, it cannot run successfully until the object exists.
    The module ‘p_InstallTypesAndReltypes’ depends on the missing object ‘dbo.p_TypeDeletePermanent’. The module will still be created; however, it cannot run successfully until the object exists.
    The module ‘p_InstallTypesAndReltypes’ depends on the missing object ‘dbo.p_MakeBulkInsertSprocForType’. The module will still be created; however, it cannot run successfully until the object exists.
    The module ‘p_InstallTypesAndReltypes’ depends on the missing object ‘p_AddBlobTriggersToTable’. The module will still be created; however, it cannot run successfully until the object exists.
    The module ‘p_InstallTypesAndReltypes’ depends on the missing object ‘dbo.p_CreateViewForRelationshipType’. The module will still be created; however, it cannot run successfully until the object exists.
    Msg 208, Level 16, State 1, Line 11
    Invalid object name ‘PartitionAndGroomingSettings’.
    Msg 208, Level 16, State 1, Line 32
    Invalid object name ‘ManagedType’.

  32. Dennie says:

    Did anyone ever find the solution for :

    Msg 208, Level 16, State 1, Procedure GetStateDaily, Line 63
    Invalid object name ‘vManagedEntityMonitor’.
    Msg 208, Level 16, State 1, Procedure GetStateHourly, Line 63
    Invalid object name ‘vManagedEntityMonitor’.

  33. Dennie says:

    Let me clarify, I get these errors when I’m trying to run the Data Warehouse sql command