Rebuild a management group (disaster recovery)

Occasionally an Operations Manager 2007 management group might need to be completely recovered for some reason. This would be required in the case of unrecoverable systems due to database corruption, or complete loss of database and/or other key components, and there are no backups to recover these systems or components. I have found this happens more often in lab environments, where there is a lot of unmanaged testing and corruption is more likely to occur.

This article provides a high-level, step-by-step process to completely remove the core Operations Manager server roles and components in order to start fresh with a new installation of the management group. It is a two-phased approach, which includes decommissioning the current installation and then installing a new Management Group. The process outlined here offers a relatively short recovery time, since we will not need to re-install any agents on the computers that were previously managed.

Exclamation

I have been successful using this process on more than a couple occasions.  However, these procedures have not been tested and approved by the product group.  Using this guidance should be a last resort, after calling into Premier support and attempting to resolve any corruption issues (especially in a production environment).

Also, these procedures have only been tested in an Operations Manager 2007 R2 environment.  Although Operations Manager 2007 R2 is mentioned throughout this guidance, the procedure should work the same on SP1 and RTM versions.

 

Phase 1

If you plan to use all or some of the computers that were previously used in the inoperable management group, we need to first ensure that all Operations Manager 2007 related components are removed. This will provide a clean slate for installing a new management group in phase 2.

Decommissioning the inoperable management group

Since we’ll be keeping the same computer names in this guidance, and assuming the management group name and computer names of core server roles will remain the same, we do not need to uninstall any agents. After installing the new management group, all agents should continue to report into the same management servers. 

Exclamation

If you plan to rename the management group or computer names of core server roles, there will be additional configuration necessary on the agents. This is not the easiest recovery path, especially if under time constraints. These additional configuration steps are not outlined in this document.

 

High-Level steps

  • Uninstall the Operations Manager 2007 Report Server role from Add/Remove Programs.
    • If you would like to retain the data in the Data Warehouse database for reporting purposes, do not delete the Data Warehouse database. When you reinstall the new management group, you can reconnect to the same Data Warehouse and report on data previously collected. Otherwise, the Data Warehouse database can be deleted after uninstalling the Report Server role component.
  • Uninstall the Operations Manager 2007 Gateway role from any computer hosting this component in the management group. This can be done in Add/Remove Programs.
  • Uninstall each Operations Manager 2007 Management Server role from any computer hosting this component in the management group. This can be done in Add/Remove Programs.
  • Uninstall each Operations Manager 2007 Web Console role from any computer hosting this component in the management group. This can be done in Add/Remove Programs.
  • Uninstall the Operations Manager 2007 Root Management Server role from the computer hosting this component in the management group. This can be done in Add/Remove Programs.
  • Uninstall the Operations Manager 2007 Operational Database role from the computer hosting the operational database in the management group. This can be done in Add/Remove Programs.
  • After uninstalling the Operational Database component, delete the database using SQL Management Studio.
    • Also check the file system to ensure the .mdf and .ldf file for the operations database has been deleted.

Exclamation

If there is any issue uninstalling any of the above Operations Manager 2007 components from any computer, refer to the article reference in the link below for instructions on how to do a brute force uninstall. https://blogs.technet.com/b/jonathanalmquist/archive/2010/07/22/brute-force-uninstall-of-management-group.aspx

Phase 2

After phase 1 is complete, we can continue to phase 2 by installing a new management group.

Installing the new management group

This section is provided as a high-level overview of installation steps. It is not intended to replace the installation procedures outlined in Microsoft official documentation. To understand requirements, supported configuration and details installation steps, please refer to the official documentation referenced in the link below.

https://technet.microsoft.com/en-us/systemcenter/om/bb498235.aspx

High-level steps          

  • Prepare for installation by gathering required information
    • Computer names and which component each computer will host.
    • Service account names and passwords.
      • SDK Account
      • Management Server Action Account
      • Data Reader Account
      • Data Warehouse Write Action account
  • Perform all installation activities with a domain user account that is a member of your “SCOM Admins” domain security group.
  • Ensure the account that you’ll use for installation procedures is a member of the Administrators local group on the server where you are installing each component.
    • A common configuration that provides the necessary rights and privileges throughout these tasks is to:
      • Add the “SCOM Admins” domain security group to the Administrators local group on each computer where you’ll be installing an Operations Manager 2007 role.
      • Give your account SA access to the SQL instance hosting the operations and data warehouse databases.
  • Logon to the computer that will host the operational database. Run SetupOM.exe from the Operations Manager 2007 R2 installation media. Choose to install Operations Manager 2007 R2. When the component selection screen appears, select only the Operational Database component. Setup will create the database objects in the selected SQL Server instance.
  • After the operational database has been created, logon to the computer that will host the Root Management Server component and run SetupOM.exe from the SCOM R2 installation media. Choose to install Operations Manager 2007 R2. When the component selection screen appears, select only the Management Server, Console and Command Shell options (Command Shell is optional, and requires Powershell to be installed).
  • After the Root Management Server has been installed, logon to a computer that will host a Management Server component and run SetupOM.exe from the SCOM R2 installation media. Choose to install Operations Manager 2007 R2. When the component selection screen appears, select only the Management Server, Console and Command Shell options (Command Shell is optional, and requires Powershell to be installed).
  • Repeat this step until all Management Server roles are installed.
  • If there will be a Gateway Role installed in the management group, follow the installation procedures outlined in our official installation documentation. This includes:
    • Installing a certificate on the computer hosting the Gateway Role and on the computer hosting the Management Server role which this Gateway Role will report to.
    • Run the Gateway Approval Tool.
    • Logon to the computer that will host the Gateway Role and run SetupOM.exe from the Operations Manager 2007 R2 installation media. Choose to install Operations Manager 2007 R2 Gateway Role.
    • Repeat these steps for each Gateway Role that will be installed in the management group.
  • If you have used the same computer and management group names in this rebuild procedure, the existing agent installations should continue to report into the management group. In order for these agents to start communicating with the management group, you’ll need to change the security settings. This is done by launching the Operations Console and navigating to Administration > Settings > Security, and selecting “Review new manual agent installations in pending management view”. Existing agent installation should begin reporting into the Administration > Pending Management node for approval.

Exclamation

The default settings are to review new manual agent installations. In the case of a management group rebuild, the previously installed agents will be considered manually installed. You can either keep this setting and approve agents as they appear in Pending Management, or you can check the security setting option under Administration to automatically approve new manually installed agents. Checking this option will allow the previously agent-managed computers to automatically go into the Agent-Managed node without further administrative action.

  • Logon to the computer that will host the Operations Manager 2007 Report Server role.
    • Launch Internet Explorer and navigate to https://localhost/reports. If the default SRS page does not appear, you’ll need to reset SRS to its default state.
    • After SRS is restored to its default state, run SetupOM.exe from the SCOM R2 installation media. Choose to install the Operations Manager 2007 R2 Reporting Role. When the component selection screen appears, select only the Report Server role component (not the Data Warehouse, since this is already installed from the previous Management Group).

Exclamation

If you chose to delete the data warehouse in this recovery procedure, you will need to install the data warehouse component on the computer hosting the SQL instance for the data warehouse before proceeding with the Report Server role installation.

    • Follow the prompts to install the Report Server role, selecting the data warehouse database that was previously installed in the management group.
  • When installation of all components has completed, install the latest Operations Manager 2007 cumulative update and import the latest Operations Manager 2007 core management packs. Check the catalog for the latest SCOM MP.  Check Kevin Holman’s hotfix page for up to date information about cumulative updates and important hotfixes.