Automation – Orchestrator Back to Basics – Use Cases Spotlight (2 of 5)

This is post #2 out of 5 in the “Orchestrator – Back to Basics – Use Cases” series, full series posts are located here:

  • #1 : Alert Remediation, where automation is used to monitor specific situations, and react automatically. This is also the introduction post to the series.
  • #2 : Maintenance tasks, where recurring tasks are being handled in a consistent and automated manner, triggered manually or when a specific condition is met (this post!)
  • #3 : Provisioning and Change Management Automation, where automation handles backend processing or user requests from a service catalog or any provisioning process worth automating in your context
  • #4 : Cross-technology integration . Here automation can be used to integrate otherwise silo’d technologies, or help in better together and migration scenarios (integrate a monitoring solution to a manager of manager, or into a ticketing solution)
  • #5 : “Miscellaneous” scenarios, like dynamic resource allocation, and new user onboarding as examples

Use Case #2 : Maintenance tasks

In this post, we’ll see how automation can help manage maintenance tasks that may be having an important daily or weekly burden on operational teams. Benefits include freeing up some useful time for the associated teams, and handling these recurring situations in a consistent manner. For illustration purposes, two scenarios will be covered :

  • Using automation for “advanced” patching (executing pre-flight and post-patching checks, restarting servers in the right order,…)
  • SQL Server maintenance tasks

Scenario #1 : Advanced Patching

As you know, patching is generally more complex than just applying a hotfix on a single server. It’s also about:

  • Making sure the server restarts just fine, and its critical services are coming back up too
  • Working through other servers dependencies, for instance by patching backend servers before frontend servers (or the other way round)
  • Disabling/enabling monitoring to avoid affecting Service Level Agreements (SLAs)
  • Creating a ticket if anything goes wrong along the way
  • Handling clustered installations
  • [Place your own requirement here!]

Runbook 1 : Patching a single server with a single patch

Let’s go through a first Runbook showing how we could patch a single server with a single patch, and handle some of these prerequisites:

image

Some important parts of this sample Runbook:

  • The first activity (1) shows the parameters for the Runbook : A patch number and a server name

image

  • After starting maintenance mode for the target server in Operations Manager, the group of activities (2) add the server in a Configuration Manager collection named according to a defined standard. This would be different in every organization. We could also check if a collection exists and create it on the fly. Actual steps would be different if you were using another patching solution. In the case of Configuration Manager, the collection membership also needs to be updated, and the policy needs to be refreshed on the target machine. While not mandatory, this explains why these steps are in the Runbook. They speed up the process, assuming we have a narrow maintenance window to do the patching.

image

image

  • This activity (3) checks deployment status, with a loop (maximum number of loops or exit conditions might also differ in our environment – this is just for illustration purposes). An error is being returned if the deployment failed, which could be picked up by another Runbook to handle ticketing through a consistent process.

image

  • Activties (4) are then handling a potential reboot. It might actually be better to let Configuration Manager handle this, so that the Runbook could just wait for the last compliance status instead of tracking deployment. With this approach, the Runbook would be modified to go directly from “Get Deployment Status” to “Get Service Status”. It is always a good idea idea to let the orchestrated solutions handle as much intelligence as they can, to keep Runbooks simple.
  • In activities (5), a specific Windows service or group of Windows services is being checked. If everything goes fine, the Operations Manager agent resumes standard monitoring.

 

The previous Runbook could be called through a simple self-service interface for the application owners, or called by a larger patching process as we will now see with the second Runbook.

Runbook 2 : Patching an application being made of multiple servers and two tiers

The second Runbook below shows how you could patch a service in sequence, calling the first Runbook for individual server patching. In this scenario, an application has two tiers, conveniently called “Frontends” and “Backends”. Assuming we need to patch the backend tier before the frontend tier, the following Runbook just retrieves the right servers in Orchestrator groups, and then calls the previous Runbook to go through patching. Junction activities are being used to wait for all branches in each tier, before continuing processing.

image

Because this is just a sample for illustration purposes, this example only takes a patch number as the parameter, and assumes the same application or group of servers is being patch every time:

image

In reality, you would likely want this to be more dynamic, and you could have two parameters : the patch and the application to update. The Runbook could then query a CMDB to retrieve the different tiers and patching orders, instead of the basic and static Orchestrator groups used in this example.

Note : While this example was illustrated using Orchestrator, Service Management Automation (SMA) could also be used. Thispost from Thomas Roettinger shows how you can use SMA to achieve offline patching of virtual machines, which is another advanced patching scenario.

Scenario #2 : SQL Server Maintenance Taks

When discussing with Database Administrators, there are recurring tasks that can take a fair amount of time, when managing a large number of databases. One of these tasks include managing the health of indexes. The following Runbook showcases how this process could be automated, or “semi-automated” if you wanted to include approvals for some of the steps.

Note : In the download package, this is one of the Runbooks still provided as a “design sample” and the SQL scripts are not provided. We are looking if we could fine tune and document this scenario and scripts better so that we can publish them. But hopefully this will already give you a good idea of what's possible!

image

Some important parts of this sample Runbook:

  • (1) The “Initialize Data” activity just uses a single parameter, the server name hosting the databases to analyze. The Runbook then retrieves the list of databases on this server:

image

image

image

  • For each database, a SQL script is being excuted through SQLCMD, to check indexes. This could be triggered in other ways, through SQL Server PowershellShell (SQLPS) for example.

image

  • Based on the output of the command line, activities in group (3) handle the situation :
    • If a Reorganize is needed, it can be carried right away with the associated script.
    • If you look at the “Rebuild” branch, you can see the rebuild process is not carried away directly, but a change request is being created. This is why I mentionned earlier that this could be a “semi-automated” process : You might still want some of the actions to require some sort of approval. In this situation, the approval would be done in the IT Service Management (ITSM) solution, per the existing processes. And then, another Runbook could be monitoring approvals to trigger the rebuild process when it’s been approved. Actual server and database names used in the post-approval Runbook would be retrieved from the Change Request, as the first Runbook would have stored them here. Notifications could also be directly handled by the ITSM solution, depending on your processes.

              image


  • Previous post in the series : #1 : Alert Remediation, where automation is used to monitor specific situations, and react automatically. This is also the introduction post to the series.
  • Next post in the series : #3 : Provisioning and Change Management Automation, where automation handles backend processing or user requests from a service catalog or any provisioning process worth automating in your context