Automation–Orchestrator Architecture and Runbook Deployment Process

Hello Readers and Viewers!

Today’s post will not have a video, but it will have some very popular and hopefully informative information about some “Best Practices” around Orchestrator Architecture and Runbook Deployment.

For those of you that have been following me for the past three or so years, this information may look very familiar. Well, that is because this is a “refresh” of the old “Opalis Architecture & Workflow Deployment Process ‘Best Practice Guide’” created and published back in November 2010.

To date, this information (available previously as .DOCX and .PDF) has been downloaded over 2200 times! This is why I believe it deserves that refresh and promotion to an actual blog post here on Building Clouds!

Let’s begin…

Orchestrator Architecture

Orchestrator Architecture Notes

• Network Connectivity between the Database(s) and connected Runbook Servers should never exceed 30ms for an optimal state of 10ms with zero packet loss.
• Network Connectivity between connected Runbook Servers and the Managed Devices they perform work on can have a higher latency value, but standard network traffic rules still apply.
  • Example: For large VHD file migrations between sites, try using FTP, instead of SMB file copy over the network.
• Connectivity and Runbook Execution between “Primary” and “Remote” Production sites can be achieved via the available REST Based oData Web Service.
• By default the maximum concurrent Runbook Instances is set to 50 – this is an extremely “safe limit” based on legacy settings. This limit can be expanded based on need and infrastructure capability. For more information about increasing this maximum, reference this TechNet Article.
• Click here for more information about “How to Configure the Orchestration Console Web Site or the Web Service for Load Balancing”.
• The following are great resources as well (from the TechNet Library): Orchestrator Architecture, Deploying System Center 2012 – Orchestrator, Administering System Center 2012 – Orchestrator, and Orchestrator (main landing page)

Runbook Deployment Process

A standard Software Development Lifecycle Process can be implemented/followed as far as Dev/Test to Production promotion is concerned. The following is an example process.

1. One or more “Dev/Test” Runbook Designers Develop and Test Runbooks outside of Production (possibly in “Dev/Test Pods”).
2. Developed, Tested and “Sanitized” Runbook Exports to be Promoted are submitted via a Change Control System as a Change Request to a CAB (or a resource where yes/no decisions can be made based on design, process, impact, etc.)
   • NOTE: “Sanitized” refers to the need for “Dev/Test” “Global Settings/Connections” to be removed from subsequent exports of the Runbooks to be promoted.
3. As a part of the Approval Process, the “Sanitized” Runbook file is imported into a Production Staging Environment (that mimics Production). Here, it is tested against a “Production Like” infrastructure.
   • NOTE: If changes are required, the Runbook should be sent back to Dev/Test and resubmitted for approval.
4. Upon Approval of the Change Request to Promote the Developed and Tested Runbook to Production, the file should be “Checked In” to a Source Control System (TFS is recommended). During this time, the existing Production Environment Runbooks should also be Exported to a file and “Checked In” to the Source Control System.
5. Once the current environment has been backed up, and the newly Approved Runbooks have been stored, they can be scheduled for deployment (Import) to the Primary and Remote Production Environment(s).
   • NOTE: It is critical that if more than one Production Environment (Remote or otherwise) exists that each one of them is backed up and updated at the same time, to keep the environments in sync.

Runbook Deployment Process Notes

• “*.ois_export” files should be treated as “Managed Code” and should be “Checked In / Out” from a Source Control System outside Orchestrator (TFS is recommended).
• Orchestrator DOES NOT have automatic “rollback” or version history built-in. These processes are manual and require storage and management of the “*.ois_export” file generated during the change process.
• Global Settings/Connections are in fact global and are a part of each export, regardless of Runbook usage. It is important to be mindful of what is being exported and imported at any given time. It is advised that a “Cleanup” Orchestrator Instance is created/used to “Sanitize” the Runbooks coming from the “Dev/Test Pods”.
• Runbook Changes (once implementation is in “steady-state”) should not occur often enough for the above process to be cumbersome. Because “*.ois_export” files should be treated as “Managed Code”, a Change Control Process should not be out of the question for any organization. In fact, it significantly decreases negative impact on the Orchestrator environment(s).
• It depends on the organization, but inclusion of ALL the Runbooks (regardless of which target environment they will execute against) may be desired in the “GOLD copy” for the Production Runbooks. Alternatively, the “GOLD copy” Production Runbooks could be broken up by “Target Execution Environment”.
• The above Architecture and Runbook Deployment Process can be applied to any size Implementation. The number of Runbook Servers, Databases, Remote Environments, Dev/Test Pods, etc. can scale as desired.

Since the old CodePlex version of this documentation was such a hit, I decided to make all the documentation available as a TechNet Gallery Contribution as well!

Click here for the TechNet Gallery Contribution for this Documentation!

For more information and tips/tricks for Orchestrator, be sure to watch for blog posts in the Automation Track!

enJOY!