Understanding Sequential vs. Parallel Processing of Runbook Activities

I saw a question in the forums the other day by a person who asked how you should expect activities in a workflow to be executed when one of the activities generates multi-value data. In other words, one of my activities returns a list of things. How do the next activities react? Will they run in parallel or will they run sequentially? The answer, like so many things, is that it depends. It depends on how you have the activities in your runbook structured.

Let’s start out with the simplest case. I have a Run .Net Script activity that runs the following command:

$collection = gci C:\ | Select Name

It then publishes the variable Collection to the data bus so that subsequent activities can access the data. When this activity runs, it returns a list of files and directory names, which is returned as multi-value data (a collection or list of data items). In the simplest scenario, there is just one activity following the Run .Net Script activity. I’ll use a “Send Event Log Message” activity because I won’t have to deal with any file locking issues if I was writing to a file and activities are running in parallel.

image

Now let’s assume that the script returns a list of 10 items. This causes the next activity to run 10 times, but does it run sequentially or in parallel? The answer is that they run sequentially.

Within a single runbook, you can think of the process flow like passing a token from activity to activity. Generically, an activity can only handle one token at a time. Some activities take in one token and generate multiple tokens, as in the above example, the Initialize Data activity sends one token to the Run .Net Script activity, which generates 10 tokens from the result of the script (or it could generate just 1 token if you had the “flatten” setting checked). However, it doesn’t pass all 10 tokens off to the next activity at once, it has to pass them one at a time. When the next activity passed on the token to the activity after that (or it “falls off the end of the runbook”), it can accept another token.

So what happens when you have even more activities following the one that generates multiple values? Take the following example:

image

What happens here is not quite what you might expect from the previous description. When these activities run, the Run .Net Script activity generates 10 tokens and passes them off, one by one to the first Send Event Log Message activity as that activity processes them. The interesting thing is that the first Send Event Log Message activity doesn’t immediately pass off the token to the next activity. It holds on to all of them until it’s done receiving them from the Run .Net Script activity and after it processes the last one, then it starts sending to the next activity. So the process looks like this.

  1. Initialize Data
  2. Run .Net Script
  3. Send Event Log Message
  4. Send Event Log Message
  5. Send Event Log Message
  6. Send Event Log Message
  7. Send Event Log Message
  8. Send Event Log Message
  9. Send Event Log Message
  10. Send Event Log Message
  11. Send Event Log Message
  12. Send Event Log Message
  13. Send Event Log Message (2)
  14. Send Event Log Message (2)
  15. Send Event Log Message (2)
  16. Send Event Log Message (2)
  17. Send Event Log Message (2)
  18. Send Event Log Message (2)
  19. Send Event Log Message (2)
  20. Send Event Log Message (2)
  21. Send Event Log Message (2)
  22. Send Event Log Message (2)

As long as you understand that this is the way it works, then this isn’t a surprise to you and you can build your runbooks appropriately around this functionality. What about branching in multiple directions from an activity at the same time? Do those operate in parallel? Well, because they’re still in the same runbook, the answer is no. Take the following runbook:

image

When this runbook executes, the activities run in this order:

  1. Initialize Data
  2. Run .Net Script
  3. Send Event Log Message
  4. Send Event Log Message (2)
  5. Send Event Log Message
  6. Send Event Log Message (2)
  7. Send Event Log Message
  8. Send Event Log Message (2)
  9. Send Event Log Message
  10. Send Event Log Message (2)
  11. Send Event Log Message
  12. Send Event Log Message (2)
  13. Send Event Log Message
  14. Send Event Log Message (2)
  15. Send Event Log Message
  16. Send Event Log Message (2)
  17. Send Event Log Message
  18. Send Event Log Message (2)
  19. Send Event Log Message
  20. Send Event Log Message (2)
  21. Send Event Log Message
  22. Send Event Log Message (2)

In this instance, the Run .Net Script activity alternates the tokens it sends out to each direction in the outbound links. Ok, you might be asking what happens when you combine the two versions and have a branch structure with multiple subsequent activities? Something like this:

image

In this example, I am using Run .Net Script activities to write to the event log because I can control the source name better and provide more readable output when looking at a lot of events. So what happens when I run this? Here’s what the event log looks like:

image

As you can see, the pattern follows the original branched runbook model I showed before, alternating back and forth from branch 1 to branch 2 until it finishes sending tokens out from the main script activity. It then combines the functionality of the previous multiple following activities example in that each “A” activity (1a and 2a) collects all the tokens until there are no more being received, and then sends them to the next activity, and the branches again alternate so that activities 1b and 2b are run in alternating fashion until the entire runbook is done.

So what you can expect from this is that when you have a non-exclusive branch (also called a split) where both sides are run without conditions or where both conditions are true, you will end up running activities in an alternating manner up until they rejoin via a Junction activity.

Just to take it one step further (because I wanted to know), what about when you have multiple branches in a runbook? How does that work? Here’s an example:

image

Now look at the event log:

image

The same pattern emerges that all the “A” activities are done first, then the “B” activities are done. However, note how the second branch is handled. When it hits the second branch, “branch 1” now does two activities for every one activity “branch 2 runs”. This means that when an activity runs and passes on its token(s) to the next item in the runbook, it will actually evaluate all the outgoing link conditions and process those as a set before relinquishing command to the other branch.

Parallel Processing

So how do you get the runbook process things in parallel? Well, within a single runbook, you can’t do it. It takes a combination of a parent and child runbook, and even then, it’s not *truly* parallel, it’s *mostly* parallel. I’ll explain…What you have to do is place the activities you want to run in parallel in a separate runbook, and then use the Invoke Runbook activity to call it.

image

image

You’ll then need to do two more things. First, you need to make sure that the Invoke Runbook activity has the “Wait for Completion” box unchecked.

image

Next, you’ll need to set the properties of the child runbook to allow multiple concurrent runbooks to be executed. Right-click on the runbook tab and select Properties, and then click on the Job Concurrency tab and set the number of concurrent runbooks allowed to a higher number.

image

Note: the number you set here can still be limited to a maximum number of overall running runbooks on the Runbook Server as determined by the throttling limit. For more info on that setting, see How to Configure Runbook Throttling.

When you leave the “Wait for completion” box unchecked, it basically triggers the other runbook to start, and as soon as it does, it considers the action complete and starts the next one, regardless of where the previous one is in its execution. So while the starting of the child runbooks happens sequentially, it happens so fast that it’s nearly parallel, and because each of the child runbooks runs in its own policymodule process, it can run independently of the others, and can run in parallel. Here is the event log output from the above runbooks:

image

As you can see, every instance of the child runbook ran and created an event log entry within the same second. If that’s not parallel execution, I don’t know what is Smile

So hopefully this has given you a better understanding of how runbook activities flow and how data passes across them in a runbook. Feel free to create your own test runbooks and try out your own scenarios.