Parallel Processing with jobs in PowerShell – Juggling multiple chainsaws


I have to be honest, back in 2010 I was firmly entrenched in VBScript and had no interest in learning anything new.  Then fate threw me a curve ball when I was presented with a task to fix over 9,000 workstations that had lost their "parent" antivirus server.  Each machine needed to have the exact same set of commands executed on it, a new file copied and service restarted, to fix them so I created a quick  VBScript to connect to each machine and perform the fix actions. The only bad part about this script was that it took days to run against the machines, processing each one sequentially in a For loop before moving on to the next. I did some research on multi-threading with VBScript and kept getting results that referred to PowerShell "jobs", which gave me the motivation to start learning PowerShell.

For those of you who come from a Unix or Linux background, jobs will be a familiar concept since you can background any command by placing an "&" at the end of the command.  PowerShell Cmdlets sometimes include the "-AsJob" parameter that will allow the command to submitted in the background, releasing the console to be used for other commands. Alternatively, you can use the Start-Job CmdLet to run commands as Jobs. This also means that you can submit multiple jobs in the background and continue processing without waiting for each job to complete.  Keep in mind that there are limits to everything so keep your machine's resource consumption in mind as you test this concept out.  Submitting a large number of background commands could be resource intensive.

Below are two different ways to do a WMI Query as a job:

Get-WMIObject Win32_OperatingSystem -AsJob

or

Start-Job {Get-WMIObject Win32_OperatingSystem}

These jobs do all of their processing in the background and store the output until you receive them.  You can check on the status of a job by running Get-Job, the status is in the "State" column, which will show if the command is Running, Completed, or Failed.  Also notice the HasMoreData column.  This indicates that there is output to be retrieved from the job.  In the example output below, notice that the job is still running.

Id     Name            PSJobTypeName   State         HasMoreData     Location             Command

1       Job1             WmiJob                  Running    True                     localhost            get-wmiobject win32_op...

 

After the job has been kicked off, you can check on the status of the job by running the Get-Job command, noting the State and HasMoreData values. The State will change to Completed when the job has finished and the  HasMoreData value will indicate if there is output.

Get-Job

Id     Name            PSJobTypeName   State             HasMoreData     Location             Command

1       Job1              WmiJob                 Completed    True                    localhost            get-wmiobject win32_op...

 

Once the job has completed, you can use the Receive-Job cmdLet to get the data from the command.

Receive-Job -Id 1       <-----  NOTE, you could also do -Name Job1 here, either will work

The output of the command is now delivered to the console:

SystemDirectory : C:\Windows\system32

Organization:         Contoso

Build Number:       15063

Registered User:    ContosoUser

SerialNumber:

Version:                      10.0.15063

 

Note that the HasMoreData value has now changed to False after running the Receive-Job command:

Get-Job

Id     Name            PSJobTypeName   State             HasMoreData     Location             Command

1       Job1              WmiJob                 Completed    False                    localhost            get-wmiobject win32_op...

The important thing to remember here is that you have one chance to get the information from the job so make sure that you capture it in a variable if the output needs to be evaluated.

$JobOutput = Receive-Job -Id 1

Once you are done getting the job's output, the job will basically hang out there until you remove it by running the Remove-Job Cmdlet.

Remove-Job -Id 1

As you can tell, there are a lot of moving parts to this.  The PowerShell script that I created to address the antivirus client remediation task had to have a mechanism to control how many jobs I could submit at a time, as well as constant monitoring of the queue to receive input from the completed jobs, remove them, and refill the queue with new jobs.  To top it off, once I finished processing the queue of items, I had to write some code to wait for the last batch of jobs was completed to make sure that I received the output from all of them.  That is a lot of overhead.

Fast forward to today and we have PowerShell WorkFlows that make this process much easier to manage which I'll cover in another post.

Comments (0)

Skip to main content