Background Jobs (PowerShell)

Overview

This feature allows Cmdlets to run as background jobs. Its fairly easy to work out how to make individual Cmdlet's or PowerShell scripts to run as jobs (see about_Jobs, but running functions are a bit more of a pain as the job runs in a new scope (so any functions, variables etc that are defined in your scripts scope, have no meaning in the background job's scope). Therefore functions, etc, have to explicitly included as a script block in the background job...

# First define the script block and function
$funky = {
    function TestJob {
        $processList = Get-Process
        Return $processList                          # Yes could be achieved in one line, but wouldn't be much of a function!
    }
}

# Start job
$job = Start-Job -ScriptBlock {TestJob} -InitializationScript $funky

$job | Format-List *                                 # Displays created $job object

Wait-Job -job $job                                   # Wait on completion of job
Receive-Job -job $job                                # Gets result of $job (ie result of Get-Process)
Get-Job                                              # Shows list of jobs (current and completed)

Arguments have to be passed through to the job through the -InputObject parameter, which isn't particularly pretty. For further info see http://robertrobelo.wordpress.com/2010/03/14/background-jobs-input-gotcha/ for a decent explanation, though I do kind of cover this below.

Background jobs normally run with Below Normal priority (most processes, including PowerShell normally run as Normal priority. For more info on checking or changing see PowerShell Process Priority.

Script Block or Script File ?

Your background task can either take the form of a script block, or a script file. Personally I prefer to keep everything in one script as it makes organisation and maintenance easier, up to an extent. However, there is a limit to size of a script block, though I've no idea what it is (it may relate to maximum script line length that PowerShell can handle).

A script I was trying to mangle into running as a background job was large (20 KB, nearly 1000 lines), and it wouldn't run. Unfortunately I was short on time and didn't have a convenient way to test where the breaking point was. But your jobs will fail if they're too big.

Job Control

Below is a fuller example of using background jobs to manage multiple work streams

Further reading...

• http://ryan.witschger.net/?p=22 - Multi-Threading in PowerShell V2

$funky = {
    function TestJob {
        $var = $Input.‘<>4__this’.Read()
        Write-Host "This is job " $var[0]
        Start-Sleep $var[1]
    }
}

$jobs = @()
$job = "" | Select Name, Vars, State, Obj
$job.Name = "Job1"
$job.Vars = ($job.Name, 10)
$jobs += $job
$job = "" | Select Name, Vars, State, Obj
$job.Name = "Job2"
$job.Vars = ($job.Name, 5)
$jobs += $job

foreach ($job in $jobs) {
    Write-Host ("Starting " + $job.Name)
    $job.Obj = Start-Job -ScriptBlock {TestJob} -InitializationScript $funky -Name $job.Name -InputObject $job.Vars
    $job.State = "Running"
}

# Idle loop
While (1) {
    $JobsRunning = 0
    foreach ($job in $jobs) {
        if ($job.State -ne $job.Obj.JobStateInfo.state) {
            Write-Host ($job.Name + " state now " + $job.Obj.JobStateInfo.state)
            $job.State = $job.Obj.JobStateInfo.state
        }
        if ($job.State -eq "Running") {
            $JobsRunning += 1
        }
   }
   Write-Host ("$JobsRunning of " + $jobs.count + " jobs still running")
   if ($JobsRunning -eq 0) {
       Break
   }
   Start-Sleep 1
}
    
Write-Host "All finished...!"

# To see output from jobs
# Get-Jobs              - shows list of jobs
# Receive-Job -Id x     - shows the data returned from the job

Transcripts / Logging

I haven't been able to get transcripts working properly with background jobs, at least, not to my liking. It is possible to capture the console output from a background job into the transcript of the parent or master script. But if you're running a large background job script and want to capture the transcript/logging from job separately you have to faff around - you can't just start and stop transcribing from within the (child) background job script, it won't write anything to disk.

Similarly, you can't redirect the output from Receive-Job to a file, you'll lose some of the output (I think this may only capture StdErr and/or explicitly returned objects, standard Write-Host output is dropped).

One way around this is to stop the transcript for your master/parent script, then start a temporary trasncript to capture then return from your child job once its finished, so so something like...

if ($job.State.ToString() -eq "Completed") {
    Write-Host ($job.Name + " writing log to job-" + $job.Name + ".log")
    # Nasty logging handling (Receive-Job StdOut to console only, can't redirect to file, can only catch StdErr to file)
    Stop-Transcript
    Start-Transcript -Path ("job-" + $job.Name + ".log")
    Receive-Job -Id $job.job.Id
    Stop-Transcript
    Start-Transcript -Path $Logfile -Append
}

Gotchas

• Working Directory
  • The background job script runs in a new context, therefore it runs in the default path. If you tend to run your scripts from a non-default path, and need to read/write files, get the current directory using Get-Location and pass it to your job as a parameter, then use Set-Location in your job.
• VMware PowerCLI 64 bit
  • Due to a bug you can't launch VMware PowerCLI background jobs when running in a 64-bit environment. Doing so cause launched PowerShell instances to exhibit high CPU and memory usage and eventually PowerShell will crash. Whether its a problem in PowerShell or in PowerCLI is unclear (though I suspect PowerShell), the problem definitely effects PowerCLI v4.1u1 and v5, and probably affects all versions.
  • Therefore launch from 32-bit PowerCLI shell. To run scripts in 32-bit see Force 32-Bit, to detect in a script see Useful One-Liners.