parallelism or multiple threads in BASH

I was working on a Nagios check the other day, which was looping over a number of items and assessing their status. I found that the check would routinely time out if the number of items was large. So, to make it scale I thought spawning the individual steps to run in parallel and then evaluating the status at the end would be a nice way to do it. There are many programming languagues that make this quite easy, but BASH isn't really know as one of those. Since the script was running pretty well as written in BASH I set out to try and stick with it rather than redoing the whole script. Here is what seems to work for this case.

Basically I rely on starting a background job, named pipe and the bash built in wait. The named pipe provides the ability to not have many individual output files, since I don't have much output to contend with.

Here is what the skeleton script looks like:

#!/bin/bash

OUTPUT="/tmp/.test"
mkfifo $OUTPUT &> /dev/null 

# this is the slow running function
slow () {
  echo "begin $1"
  sleep 3;
  echo "slow $1" > $OUTPUT &
  echo "end $1"
}

# here the function is spwaned
for x in {1..3} ; do
  echo "spawn $x";
  slow $x &
done

# we wait for all the child processes to finish
echo waiting
wait
echo done waiting

# then we read the output
cat $OUTPUT
rm $OUTPUT

If you run the script you'll see something like this:

# ./foo.sh 
spawn 1
spawn 2
begin 1
spawn 3
waiting
begin 3
begin 2
end 1
end 3
end 2
done waiting
slow 1
slow 2
slow 3
#

You can see that the processes don't even finish in order, which is exactly what I needed to deal with.

The interesting bits are that the function call must be backgrounded and the same with the output to the named pipe. It's also possible trap the exit code of the slow script.

One possible negative is that if the script is interrupted you might end up with some or all of the output from the last time the script was run. Then again, that may also be desirable outcome.

Here's to named pipes!

\@matthias

SOURCE URL : http://opennomad.com/content/parallelism-or-multiple-threads-bash