Friday, February 15, 2013

simple parallel processing with make

I've used xargs a fair bit for some simple, local, parallel processing. I was just recently reminded of another clever and simple solution that a friend came up with.

I'll let the code do the talking; here's the basic bash script (stored as an executable):
#!/bin/bash

if [[ $# -ne 1 ]]; then
   echo "Usage: cat commands.txt | $(basename $0) <num processes>"
   exit 1
fi

(while read line; do
  echo -e "$((++i)):\n\t$line";
done; 
echo "all:" $(seq 1 $i)) | make -B -j $1 -f <(cat -) all
This uses a couple of clever tricks. I especially like the use of process substitution in the make command (substituting the 'cat -' for the input makefile).

This approach allows the commands in commands.txt to redirect their own output as they need to (using '>', '2>', '&>', etc.)

No comments:

Post a Comment