Dec 30, 2011

Multi-processor Work Queue script

Keywords:  bash
Depends on:  Nothing
Download:  thread_task

Certain processes are not multi-threaded.  For example, bzip2.  If we have a set of files which needs to be compressed, bzip2 can be handed the list of files and it'll compress the files one after another.  On a single processor machine, this works well.  On a multi-processor machine though, all but one processor are idle.

What's required is for the files to be placed in a queue and each bzip2 process is handed one file to compress.  When it's finished, the next file in the work queue is processed until all the files are processed.

The thread_task shell script is a generic work queue script.  It will run any command against a set of files.  By default, it will determine how many processors are on your machine and run at most that many simultaneous processes.  The script has a nice switch as well.

Todo
  • Enable the list of files to come from stdin so we can pipe the output of, say, find.

No comments:

Post a Comment