An easy to use self organising system to run script based jobs in parallel over many machines.
Did you ever have the problem that you need to run many python (or matlab or octave) skripts in parallel but never got around to install a fully fledged parallel computing system because that is just too much effort? Then this system is for you!
All you need to do is share one folder on all the machines that should run your jobs, e.g. using sshfs. Run as many client.py jobs in that folder on your machines as you want to have workers (for single-threaded workers as many as cores for example). Each worker will monitor the folder and run any j[xxxx].py skript in it and write the output into a j[xxx].log file. All jobs that already have a log file will be igored. Adaption to run other skripts is easy - just set the correct extention in line 26. ;-)
You can let the system create job files out of your original scripts with create_job.py.
A more detailed description can be found in the included pdf.
I developed this job system for my doctoral thesis around 2005/2006. So please excuse the old fashioned coding style. Obviously it runs with python 2 though adaption to python 3 will be minimal.
- Ingo R. Keck, ICA Incompleto Paralelo: Una nueva herramienta para el análisis de datos fMRI; Parallel Incomplete ICA – A New Tool to Analyse fMRI Data Sets, Doctoral thesis, University of Granada, 2006.
A copy of the relevant parts of the thesis is included as inICA.pdf.
Copyright 2006 Ingo R. Keck
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.