Makeflow is Copyright (C) 2009 The University of Notre Dame. This software is distributed under a BSD-style license. See the file COPYING for details.
You can run a Makeflow on your local machine to test it out. If you have a multi-core machine, then you can run multiple tasks simultaneously. If you have a Condor pool, then you can tell Makeflow to run your jobs on the Condor pool. You can also submit Makeflow itself as a Condor job, making it robust to system failures. Finally, if you don't already have a batch system, Makeflow comes with a system called Work Queue that will let you distribute the load across any collection of machines, large or small.
Makeflow is part of the Cooperating Computing Tools. You can download the CCTools from this web page, follow the installation instructions, and you are ready to go.
Makeflow attempts to generate all of the target files in a script. It examines all of the rules and determines which rules must run before others. Where possible, it runs commands in parallel to reduce the execution time.
Here is a real Makeflow that does something more complex: it downloads an image from the web, creates four variations of the image, and then combines them back together into an animation. The first and the last task are marked as LOCAL to force them to run on the controlling machine.
# This is a more complex example of Makeflow. CURL=/usr/bin/curl CONVERT=/usr/bin/convert URL=http://www.cse.nd.edu/~ccl/images/capitol.jpg capitol.montage.gif: capitol.jpg capitol.90.jpg capitol.180.jpg capitol.270.jpg capitol.360.jpg LOCAL $CONVERT -delay 10 -loop 0 capitol.jpg capitol.90.jpg capitol.180.jpg capitol.270.jpg capitol.360.jpg capitol.270.jpg capitol.180.jpg capitol.90.jpg capitol.montage.gif capitol.90.jpg: capitol.jpg $CONVERT -swirl 90 capitol.jpg capitol.90.jpg capitol.180.jpg: capitol.jpg $CONVERT -swirl 180 capitol.jpg capitol.180.jpg capitol.270.jpg: capitol.jpg $CONVERT -swirl 270 capitol.jpg capitol.270.jpg capitol.360.jpg: capitol.jpg $CONVERT -swirl 360 capitol.jpg capitol.360.jpg capitol.jpg: LOCAL $CURL -o capitol.jpg $URLNote that Makeflow differs from Make in a few important ways. Read section 4 below to get all of the details.
% makeflow example.makeflowNote that if you run it a second time, nothing will happen, because all of the files are built:
% makeflow example.makeflow makeflow: nothing left to doUse the -c option to clean everything up before trying it again:
% makeflow -c example.makeflowIf you have access to a Condor Pool, then you can direct Makeflow to run your jobs there:
% makeflow -T condor example.makeflowTo submit Makeflow as a Condor job that submits more Condor jobs:
% condor_submit_makeflow example.makeflowYou will notice that a workflow can run very slowly if you submit each batch job to a system like Condor, because it typically takes 30 seconds or so to start each batch job running. To get around this limitation, we provide the Work Queue system. This allows Makeflow to function as a master process that quickly dispatches work to remote worker processes.
To begin, let's assume that you are logged into a machine named barney.nd.edu. start your Makeflow like this:
% makeflow -T wq example.makeflowThen, submit 10 worker processes to Condor like this:
% condor_submit_workers barney.nd.edu 9123 10 Submitting job(s).......... Logging submit event(s).......... 10 job(s) submitted to cluster 298.Or, you can start workers manually on any other machine you can log into:
% worker barney.nd.edu 9123Once the workers begin running, Makeflow will dispatch multiple tasks to each one very quickly. If a worker should fail, Makeflow will retry the work elsewhere, so it is safe to submit many workers to an unreliable system.
When the Makeflow completes, your workers will still be available, so you can either run another Makeflow with the same workers, or you can remove the workers like this:
% condor_rm 298
# This is an incorrect rule. output.txt: ./mysim.exe -c calib.data -o output.txtHowever, the following is correct, because the rule states all of the files needed to run the simulation. Makeflow will use this information to construct a batch job that consists of mysim.exe and calib.data and uses it to produce output.txt:
# This is a correct rule. output.txt: mysim.exe calib.data ./mysim.exe -c calib.data -o output.txt
BATCH_OPTIONS=Requirements = (Memory>1024)\nRank = Mips
% makeflow -D example.makeflow | dot -T gif > example.gif