makeflow - workflow engine for executing distributed workflows


makeflow [options] <dagfile>


Makeflow is a workflow engine for distributed computing. It accepts a specification of a large amount of work to be performed, and runs it on remote machines in parallel where possible. In addition, Makeflow is fault-tolerant, so you can use it to coordinate very large tasks that may run for days or weeks in the face of failures. Makeflow is designed to be similar to Make, so if you can write a Makefile, then you can write a Makeflow.

You can run a Makeflow on your local machine to test it out. If you have a multi-core machine, then you can run multiple tasks simultaneously. If you have a Condor pool or a Sun Grid Engine batch system, then you can send your jobs there to run. If you don't already have a batch system, Makeflow comes with a system called Work Queue that will let you distribute the load across any collection of machines, large or small.


When makeflow is ran without arguments, it will attempt to execute the workflow specified by the Makeflow dagfile using the local execution engine.


-b,--bundle-dir <directory>
Create portable bundle of workflow.
-c, --clean Clean up: remove logfile and all targets.
-f,--summary-log <file>
Write summary of workflow to file.
-h, --help Show this help screen.
-I, --show-input Show input files.
-m,--email <email>
Email summary of workflow to address.
-O, --show-output Show output files.
-v, --version Show version string.

Batch Options

-B,--batch-options <options>
Add these options to all batch submit files.
-j,--max-local <#>
Max number of local jobs to run at once. (default is # of cores)
-J,--max-remote <#>
Max number of remote jobs to run at once. (default is 1000 for -Twq, 100 otherwise)
-l,--makeflow-log <logfile>
Use this file for the makeflow log. (default is X.makeflowlog)
-L,--batch-log <logfile>
Use this file for the batch system log. (default is X.<type>log)
-R, --retry Automatically retry failed batch jobs up to 100 times.
-r,--retry-count <n>
Automatically retry failed batch jobs up to n times.
-S,--submission-timeout <timeout>
Time to retry failed batch job submission. (default is 3600s)
-T,--batch-type <type>
Batch system type: local, condor, sge, moab, cluster, wq, hadoop, mpi-queue. (default is local)

Debugging Options

-d,--debug <subsystem>
Enable debugging for this subsystem.
-o,--debug-file <file>
Send debugging to this file.
--verbose Display runtime progress on stdout.

WorkQueue Options

-a, --advertise Advertise the master information to a catalog server.
-C,--catalog-server <catalog>
Set catalog server to <catalog>. Format: HOSTNAME:PORT
-F,--wq-fast-abort <#>
WorkQueue fast abort multiplier. (default is deactivated)
-N,--project-name <project>
Set the project name to <project>.
-p,--port <port>
Port number to use with WorkQueue. (default is 9123, 0=arbitrary)
-Z,--port-file <file>
Select port at random and write it to this file. (default is disabled)
-P,--priority <integer>
Priority. Higher the value, higher the priority.
-W,--wq-schedule <mode>
WorkQueue scheduling algorithm. (time|files|fcfs)
-s,--password <pwfile>
Password file for authenticating workers.
--disable-cache Disable file caching (currently only Work Queue, default is false)

Monitor Options

-M,--monitor <dir>
Enable the resource monitor, and write the monitor logs to
--monitor-limits <file>
Use as value-pairs for resource limits.
--monitor-with-time-series Enable monitor time series. (default is disabled)
--monitor-with-opened-files Enable monitoring of openened files. (default is disabled)
--monitor-interval <#>
Set monitor interval to <#> seconds. (default 1 second)
--monitor-log-fmt <fmt>
Format for monitor logs. (default resource-rule-%06.6d, %d -> rule number)

Other Options

-A, --disable-afs-check Disable the check for AFS. (experts only)
-e, --export-as-dax Export the DAG in DAX format. (Pegasus)
-k, --syntax-check Syntax check.
-K, --preserve-links Preserve (i.e., do not clean) intermediate symbolic links.
-z, --zero-length-error Force failure on zero-length output files.

Display Options

-D,--display <opt>
Display the Makefile as a Dot graph or a PPM completion graph. is one of: dot Standard Dot graph file Display the file as interpreted by Makeflow ppm Display a completion graph in PPM format
--dot-merge-similar Condense similar boxes
--dot-proportional Change the size of the boxes proportional to file size
The following options for ppm generation are mutually exclusive:
--ppm-highlight-row <row>
Highlight row in completion grap
--ppm-highlight-file <file>
Highlight node that creates file in completion graph
--ppm-highlight-executable <exe>
Highlight executable in completion grap
--ppm-show-levels Display different levels of depth in completion graph


The following environment variables will affect the execution of your Makeflow:


This corresponds to the -B <options> parameter and will pass extra batch options to the underlying execution engine.


This corresponds to the -j <#> parameter and will set the maximum number of local batch jobs. If a -j <#> parameter is specified, the minimum of the argument and the environment variable is used.


This corresponds to the -J <#> parameter and will set the maximum number of local batch jobs. If a -J <#> parameter is specified, the minimum of the argument and the environment variable is used.

Note that variables defined in your Makeflow are exported to the environment.


Inclusive low port in range used with -p 0.


) Inclusive high port in range used with -p 0.


On success, returns zero. On failure, returns non-zero.


Run makeflow locally with debugging:
makeflow -d all Makeflow
Run makeflow on Condor will special requirements:
makeflow -T condor -B "requirements = MachineGroup == 'ccl'" Makeflow
Run makeflow with WorkQueue using named workers:
makeflow -T wq -a -N Makeflow
Create a directory containing all of the dependencies required to run the specified makeflow
makeflow -b bundle Makeflow


The Cooperative Computing Tools are Copyright (C) 2003-2004 Douglas Thain and Copyright (C) 2005-2011 The University of Notre Dame. This software is distributed under the GNU General Public License. See the file COPYING for details.


  • Cooperative Computing Tools Documentation
  • Makeflow User Manual
  • makeflow(1)
  • makeflow_monitor(1)
  • starch(1)

  • CCTools 4.1.4rc5 released on 04/10/2014