makeflow(1)

NAME

makeflow - workflow engine for executing distributed workflows

SYNOPSIS

makeflow [options] <dagfile>

DESCRIPTION

Makeflow is a workflow engine for distributed computing. It accepts a specification of a large amount of work to be performed, and runs it on remote machines in parallel where possible. In addition, Makeflow is fault-tolerant, so you can use it to coordinate very large tasks that may run for days or weeks in the face of failures. Makeflow is designed to be similar to Make, so if you can write a Makefile, then you can write a Makeflow.

You can run a Makeflow on your local machine to test it out. If you have a multi-core machine, then you can run multiple tasks simultaneously. If you have a Condor pool or a Sun Grid Engine batch system, then you can send your jobs there to run. If you don't already have a batch system, Makeflow comes with a system called Work Queue that will let you distribute the load across any collection of machines, large or small.

OPTIONS

When makeflow is ran without arguments, it will attempt to execute the workflow specified by the Makeflow dagfile using the local execution engine.

Commands

-c Clean up: remove logfile and all targets.
-D Display the Makefile as a Dot graph.
-k Syntax check.
-h Show this help screen.
-I Show input files.
-O Show output files.
-v Show version string.

Batch Options

-B <options>
Add these options to all batch submit files.
-j <#>
Max number of local jobs to run at once. (default is # of cores)
-J <#>
Max number of remote jobs to run at once. (default is 100)
-l <logfile>
Use this file for the makeflow log. (default is X.makeflowlog)
-L <logfile>
Use this file for the batch system log. (default is X.<type>log)
-R Automatically retry failed batch jobs up to 100 times.
-r <n>
Automatically retry failed batch jobs up to n times.
-S <timeout>
Time to retry failed batch job submission. (default is 3600s)
-T <type>
Batch system type: local, condor, sge, moab, cluster, wq, hadoop, mpi-queue. (default is local)

Debugging Options

-d <subsystem>
Enable debugging for this subsystem.
-o <file>
Send debugging to this file.

WorkQueue Options

-a Advertise the master information to a catalog server.
-C <catalog>
Set catalog server to <catalog>. Format: HOSTNAME:PORT
-F <#>
WorkQueue fast abort multiplier. (default is deactivated)
-N <project>
Set the project name to <project>.
-p <port>
Port number to use with WorkQueue. (default is 9123, 0=arbitrary)
-Z <file>
Select port at random and write it to this file. (default is disabled)
-P <integer>
Priority. Higher the value, higher the priority.
-w <mode>
Auto WorkQueue mode. Mode is either 'width' or 'group' (DAG [width] or largest [group] of tasks).
-W <mode>
WorkQueue scheduling algorithm. (time|files|fcfs)

Other Options

-A Disable the check for AFS. (experts only)
-K Preserve (i.e., do not clean) intermediate symbolic links.
-z Force failure on zero-length output files.

ENVIRONMENT VARIABLES

The following environment variables will affect the execution of your Makeflow:

BATCH_OPTIONS

This corresponds to the -B <options> parameter and will pass extra batch options to the underlying execution engine.

MAKEFLOW_MAX_LOCAL_JOBS

This corresponds to the -j <#> parameter and will set the maximum number of local batch jobs. If a -j <#> parameter is specified, the minimum of the argument and the environment variable is used.

MAKEFLOW_MAX_REMOTE_JOBS

This corresponds to the -J <#> parameter and will set the maximum number of local batch jobs. If a -J <#> parameter is specified, the minimum of the argument and the environment variable is used.

Note that variables defined in your Makeflow are exported to the environment.

TCP_LOW_PORT

Inclusive low port in range used with -p 0.

TCP_HIGH_PORT

) Inclusive high port in range used with -p 0.

EXIT STATUS

On success, returns zero. On failure, returns non-zero.

EXAMPLES

Run makeflow locally with debugging:
makeflow -d all Makeflow
Run makeflow on Condor will special requirements:
makeflow -T condor -B "requirements = MachineGroup == 'ccl'" Makeflow
Run makeflow with WorkQueue using named workers:
makeflow -T wq -a -N project.name Makeflow

COPYRIGHT

The Cooperative Computing Tools are Copyright (C) 2003-2004 Douglas Thain and Copyright (C) 2005-2011 The University of Notre Dame. This software is distributed under the GNU General Public License. See the file COPYING for details.

SEE ALSO

  • Cooperative Computing Tools Documentation
  • Makeflow User Manual
  • makeflow(1)
  • makeflow_monitor(1)
  • starch(1)

  • CCTools 3.7.3 released on 06/12/2013