makeflow(1)

NAME

makeflow - workflow engine for executing distributed workflows

SYNOPSIS

makeflow [options] <dagfile>

DESCRIPTION

Makeflow is a workflow engine for distributed computing. It accepts a specification of a large amount of work to be performed, and runs it on remote machines in parallel where possible. In addition, Makeflow is fault-tolerant, so you can use it to coordinate very large tasks that may run for days or weeks in the face of failures. Makeflow is designed to be similar to Make, so if you can write a Makefile, then you can write a Makeflow.

You can run a Makeflow on your local machine to test it out. If you have a multi-core machine, then you can run multiple tasks simultaneously. If you have a Condor pool or a Sun Grid Engine batch system, then you can send your jobs there to run. If you don't already have a batch system, Makeflow comes with a system called Work Queue that will let you distribute the load across any collection of machines, large or small.

OPTIONS

When makeflow is ran without arguments, it will attempt to execute the workflow specified by the Makeflow dagfile using the local execution engine.

Commands

-c, --clean Clean up: remove logfile and all targets.
-f,--summary-log <file>
Write summary of workflow to file.
-h, --help Show this help screen.
-m,--email <email>
Email summary of workflow to address.
-v, --version Show version string.

Batch Options

-B,--batch-options <options>
Add these options to all batch submit files.
-j,--max-local <#>
Max number of local jobs to run at once. (default is # of cores)
-J,--max-remote <#>
Max number of remote jobs to run at once. (default is 1000 for -Twq, 100 otherwise)
-l,--makeflow-log <logfile>
Use this file for the makeflow log. (default is X.makeflowlog)
-L,--batch-log <logfile>
Use this file for the batch system log. (default is X.<type>log)
-R, --retry Automatically retry failed batch jobs up to 100 times.
-r,--retry-count <n>
Automatically retry failed batch jobs up to n times.
-S,--submission-timeout <timeout>
Time to retry failed batch job submission. (default is 3600s)
-T,--batch-type <type>
Batch system type: local, condor, sge, moab, cluster, wq, hadoop, mpi-queue. (default is local)

Debugging Options

-d,--debug <subsystem>
Enable debugging for this subsystem.
-o,--debug-file <file>
Write debugging output to this file. By default, debugging is sent to stderr (":stderr"). You may specify logs be sent to stdout (":stdout"), to the system syslog (":syslog"), or to the systemd journal (":journal").
--verbose Display runtime progress on stdout.

WorkQueue Options

-a, --advertise Advertise the master information to a catalog server.
-C,--catalog-server <catalog>
Set catalog server to <catalog>. Format: HOSTNAME:PORT
-F,--wq-fast-abort <#>
WorkQueue fast abort multiplier. (default is deactivated)
-N,--project-name <project>
Set the project name to <project>.
-p,--port <port>
Port number to use with WorkQueue. (default is 9123, 0=arbitrary)
-Z,--port-file <file>
Select port at random and write it to this file. (default is disabled)
-P,--priority <integer>
Priority. Higher the value, higher the priority.
-W,--wq-schedule <mode>
WorkQueue scheduling algorithm. (time|files|fcfs)
-s,--password <pwfile>
Password file for authenticating workers.
--disable-cache Disable file caching (currently only Work Queue, default is false)

Monitor Options

-M,--monitor <dir>
Enable the resource monitor, and write the monitor logs to
--monitor-limits <file>
Use as value-pairs for resource limits.
--monitor-with-time-series Enable monitor time series. (default is disabled)
--monitor-with-opened-files Enable monitoring of openened files. (default is disabled)
--monitor-interval <#>
Set monitor interval to <#> seconds. (default 1 second)
--monitor-log-fmt <fmt>
Format for monitor logs. (default resource-rule-%06.6d, %d -> rule number)

Other Options

-A, --disable-afs-check Disable the check for AFS. (experts only)
-K, --preserve-links Preserve (i.e., do not clean) intermediate symbolic links.
-z, --zero-length-error Force failure on zero-length output files.

ENVIRONMENT VARIABLES

The following environment variables will affect the execution of your Makeflow:

BATCH_OPTIONS

This corresponds to the -B <options> parameter and will pass extra batch options to the underlying execution engine.

MAKEFLOW_MAX_LOCAL_JOBS

This corresponds to the -j <#> parameter and will set the maximum number of local batch jobs. If a -j <#> parameter is specified, the minimum of the argument and the environment variable is used.

MAKEFLOW_MAX_REMOTE_JOBS

This corresponds to the -J <#> parameter and will set the maximum number of local batch jobs. If a -J <#> parameter is specified, the minimum of the argument and the environment variable is used.

Note that variables defined in your Makeflow are exported to the environment.

TCP_LOW_PORT

Inclusive low port in range used with -p 0.

TCP_HIGH_PORT

) Inclusive high port in range used with -p 0.

EXIT STATUS

On success, returns zero. On failure, returns non-zero.

EXAMPLES

Run makeflow locally with debugging:
makeflow -d all Makeflow
Run makeflow on Condor will special requirements:
makeflow -T condor -B "requirements = MachineGroup == 'ccl'" Makeflow
Run makeflow with WorkQueue using named workers:
makeflow -T wq -a -N project.name Makeflow
Create a directory containing all of the dependencies required to run the specified makeflow
makeflow -b bundle Makeflow

COPYRIGHT

The Cooperative Computing Tools are Copyright (C) 2003-2004 Douglas Thain and Copyright (C) 2005-2011 The University of Notre Dame. This software is distributed under the GNU General Public License. See the file COPYING for details.

SEE ALSO

  • Cooperative Computing Tools Documentation
  • Makeflow User Manual
  • makeflow(1)
  • makeflow_monitor(1)
  • makeflow_analyze(1)
  • makeflow_viz(1)
  • makeflow_graph_log(1)
  • starch(1)

  • CCTools 4.2.0rc3 released on 06/18/2014