You can run a Makeflow on your local machine to test it out. If you have a multi-core machine, then you can run multiple tasks simultaneously. If you have a Condor pool or a Sun Grid Engine batch system, then you can send your jobs there to run. If you don't already have a batch system, Makeflow comes with a system called Work Queue that will let you distribute the load across any collection of machines, large or small.
-c, --clean | Clean up: remove logfile and all targets. |
-f,--summary-log <file> | |
Write summary of workflow to file. | |
-h, --help | Show this help screen. |
-m,--email <email> | |
Email summary of workflow to address. | |
-v, --version | Show version string. |
-X,--chdir <directory> | |
Chdir to enable executing the Makefile in other directory. |
-B,--batch-options <options> | |
Add these options to all batch submit files. | |
-j,--max-local <#> | |
Max number of local jobs to run at once. (default is # of cores) | |
-J,--max-remote <#> | |
Max number of remote jobs to run at once. (default is 1000 for -Twq, 100 otherwise) | |
-l,--makeflow-log <logfile> | |
Use this file for the makeflow log. (default is X.makeflowlog) | |
-L,--batch-log <logfile> | |
Use this file for the batch system log. (default is X.<type>log) | |
-R, --retry | Automatically retry failed batch jobs up to 100 times. |
-r,--retry-count <n> | |
Automatically retry failed batch jobs up to n times. | |
--wait-for-files-upto <#> | |
Wait for output files to be created upto this many seconds (e.g., to deal with NFS semantics). | |
-S,--submission-timeout <timeout> | |
Time to retry failed batch job submission. (default is 3600s) | |
-T,--batch-type <type> | |
Batch system type: local, condor, sge, moab, cluster, wq, hadoop, mpi-queue. (default is local) |
-d,--debug <subsystem> | |
Enable debugging for this subsystem. | |
-o,--debug-file <file> | |
Write debugging output to this file. By default, debugging is sent to stderr (":stderr"). You may specify logs be sent to stdout (":stdout"), to the system syslog (":syslog"), or to the systemd journal (":journal"). | |
--verbose | Display runtime progress on stdout. |
-a, --advertise | Advertise the master information to a catalog server. |
-C,--catalog-server <catalog> | |
Set catalog server to <catalog>. Format: HOSTNAME:PORT | |
-F,--wq-fast-abort <#> | |
WorkQueue fast abort multiplier. (default is deactivated) | |
-N,--project-name <project> | |
Set the project name to <project>. | |
-p,--port <port> | |
Port number to use with WorkQueue. (default is 9123, 0=arbitrary) | |
-Z,--port-file <file> | |
Select port at random and write it to this file. (default is disabled) | |
-P,--priority <integer> | |
Priority. Higher the value, higher the priority. | |
-W,--wq-schedule <mode> | |
WorkQueue scheduling algorithm. (time|files|fcfs) | |
-s,--password <pwfile> | |
Password file for authenticating workers. | |
--disable-cache | Disable file caching (currently only Work Queue, default is false) |
-M,--monitor <dir> | |||||||||||||||||
Enable the resource monitor, and write the monitor logs to --monitor-limits <file> | Use | --monitor-with-time-series | Enable monitor time series. (default is disabled)
| --monitor-with-opened-files | Enable monitoring of openened files. (default is disabled)
| --monitor-interval <#> | Set monitor interval to <#> seconds. (default 1 second)
| --monitor-log-fmt <fmt> | Format for monitor logs. (default resource-rule-%06.6d, %d -> rule number)
| |
-A, --disable-afs-check | Disable the check for AFS. (experts only) |
-z, --zero-length-error | Force failure on zero-length output files. |
--wrapper <command> | |
Wrap all commands with this prefix. | |
--wrapper-input <file> | |
Wrapper command requires this input file. | |
--wrapper-input <file> | |
Wrapper command produces this output file. | |
--docker <image> | |
Run each task with a container based on this docker image. | |
--docker-tar <tar> | |
Load docker image from tar file |
Note that variables defined in your Makeflow are exported to the environment.
makeflow -d all MakeflowRun makeflow on Condor will special requirements:
makeflow -T condor -B "requirements = MachineGroup == 'ccl'" MakeflowRun makeflow with WorkQueue using named workers:
makeflow -T wq -a -N project.name MakeflowCreate a directory containing all of the dependencies required to run the specified makeflow
makeflow -b bundle Makeflow