NAME

Work_Queue - Perl Work Queue bindings.


SYNOPSIS

The objects and methods provided by this package correspond to the native C API in work_queue.h. See also Work_Queue::Task, which is automatically loaded with this module.

The SWIG-based Perl bindings provide a higher-level interface, such as:

                use Work_Queue;
                my $q = Work_Queue->new( port => $port, name => 'my_queue_name');
                my $t = Work_Queue::Task->new($command);
                $t->specify_input_file(local_name => 'some_name', remote_name => 'some_other_name');
                $t->specify_output_file('some_name');
                $q->submit($t);
                my $stats = $q->stats;
                print $stats->{tasks_running}, '\n';
                $t = $q->wait(5);
                if($t) {
                                my $resources = $t->resources_measured;
                                print $resources->{resident_memory}, '\n';
                }


METHODS

Work_Queue

Work_Queue::new ( )

Work_Queue::new ( $port )

Work_Queue::new ( port => ..., name => ..., catalog => ..., shutdown => ...)

Create a new work queue.

port

The port number to listen on. If zero is specified, then the default is chosen, and if -1 is specified, a random port is chosen.

name

The project name to use.

catalog

Whether or not to enable catalog mode.

shutdown

Automatically shutdown workers when queue is finished. Disabled by default.

                my $q = Work_Queue->new( port => 0, name => 'my_queue' );

See work_queue_create in the C API for more information about environmental variables that affect the behavior this method.

name

Get the project name of the queue.

                 print $q->name;

port

Get the listening port of the queue.

                 print $q->port

stats

Get the master's queue statistics.

                 print $q->stats->{workers_busy};

stats_hierarchy

Get the queue statistics, including master and foremen.

                 print $q->stats_hierarchy->{workers_busy};

stats_category

Get the tasks statistics from the particular category.

                 $s = $q->stats_category("my_category")
                 print $s->{tasks_waiting}

enable_monitoring($dir_name)

Enables resource monitoring of tasks in the queue, and writes a summary per task to the directory given. Additionally, all summaries are consolidate into the file all_summaries-PID.log

Returns 1 on success, 0 on failure (i.e., monitoring was not enabled).

dirname Directory name for the monitor output.

enable_monitoring_full($dir_name)

As @ref enable_monitoring, but it also generates a time series and a debug file. WARNING: Such files may reach gigabyte sizes for long running tasks.

Returns 1 on success, 0 on failure (i.e., monitoring was not enabled).

dirname Directory name for the monitor output.

activate_fast_abort

Turn on or off fast abort functionality for a given queue for tasks without an explicit category. Given the multiplier, abort a task which running time is larger than the average times the multiplier. Fast-abort is computed per task category. The value specified here applies to all the categories for which @ref activate_fast_abort_category was not explicitely called.

multiplier

The multiplier of the average task time at which point to abort; if less than zero, fast_abort is deactivated (the default).

activate_fast_abort_category

Turn on or off fast abort functionality for a given category. Given the multiplier, abort a task which running time is larger than the average times the multiplier. The value specified here applies only to tasks in the given category. (Note: work_queue_activate_fast_abort_category(q, "default", n) is the same as work_queue_activate_fast_abort(q, n).)

name

The name of the category.

multiplier

The multiplier of the average task time at which point to abort; if zero, fast_abort is deactivated. If less than zero (default), use the fast abort of the "default" category.

empty

Determine whether there are any known tasks queued, running, or waiting to be collected.

Returns 0 if there are tasks remaining in the system, 1 if the system is "empty".

hungry

Determine whether the queue can support more tasks.

Returns the number of additional tasks it can support if "hungry" and 0 if "sated".

specify_algorithm

Set the worker selection algorithm for queue.

algorithm

One of the following algorithms to use in assigning a task to a worker:

$Work_Queue::WORK_QUEUE_SCHEDULE_FCFS
$Work_Queue::WORK_QUEUE_SCHEDULE_FILES
$Work_Queue::WORK_QUEUE_SCHEDULE_TIME
$Work_Queue::WORK_QUEUE_SCHEDULE_RAND

specify_task_order

Set the order for dispatching submitted tasks in the queue to workers:

order

One of the following algorithms to use in dispatching

$Work_Queue::WORK_QUEUE_TASK_ORDER_FIFO
$Work_Queue::WORK_QUEUE_TASK_ORDER_LIFO

specify_name

Change the project name for the given queue.

name

The new project name.

specify_priority

Change the project priority for the given queue.

priority

An integer that presents the priorty of this work queue master. The higher the value, the higher the priority.

specify_num_tasks_left

Specify the number of tasks not yet submitted to the queue. It is used by work_queue_factory to determine the number of workers to launch. If not specified, it defaults to 0. work_queue_factory considers the number of tasks as: num tasks left + num tasks running + num tasks read.

ntasks

ntasks Number of tasks yet to be submitted.

specify_master_mode

Specify the master mode for the given queue.

mode

This may be one of the following values:

$Work_Queue::WORK_QUEUE_MASTER_MODE_STANDALONE
$Work_Queue::WORK_QUEUE_MASTER_MODE_CATALOG.

specify_catalog_server

Specify the catalog server the master should report to.

hostname

The hostname of the catalog server.

port

The port the catalog server is listening on.

specify_log

Specify a log file that records the states of connected workers and submitted tasks.

logfile

Name of the file to write the log. If the file exists, then new records are appended.

specify_password

Add a mandatory password that each worker must present.

password

The password, as a string.

specify_password_file

Add a mandatory password file that each worker must present.

file

Name of the file containing the password.

cancel_by_taskid

Cancel task identified by its taskid and remove from the given queue.

id

The taskid returned from Work_Queue->submit.

cancel_by_tasktag

Cancel task identified by its tag and remove from the given queue.

tag

The tag assigned to task using $t->speficy_tag($tag);

shutdown_workers

Shutdown workers connected to queue. Gives a best effort and then returns the number of workers given the shutdown order.

n

The number to shutdown. To shut down all workers, specify 0.

blacklist

Blacklist workers running on host.

host

The hostname the host running the workers.

blacklist_clear

Remove host from blacklist. Clear all blacklist if host not provided.

host

The of the hostname the host.

specify_keepalive_interval

Change keepalive interval for a given queue.

interval Minimum number of seconds to wait before sending new keepalive checks to workers.

specify_keepalive_timeout

Change keepalive timeout for a given queue.

timeout

Minimum number of seconds to wait for a keepalive response from worker before marking it as dead.

estimate_capacity

Turn on master capacity measurements.

activate_worker_waiting

Wait for at least n workers to connect before continuing.

n

Number of workers.

tune

Tune advanced parameters for work queue. Return 0 on succes, -1 on failure.

name The name fo the parameter to tune. Can be one of following:
"asynchrony-multiplier"

Treat each worker as having (actual_cores * multiplier) total cores. (default = 1.0)

"asynchrony-modifier"

Treat each worker as having an additional "modifier" cores. (default=0)

"min-transfer-timeout"

Set the minimum number of seconds to wait for files to be transferred to or from a worker. (default=300)

"foreman-transfer-timeout"

Set the minimum number of seconds to wait for files to be transferred to or from a foreman. (default=3600)

"fast-abort-multiplier"

Set the multiplier of the average task time at which point to abort; if negative or zero fast_abort is deactivated. (default=0)

"keepalive-interval"

Set the minimum number of seconds to wait before sending new keepalive checks to workers. (default=300)

"keepalive-timeout"

Set the minimum number of seconds to wait for a keepalive response from worker before marking it as dead. (default=30)

value The value to set the parameter to.

specify_max_resources

Enables resource autolabeling for tasks without an explicit category ("default" category). rm specifies the maximum resources a task in the default category may use. If rm is undefined, disable autolabeling for the default category.

rm

Hash reference indicating maximum values. See @resources_measured for possible fields.

A maximum of 4 cores is found on any worker:

                q->specify_max_resources({'cores' => 4});

A maximum of 8 cores, 1GB of memory, and 10GB disk are found on any worker:

                q->specify_max_resources({'cores' => 8, 'memory' => 1024, 'disk' => 10240});

specify_max_category_resources

Enables resource autolabeling for tasks in the given category. rm specifies the maximum resources a task in the category may use. If rm is undefined, disable autolabeling for that category.

category

Name of the category

rm

Hash reference indicating maximum values. See @resources_measured for possible fields.

A maximum of 4 cores is found on any worker:

                q->specify_max_category_resources('my_category', {'cores' => 4});

A maximum of 8 cores, 1GB of memory, and 10GB disk are found on any worker:

                q->specify_max_category_resources('my_category', {'cores' => 8, 'memory' => 1024, 'disk' => 10240});

initialize_categories

Initialize first value of categories

rm

Hash reference indicating maximum values. See @resources_measured for possible fields.

filename

JSON file with resource summaries.

submit

Submit a task to the queue.

task

A task description created from Work_Queue::Task.

                $q->submit($task);

wait

Wait for tasks to complete.

This call will block until the timeout has elapsed

timeout

The number of seconds to wait for a completed task back before returning. Use an integer to set the timeout or the constant $Work_Queue::WORK_QUEUE_WAITFORTASK to block until a task has completed.

                while( !$q->empty ) {
                                ...
                                $task = $q->wait($seconds);
                                if($task) {
                                        ...
                                }
                                ...
                }
=cut