NAME

Work_Queue - Perl Work Queue bindings.


SYNOPSIS

The objects and methods provided by this package correspond to the native C API in work_queue.h. See also Work_Queue::Task, which is automatically loaded with this module.

The SWIG-based Perl bindings provide a higher-level interface, such as:

                use Work_Queue;
                my $q = Work_Queue->new( port => $port, name => 'my_queue_name');
                my $t = Work_Queue::Task->new($command);
                $t->specify_input_file(local_name => 'some_name', remote_name => 'some_other_name');
                $t->specify_output_file('some_name');
                $q->submit($t);
                my $stats = $q->stats;
                print $stats->{tasks_running}, '\n';
                $t = $q->wait(5);
                if($t) {
                                my $resources = $t->resources_measured;
                                print $resources->{resident_memory}, '\n';
                }


METHODS

Work_Queue

Work_Queue::new ( )

Work_Queue::new ( $port )

Work_Queue::new ( port => ..., name => ..., catalog => ..., shutdown => ...)

Create a new work queue.

port

The port number to listen on. If zero is specified, then the default is chosen, and if -1 is specified, a random port is chosen.

name

The project name to use.

catalog

Whether or not to enable catalog mode.

shutdown

Automatically shutdown workers when queue is finished. Disabled by default.

                my $q = Work_Queue->new( port => 0, name => 'my_queue' );

See work_queue_create in the C API for more information about environmental variables that affect the behavior this method.

name

Get the project name of the queue.

                 print $q->name;

port

Get the listening port of the queue.

                 print $q->port

stats

Get the master's queue statistics.

                 print $q->stats->{workers_busy};

stats_hierarchy

Get the queue statistics, including master and foremen.

                 print $q->stats_hierarchy->{workers_busy};

enable_monitoring($dir_name)

Enables resource monitoring of tasks in the queue, and writes a summary per task to the directory given. Additionally, all summaries are consolidate into the file all_summaries-PID.log

Returns 1 on success, 0 on failure (i.e., monitoring was not enabled).

dirname Directory name for the monitor output.

enable_monitoring_full($dir_name)

As @ref enable_monitoring, but it also generates a time series and a debug file. WARNING: Such files may reach gigabyte sizes for long running tasks.

Returns 1 on success, 0 on failure (i.e., monitoring was not enabled).

dirname Directory name for the monitor output.

fast_abort

Turn on or off fast abort functionality for a given queue.

multiplier

The multiplier of the average task time at which point to abort; if negative (the default) fast_abort is deactivated.

empty

Determine whether there are any known tasks queued, running, or waiting to be collected.

Returns 0 if there are tasks remaining in the system, 1 if the system is "empty".

hungry

Determine whether the queue can support more tasks.

Returns the number of additional tasks it can support if "hungry" and 0 if "sated".

specify_algorithm

Set the worker selection algorithm for queue.

algorithm

One of the following algorithms to use in assigning a task to a worker:

$Work_Queue::WORK_QUEUE_SCHEDULE_FCFS
$Work_Queue::WORK_QUEUE_SCHEDULE_FILES
$Work_Queue::WORK_QUEUE_SCHEDULE_TIME
$Work_Queue::WORK_QUEUE_SCHEDULE_RAND

specify_task_order

Set the order for dispatching submitted tasks in the queue to workers:

order

One of the following algorithms to use in dispatching

$Work_Queue::WORK_QUEUE_TASK_ORDER_FIFO
$Work_Queue::WORK_QUEUE_TASK_ORDER_LIFO

specify_name

Change the project name for the given queue.

name

The new project name.

specify_priority

Change the project priority for the given queue.

priority

An integer that presents the priorty of this work queue master. The higher the value, the higher the priority.

specify_num_tasks_left

Specify the number of tasks not yet submitted to the queue. It is used by work_queue_pool to determine the number of workers to launch. If not specified, it defaults to 0. work_queue_pool considers the number of tasks as: num tasks left + num tasks running + num tasks read.

ntasks

ntasks Number of tasks yet to be submitted.

specify_master_mode

Specify the master mode for the given queue.

mode

This may be one of the following values:

$Work_Queue::WORK_QUEUE_MASTER_MODE_STANDALONE
$Work_Queue::WORK_QUEUE_MASTER_MODE_CATALOG.

specify_catalog_server

Specify the catalog server the master should report to.

hostname

The hostname of the catalog server.

port

The port the catalog server is listening on.

specify_log

Specify a log file that records the states of connected workers and submitted tasks.

logfile

Name of the file to write the log. If the file exists, then new records are appended.

specify_password

Add a mandatory password that each worker must present.

password

The password, as a string.

specify_password_file

Add a mandatory password file that each worker must present.

file

Name of the file containing the password.

cancel_by_taskid

Cancel task identified by its taskid and remove from the given queue.

id

The taskid returned from Work_Queue->submit.

cancel_by_tasktag

Cancel task identified by its tag and remove from the given queue.

tag

The tag assigned to task using $t->speficy_tag($tag);

shutdown_workers

Shutdown workers connected to queue. Gives a best effort and then returns the number of workers given the shutdown order.

n

The number to shutdown. To shut down all workers, specify 0.

blacklist

Blacklist workers running on host.

host

The hostname the host running the workers.

blacklist_clear

Remove host from blacklist. Clear all blacklist if host not provided.

host

The of the hostname the host.

specify_keepalive_interval

Change keepalive interval for a given queue.

interval Minimum number of seconds to wait before sending new keepalive checks to workers.

specify_keepalive_timeout

Change keepalive timeout for a given queue.

timeout

Minimum number of seconds to wait for a keepalive response from worker before marking it as dead.

estimate_capacity

Turn on master capacity measurements.

activate_worker_waiting

Wait for at least n workers to connect before continuing.

n

Number of workers.

tune

Tune advanced parameters for work queue.

name The name fo the parameter to tune. Can be one of following:
"asynchrony-multiplier"

Treat each worker as having (actual_cores * multiplier) total cores. (default = 1.0)

"asynchrony-modifier"

Treat each worker as having an additional "modifier" cores. (default=0)

"min-transfer-timeout"

Set the minimum number of seconds to wait for files to be transferred to or from a worker. (default=300)

"foreman-transfer-timeout"

Set the minimum number of seconds to wait for files to be transferred to or from a foreman. (default=3600)

"fast-abort-multiplier"

Set the multiplier of the average task time at which point to abort; if negative or zero fast_abort is deactivated. (default=0)

"keepalive-interval"

Set the minimum number of seconds to wait before sending new keepalive checks to workers. (default=300)

"keepalive-timeout"

Set the minimum number of seconds to wait for a keepalive response from worker before marking it as dead. (default=30)

value The value to set the parameter to.

Return 0 on succes, -1 on failure.

submit

Submit a task to the queue.

task

A task description created from Work_Queue::Task.

                $q->submit($task);

wait

Wait for tasks to complete.

This call will block until the timeout has elapsed

timeout

The number of seconds to wait for a completed task back before returning. Use an integer to set the timeout or the constant $Work_Queue::WORK_QUEUE_WAITFORTASK to block until a task has completed.

                while( !$q->empty ) {
                                ...
                                $task = $q->wait($seconds);
                                if($task) {
                                        ...
                                }
                                ...
                }
=cut