Work_Queue - Perl Work Queue bindings.
The objects and methods provided by this package correspond to the native C API in work_queue.h. See also Work_Queue::Task, which is automatically loaded with this module.
The SWIG-based Perl bindings provide a higher-level interface, such as:
use Work_Queue;
my $q = Work_Queue::Queue->new( port => $port, name => 'my_queue_name');
my $t = Work_Queue::Task->new($command);
$t->specify_input_file(local_name => 'some_name', remote_name => 'some_other_name');
$t->specify_output_file('some_name');
$q->submit($t);
my $stats = $q->stats;
print $stats->{tasks_running}, '\n';
$t = $q->wait(5);
if($t) {
my $resources = $t->resources_measured;
print $resources->{resident_memory}, '\n';
}
Work_Queue::new ( )
Work_Queue::new ( $port )
Work_Queue::new ( port => ..., name => ..., catalog => ..., shutdown => ...)
Create a new work queue.
The port number to listen on. If zero is specified, then the default is chosen, and if -1 is specified, a random port is chosen.
The project name to use.
Whether or not to enable catalog mode.
Automatically shutdown workers when queue is finished. Disabled by default.
my $q = Work_Queue->new( port => 0, name => 'my_queue' );
See work_queue_create in the C API for more information about environmental variables that affect the behavior this method.
name
Get the project name of the queue.
print $q->name;
port
Get the listening port of the queue.
print $q->port
stats
Get the master's queue statistics.
print $q->stats->{workers_busy};
stats_hierarchy
Get the queue statistics, including master and foremen.
print $q->stats_hierarchy->{workers_busy};
enable_monitoring($summary_file)
Enables resource monitoring of tasks in the queue. And writes a summary of the monitored information to a file.
Returns 1 on success, 0 on failure (i.e., monitoring was not enabled).
Filename for the summary log (If NULL, writes to wq-\<pid\>-resource-usage).
fast_abort
Turn on or off fast abort functionality for a given queue.
The multiplier of the average task time at which point to abort; if negative (the default) fast_abort is deactivated.
empty
Determine whether there are any known tasks queued, running, or waiting to be collected.
Returns 0 if there are tasks remaining in the system, 1 if the system is "empty".
hungry
Determine whether the queue can support more tasks.
Returns the number of additional tasks it can support if "hungry" and 0 if "sated".
specify_algorithm
Set the worker selection algorithm for queue.
One of the following algorithms to use in assigning a task to a worker:
specify_task_order
Set the order for dispatching submitted tasks in the queue to workers:
One of the following algorithms to use in dispatching
specify_name
Change the project name for the given queue.
The new project name.
specify_priority
Change the project priority for the given queue.
An integer that presents the priorty of this work queue master. The higher the value, the higher the priority.
specify_master_mode
Specify the master mode for the given queue.
This may be one of the following values:
specify_catalog_server
Specify the catalog server the master should report to.
The hostname of the catalog server.
The port the catalog server is listening on.
specify_log
Specify a log file that records the states of connected workers and submitted tasks.
Name of the file to write the log. If the file exists, then new records are appended.
specify_password
Add a mandatory password that each worker must present.
The password, as a string.
specify_password_file
Add a mandatory password file that each worker must present.
Name of the file containing the password.
cancel_by_taskid
Cancel task identified by its taskid and remove from the given queue.
The taskid returned from Work_Queue->submit.
cancel_by_tasktag
Cancel task identified by its tag and remove from the given queue.
The tag assigned to task using $t->speficy_tag($tag);
shutdown_workers
Shutdown workers connected to queue. Gives a best effort and then returns the number of workers given the shutdown order.
The number to shutdown. To shut down all workers, specify 0.
specify_keepalive_interval
Change keepalive interval for a given queue.
specify_keepalive_timeout
Change keepalive timeout for a given queue.
Minimum number of seconds to wait for a keepalive response from worker before marking it as dead.
estimate_capacity
Turn on master capacity measurements.
activate_worker_waiting
Wait for at least n workers to connect before continuing.
Number of workers.
tune
Tune advanced parameters for work queue.
Treat each worker as having (actual_cores * multiplier) total cores. (default = 1.0)
Treat each worker as having an additional "modifier" cores. (default=0)
Set the minimum number of seconds to wait for files to be transferred to or from a worker. (default=300)
Set the minimum number of seconds to wait for files to be transferred to or from a foreman. (default=3600)
Set the multiplier of the average task time at which point to abort; if negative or zero fast_abort is deactivated. (default=0)
Set the minimum number of seconds to wait before sending new keepalive checks to workers. (default=300)
Set the minimum number of seconds to wait for a keepalive response from worker before marking it as dead. (default=30)
Return 0 on succes, -1 on failure.
submit
Submit a task to the queue.
A task description created from Work_Queue::Task.
$q->submit($task);
wait
Wait for tasks to complete.
This call will block until the timeout has elapsed
The number of seconds to wait for a completed task back before returning. Use an integer to set the timeout or the constant $WORK_QUEUE_WAITFORTASK to block until a task has completed.
while( !$q->empty ) {
...
$task = $q->wait($seconds);
if($task) {
...
}
...
}