Work_Queue::new ( )
Work_Queue::new ( $port )
Work_Queue::new ( port => ..., name => ..., catalog => ..., shutdown => ...)
name
port
stats
stats_hierarchy
enable_monitoring($summary_file)
fast_abort
empty
hungry
specify_algorithm
specify_task_order
specify_name
specify_priority
specify_master_mode
specify_catalog_server
specify_log
specify_password
specify_password_file
cancel_by_taskid
cancel_by_tasktag
shutdown_workers
blacklist
blacklist_clear
specify_keepalive_interval
specify_keepalive_timeout
estimate_capacity
activate_worker_waiting
tune
submit
wait
Work_Queue - Perl Work Queue bindings.
The objects and methods provided by this package correspond to the native C API in work_queue.h. See also Work_Queue::Task, which is automatically loaded with this module.
The SWIG-based Perl bindings provide a higher-level interface, such as:
use Work_Queue;
my $q = Work_Queue->new( port => $port, name => 'my_queue_name');
my $t = Work_Queue::Task->new($command); $t->specify_input_file(local_name => 'some_name', remote_name => 'some_other_name'); $t->specify_output_file('some_name');
$q->submit($t);
my $stats = $q->stats; print $stats->{tasks_running}, '\n';
$t = $q->wait(5);
if($t) { my $resources = $t->resources_measured; print $resources->{resident_memory}, '\n'; }
Work_Queue::new ( )
Work_Queue::new ( $port )
Work_Queue::new ( port => ..., name => ..., catalog => ..., shutdown => ...)
Create a new work queue.
The port number to listen on. If zero is specified, then the default is chosen, and if -1 is specified, a random port is chosen.
The project name to use.
Whether or not to enable catalog mode.
Automatically shutdown workers when queue is finished. Disabled by default.
my $q = Work_Queue->new( port => 0, name => 'my_queue' );
See work_queue_create in the C API for more information about environmental variables that affect the behavior this method.
name
Get the project name of the queue.
print $q->name;
port
Get the listening port of the queue.
print $q->port
stats
Get the master's queue statistics.
print $q->stats->{workers_busy};
stats_hierarchy
Get the queue statistics, including master and foremen.
print $q->stats_hierarchy->{workers_busy};
enable_monitoring($summary_file)
Enables resource monitoring of tasks in the queue. And writes a summary of the monitored information to a file.
Returns 1 on success, 0 on failure (i.e., monitoring was not enabled).
Filename for the summary log (If NULL, writes to wq-\<pid\>-resource-usage).
fast_abort
Turn on or off fast abort functionality for a given queue.
The multiplier of the average task time at which point to abort; if negative (the default) fast_abort is deactivated.
empty
Determine whether there are any known tasks queued, running, or waiting to be collected.
Returns 0 if there are tasks remaining in the system, 1 if the system is ``empty''.
hungry
Determine whether the queue can support more tasks.
Returns the number of additional tasks it can support if ``hungry'' and 0 if ``sated''.
specify_algorithm
Set the worker selection algorithm for queue.
One of the following algorithms to use in assigning a task to a worker:
specify_task_order
Set the order for dispatching submitted tasks in the queue to workers:
One of the following algorithms to use in dispatching
specify_name
Change the project name for the given queue.
The new project name.
specify_priority
Change the project priority for the given queue.
An integer that presents the priorty of this work queue master. The higher the value, the higher the priority.
specify_master_mode
Specify the master mode for the given queue.
This may be one of the following values:
specify_catalog_server
Specify the catalog server the master should report to.
The hostname of the catalog server.
The port the catalog server is listening on.
specify_log
Specify a log file that records the states of connected workers and submitted tasks.
Name of the file to write the log. If the file exists, then new records are appended.
specify_password
Add a mandatory password that each worker must present.
The password, as a string.
specify_password_file
Add a mandatory password file that each worker must present.
Name of the file containing the password.
cancel_by_taskid
Cancel task identified by its taskid and remove from the given queue.
The taskid returned from Work_Queue->submit.
cancel_by_tasktag
Cancel task identified by its tag and remove from the given queue.
The tag assigned to task using $t->speficy_tag($tag);
shutdown_workers
Shutdown workers connected to queue. Gives a best effort and then returns the number of workers given the shutdown order.
The number to shutdown. To shut down all workers, specify 0.
blacklist
Blacklist workers running on host.
The hostname the host running the workers.
blacklist_clear
Remove host from blacklist. Clear all blacklist if host not provided.
The of the hostname the host.
specify_keepalive_interval
Change keepalive interval for a given queue.
specify_keepalive_timeout
Change keepalive timeout for a given queue.
Minimum number of seconds to wait for a keepalive response from worker before marking it as dead.
estimate_capacity
Turn on master capacity measurements.
activate_worker_waiting
Wait for at least n workers to connect before continuing.
Number of workers.
tune
Tune advanced parameters for work queue.
Treat each worker as having (actual_cores * multiplier) total cores. (default = 1.0)
Treat each worker as having an additional ``modifier'' cores. (default=0)
Set the minimum number of seconds to wait for files to be transferred to or from a worker. (default=300)
Set the minimum number of seconds to wait for files to be transferred to or from a foreman. (default=3600)
Set the multiplier of the average task time at which point to abort; if negative or zero fast_abort is deactivated. (default=0)
Set the minimum number of seconds to wait before sending new keepalive checks to workers. (default=300)
Set the minimum number of seconds to wait for a keepalive response from worker before marking it as dead. (default=30)
Return 0 on succes, -1 on failure.
submit
Submit a task to the queue.
A task description created from Work_Queue::Task.
$q->submit($task);
wait
Wait for tasks to complete.
This call will block until the timeout has elapsed
The number of seconds to wait for a completed task back before returning. Use an integer to set the timeout or the constant $Work_Queue::WORK_QUEUE_WAITFORTASK to block until a task has completed.
while( !$q->empty ) { ... $task = $q->wait($seconds);
if($task) { ... } ... } =cut