work_queue::WorkQueue Class Reference
Python Work Queue object.
More...
Detailed Description
Python Work Queue object.
This class uses a dictionary to map between the task pointer objects and the work_queue::Task.
Member Function Documentation
def work_queue::WorkQueue::__init__ |
( |
|
self, |
|
|
|
port = WORK_QUEUE_DEFAULT_PORT , |
|
|
|
name = None , |
|
|
|
catalog = False , |
|
|
|
exclusive = True , |
|
|
|
shutdown = False | |
|
) |
| | |
Create a new work queue.
- Parameters:
-
| self | Reference to the current work queue object. |
| port | The port number to listen on. If zero is specified, then the default is chosen, and if -1 is specified, a random port is chosen. |
| name | The project name to use. |
| catalog | Whether or not to enable catalog mode. |
| exclusive | Whether or not the workers should be exclusive. |
| shutdown | Automatically shutdown workers when queue is finished. Disabled by default. |
- See also:
- work_queue_create - For more information about environmental variables that affect the behavior this method.
def work_queue::WorkQueue::name |
( |
|
self |
) |
|
Get the project name of the queue.
Note: This is defined using property decorator. So it must be called without parentheses (). For example:
def work_queue::WorkQueue::port |
( |
|
self |
) |
|
Get the listening port of the queue.
Note: This is defined using property decorator. So it must be called without parentheses (). For example:
def work_queue::WorkQueue::stats |
( |
|
self |
) |
|
Get queue statistics.
Note: This is defined using property decorator. So it must be called without parentheses (). For example:
The fields in work_queue_stats can also be individually accessed through this call. For example:
>>> print q.stats.workers_busy
def work_queue::WorkQueue::stats_hierarchy |
( |
|
self |
) |
|
Get worker hierarchy statistics.
Note: This is defined using property decorator. So it must be called without parentheses (). For example:
The fields in work_queue_stats can also be individually accessed through this call. For example:
>>> print q.stats.workers_busy
def work_queue::WorkQueue::task_state |
( |
|
self, |
|
|
|
taskid | |
|
) |
| | |
def work_queue::WorkQueue::enable_monitoring |
( |
|
self, |
|
|
|
dirname | |
|
) |
| | |
Enables resource monitoring of tasks in the queue, and writes a summary per task to the directory given.
Additionally, all summaries are consolidate into the file all_summaries-PID.log
Returns 1 on success, 0 on failure (i.e., monitoring was not enabled).
- Parameters:
-
| self | Reference to the current work queue object. |
| dirname | Directory name for the monitor output. |
def work_queue::WorkQueue::enable_monitoring_full |
( |
|
self, |
|
|
|
dirname | |
|
) |
| | |
As enable_monitoring, but it also generates a time series and a debug file.
WARNING: Such files may reach gigabyte sizes for long running tasks.
Returns 1 on success, 0 on failure (i.e., monitoring was not enabled).
- Parameters:
-
| self | Reference to the current work queue object. |
| dirname | Directory name for the monitor output. |
def work_queue::WorkQueue::activate_fast_abort |
( |
|
self, |
|
|
|
multiplier | |
|
) |
| | |
Turn on or off fast abort functionality for a given queue.
- Parameters:
-
| self | Reference to the current work queue object. |
| multiplier | The multiplier of the average task time at which point to abort; if negative (the default) fast_abort is deactivated. |
def work_queue::WorkQueue::empty |
( |
|
self |
) |
|
Determine whether there are any known tasks queued, running, or waiting to be collected.
Returns 0 if there are tasks remaining in the system, 1 if the system is "empty".
- Parameters:
-
| self | Reference to the current work queue object. |
def work_queue::WorkQueue::hungry |
( |
|
self |
) |
|
Determine whether the queue can support more tasks.
Returns the number of additional tasks it can support if "hungry" and 0 if "sated".
- Parameters:
-
| self | Reference to the current work queue object. |
def work_queue::WorkQueue::specify_algorithm |
( |
|
self, |
|
|
|
algorithm | |
|
) |
| | |
Set the worker selection algorithm for queue.
- Parameters:
-
| self | Reference to the current work queue object. |
| algorithm | One of the following algorithms to use in assigning a task to a worker. See work_queue_schedule_t for possible values. |
def work_queue::WorkQueue::specify_task_order |
( |
|
self, |
|
|
|
order | |
|
) |
| | |
Set the order for dispatching submitted tasks in the queue.
- Parameters:
-
| self | Reference to the current work queue object. |
| order | One of the following algorithms to use in dispatching submitted tasks to workers:
|
def work_queue::WorkQueue::specify_name |
( |
|
self, |
|
|
|
name | |
|
) |
| | |
Change the project name for the given queue.
- Parameters:
-
| self | Reference to the current work queue object. |
| name | The new project name. |
def work_queue::WorkQueue::specify_priority |
( |
|
self, |
|
|
|
priority | |
|
) |
| | |
Change the project priority for the given queue.
- Parameters:
-
| self | Reference to the current work queue object. |
| priority | An integer that presents the priorty of this work queue master. The higher the value, the higher the priority. |
def work_queue::WorkQueue::specify_num_tasks_left |
( |
|
self, |
|
|
|
ntasks | |
|
) |
| | |
Specify the number of tasks not yet submitted to the queue.
It is used by work_queue_pool to determine the number of workers to launch. If not specified, it defaults to 0. work_queue_pool considers the number of tasks as: num tasks left + num tasks running + num tasks read.
- Parameters:
-
| q | A work queue object. |
| ntasks | Number of tasks yet to be submitted. |
def work_queue::WorkQueue::specify_master_mode |
( |
|
self, |
|
|
|
mode | |
|
) |
| | |
Specify the master mode for the given queue.
- Parameters:
-
def work_queue::WorkQueue::specify_catalog_server |
( |
|
self, |
|
|
|
hostname, |
|
|
|
port | |
|
) |
| | |
Specify the catalog server the master should report to.
- Parameters:
-
| self | Reference to the current work queue object. |
| hostname | The hostname of the catalog server. |
| port | The port the catalog server is listening on. |
def work_queue::WorkQueue::specify_log |
( |
|
self, |
|
|
|
logfile | |
|
) |
| | |
Specify a log file that records the states of connected workers and submitted tasks.
- Parameters:
-
| self | Reference to the current work queue object. |
| logfile | Filename. |
def work_queue::WorkQueue::specify_password |
( |
|
self, |
|
|
|
password | |
|
) |
| | |
Add a mandatory password that each worker must present.
- Parameters:
-
| self | Reference to the current work queue object. |
| password | The password. |
def work_queue::WorkQueue::specify_password_file |
( |
|
self, |
|
|
|
file | |
|
) |
| | |
Add a mandatory password file that each worker must present.
- Parameters:
-
| self | Reference to the current work queue object. |
| file | Name of the file containing the password. |
def work_queue::WorkQueue::cancel_by_taskid |
( |
|
self, |
|
|
|
id | |
|
) |
| | |
Cancel task identified by its taskid and remove from the given queue.
- Parameters:
-
| self | Reference to the current work queue object. |
| id | The taskid returned from submit. |
def work_queue::WorkQueue::cancel_by_tasktag |
( |
|
self, |
|
|
|
tag | |
|
) |
| | |
Cancel task identified by its tag and remove from the given queue.
- Parameters:
-
def work_queue::WorkQueue::shutdown_workers |
( |
|
self, |
|
|
|
n | |
|
) |
| | |
Shutdown workers connected to queue.
Gives a best effort and then returns the number of workers given the shutdown order.
- Parameters:
-
| self | Reference to the current work queue object. |
| n | The number to shutdown. To shut down all workers, specify "0". |
def work_queue::WorkQueue::blacklist |
( |
|
self, |
|
|
|
host | |
|
) |
| | |
Blacklist workers running on host.
- Parameters:
-
| self | Reference to the current work queue object. |
| host | The hostname the host running the workers. |
def work_queue::WorkQueue::blacklist_clear |
( |
|
self, |
|
|
|
host = None | |
|
) |
| | |
Remove host from blacklist.
Clear all blacklist if host not provided.
- Parameters:
-
| self | Reference to the current work queue object. |
| host | The of the hostname the host. |
def work_queue::WorkQueue::specify_keepalive_interval |
( |
|
self, |
|
|
|
interval | |
|
) |
| | |
Change keepalive interval for a given queue.
- Parameters:
-
| self | Reference to the current work queue object. |
| interval | Minimum number of seconds to wait before sending new keepalive checks to workers. |
def work_queue::WorkQueue::specify_keepalive_timeout |
( |
|
self, |
|
|
|
timeout | |
|
) |
| | |
Change keepalive timeout for a given queue.
- Parameters:
-
| self | Reference to the current work queue object. |
| timeout | Minimum number of seconds to wait for a keepalive response from worker before marking it as dead. |
def work_queue::WorkQueue::estimate_capacity |
( |
|
self |
) |
|
Turn on master capacity measurements.
- Parameters:
-
| self | Reference to the current work queue object. |
def work_queue::WorkQueue::tune |
( |
|
self, |
|
|
|
name, |
|
|
|
value | |
|
) |
| | |
Tune advanced parameters for work queue.
- Parameters:
-
| self | Reference to the current work queue object. |
| name | The name fo the parameter to tune. Can be one of following:
- "asynchrony-multiplier" Treat each worker as having (actual_cores * multiplier) total cores. (default = 1.0)
- "asynchrony-modifier" Treat each worker as having an additional "modifier" cores. (default=0)
- "min-transfer-timeout" Set the minimum number of seconds to wait for files to be transferred to or from a worker. (default=300)
- "foreman-transfer-timeout" Set the minimum number of seconds to wait for files to be transferred to or from a foreman. (default=3600)
- "fast-abort-multiplier" Set the multiplier of the average task time at which point to abort; if negative or zero fast_abort is deactivated. (default=0)
- "keepalive-interval" Set the minimum number of seconds to wait before sending new keepalive checks to workers. (default=300)
- "keepalive-timeout" Set the minimum number of seconds to wait for a keepalive response from worker before marking it as dead. (default=30)
|
| value | The value to set the parameter to. |
- Returns:
- 0 on succes, -1 on failure.
def work_queue::WorkQueue::submit |
( |
|
self, |
|
|
|
task | |
|
) |
| | |
Submit a task to the queue.
It is safe to re-submit a task returned by wait.
- Parameters:
-
| self | Reference to the current work queue object. |
| task | A task description created from work_queue::Task. |
def work_queue::WorkQueue::wait |
( |
|
self, |
|
|
|
timeout = WORK_QUEUE_WAITFORTASK | |
|
) |
| | |
Wait for tasks to complete.
This call will block until the timeout has elapsed
- Parameters:
-
| self | Reference to the current work queue object. |
| timeout | The number of seconds to wait for a completed task before returning. Use an integer to set the timeout or the constant WORK_QUEUE_WAITFORTASK to block until a task has completed. |
The documentation for this class was generated from the following file: