This class uses a dictionary to map between the task pointer objects and the work_queue::Task.
Public Member Functions | |
def | __init__ |
Create a new work queue. | |
def | name |
Get the project name of the queue. | |
def | port |
Get the listening port of the queue. | |
def | stats |
Get queue statistics. | |
def | stats_hierarchy |
Get worker hierarchy statistics. | |
def | stats_category |
Get the task statistics for the given category. | |
def | task_state |
Get current task state. | |
def | enable_monitoring |
Enables resource monitoring of tasks in the queue, and writes a summary per task to the directory given. | |
def | enable_monitoring_full |
As enable_monitoring, but it also generates a time series and a debug file. | |
def | activate_fast_abort |
Turn on or off fast abort functionality for a given queue for tasks in the "default" category, and for task which category does not set an explicit multiplier. | |
def | activate_fast_abort_category |
Turn on or off fast abort functionality for a given queue. | |
def | empty |
Determine whether there are any known tasks queued, running, or waiting to be collected. | |
def | hungry |
Determine whether the queue can support more tasks. | |
def | specify_algorithm |
Set the worker selection algorithm for queue. | |
def | specify_task_order |
Set the order for dispatching submitted tasks in the queue. | |
def | specify_name |
Change the project name for the given queue. | |
def | specify_priority |
Change the project priority for the given queue. | |
def | specify_num_tasks_left |
Specify the number of tasks not yet submitted to the queue. | |
def | specify_master_mode |
Specify the master mode for the given queue. | |
def | specify_catalog_server |
Specify the catalog server the master should report to. | |
def | specify_log |
Specify a log file that records the states of connected workers and submitted tasks. | |
def | specify_password |
Add a mandatory password that each worker must present. | |
def | specify_password_file |
Add a mandatory password file that each worker must present. | |
def | specify_max_resources |
Enables resource autolabeling for tasks without an explicit category ("default" category). | |
def | specify_max_category_resources |
Enables resource autolabeling for tasks in the given category. | |
def | initialize_categories |
Initialize first value of categories. | |
def | cancel_by_taskid |
Cancel task identified by its taskid and remove from the given queue. | |
def | cancel_by_tasktag |
Cancel task identified by its tag and remove from the given queue. | |
def | shutdown_workers |
Shutdown workers connected to queue. | |
def | blacklist |
Blacklist workers running on host. | |
def | blacklist_clear |
Remove host from blacklist. | |
def | specify_keepalive_interval |
Change keepalive interval for a given queue. | |
def | specify_keepalive_timeout |
Change keepalive timeout for a given queue. | |
def | estimate_capacity |
Turn on master capacity measurements. | |
def | tune |
Tune advanced parameters for work queue. | |
def | submit |
Submit a task to the queue. | |
def | wait |
Wait for tasks to complete. |
def work_queue::WorkQueue::__init__ | ( | self, | ||
port = WORK_QUEUE_DEFAULT_PORT , |
||||
name = None , |
||||
catalog = False , |
||||
exclusive = True , |
||||
shutdown = False | ||||
) |
Create a new work queue.
self | Reference to the current work queue object. | |
port | The port number to listen on. If zero is specified, then the default is chosen, and if -1 is specified, a random port is chosen. | |
name | The project name to use. | |
catalog | Whether or not to enable catalog mode. | |
exclusive | Whether or not the workers should be exclusive. | |
shutdown | Automatically shutdown workers when queue is finished. Disabled by default. |
def work_queue::WorkQueue::name | ( | self | ) |
Get the project name of the queue.
Note: This is defined using property decorator. So it must be called without parentheses (). For example:
>>> print q.name
def work_queue::WorkQueue::port | ( | self | ) |
Get the listening port of the queue.
Note: This is defined using property decorator. So it must be called without parentheses (). For example:
>>> print q.port
def work_queue::WorkQueue::stats | ( | self | ) |
Get queue statistics.
Note: This is defined using property decorator. So it must be called without parentheses (). For example:
>>> print q.stats
>>> print q.stats.workers_busy
def work_queue::WorkQueue::stats_hierarchy | ( | self | ) |
Get worker hierarchy statistics.
Note: This is defined using property decorator. So it must be called without parentheses (). For example:
>>> print q.stats_hierarchy
>>> print q.stats_hierarchy.workers_busy
def work_queue::WorkQueue::stats_category | ( | self, | ||
category | ||||
) |
Get the task statistics for the given category.
self | Reference to the current work queue object. | |
category | A category name. For example: s = q.stats_category("my_category") >>> print s >>> print s.tasks_waiting
|
def work_queue::WorkQueue::task_state | ( | self, | ||
taskid | ||||
) |
Get current task state.
See work_queue_task_state_t for possible values.
>>> print q.task_state(taskid)
def work_queue::WorkQueue::enable_monitoring | ( | self, | ||
dirname | ||||
) |
Enables resource monitoring of tasks in the queue, and writes a summary per task to the directory given.
Additionally, all summaries are consolidate into the file all_summaries-PID.log
Returns 1 on success, 0 on failure (i.e., monitoring was not enabled).
self | Reference to the current work queue object. | |
dirname | Directory name for the monitor output. |
def work_queue::WorkQueue::enable_monitoring_full | ( | self, | ||
dirname | ||||
) |
As enable_monitoring, but it also generates a time series and a debug file.
WARNING: Such files may reach gigabyte sizes for long running tasks.
Returns 1 on success, 0 on failure (i.e., monitoring was not enabled).
self | Reference to the current work queue object. | |
dirname | Directory name for the monitor output. |
def work_queue::WorkQueue::activate_fast_abort | ( | self, | ||
multiplier | ||||
) |
Turn on or off fast abort functionality for a given queue for tasks in the "default" category, and for task which category does not set an explicit multiplier.
self | Reference to the current work queue object. | |
multiplier | The multiplier of the average task time at which point to abort; if negative (the default) fast_abort is deactivated. |
def work_queue::WorkQueue::activate_fast_abort_category | ( | self, | ||
name, | ||||
multiplier | ||||
) |
Turn on or off fast abort functionality for a given queue.
self | Reference to the current work queue object. | |
name | Name of the category. | |
multiplier | The multiplier of the average task time at which point to abort; if zero, deacticate for the category, negative (the default), use the one for the "default" category (see fast_abort) |
def work_queue::WorkQueue::empty | ( | self | ) |
Determine whether there are any known tasks queued, running, or waiting to be collected.
Returns 0 if there are tasks remaining in the system, 1 if the system is "empty".
self | Reference to the current work queue object. |
def work_queue::WorkQueue::hungry | ( | self | ) |
Determine whether the queue can support more tasks.
Returns the number of additional tasks it can support if "hungry" and 0 if "sated".
self | Reference to the current work queue object. |
def work_queue::WorkQueue::specify_algorithm | ( | self, | ||
algorithm | ||||
) |
Set the worker selection algorithm for queue.
self | Reference to the current work queue object. | |
algorithm | One of the following algorithms to use in assigning a task to a worker. See work_queue_schedule_t for possible values. |
def work_queue::WorkQueue::specify_task_order | ( | self, | ||
order | ||||
) |
Set the order for dispatching submitted tasks in the queue.
self | Reference to the current work queue object. | |
order | One of the following algorithms to use in dispatching submitted tasks to workers: |
def work_queue::WorkQueue::specify_name | ( | self, | ||
name | ||||
) |
Change the project name for the given queue.
self | Reference to the current work queue object. | |
name | The new project name. |
def work_queue::WorkQueue::specify_priority | ( | self, | ||
priority | ||||
) |
Change the project priority for the given queue.
self | Reference to the current work queue object. | |
priority | An integer that presents the priorty of this work queue master. The higher the value, the higher the priority. |
def work_queue::WorkQueue::specify_num_tasks_left | ( | self, | ||
ntasks | ||||
) |
Specify the number of tasks not yet submitted to the queue.
It is used by work_queue_factory to determine the number of workers to launch. If not specified, it defaults to 0. work_queue_factory considers the number of tasks as: num tasks left + num tasks running + num tasks read.
q | A work queue object. | |
ntasks | Number of tasks yet to be submitted. |
def work_queue::WorkQueue::specify_master_mode | ( | self, | ||
mode | ||||
) |
Specify the master mode for the given queue.
self | Reference to the current work queue object. | |
mode | This may be one of the following values: WORK_QUEUE_MASTER_MODE_STANDALONE or WORK_QUEUE_MASTER_MODE_CATALOG. |
def work_queue::WorkQueue::specify_catalog_server | ( | self, | ||
hostname, | ||||
port | ||||
) |
Specify the catalog server the master should report to.
self | Reference to the current work queue object. | |
hostname | The hostname of the catalog server. | |
port | The port the catalog server is listening on. |
def work_queue::WorkQueue::specify_log | ( | self, | ||
logfile | ||||
) |
Specify a log file that records the states of connected workers and submitted tasks.
self | Reference to the current work queue object. | |
logfile | Filename. |
def work_queue::WorkQueue::specify_password | ( | self, | ||
password | ||||
) |
Add a mandatory password that each worker must present.
self | Reference to the current work queue object. | |
password | The password. |
def work_queue::WorkQueue::specify_password_file | ( | self, | ||
file | ||||
) |
Add a mandatory password file that each worker must present.
self | Reference to the current work queue object. | |
file | Name of the file containing the password. |
def work_queue::WorkQueue::specify_max_resources | ( | self, | ||
rmd | ||||
) |
Enables resource autolabeling for tasks without an explicit category ("default" category).
rm specifies the maximum resources a task in the default category may use. If rm is None, disable autolabeling for the default category.
self | Reference to the current work queue object. | |
rm | Dictionary indicating maximum values. See for possible fields. For example: >>> # A maximum of 4 cores is found on any worker: >>> q.specify_max_resources({'cores': 4}) >>> # A maximum of 8 cores, 1GB of memory, and 10GB disk are found on any worker: >>> q.specify_max_resources({'cores': 8, 'memory': 1024, 'disk': 10240}) |
def work_queue::WorkQueue::specify_max_category_resources | ( | self, | ||
category, | ||||
rmd | ||||
) |
Enables resource autolabeling for tasks in the given category.
rm specifies the maximum resources a task in the category may use. If rm is None, disable autolabeling for that category.
self | Reference to the current work queue object. | |
category | Name of the category. | |
rm | Dictionary indicating maximum values. See for possible fields. For example: >>> # A maximum of 4 cores may be used by a task in the category: >>> q.specify_max_category_resources("my_category", {'cores': 4}) >>> # A maximum of 8 cores, 1GB of memory, and 10GB may be used by a task: >>> q.specify_max_category_resources("my_category", {'cores': 8, 'memory': 1024, 'disk': 10240}) |
def work_queue::WorkQueue::initialize_categories | ( | filename, | ||
rm | ||||
) |
Initialize first value of categories.
self | Reference to the current work queue object. | |
rm | Dictionary indicating maximum values. See for possible fields. | |
filename | JSON file with resource summaries. |
def work_queue::WorkQueue::cancel_by_taskid | ( | self, | ||
id | ||||
) |
Cancel task identified by its taskid and remove from the given queue.
self | Reference to the current work queue object. | |
id | The taskid returned from submit. |
def work_queue::WorkQueue::cancel_by_tasktag | ( | self, | ||
tag | ||||
) |
Cancel task identified by its tag and remove from the given queue.
self | Reference to the current work queue object. | |
tag | The tag assigned to task using work_queue_task_specify_tag. |
def work_queue::WorkQueue::shutdown_workers | ( | self, | ||
n | ||||
) |
Shutdown workers connected to queue.
Gives a best effort and then returns the number of workers given the shutdown order.
self | Reference to the current work queue object. | |
n | The number to shutdown. To shut down all workers, specify "0". |
def work_queue::WorkQueue::blacklist | ( | self, | ||
host | ||||
) |
Blacklist workers running on host.
self | Reference to the current work queue object. | |
host | The hostname the host running the workers. |
def work_queue::WorkQueue::blacklist_clear | ( | self, | ||
host = None | ||||
) |
Remove host from blacklist.
Clear all blacklist if host not provided.
self | Reference to the current work queue object. | |
host | The of the hostname the host. |
def work_queue::WorkQueue::specify_keepalive_interval | ( | self, | ||
interval | ||||
) |
Change keepalive interval for a given queue.
self | Reference to the current work queue object. | |
interval | Minimum number of seconds to wait before sending new keepalive checks to workers. |
def work_queue::WorkQueue::specify_keepalive_timeout | ( | self, | ||
timeout | ||||
) |
Change keepalive timeout for a given queue.
self | Reference to the current work queue object. | |
timeout | Minimum number of seconds to wait for a keepalive response from worker before marking it as dead. |
def work_queue::WorkQueue::estimate_capacity | ( | self | ) |
Turn on master capacity measurements.
self | Reference to the current work queue object. |
def work_queue::WorkQueue::tune | ( | self, | ||
name, | ||||
value | ||||
) |
Tune advanced parameters for work queue.
self | Reference to the current work queue object. | |
name | The name fo the parameter to tune. Can be one of following:
| |
value | The value to set the parameter to. |
def work_queue::WorkQueue::submit | ( | self, | ||
task | ||||
) |
Submit a task to the queue.
It is safe to re-submit a task returned by wait.
self | Reference to the current work queue object. | |
task | A task description created from work_queue::Task. |
def work_queue::WorkQueue::wait | ( | self, | ||
timeout = WORK_QUEUE_WAITFORTASK | ||||
) |
Wait for tasks to complete.
This call will block until the timeout has elapsed
self | Reference to the current work queue object. | |
timeout | The number of seconds to wait for a completed task before returning. Use an integer to set the timeout or the constant WORK_QUEUE_WAITFORTASK to block until a task has completed. |