cctools
|
Python Work Queue object. More...
Public Member Functions | |
def | __init__ |
Create a new work queue. More... | |
def | name |
Get the project name of the queue. More... | |
def | port |
Get the listening port of the queue. More... | |
def | stats |
Get queue statistics. More... | |
def | stats_hierarchy |
Get worker hierarchy statistics. More... | |
def | enable_monitoring |
Enables resource monitoring of tasks in the queue. More... | |
def | activate_fast_abort |
Turn on or off fast abort functionality for a given queue. More... | |
def | empty |
Determine whether there are any known tasks queued, running, or waiting to be collected. More... | |
def | hungry |
Determine whether the queue can support more tasks. More... | |
def | specify_algorithm |
Set the worker selection algorithm for queue. More... | |
def | specify_task_order |
Set the order for dispatching submitted tasks in the queue. More... | |
def | specify_name |
Change the project name for the given queue. More... | |
def | specify_priority |
Change the project priority for the given queue. More... | |
def | specify_master_mode |
Specify the master mode for the given queue. More... | |
def | specify_catalog_server |
Specify the catalog server the master should report to. More... | |
def | specify_log |
Specify a log file that records the states of connected workers and submitted tasks. More... | |
def | specify_password |
Add a mandatory password that each worker must present. More... | |
def | specify_password_file |
Add a mandatory password file that each worker must present. More... | |
def | cancel_by_taskid |
Cancel task identified by its taskid and remove from the given queue. More... | |
def | cancel_by_tasktag |
Cancel task identified by its tag and remove from the given queue. More... | |
def | shutdown_workers |
Shutdown workers connected to queue. More... | |
def | blacklist |
Blacklist workers running on host. More... | |
def | blacklist_clear |
Remove host from blacklist. More... | |
def | specify_keepalive_interval |
Change keepalive interval for a given queue. More... | |
def | specify_keepalive_timeout |
Change keepalive timeout for a given queue. More... | |
def | estimate_capacity |
Turn on master capacity measurements. More... | |
def | tune |
Tune advanced parameters for work queue. More... | |
def | submit |
Submit a task to the queue. More... | |
def | wait |
Wait for tasks to complete. More... | |
Python Work Queue object.
This class uses a dictionary to map between the task pointer objects and the work_queue.Task.
def work_queue.WorkQueue.__init__ | ( | self, | |
port = WORK_QUEUE_DEFAULT_PORT , |
|||
name = None , |
|||
catalog = False , |
|||
exclusive = True , |
|||
shutdown = False |
|||
) |
Create a new work queue.
self | Reference to the current work queue object. |
port | The port number to listen on. If zero is specified, then the default is chosen, and if -1 is specified, a random port is chosen. |
name | The project name to use. |
catalog | Whether or not to enable catalog mode. |
exclusive | Whether or not the workers should be exclusive. |
shutdown | Automatically shutdown workers when queue is finished. Disabled by default. |
References work_queue.WorkQueue.__free_queue(), work_queue.WorkQueue._shutdown, work_queue.WorkQueue._stats, work_queue.WorkQueue._stats_hierarchy, work_queue.WorkQueue._task_table, work_queue.WorkQueue._work_queue, work_queue.WorkQueue.shutdown_workers(), work_queue_create(), work_queue_delete(), work_queue_specify_master_mode(), and work_queue_specify_name().
def work_queue.WorkQueue.name | ( | self | ) |
Get the project name of the queue.
Note: This is defined using property decorator. So it must be called without parentheses (). For example:
References work_queue.WorkQueue._work_queue, and work_queue_name().
def work_queue.WorkQueue.port | ( | self | ) |
Get the listening port of the queue.
Note: This is defined using property decorator. So it must be called without parentheses (). For example:
References work_queue.WorkQueue._work_queue, and work_queue_port().
def work_queue.WorkQueue.stats | ( | self | ) |
Get queue statistics.
Note: This is defined using property decorator. So it must be called without parentheses (). For example:
The fields in work_queue_stats can also be individually accessed through this call. For example:
References work_queue.WorkQueue._stats, work_queue.WorkQueue._work_queue, and work_queue_get_stats().
def work_queue.WorkQueue.stats_hierarchy | ( | self | ) |
Get worker hierarchy statistics.
Note: This is defined using property decorator. So it must be called without parentheses (). For example:
The fields in work_queue_stats can also be individually accessed through this call. For example:
References work_queue.WorkQueue._stats_hierarchy, work_queue.WorkQueue._work_queue, and work_queue_get_stats_hierarchy().
def work_queue.WorkQueue.enable_monitoring | ( | self, | |
summaryfile | |||
) |
Enables resource monitoring of tasks in the queue.
And writes a summary of the monitored information to a file.
Returns 1 on success, 0 on failure (i.e., monitoring was not enabled).
self | Reference to the current work queue object. |
summaryfile | Filename for the summary log (If NULL, writes to wq-<pid>-resource-usage). |
References work_queue.WorkQueue._work_queue, and work_queue_enable_monitoring().
def work_queue.WorkQueue.activate_fast_abort | ( | self, | |
multiplier | |||
) |
Turn on or off fast abort functionality for a given queue.
self | Reference to the current work queue object. |
multiplier | The multiplier of the average task time at which point to abort; if negative (the default) fast_abort is deactivated. |
References work_queue.WorkQueue._work_queue, and work_queue_activate_fast_abort().
def work_queue.WorkQueue.empty | ( | self | ) |
Determine whether there are any known tasks queued, running, or waiting to be collected.
Returns 0 if there are tasks remaining in the system, 1 if the system is "empty".
self | Reference to the current work queue object. |
References work_queue.WorkQueue._work_queue, and work_queue_empty().
def work_queue.WorkQueue.hungry | ( | self | ) |
Determine whether the queue can support more tasks.
Returns the number of additional tasks it can support if "hungry" and 0 if "sated".
self | Reference to the current work queue object. |
References work_queue.WorkQueue._work_queue, and work_queue_hungry().
def work_queue.WorkQueue.specify_algorithm | ( | self, | |
algorithm | |||
) |
Set the worker selection algorithm for queue.
self | Reference to the current work queue object. |
algorithm | One of the following algorithms to use in assigning a task to a worker: |
References work_queue.WorkQueue._work_queue, and work_queue_specify_algorithm().
def work_queue.WorkQueue.specify_task_order | ( | self, | |
order | |||
) |
Set the order for dispatching submitted tasks in the queue.
self | Reference to the current work queue object. |
order | One of the following algorithms to use in dispatching submitted tasks to workers: |
References work_queue.WorkQueue._work_queue, and work_queue_specify_task_order().
def work_queue.WorkQueue.specify_name | ( | self, | |
name | |||
) |
Change the project name for the given queue.
self | Reference to the current work queue object. |
name | The new project name. |
References work_queue.WorkQueue._work_queue, and work_queue_specify_name().
def work_queue.WorkQueue.specify_priority | ( | self, | |
priority | |||
) |
Change the project priority for the given queue.
self | Reference to the current work queue object. |
priority | An integer that presents the priorty of this work queue master. The higher the value, the higher the priority. |
References work_queue.WorkQueue._work_queue, and work_queue_specify_priority().
def work_queue.WorkQueue.specify_master_mode | ( | self, | |
mode | |||
) |
Specify the master mode for the given queue.
self | Reference to the current work queue object. |
mode | This may be one of the following values: WORK_QUEUE_MASTER_MODE_STANDALONE or WORK_QUEUE_MASTER_MODE_CATALOG. |
References work_queue.WorkQueue._work_queue, and work_queue_specify_master_mode().
def work_queue.WorkQueue.specify_catalog_server | ( | self, | |
hostname, | |||
port | |||
) |
Specify the catalog server the master should report to.
self | Reference to the current work queue object. |
hostname | The hostname of the catalog server. |
port | The port the catalog server is listening on. |
References work_queue.WorkQueue._work_queue, and work_queue_specify_catalog_server().
def work_queue.WorkQueue.specify_log | ( | self, | |
logfile | |||
) |
Specify a log file that records the states of connected workers and submitted tasks.
self | Reference to the current work queue object. |
logfile | Filename. |
References work_queue.WorkQueue._work_queue, and work_queue_specify_log().
def work_queue.WorkQueue.specify_password | ( | self, | |
password | |||
) |
Add a mandatory password that each worker must present.
self | Reference to the current work queue object. |
password | The password. |
References work_queue.WorkQueue._work_queue, and work_queue_specify_password().
def work_queue.WorkQueue.specify_password_file | ( | self, | |
file | |||
) |
Add a mandatory password file that each worker must present.
self | Reference to the current work queue object. |
file | Name of the file containing the password. |
References work_queue.WorkQueue._work_queue, and work_queue_specify_password_file().
def work_queue.WorkQueue.cancel_by_taskid | ( | self, | |
id | |||
) |
Cancel task identified by its taskid and remove from the given queue.
self | Reference to the current work queue object. |
id | The taskid returned from submit. |
References work_queue.WorkQueue._work_queue, and work_queue_cancel_by_taskid().
def work_queue.WorkQueue.cancel_by_tasktag | ( | self, | |
tag | |||
) |
Cancel task identified by its tag and remove from the given queue.
self | Reference to the current work queue object. |
tag | The tag assigned to task using work_queue_task_specify_tag. |
References work_queue.WorkQueue._work_queue, and work_queue_cancel_by_tasktag().
def work_queue.WorkQueue.shutdown_workers | ( | self, | |
n | |||
) |
Shutdown workers connected to queue.
Gives a best effort and then returns the number of workers given the shutdown order.
self | Reference to the current work queue object. |
n | The number to shutdown. To shut down all workers, specify "0". |
References work_queue.WorkQueue._work_queue, and work_queue_shut_down_workers().
Referenced by work_queue.WorkQueue.__init__().
def work_queue.WorkQueue.blacklist | ( | self, | |
host | |||
) |
Blacklist workers running on host.
self | Reference to the current work queue object. |
host | The hostname the host running the workers. |
References work_queue.WorkQueue._work_queue, and work_queue_blacklist_add().
def work_queue.WorkQueue.blacklist_clear | ( | self, | |
host = None |
|||
) |
Remove host from blacklist.
Clear all blacklist if host not provided.
self | Reference to the current work queue object. |
host | The of the hostname the host. |
References work_queue.WorkQueue._work_queue, work_queue_blacklist_clear(), and work_queue_blacklist_remove().
def work_queue.WorkQueue.specify_keepalive_interval | ( | self, | |
interval | |||
) |
Change keepalive interval for a given queue.
self | Reference to the current work queue object. |
interval | Minimum number of seconds to wait before sending new keepalive checks to workers. |
References work_queue.WorkQueue._work_queue, and work_queue_specify_keepalive_interval().
def work_queue.WorkQueue.specify_keepalive_timeout | ( | self, | |
timeout | |||
) |
Change keepalive timeout for a given queue.
self | Reference to the current work queue object. |
timeout | Minimum number of seconds to wait for a keepalive response from worker before marking it as dead. |
References work_queue.WorkQueue._work_queue, and work_queue_specify_keepalive_timeout().
def work_queue.WorkQueue.estimate_capacity | ( | self | ) |
Turn on master capacity measurements.
self | Reference to the current work queue object. |
References work_queue.WorkQueue._work_queue, and work_queue_specify_estimate_capacity_on().
def work_queue.WorkQueue.tune | ( | self, | |
name, | |||
value | |||
) |
Tune advanced parameters for work queue.
self | Reference to the current work queue object. |
name | The name fo the parameter to tune. Can be one of following:
|
value | The value to set the parameter to. |
References work_queue.WorkQueue._work_queue, and work_queue_tune().
def work_queue.WorkQueue.submit | ( | self, | |
task | |||
) |
Submit a task to the queue.
It is safe to re-submit a task returned by wait.
self | Reference to the current work queue object. |
task | A task description created from work_queue.Task. |
References work_queue.WorkQueue._task_table, work_queue.WorkQueue._work_queue, and work_queue_submit().
def work_queue.WorkQueue.wait | ( | self, | |
timeout = WORK_QUEUE_WAITFORTASK |
|||
) |
Wait for tasks to complete.
This call will block until the timeout has elapsed
self | Reference to the current work queue object. |
timeout | The number of seconds to wait for a completed task before returning. Use an integer to set the timeout or the constant WORK_QUEUE_WAITFORTASK to block until a task has completed. |
References work_queue.WorkQueue._task_table, work_queue.WorkQueue._work_queue, and work_queue_wait().