work_queue::WorkQueue Class Reference

Python Work Queue object. More...

Public Member Functions

def __init__
 Create a new work queue.
def name
 Get the project name of the queue.
def port
 Get the listening port of the queue.
def stats
 Get queue statistics.
def stats_hierarchy
 Get worker hierarchy statistics.
def task_state
 Get current task state.
def enable_monitoring
 Enables resource monitoring of tasks in the queue.
def activate_fast_abort
 Turn on or off fast abort functionality for a given queue.
def empty
 Determine whether there are any known tasks queued, running, or waiting to be collected.
def hungry
 Determine whether the queue can support more tasks.
def specify_algorithm
 Set the worker selection algorithm for queue.
def specify_task_order
 Set the order for dispatching submitted tasks in the queue.
def specify_name
 Change the project name for the given queue.
def specify_priority
 Change the project priority for the given queue.
def specify_master_mode
 Specify the master mode for the given queue.
def specify_catalog_server
 Specify the catalog server the master should report to.
def specify_log
 Specify a log file that records the states of connected workers and submitted tasks.
def specify_password
 Add a mandatory password that each worker must present.
def specify_password_file
 Add a mandatory password file that each worker must present.
def cancel_by_taskid
 Cancel task identified by its taskid and remove from the given queue.
def cancel_by_tasktag
 Cancel task identified by its tag and remove from the given queue.
def shutdown_workers
 Shutdown workers connected to queue.
def blacklist
 Blacklist workers running on host.
def blacklist_clear
 Remove host from blacklist.
def specify_keepalive_interval
 Change keepalive interval for a given queue.
def specify_keepalive_timeout
 Change keepalive timeout for a given queue.
def estimate_capacity
 Turn on master capacity measurements.
def tune
 Tune advanced parameters for work queue.
def submit
 Submit a task to the queue.
def wait
 Wait for tasks to complete.

Detailed Description

Python Work Queue object.

This class uses a dictionary to map between the task pointer objects and the work_queue::Task.


Member Function Documentation

def work_queue::WorkQueue::__init__ (   self,
  port = WORK_QUEUE_DEFAULT_PORT,
  name = None,
  catalog = False,
  exclusive = True,
  shutdown = False 
)

Create a new work queue.

Parameters:
self Reference to the current work queue object.
port The port number to listen on. If zero is specified, then the default is chosen, and if -1 is specified, a random port is chosen.
name The project name to use.
catalog Whether or not to enable catalog mode.
exclusive Whether or not the workers should be exclusive.
shutdown Automatically shutdown workers when queue is finished. Disabled by default.
See also:
work_queue_create - For more information about environmental variables that affect the behavior this method.
def work_queue::WorkQueue::name (   self  ) 

Get the project name of the queue.

Note: This is defined using property decorator. So it must be called without parentheses (). For example:

 >>> print q.name
def work_queue::WorkQueue::port (   self  ) 

Get the listening port of the queue.

Note: This is defined using property decorator. So it must be called without parentheses (). For example:

 >>> print q.port
def work_queue::WorkQueue::stats (   self  ) 

Get queue statistics.

Note: This is defined using property decorator. So it must be called without parentheses (). For example:

 >>> print q.stats

The fields in work_queue_stats can also be individually accessed through this call. For example:

 >>> print q.stats.workers_busy
def work_queue::WorkQueue::stats_hierarchy (   self  ) 

Get worker hierarchy statistics.

Note: This is defined using property decorator. So it must be called without parentheses (). For example:

 >>> print q.stats

The fields in work_queue_stats can also be individually accessed through this call. For example:

 >>> print q.stats.workers_busy
def work_queue::WorkQueue::task_state (   self,
  taskid 
)

Get current task state.

 >>> print q.task_state(taskid)
def work_queue::WorkQueue::enable_monitoring (   self,
  summaryfile 
)

Enables resource monitoring of tasks in the queue.

And writes a summary of the monitored information to a file.

Returns 1 on success, 0 on failure (i.e., monitoring was not enabled).

Parameters:
self Reference to the current work queue object.
summaryfile Filename for the summary log (If NULL, writes to wq-<pid>-resource-usage).
def work_queue::WorkQueue::activate_fast_abort (   self,
  multiplier 
)

Turn on or off fast abort functionality for a given queue.

Parameters:
self Reference to the current work queue object.
multiplier The multiplier of the average task time at which point to abort; if negative (the default) fast_abort is deactivated.
def work_queue::WorkQueue::empty (   self  ) 

Determine whether there are any known tasks queued, running, or waiting to be collected.

Returns 0 if there are tasks remaining in the system, 1 if the system is "empty".

Parameters:
self Reference to the current work queue object.
def work_queue::WorkQueue::hungry (   self  ) 

Determine whether the queue can support more tasks.

Returns the number of additional tasks it can support if "hungry" and 0 if "sated".

Parameters:
self Reference to the current work queue object.
def work_queue::WorkQueue::specify_algorithm (   self,
  algorithm 
)

Set the worker selection algorithm for queue.

Parameters:
self Reference to the current work queue object.
algorithm One of the following algorithms to use in assigning a task to a worker:

def work_queue::WorkQueue::specify_task_order (   self,
  order 
)

Set the order for dispatching submitted tasks in the queue.

Parameters:
self Reference to the current work queue object.
order One of the following algorithms to use in dispatching submitted tasks to workers:

def work_queue::WorkQueue::specify_name (   self,
  name 
)

Change the project name for the given queue.

Parameters:
self Reference to the current work queue object.
name The new project name.
def work_queue::WorkQueue::specify_priority (   self,
  priority 
)

Change the project priority for the given queue.

Parameters:
self Reference to the current work queue object.
priority An integer that presents the priorty of this work queue master. The higher the value, the higher the priority.
def work_queue::WorkQueue::specify_master_mode (   self,
  mode 
)

Specify the master mode for the given queue.

Parameters:
self Reference to the current work queue object.
mode This may be one of the following values: WORK_QUEUE_MASTER_MODE_STANDALONE or WORK_QUEUE_MASTER_MODE_CATALOG.
def work_queue::WorkQueue::specify_catalog_server (   self,
  hostname,
  port 
)

Specify the catalog server the master should report to.

Parameters:
self Reference to the current work queue object.
hostname The hostname of the catalog server.
port The port the catalog server is listening on.
def work_queue::WorkQueue::specify_log (   self,
  logfile 
)

Specify a log file that records the states of connected workers and submitted tasks.

Parameters:
self Reference to the current work queue object.
logfile Filename.
def work_queue::WorkQueue::specify_password (   self,
  password 
)

Add a mandatory password that each worker must present.

Parameters:
self Reference to the current work queue object.
password The password.
def work_queue::WorkQueue::specify_password_file (   self,
  file 
)

Add a mandatory password file that each worker must present.

Parameters:
self Reference to the current work queue object.
file Name of the file containing the password.
def work_queue::WorkQueue::cancel_by_taskid (   self,
  id 
)

Cancel task identified by its taskid and remove from the given queue.

Parameters:
self Reference to the current work queue object.
id The taskid returned from submit.
def work_queue::WorkQueue::cancel_by_tasktag (   self,
  tag 
)

Cancel task identified by its tag and remove from the given queue.

Parameters:
self Reference to the current work queue object.
tag The tag assigned to task using work_queue_task_specify_tag.
def work_queue::WorkQueue::shutdown_workers (   self,
  n 
)

Shutdown workers connected to queue.

Gives a best effort and then returns the number of workers given the shutdown order.

Parameters:
self Reference to the current work queue object.
n The number to shutdown. To shut down all workers, specify "0".
def work_queue::WorkQueue::blacklist (   self,
  host 
)

Blacklist workers running on host.

Parameters:
self Reference to the current work queue object.
host The hostname the host running the workers.
def work_queue::WorkQueue::blacklist_clear (   self,
  host = None 
)

Remove host from blacklist.

Clear all blacklist if host not provided.

Parameters:
self Reference to the current work queue object.
host The of the hostname the host.
def work_queue::WorkQueue::specify_keepalive_interval (   self,
  interval 
)

Change keepalive interval for a given queue.

Parameters:
self Reference to the current work queue object.
interval Minimum number of seconds to wait before sending new keepalive checks to workers.
def work_queue::WorkQueue::specify_keepalive_timeout (   self,
  timeout 
)

Change keepalive timeout for a given queue.

Parameters:
self Reference to the current work queue object.
timeout Minimum number of seconds to wait for a keepalive response from worker before marking it as dead.
def work_queue::WorkQueue::estimate_capacity (   self  ) 

Turn on master capacity measurements.

Parameters:
self Reference to the current work queue object.
def work_queue::WorkQueue::tune (   self,
  name,
  value 
)

Tune advanced parameters for work queue.

Parameters:
self Reference to the current work queue object.
name The name fo the parameter to tune. Can be one of following:

  • "asynchrony-multiplier" Treat each worker as having (actual_cores * multiplier) total cores. (default = 1.0)
  • "asynchrony-modifier" Treat each worker as having an additional "modifier" cores. (default=0)
  • "min-transfer-timeout" Set the minimum number of seconds to wait for files to be transferred to or from a worker. (default=300)
  • "foreman-transfer-timeout" Set the minimum number of seconds to wait for files to be transferred to or from a foreman. (default=3600)
  • "fast-abort-multiplier" Set the multiplier of the average task time at which point to abort; if negative or zero fast_abort is deactivated. (default=0)
  • "keepalive-interval" Set the minimum number of seconds to wait before sending new keepalive checks to workers. (default=300)
  • "keepalive-timeout" Set the minimum number of seconds to wait for a keepalive response from worker before marking it as dead. (default=30)
value The value to set the parameter to.
Returns:
0 on succes, -1 on failure.
def work_queue::WorkQueue::submit (   self,
  task 
)

Submit a task to the queue.

It is safe to re-submit a task returned by wait.

Parameters:
self Reference to the current work queue object.
task A task description created from work_queue::Task.
def work_queue::WorkQueue::wait (   self,
  timeout = WORK_QUEUE_WAITFORTASK 
)

Wait for tasks to complete.

This call will block until the timeout has elapsed

Parameters:
self Reference to the current work queue object.
timeout The number of seconds to wait for a completed task before returning. Use an integer to set the timeout or the constant WORK_QUEUE_WAITFORTASK to block until a task has completed.

The documentation for this class was generated from the following file:

Generated on 6 May 2015 for cctools by  doxygen 1.6.1