Main Page | Alphabetical List | Data Structures | File List | Data Fields | Globals | Related Pages

work_queue.h File Reference


Detailed Description

A master-worker library.

The work queue provides an implementation of the master-worker computing model using TCP sockets, Unix applications, and files as intermediate buffers. A master process uses work_queue_create to create a queue, then work_queue_submit to submit tasks. Once tasks are running, call work_queue_wait to wait for completion. The generic worker program can be run on any machine, and simply needs to be told the host and port of the master.

#include "timestamp.h"

Go to the source code of this file.

Data Structures

struct  work_queue_task
 A task description. More...
struct  work_queue_stats
 Statistics describing a work queue. More...

Functions

work_queue * work_queue_create (int port, time_t stoptime)
 Create a new work queue.
void work_queue_delete (struct work_queue *q)
 Delete a work queue.
void work_queue_get_stats (struct work_queue *q, struct work_queue_stats *s)
 Get queue statistics.
int work_queue_activate_fast_abort (struct work_queue *q, double multiplier)
 Turn on or off fast abort functionality for a given queue.
int work_queue_specify_scheduler (struct work_queue *q, int alg)
 Change the worker selection algorithm for a given queue.
work_queue_taskwork_queue_wait (struct work_queue *q, int timeout)
 Wait for tasks to complete.
void work_queue_submit (struct work_queue *q, struct work_queue_task *t)
 Submit a job to a work queue.
void work_queue_task_delete (struct work_queue_task *t)
 Delete a task specification.
work_queue_taskwork_queue_task_create (const char *full_command)
 Create a new task specification.
void work_queue_task_specify_tag (struct work_queue_task *t, const char *tag)
 Attach a user defined logical name to the task.
int work_queue_task_specify_scheduler (struct work_queue_task *t, int alg)
 Further define a task specification.
void work_queue_task_specify_input_buf (struct work_queue_task *t, const char *buf, int length, const char *rname)
 Further define a task specification.
void work_queue_task_specify_input_file (struct work_queue_task *t, const char *fname, const char *rname)
 Further define a task specification.
void work_queue_task_specify_output_file (struct work_queue_task *t, const char *rname, const char *fname)
 Further define a task specification.
int work_queue_hungry (struct work_queue *q)
 Determine whether the queue can support more tasks.
int work_queue_empty (struct work_queue *q)
 Determine whether there are any known tasks queued, running, or waiting to be collected.
int work_queue_shut_down_workers (struct work_queue *q, int n)
 Shut down workers connected to the work_queue system.

Variables

double wq_option_fast_abort_multiplier
 Initial setting for fast abort multiplier upon creating queue.
int wq_option_scheduler
 Initial setting for algorithm to assign tasks to workers upon creating queue.


Function Documentation

struct work_queue* work_queue_create int  port,
time_t  stoptime
 

Create a new work queue.

Parameters:
port The port number to listen on, or zero to choose a default. The default port is 9123, but can be overridden by the environment variable WORK_QUEUE_PORT.
stoptime The time at which to return null if not yet able to be created.
Returns:
A new work queue, or null if it could not be created.

void work_queue_delete struct work_queue *  q  ) 
 

Delete a work queue.

Parameters:
q The work queue to delete.

void work_queue_get_stats struct work_queue *  q,
struct work_queue_stats s
 

Get queue statistics.

Parameters:
q The queue to query.
s A pointer to a buffer that will be filed with statistics.

int work_queue_activate_fast_abort struct work_queue *  q,
double  multiplier
 

Turn on or off fast abort functionality for a given queue.

Parameters:
q A pointer to the queue to modify.
multiplier The multiplier of the average task time at which point to abort; if negative (and by default) fast_abort is deactivated.
Returns:
0 if activated or deactivated with an appropriate multiplier, 1 if deactivated due to inappropriate multiplier.

int work_queue_specify_scheduler struct work_queue *  q,
int  alg
 

Change the worker selection algorithm for a given queue.

Parameters:
q A pointer to the queue to modify.
alg The algorithm to use in assigning a task to a worker. Valid possibilities are defined in this file as "WORK_QUEUE_SCHEDULE_X" values.

struct work_queue_task* work_queue_wait struct work_queue *  q,
int  timeout
 

Wait for tasks to complete.

This call will block until the timeout has elapsed.

Parameters:
q The work queue to wait on.
timeout The number of seconds to wait for a completed task before returning. Use an integer time to set the timeout or the constant WAITFORTASK to block until a task has completed.
Returns:
A completed task description, or null if the queue is empty or the timeout was reached without a completed task. The returned task must be deleted with work_queue_task_delete or resubmitted with work_queue_submit.

void work_queue_submit struct work_queue *  q,
struct work_queue_task t
 

Submit a job to a work queue.

It is safe to re-submit a task returned by work_queue_wait.

Parameters:
q A work queue returned from work_queue_create.
t A task description returned from work_queue_task_create.

void work_queue_task_delete struct work_queue_task t  ) 
 

Delete a task specification.

This may be called on tasks after they are returned from work_queue_wait.

Parameters:
t The task specification to delete.

struct work_queue_task* work_queue_task_create const char *  full_command  ) 
 

Create a new task specification.

Once created, the task may be passed to work_queue_submit.

Parameters:
full_command The shell command line to be executed by the task.

void work_queue_task_specify_tag struct work_queue_task t,
const char *  tag
 

Attach a user defined logical name to the task.

This field is not interpreted by the work queue, but simply maintained to help the user track tasks.

Parameters:
t The task to which to add parameters
tag The tag to attach to task t.

int work_queue_task_specify_scheduler struct work_queue_task t,
int  alg
 

Further define a task specification.

Once completed, the task may be passed to work_queue_submit.

Parameters:
t The task to which to add parameters
alg The algorithm to use in assigning a task to a worker. Valid possibilities are defined in this file as "WORK_QUEUE_SCHEDULE_X" values.

void work_queue_task_specify_input_buf struct work_queue_task t,
const char *  buf,
int  length,
const char *  rname
 

Further define a task specification.

Once completed, the task may be passed to work_queue_submit.

Parameters:
t The task to which to add parameters
buf A pointer to the data buffer to send to the worker to be available to the commands.
length The number of bytes of data in the buffer
rname The name of the file in which to store the buffer data on the worker

void work_queue_task_specify_input_file struct work_queue_task t,
const char *  fname,
const char *  rname
 

Further define a task specification.

Once completed, the task may be passed to work_queue_submit.

Parameters:
t The task to which to add parameters
fname The name of the data file to send to the worker to be available to the commands.
rname The name of the file in which to store the buffer data on the worker

void work_queue_task_specify_output_file struct work_queue_task t,
const char *  rname,
const char *  fname
 

Further define a task specification.

Once completed, the task may be passed to work_queue_submit. If no file is defined, the program will have default (no) output files retrieved.

Parameters:
t The task to which to add parameters
rname The name of a file created by the program when it runs.
fname The name of the file local target for copying rname back.

int work_queue_hungry struct work_queue *  q  ) 
 

Determine whether the queue can support more tasks.

Returns the number of additional tasks it can support if "hungry" and 0 if "sated".

Parameters:
q A pointer to the queue to query.

int work_queue_empty struct work_queue *  q  ) 
 

Determine whether there are any known tasks queued, running, or waiting to be collected.

Returns 0 if there are tasks remaining in the system, 1 if the system is "empty".

Parameters:
q A pointer to the queue to query.

int work_queue_shut_down_workers struct work_queue *  q,
int  n
 

Shut down workers connected to the work_queue system.

Gives a best effort and then returns the number of workers given the shut down order.

Parameters:
q A pointer to the queue to query.
n The number to shut down. All workers if given "0".


Variable Documentation

double wq_option_fast_abort_multiplier
 

Initial setting for fast abort multiplier upon creating queue.

Turned off if less than 0. Change prior to calling work_queue_create, after queue is created this variable is not considered and changes must be made through the API calls.

int wq_option_scheduler
 

Initial setting for algorithm to assign tasks to workers upon creating queue.

Change prior to calling work_queue_create, after queue is created this variable is not considered and changes must be made through the API calls.


Generated on Fri Jun 4 09:58:46 2010 for cctools by  doxygen 1.3.9.1