Main Page | Alphabetical List | Data Structures | File List | Data Fields | Globals | Related Pages

work_queue.h File Reference


Detailed Description

A master-worker library.

The work queue provides an implementation of the master-worker computing model using TCP sockets, Unix applications, and files as intermediate buffers. A master process uses work_queue_create to create a queue, then work_queue_submit to submit tasks. Once tasks are running, call work_queue_wait to wait for completion. The generic worker program can be run on any machine, and simply needs to be told the host and port of the master.

#include "timestamp.h"

Go to the source code of this file.

Data Structures

struct  work_queue_task
 A task description. More...
struct  work_queue_stats
 Statistics describing a work queue. More...

Functions

work_queue * work_queue_create (int port, time_t stoptime)
 Create a new work queue.
void work_queue_delete (struct work_queue *q)
 Delete a work queue.
void work_queue_get_stats (struct work_queue *q, struct work_queue_stats *s)
 Get queue statistics.
int work_queue_activate_fast_abort (struct work_queue *q, double multiplier)
 Turn on fast abort functionality for a given queue.
work_queue_taskwork_queue_wait (struct work_queue *q, int timeout)
 Wait for tasks to complete.
void work_queue_submit (struct work_queue *q, struct work_queue_task *t)
 Submit a job to a work queue.
void work_queue_task_delete (struct work_queue_task *t)
 Delete a task specification.
work_queue_taskwork_queue_task_create (const char *full_command)
 Create a new task specification.
INT64_T work_queue_task_specify_tag (struct work_queue_task *t, const char *tag)
 Further define a task specification.
INT64_T work_queue_task_specify_input_buf (struct work_queue_task *t, const char *buf, int length, const char *rname)
 Further define a task specification.
INT64_T work_queue_task_specify_input_file (struct work_queue_task *t, const char *fname, const char *rname)
 Further define a task specification.
INT64_T work_queue_task_specify_output_file (struct work_queue_task *t, const char *rname, const char *fname)
 Further define a task specification.
int work_queue_hungry (struct work_queue *q)
 Determine whether the queue can support more tasks.
int work_queue_empty (struct work_queue *q)
 Determine whether there are any known tasks queued, running, or waiting to be collected.
int work_queue_shut_down_workers (struct work_queue *q, int n)
 Shut down workers connected to the work_queue system.


Function Documentation

struct work_queue* work_queue_create int  port,
time_t  stoptime
 

Create a new work queue.

Parameters:
port The port number to listen on, or zero to choose a default. The default port is 9123, but can be overridden by the environment variable WORK_QUEUE_PORT.
stoptime The time at which to return null if not yet able to be created.
Returns:
A new work queue, or null if it could not be created.

void work_queue_delete struct work_queue *  q  ) 
 

Delete a work queue.

Parameters:
q The work queue to delete.

void work_queue_get_stats struct work_queue *  q,
struct work_queue_stats s
 

Get queue statistics.

Parameters:
q The queue to query.
s A pointer to a buffer that will be filed with statistics.

int work_queue_activate_fast_abort struct work_queue *  q,
double  multiplier
 

Turn on fast abort functionality for a given queue.

Parameters:
q A pointer to the queue to modify.
multiplier The multiplier of the average task time at which point to abort.
Returns:
0 if activated with appropriate multiplier, 1 if activated with the default multiplier.

struct work_queue_task* work_queue_wait struct work_queue *  q,
int  timeout
 

Wait for tasks to complete.

This call will block until the timeout has elapsed.

Parameters:
q The work queue to wait on.
timeout The number of seconds to wait for a completed task before returning. Use an integer time to set the timeout or the constant WAITFORTASK to block until a task has completed.
Returns:
A completed task description, or null if the queue is empty or the timeout was reached without a completed task.

void work_queue_submit struct work_queue *  q,
struct work_queue_task t
 

Submit a job to a work queue.

Parameters:
q A work queue returned from work_queue_create.
t A task description returned from work_queue_task_create.

void work_queue_task_delete struct work_queue_task t  ) 
 

Delete a task specification.

This may be called on tasks after they are returned from work_queue_wait.

Parameters:
t The task specification to delete.

struct work_queue_task* work_queue_task_create const char *  full_command  ) 
 

Create a new task specification.

Once created, the task may be passed to work_queue_submit.

Parameters:
full_command The shell command line to be executed by the task.

INT64_T work_queue_task_specify_tag struct work_queue_task t,
const char *  tag
 

Further define a task specification.

Once completed, the task may be passed to work_queue_submit.

Parameters:
t The task to which to add parameters
tag The tag to attatch to tast t.

INT64_T work_queue_task_specify_input_buf struct work_queue_task t,
const char *  buf,
int  length,
const char *  rname
 

Further define a task specification.

Once completed, the task may be passed to work_queue_submit.

Parameters:
t The task to which to add parameters
buf A pointer to the data buffer to send to the worker to be available to the commands.
length The number of bytes of data in the buffer
rname The name of the file in which to store the buffer data on the worker

INT64_T work_queue_task_specify_input_file struct work_queue_task t,
const char *  fname,
const char *  rname
 

Further define a task specification.

Once completed, the task may be passed to work_queue_submit.

Parameters:
t The task to which to add parameters
fname The name of the data file to send to the worker to be available to the commands.
rname The name of the file in which to store the buffer data on the worker

INT64_T work_queue_task_specify_output_file struct work_queue_task t,
const char *  rname,
const char *  fname
 

Further define a task specification.

Once completed, the task may be passed to work_queue_submit. If no file is defined, the program will have default (no) output files retrieved.

Parameters:
t The task to which to add parameters
rname The name of a file created by the program when it runs.
fname The name of the file local target for copying rname back.

int work_queue_hungry struct work_queue *  q  ) 
 

Determine whether the queue can support more tasks.

Returns the number of additional tasks it can support if "hungry" and 0 if "sated".

Parameters:
q A pointer to the queue to query.

int work_queue_empty struct work_queue *  q  ) 
 

Determine whether there are any known tasks queued, running, or waiting to be collected.

Returns 1 if so and 0 if "empty".

Parameters:
q A pointer to the queue to query.

int work_queue_shut_down_workers struct work_queue *  q,
int  n
 

Shut down workers connected to the work_queue system.

Gives a best effort and then returns the number of workers given the shut down order.

Parameters:
q A pointer to the queue to query.
n The number to shut down. All workers if given "0".


Generated on Thu Jul 23 09:26:51 2009 for cctools by  doxygen 1.3.9.1