Main Page | Alphabetical List | Data Structures | File List | Data Fields | Globals | Related Pages

work_queue.h File Reference


Detailed Description

A master-worker library.

The work queue provides an implementation of the master-worker computing model using TCP sockets, Unix applications, and files as intermediate buffers. A master process uses work_queue_create to create a queue, then work_queue_submit to submit tasks. Once tasks are running, call work_queue_wait to wait for completion. The generic worker program can be run on any machine, and simply needs to be told the host and port of the master.

#include "timestamp.h"

Go to the source code of this file.

Data Structures

struct  task_file
 A file for input to a task. More...
struct  work_queue_task
 A task description. More...
struct  work_queue_stats
 Statistics describing a work queue. More...

Functions

work_queue * work_queue_create (int port, time_t stoptime)
 Create a new work queue.
void work_queue_delete (struct work_queue *q)
 Delete a work queue.
void work_queue_get_stats (struct work_queue *q, struct work_queue_stats *s)
 Get queue statistics.
work_queue_taskwork_queue_wait (struct work_queue *q, time_t timeout)
 Wait for tasks to complete.
void work_queue_submit (struct work_queue *q, struct work_queue_task *t)
 Submit a job to a work queue.
void work_queue_task_delete (struct work_queue_task *t)
 Delete a task specification.
work_queue_taskwork_queue_task_create (const char *program, const char *args)
 Create a new task specification.
INT64_T work_queue_task_add_tag (struct work_queue_task *t, const char *tag)
 Further define a task specification.
INT64_T work_queue_task_add_standard_input_buf (struct work_queue_task *t, const char *buf, int length)
 Further define a task specification.
INT64_T work_queue_task_add_standard_input_file (struct work_queue_task *t, const char *fname)
 Further define a task specification.
INT64_T work_queue_task_add_extra_created_file (struct work_queue_task *t, const char *rname, const char *fname)
 Further define a task specification.
INT64_T work_queue_task_add_extra_staged_buf (struct work_queue_task *t, const char *buf, int length, const char *rname)
 Further define a task specification.
INT64_T work_queue_task_add_extra_staged_file (struct work_queue_task *t, const char *fname, const char *rname)
 Further define a task specification.
int work_queue_hungry (struct work_queue *q)
 Determine whether the queue can support more tasks.
int work_queue_empty (struct work_queue *q)
 Determine whether there are any known tasks queued, running, or waiting to be collected.
int work_queue_shut_down_workers (struct work_queue *q, int n)
 Shut down workers connected to the work_queue system.


Function Documentation

struct work_queue* work_queue_create int  port,
time_t  stoptime
 

Create a new work queue.

Parameters:
port The port number to listen on, or zero to choose a default.
Returns:
A new work queue, or zero if it could not be created.

void work_queue_delete struct work_queue *  q  ) 
 

Delete a work queue.

Parameters:
q The work queue to delete.

void work_queue_get_stats struct work_queue *  q,
struct work_queue_stats s
 

Get queue statistics.

Parameters:
q The queue to query.
s A pointer to a buffer that will be filed with statistics.

struct work_queue_task* work_queue_wait struct work_queue *  q,
time_t  timeout
 

Wait for tasks to complete.

This call will block until the timeout has elapsed.

Parameters:
q The work queue to wait on.
timeout The length of time to wait for a completed task before returning. Use an integer time to set the timeout or the constant WAITFORTASK to block until a task has completed.
Returns:
A completed task description, or null if the queue is empty or the timeout was reached without a completed task.

void work_queue_submit struct work_queue *  q,
struct work_queue_task t
 

Submit a job to a work queue.

Parameters:
q A work queue returned from work_queue_create.
t A task description returned from work_queue_task_create.

void work_queue_task_delete struct work_queue_task t  ) 
 

Delete a task specification.

This may be called on tasks after they are returned from work_queue_wait.

Parameters:
t The task specification to delete.

struct work_queue_task* work_queue_task_create const char *  program,
const char *  args
 

Create a new task specification.

Once created, the task may be passed to work_queue_submit.

Parameters:
program The command to be executed.
args The arguments to the command.

INT64_T work_queue_task_add_tag struct work_queue_task t,
const char *  tag
 

Further define a task specification.

Once completed, the task may be passed to work_queue_submit.

Parameters:
t The task to which to add parameters
tag The tag to attatch to tast t.

INT64_T work_queue_task_add_standard_input_buf struct work_queue_task t,
const char *  buf,
int  length
 

Further define a task specification.

Once completed, the task may be passed to work_queue_submit. If no buffer or file is defined, the program will have default stdin.

Parameters:
t The task to which to add parameters
buf The data buffer to send to the program on standard input.

INT64_T work_queue_task_add_standard_input_file struct work_queue_task t,
const char *  fname
 

Further define a task specification.

Once completed, the task may be passed to work_queue_submit. If no buffer or file is defined, the program will have default stdin.

Parameters:
t The task to which to add parameters
buf The name of the data file to send to the program on standard input.

INT64_T work_queue_task_add_extra_created_file struct work_queue_task t,
const char *  rname,
const char *  fname
 

Further define a task specification.

Once completed, the task may be passed to work_queue_submit. If no file is defined, the program will have default (no) output files retrieved.

Parameters:
t The task to which to add parameters
rname The name of a file created by the program when it runs.
fname The name of the file local target for copying rname back.

INT64_T work_queue_task_add_extra_staged_buf struct work_queue_task t,
const char *  buf,
int  length,
const char *  rname
 

Further define a task specification.

Once completed, the task may be passed to work_queue_submit.

Parameters:
t The task to which to add parameters
buf The data buffer to send to the worker to be available to the program as a file.
rname The name of the file on the remote worker.

INT64_T work_queue_task_add_extra_staged_file struct work_queue_task t,
const char *  fname,
const char *  rname
 

Further define a task specification.

Once completed, the task may be passed to work_queue_submit.

Parameters:
t The task to which to add parameters
buf The name of the data file to send to the worker to be available to the program.
rname The name of the file on the remote worker.

int work_queue_hungry struct work_queue *  q  ) 
 

Determine whether the queue can support more tasks.

Returns the number of additional tasks it can support if "hungry" and 0 if "sated".

Parameters:
q A pointer to the queue to query.

int work_queue_empty struct work_queue *  q  ) 
 

Determine whether there are any known tasks queued, running, or waiting to be collected.

Returns 1 if so and 0 if "empty".

Parameters:
q A pointer to the queue to query.

int work_queue_shut_down_workers struct work_queue *  q,
int  n
 

Shut down workers connected to the work_queue system.

Gives a best effort and then returns the number of workers successfully shut down.

Parameters:
q A pointer to the queue to query.
n The number to shut down. All workers if given "0".


Generated on Fri Jun 26 09:34:27 2009 for cctools by  doxygen 1.3.9.1