The work queue provides an implementation of the master-worker computing model using TCP sockets, Unix applications, and files as intermediate buffers. A master process uses work_queue_create to create a queue, then work_queue_submit to submit tasks. Once tasks are running, call work_queue_wait to wait for completion. The generic worker
program can be run on any machine, and simply needs to be told the host and port of the master.
#include "timestamp.h"
Go to the source code of this file.
Data Structures | |
struct | work_queue_task |
A task description. More... | |
struct | work_queue_stats |
Statistics describing a work queue. More... | |
Functions | |
work_queue * | work_queue_create (int port, time_t stoptime) |
Create a new work queue. | |
void | work_queue_delete (struct work_queue *q) |
Delete a work queue. | |
void | work_queue_get_stats (struct work_queue *q, struct work_queue_stats *s) |
Get queue statistics. | |
int | work_queue_activate_fast_abort (struct work_queue *q, double multiplier) |
Turn on or off fast abort functionality for a given queue. | |
int | work_queue_specify_scheduler (struct work_queue *q, int alg) |
Change the worker selection algorithm for a given queue. | |
work_queue_task * | work_queue_wait (struct work_queue *q, int timeout) |
Wait for tasks to complete. | |
void | work_queue_submit (struct work_queue *q, struct work_queue_task *t) |
Submit a job to a work queue. | |
void | work_queue_task_delete (struct work_queue_task *t) |
Delete a task specification. | |
work_queue_task * | work_queue_task_create (const char *full_command) |
Create a new task specification. | |
void | work_queue_task_specify_tag (struct work_queue_task *t, const char *tag) |
Attach a user defined logical name to the task. | |
int | work_queue_task_specify_scheduler (struct work_queue_task *t, int alg) |
Further define a task specification. | |
void | work_queue_task_specify_input_buf (struct work_queue_task *t, const char *buf, int length, const char *rname) |
Further define a task specification. | |
void | work_queue_task_specify_input_file (struct work_queue_task *t, const char *fname, const char *rname) |
Further define a task specification. | |
void | work_queue_task_specify_output_file (struct work_queue_task *t, const char *rname, const char *fname) |
Further define a task specification. | |
int | work_queue_hungry (struct work_queue *q) |
Determine whether the queue can support more tasks. | |
int | work_queue_empty (struct work_queue *q) |
Determine whether there are any known tasks queued, running, or waiting to be collected. | |
int | work_queue_shut_down_workers (struct work_queue *q, int n) |
Shut down workers connected to the work_queue system. | |
Variables | |
double | wq_option_fast_abort_multiplier |
Initial setting for fast abort multiplier upon creating queue. | |
int | wq_option_scheduler |
Initial setting for algorithm to assign tasks to workers upon creating queue . |
struct work_queue* work_queue_create | ( | int | port, | |
time_t | stoptime | |||
) |
Create a new work queue.
port | The port number to listen on, or zero to choose a default. The default port is 9123, but can be overridden by the environment variable WORK_QUEUE_PORT. | |
stoptime | The time at which to return null if not yet able to be created. |
void work_queue_delete | ( | struct work_queue * | q | ) |
Delete a work queue.
q | The work queue to delete. |
void work_queue_get_stats | ( | struct work_queue * | q, | |
struct work_queue_stats * | s | |||
) |
Get queue statistics.
q | The queue to query. | |
s | A pointer to a buffer that will be filed with statistics. |
int work_queue_activate_fast_abort | ( | struct work_queue * | q, | |
double | multiplier | |||
) |
Turn on or off fast abort functionality for a given queue.
q | A pointer to the queue to modify. | |
multiplier | The multiplier of the average task time at which point to abort; if negative (and by default) fast_abort is deactivated. |
int work_queue_specify_scheduler | ( | struct work_queue * | q, | |
int | alg | |||
) |
Change the worker selection algorithm for a given queue.
q | A pointer to the queue to modify. | |
alg | The algorithm to use in assigning a task to a worker. Valid possibilities are defined in this file as "WORK_QUEUE_SCHEDULE_X" values. |
struct work_queue_task* work_queue_wait | ( | struct work_queue * | q, | |
int | timeout | |||
) |
Wait for tasks to complete.
This call will block until the timeout has elapsed.
q | The work queue to wait on. | |
timeout | The number of seconds to wait for a completed task before returning. Use an integer time to set the timeout or the constant WAITFORTASK to block until a task has completed. |
void work_queue_submit | ( | struct work_queue * | q, | |
struct work_queue_task * | t | |||
) |
Submit a job to a work queue.
It is safe to re-submit a task returned by work_queue_wait.
q | A work queue returned from work_queue_create. | |
t | A task description returned from work_queue_task_create. |
void work_queue_task_delete | ( | struct work_queue_task * | t | ) |
Delete a task specification.
This may be called on tasks after they are returned from work_queue_wait.
t | The task specification to delete. |
struct work_queue_task* work_queue_task_create | ( | const char * | full_command | ) |
Create a new task specification.
Once created, the task may be passed to work_queue_submit.
full_command | The shell command line to be executed by the task. |
void work_queue_task_specify_tag | ( | struct work_queue_task * | t, | |
const char * | tag | |||
) |
Attach a user defined logical name to the task.
This field is not interpreted by the work queue, but simply maintained to help the user track tasks.
t | The task to which to add parameters | |
tag | The tag to attach to task t. |
int work_queue_task_specify_scheduler | ( | struct work_queue_task * | t, | |
int | alg | |||
) |
Further define a task specification.
Once completed, the task may be passed to work_queue_submit.
t | The task to which to add parameters | |
alg | The algorithm to use in assigning a task to a worker. Valid possibilities are defined in this file as "WORK_QUEUE_SCHEDULE_X" values. |
void work_queue_task_specify_input_buf | ( | struct work_queue_task * | t, | |
const char * | buf, | |||
int | length, | |||
const char * | rname | |||
) |
Further define a task specification.
Once completed, the task may be passed to work_queue_submit.
t | The task to which to add parameters | |
buf | A pointer to the data buffer to send to the worker to be available to the commands. | |
length | The number of bytes of data in the buffer | |
rname | The name of the file in which to store the buffer data on the worker |
void work_queue_task_specify_input_file | ( | struct work_queue_task * | t, | |
const char * | fname, | |||
const char * | rname | |||
) |
Further define a task specification.
Once completed, the task may be passed to work_queue_submit.
t | The task to which to add parameters | |
fname | The name of the data file to send to the worker to be available to the commands. | |
rname | The name of the file in which to store the buffer data on the worker |
void work_queue_task_specify_output_file | ( | struct work_queue_task * | t, | |
const char * | rname, | |||
const char * | fname | |||
) |
Further define a task specification.
Once completed, the task may be passed to work_queue_submit. If no file is defined, the program will have default (no) output files retrieved.
t | The task to which to add parameters | |
rname | The name of a file created by the program when it runs. | |
fname | The name of the file local target for copying rname back. |
int work_queue_hungry | ( | struct work_queue * | q | ) |
Determine whether the queue can support more tasks.
Returns the number of additional tasks it can support if "hungry" and 0 if "sated".
q | A pointer to the queue to query. |
int work_queue_empty | ( | struct work_queue * | q | ) |
Determine whether there are any known tasks queued, running, or waiting to be collected.
Returns 0 if there are tasks remaining in the system, 1 if the system is "empty".
q | A pointer to the queue to query. |
int work_queue_shut_down_workers | ( | struct work_queue * | q, | |
int | n | |||
) |
Shut down workers connected to the work_queue system.
Gives a best effort and then returns the number of workers given the shut down order.
q | A pointer to the queue to query. | |
n | The number to shut down. All workers if given "0". |
Initial setting for fast abort multiplier upon creating queue.
Turned off if less than 0. Change prior to calling work_queue_create, after queue is created this variable is not considered and changes must be made through the API calls.
Initial setting for algorithm to assign tasks to workers upon creating queue .
Change prior to calling work_queue_create, after queue is created this variable is not considered and changes must be made through the API calls.