CCL Home

Research

Software Community Operations

Work Queue: A Scalable Master/Worker Framework

Work Queue is a framework for building large master-worker applications that span many computers including clusters, clouds, and grids. Work Queue applications are written in C, Perl, or Python using a simple API that allows users to define tasks, submit them to the queue, and wait for completion. Tasks are executed by a standard worker process that can run on any available machine. Each worker calls home to the master process, arranges for data transfer, and executes the tasks. The system handles a wide variety of failures, allowing for dynamically scalable and robust applications.

Work Queue has been used to write many applications that scale up to hundreds or thousands of machines. Examples include ForceBalance, Accelerated Weighted Ensemble, the SAND genome assembler, the Makeflow workflow engine, and the All-Pairs and Wavefront abstractions. The framework is easy to use, and is currently use to teach parallel and distributed programming techniques in undergraduate classes at the University of Notre Dame.

For More Information

  • Work Queue User's Manual
  • Work Queue API (C | Perl | Python)
  • Work Queue Example Program (C | Perl | Python)
  • Download Work Queue
  • Getting Help with Work Queue
  • Publications

    (Showing papers with tag workqueue. See all papers instead.)