CCL Home

Research

Software Community Operations

CCL Software

Makeflow

Makeflow is a workflow system for parallel and distributed computing that uses a language very similar to Make. Using Makeflow, you can write simple scripts that easily execute on hundreds or thousands of machines.

Work Queue

Work Queue is a system and library for creating and managing scalable master-worker style programs that scale up to thousands machines on clusters, clouds, and grids. Work Queue programs are easy to write in C, Python or Perl.

Parrot

Parrot is a transparent user-level virtual filesystem that allows any ordinary program to be attached to many different remote storage systems, including HDFS, iRODS, Chirp, and FTP.

Chirp

Chirp is a personal user-level distributed filesystem that allows unprivileged users to share space securely, efficiently, and conveniently. When combined with Parrot, Chirp allows users to create custom wide-area distributed filesystems.

Resource Monitor

Monitors the cpu, memory, io, and disk usage of applications running in distributed systems, and can optionally enforce limits on each resource. A visualizer allows you to explore the consumption of workloads consisting of thousands of tasks.

Accelerated Weighted Ensemble

Accelerated Weighted Ensemble (AWE) is a system for large scale adaptive sampling molecular dynamics. The framework decomposes the resampling computations and the molecular dynamics simulations into tasks that are dispatched to Work Queue for distribution and execution across thousands of CPUs and GPUs.

SAND

The Scalable Assembler at Notre Dame (SAND) is a set of modules that augment the Celera genome assembler to scale up to thousands of nodes using the Work Queue system.

All-Pairs

All-Pairs is a specialized framework for running large number of pairwise comparisons by harnessing hundreds to thousands of machines. All-Pairs problems are common in fields such as data mining, bioinformatics, and biometrics.

Wavefront

Wavefront is a specialized framework for running very large dynamic programming problems by harnessing hundreds to thousands of machines. Wavefront problems are common in game theory, economics, and bioinformatics.

And More

Older software not under active development:

  • FTSH - The Fault Tolerant Shell
  • AllocFS - A kernel-level filesystem with hierarchical space allocation.
  • SubID - A user-level implementation of hierarchical user identities.