Installing the Cooperative Computing Tools

November 2009

The Cooperative Computing Tools are Copyright (C) 2003-2004 Douglas Thain and Copyright (C) 2005- The University of Notre Dame. This software is distributed under the GNU General Public License See the file COPYING for details.

Overview

The Cooperative Computing Tools (cctools) are a collection of programs designed to assist users with the difficulties of building and managing complex, fault-prone distributed systems. The components of the cctools are:
  • Parrot - A personal virtual file system.
  • Chirp - A distributed file and storage system.
  • Makeflow - A workflow engine similar to Make.
  • Work Queue - A flexible master-worker library.
  • All-Pairs - A computational abstraction.
  • Wavefront - A computational abstraction.
  • Watchdog - A reliable process manager.
  • ftsh - A fault tolerant shell language.
  • FTP-Lite - A light weight FTP client library.
    • This short manual describes how to install the cctools. For more information about the tools themselves, see the links above.

    Installing

    The cctools may already be installed on your system. Simply run chirp -v. If you see something like 
    chirp version 1.0.1 built by dthain@hedwig on Sep 12 2004 at 20:04:51
    then the cctools are installed, and you are ready to go.

    Otherwise, you must install the cctools yourself.

    Installing at Notre Dame

    If you are on the Notre Dame campus, you may find a recent version of the cctools in AFS. Simply set up your path as follows: 
    % setenv PATH /afs/nd.edu/user37/ccl/software/cctools/bin:$PATH

    Installation From a Binary Package

    From any other location, we recommend that you install a binary package from the cctools web page. Simply unpack the tarball in any directory that you like, and then add the bin directory to your path. For example, to install in $HOME/cctools: 
    % cd $HOME
% gunzip cctools-xxx-yyy.tar.gz
% tar xvf cctools-xxx-yyy.tar
% setenv PATH $HOME/cctools/bin:$PATH

    Installation From a Source Package

    Building the CCTools from source is very easy. Just download a source package:
    1. CCTools Download (Required)
    And follow this recipe: 
    % gunzip cctools-xxx-src.tar.gz
% tar xvf cctools-xxx-src.tar
% cd cctools-xxx-src
% ./configure --prefix ${HOME}/cctools
% make
% make install
% setenv PATH ${HOME}/cctools/bin:$PATH
    The software will happily build and run without installing any external packages. Optionally, the cctools will interoperate with a variety of external packages for security and data access. To use these, you must download and install them separately:
    1. Kerberos V (Optional)
    2. Globus (Optional) (Tested with Globus 2.2, 2.4, and 4.0.)
    3. EGEE GLite (Optional)
    4. Hadoop (Optional)
    5. iRODS (Optional)
    Once the desired packages are correctly installed, unpack the cctools and then issue a configure command that points to all of the other installations. Then, make and install. For example: 
    % gunzip cctools-xxx.tar.gz
% tar xvf cctools-xxx.tar
% cd cctools-xxx
% ./configure --prefix $HOME/cctools --with-krb5-path /usr/kerberos --with-globus-path /usr/local/globus ...
% make
% make install
% setenv PATH ${HOME}/cctools/bin:$PATH