allpairs_master(1)

NAME

allpairs_master - executes All-Pairs workflow in parallel on distributed systems

SYNOPSIS

allparis_master [options] <set A> <set B> <compare function>

DESCRIPTION

allpairs_master computes the Cartesian product of two sets (<set A> and <set B>), generating a matrix where each cell M[i,j] contains the output of the function F (<compare function>) on objects A[i] (an item in <set A>) and B[j] (an item in <set B>). The resulting matrix is displayed on the standard output, one comparison result per line along with the associated X and Y indices.

allpairs_master uses the Work Queue system to distribute tasks among processors. Each processor utilizes the allpairs_multicore(1) program to execute the tasks in parallel if multiple cores are present. After starting allpairs_master, you must start a number of work_queue_worker(1) processes on remote machines. The workers will then connect back to the master process and begin executing tasks.

OPTIONS

-p <port>
The port that the master will be listening on.
-e <args>
Extra arguments to pass to the comparison function.
-f <file>
Extra input file needed by the comparison function. (may be given multiple times)
-t <seconds>
Estimated time to run one comparison. (default chosen at runtime)
-x <item>
Width of one work unit, in items to compare. (default chosen at runtime)
-y <items>
Height of one work unit, in items to compare. (default chosen at runtime)
-N <project>
Report the master information to a catalog server with the project name - <project>
-E <priority>
Priority. Higher the value, higher the priority.
-d <flag>
Enable debugging for this subsystem. (Try -d all to start.)
-v Show program version.
-h Display this message.

EXIT STATUS

On success, returns zero. On failure, returns non-zero.

EXAMPLES

Let's suppose you have a whole lot of files that you want to compare all to each other, named a, b, c, and so on. Suppose that you also have a program named compareit that when invoked as compareit a b will compare files a and b and produce some output summarizing the difference between the two, like this:
 a b are 45 percent similar
To use the allpairs framework, create a file called set.list that lists each of your files, one per line:
 a
 b
 c
 ...
Because allpairs_master utilizes allpairs_multicore(1), so please make sure allpairs_multicore(1) is in your PATH before you proceed.To run a All-Pairs workflow sequentially, start a single work_queue_worker(1) process in the background. Then, invoke allpairs_master.
 % work_queue_worker localhost 9123 &
 % allpairs_master set.list set.list compareit
The framework will carry out all possible comparisons of the objects, and print the results one by one (note that the first two columns are X and Y indices in the resulting matrix):
 1	1	a a are 100 percent similar
 1	2	a b are 45 percent similar
 1	3	a c are 37 percent similar
 ...
To speed up the process, run more work_queue_worker(1) processes on other machines, or use condor_submit_workers(1) or sge_submit_workers(1) to start hundreds of workers in your local batch system.

The following is an example of adding more workers to execute a All-Pairs workflow. Suppose your allpairs_master is running on a machine named barney.nd.edu. If you have access to login to other machines, you could simply start worker processes on each one, like this:

 % work_queue_worker barney.nd.edu 9123
If you have access to a batch system like Condor, you can submit multiple workers at once:
 % condor_submit_workers barney.nd.edu 9123 10
 Submitting job(s)..........
 Logging submit event(s)..........
 10 job(s) submitted to cluster 298.

COPYRIGHT

The Cooperative Computing Tools are Copyright (C) 2003-2004 Douglas Thain and Copyright (C) 2005-2011 The University of Notre Dame. This software is distributed under the GNU General Public License. See the file COPYING for details.

SEE ALSO

  • The Cooperative Computing Tools
  • All-Pairs User Manual
  • Work Queue User Manual
  • work_queue_worker(1)
  • condor_submit_workers(1)
  • sge_submit_workers(1)
  • allpairs_multicore(1)

  • CCTools 3.4.1 released on 11/11/2011