Allpairs User's Manual

Last Updated January 2010

Allpairs is Copyright (C) 2009 The University of Notre Dame. This software is distributed under a BSD-style license. See the file COPYING for details.

Overview

All-Pairs( array A[i], array B[j], function F(x,y) )
returns matrix M where
M[i,j] = F( A[i], B[j] )

The All-Pairs abstraction computes the Cartesian product of two sets, generating a matrix where each cell M[i,j] contains the output of the function F on objects A[i] and B[j]. You provide two sets of data files (as in the above figure, one set is setA = {A0, A1, A2, A3} and the other set is setB = {B0, B1, B2, B3}) and a function F that computes on them (later in the text we refer to this fuction F as either compare program or compare function. You may optionally provide additional parameters to control the actual computation, such as computing only part of the matrix, using a specified number of CPU cores. The abstraction then runs each of the functions in parallel, automatically handling load balancing, data movement, fault tolerance and so on for you.

The First Example

Before running All-Pairs, you should have:

  1. Cooperative Computing Tools installed - You can find the installation guide here
  2. Two data sets - Each data set is a directory containing all the data files in the set. In the data set directory, there should also be a file named set.list. The set.list file contains the paths (absolute paths or relative paths that are relative to the data set directory) to the corresponding data files one at a line.
  3. A Compare Program/Function - The compare program should accept two files and return a double type value that would be placed in the matrix cell. A compare function would make the computation faster compared to the compare program. The details of how to use compare function in allpairs can be found here
Now let's take a look at a real example!

We have two directories named setA and setB.

We also have a compare program func.exe which accepts two files from command line arguments and return a double type value based on certain computations on those two files.

Our goal is to generate a matrix from setA and setB using func.exe as described in the following graph:

To achieve this using allpairs abstraction, we first have to add the set.list files to the setA and setB directories.

The contents of the set.list file in setA are: 

A1
A2
...
A5
As can be seen, each line of the set.list file is a relative path to a file that would be used in the matrix generation. The order of the paths in set.list decides the indices of the files during matrix creation. In this case, A1's index is 0 and A5's index is 4. Two indices together decide the coordinate of the value computed from the corresponding two files. In the above graph, the numbers in the yellow circles are the coordinates in the output matrix.

Similarly, the contents of the set.list file in setB are: 

B1
B2
...
B5
Now, we are ready to invoke the allpairs_multicore program to perform the matrix generation task. To do this is as simple as typing the following command at the command line: 
allpairs_multicore setA setB func.exe
In general, this is how you invoke allpairs_multicore: 
allpairs_multicore [options] <set A> <set B> <compare function>
<set A> is the name of one set(a directory that has a set.list file), and <set B> is the name of the other set. In this example, they are setA and setB that we prepared in a earlier step. <compare function> is the function we wrote for each computation of a cell value. In this case it is func.exe

The output of the execution looks like this: 

1	1	234
2	3  	345
0	0 	32
...
...
4	4	987
...
3	2	184
The output is what eventually matters. So let's spend a little time to understand what does the bunch of numbers mean. The format of the output is: 
x-axis	y-axis	value
The x-axis and y-axis are the coordinates in the matrix.
The value is the value of the corresponding cell in the matrix.

Note that the program func.exe may be written in any language that you prefer. It's not restricted to just normal programming language, it can even be a script or anything as long as it can be invoked like this: 

./func.exe fileA fileB
And of course, returns a double type value to the standard output.

The Fine Details

Compute only a submatrix

The first example has no options, which by default computes the matrix using all the files listed in the two set.list files. Sometimes the matrix is very big or some parts of the matrix is already computed for some reason. Then you might just want to compute part of the matrix. You can do this by providing a few additional parameters: 
allpairs_multicore -i 2 -j 2 -k 4 -l 3 setA setB func.exe
The meaning of the options are: In this example, the start point of computation is [2, 2] and the end point is [4, 3]. The output is thus just restricted to the cells in the submatrix encompassed by the red dotted line in the graph below:

The whole matrix is a submatrix of another

The matrix created from setA and setB may be a submatrix of an another larger matrix. In this case, you can provide more parameters to inform the allpairs_multicore program where the matrix would be in the larger matrix: 
allpairs_multicore -X 10 -Y 5 setA setB func.exe
The meaning of the options are: In this example, all the files listed in the two set.list files will be used to generate a matrix, and the output coordinates would reflect the new positions as if they were in a larger matrix.

Can I do "a submatrix in a larger matrix context"?

Of course, you can combine the functionalities of the above two examples: 
allpairs_multicore -i 2 -j 1 -k 3 -l 3 -X 78 - Y 534 setA setB func.exe
This will compute the following:

For more options on allpairs_multicore, please type the following command and refer to the help message: 
allpairs_multicore -h

Work with a Compare Function to boost performance

The main difference between a compare function and a compare program in allpairs is: a compare function is an internal function in the allpairs_multicore source code written in C language, and a compare program is an external standalone program that can be written in any language.

According to our reseach on allpair multicore abstraction, invoking a compare function is more than two orders of magnitude faster than invoking a compare program. This is especially important when a compare program could be finished in a very short time, say less than 1 or 2 seconds. If this is the case, the invocatio time of the compare program would take up a large portion of the total execution time. The situation could be even worse. To invoke an external program, allpairs_multicore will also invoke the user's shell along with their complex startup scripts, which can have unbounded execution time. Consider you may have to run the compare function thousands of times when the target matrix is large, then the program invocation overhead could hardly be neglected.

Our solution to this problem is to allow the user to be able to plug a snippet of C code into the allpairs_multicore source code. To take advantage of this feature, you have to rebuild the allpairs_multicore program from source code. You can download the source code here.

There 3 steps to plug in a customized compare function:

1. Write the function and place it in allpairs_multicore.c

Your compare function should look like this: 

double function_name(const void *mmap1, size_t size1, const void *mmap2, size_t size2) {
	The C code that implements the "comparison" algorithm...
}
mmap1 and mmap2 are pointers that returned by invoking POSIX-compliant Unix system call mmap on two files. size1 and size2 are sizes of the corresponding mapped memory areas. Your algorithm of "comparing" the two files should work on those two mapped memory areas and return a double type value as a result.
To plug the function code into the allpairs_multicore source code, first search for the string "To developers: ADD YOUR CUSTOMIZED COMPARE FUNCTION" and then copy the code to the area right below the "To developers" comment. That's it! Now , you are ready to move on to the next step.

2. Register your compare function in allpairs_multicore.c

To register an inner compare function, first search for the string "To developers: REGISTER YOUR COMPARE FUNCTION" in allpairs_multicore.c. Then add code in the following format after the comment: 

register_compare_function("YOUR_COMPARE_FUNCTION'S_NAME", YOUR_COMPARE_FUNCTION);
Here YOUR_COMPARE_FUNCTION is the compare function you just added in the first step.

Let's look at an example. Suppose the following function is the one you just added in the first step: 

double my_function(const void *mmap1, size_t size1, const void *mmap2, size_t size2) {
	Some C code ...
}
Then the line you should add below the comment would be: 
register_compare_function("my_function", my_function);
3. Rebuild Type make under the directory where allpairs_multicore.c resides. No compile errors? Then HOORAY, you get a new allpairs_multicore with your own compare function embedded in.

Now you are ready to use the new allpairs_multicore with all the options remaining the same. To use the added compare function, just replace the required argument - 'function name' with the newly added one. And, enjoy the boost.

All-Pairs Work Queue

So far, we have introduced how to use All-Pairs abstraction on a single machine. But sometimes the All-Pairs problem is too big to allow a single machine to finish it in a reasonable amount of time, even if the single machine is multicore. So, we have built a Work Queue version of the All-Pairs abstraction which allows the users to easily apply the All-Pairs abstraction on clusters, grids or clouds.

To use the All-Pairs Work Queue version, you will need to start a All-Pairs master program called allpairs_master and a bunch of workers. The workers will perform the tasks distributed by the master and return the results to the master. The individual tasks that the master program distributes are sub-matrix computation tasks and all the tasks would be performed by the allpairs_multicore program on the workers. For end users, the only extra step involved here is starting the workers. Starting the All-Pairs master program is almost identical to starting the All-Pairs multicore program except the program name and a few master program exclusive options.

Starting the Master Program

Before you start the master program, the preparation same as in the All-Pairs multicore needs to be done. Basically you will need to have CCTools installed, have two data sets in the proper form and have a compare function/program. Say you have already finished the preparation as in the first example. Now, for example, you can start the All-Pairs master program like this: 
 
allpairs_master -c 4 setA setB func.exe
As you might have noticed, allpairs program's name is changed from allpairs_multicore to allpairs_master. Also, there is an extra option '-c'. The '-c' option indicates the split factor. The number of subtasks that can be executed by workers simultaneously is 2 to the power of 'split factor'. In the above example, the number of subtasks would be 16. With that being said, the number of subtasks you assigned through '-c' option had better be greater than the number of workers you would start. Otherwise some workers wouldn't get any tasks to do. A good news is that all the allpairs_multicore's options remain the same in the allpairs_master. So, all the options you have learned in the Fine Details section are still usable when you use the allpairs_master program.

Starting the Workers

To begin, let's assume that you are logged into a machine named barney.nd.edu. In order to procure workers, you can use your batch system (such as those running running SGE, or Condor), or you can execute the workers yourself. In order to make this a little easier, we have written some tools, provided in the CCTools, that submit workers to each of these two common batch systems.
This is an example of submitting 10 worker processes to Condor: 

% condor_submit_workers barney.nd.edu 9123 10
Submitting job(s)..........
Logging submit event(s)..........
10 job(s) submitted to cluster 298.
Or, submitting 10 worker processes to SGE: 
% sge_submit_workers barney.nd.edu 9123 10
In the above two examples, the first argument is the port number that the master process will be or is listening on and the second the argument is the number of workers to start. Note that 9123 is the default port number that the master process uses. If you use the '-p' option in the allpairs_master to change the listening port, you will need to modify the port argument in the starting worker command accordingly. Or, you can start workers manually on any other machine you can log into, using the worker executable built in the CCTools: 
% worker barney.nd.edu 9123
Once the workers begin running, the allpairs_master can dispatch tasks to each one very quickly. If a worker should fail, Work Queue will retry the work elsewhere, so it is safe to submit many workers to an unreliable system.

When the All-Pairs master process completes, your workers will still be available, so you can either run another master with the same workers, remove them from the batch system, or wait for them to expire. If you do nothing for 15 minutes, they will automatically exit by default. Of course, you can change this worker expiration time by setting the '-t' option.

Note that condor_submit_workers and sge_submit_workers are simple shells scripts, so you can edit them directly if you would like to change batch options or other details.

The Result

Now the workers are executing the task for you. All you need to do is waiting for the result to show up. The result will come out in the same form as in allpairs_multicore.

For more options on allpairs_master, please type the following command and refer to the help message: 

allpairs_master -h

For More Information

For the latest information about Allpairs, please visit our web site and subscribe to our mailing list.