Chirp is Copyright (C) 2003-2004 Douglas Thain and Copyright (C) 2005- The University of Notre Dame. All rights reserved. This software is distributed under the GNU General Public License. See the file COPYING for details.Please use the following citation for Chirp in a scientific publication::
Chirp is like a distributed filesystem (such as NFS) except that it can be run over wide area networks and requires no special privileges on either the client or the server end. Chirp allows the end user to set up fine-grained access control so that data can be shared (or not shared) with the right people.
Chirp is also like a file transfer system (such as FTP) that provides streaming point-to-point data transfer over the Internet. However, Chirp also provides fine-grained Unix-like data access suitable for direct access by ordinary programs.
Chirp also includes advanced features for space allocation and active storage. However, each of these features must be explicitly enabled, so you don't have to worry about them if all you want is simple storage access. Read on below for more details.
Begin by installing the cctools on your system. When you are ready, proceed below.
/tmp/mydata
% chirp_server -r /tmp/mydata &
% chirp localhost chirp:localhost:/> setacl . hostname:*.mydomain.edu write
% parrot tcsh % cd /chirp % ls angband.somewhere.edu:9094 dustpuppy.somewhere.edu:9094 peanuts.somewhere.edu:9094 ... % cd /chirp/peanuts.somewhere.edu % cp /tmp/bigfile . % ls -la total 804 drwx------ 2 fred users 4096 Sep 10 12:40 . drwx------ 2 fred users 4096 Sep 10 12:40 .. -rw-r--r-- 1 fred users 104857600 Sep 10 12:57 bigfile -rw-r--r-- 1 fred users 147 Sep 10 12:39 hosts % parrot_getacl unix:fred rwlda hostname:hedwig rl ... |
(If you are having difficulting accessing your server, have a look at "debugging hints" below.)
Parrot is certainly the most convenient way to access storage, but it has some limitations: it only works on Linux, and it imposes some performance penalty.
You can also attach to Chirp filesystems by using the FUSE package to attach Chirp as a kernel filesystem module. Unlike Parrot, this requires superuser privileges to install the FUSE package, but will likely work more reliably on a larger number of programs. You can do this with either Linux FUSE or MacFuse. Once you have downloaded and installed FUSE, simply run chirp_fuse with the name of a directory on which the filesystem should be mounted. For example:
% mkdir /tmp/chirp % chirp_fuse /tmp/chirp % cd /tmp/chirp % ls -la total 9742 dr-xr-xr-x 0 fred users 6697 Feb 22 13:54 albus.cse.nd.edu:9094 dr-xr-xr-x 0 fred users 6780 Feb 22 13:54 aluminum.helios.nd.edu:9094 dr-xr-xr-x 0 fred users 27956 Feb 22 13:54 angband.cse.nd.edu:9094 dr-xr-xr-x 0 fred users 6466 Feb 22 13:54 angelico.helios.nd.edu:9094 ... |
% chirp ... chirp::> open myhost.somewhere.edu chirp:myhost.somewhere.edu:/> put /tmp/bigfile file /tmp/bigfile -> /bigfile (11.01 MB/s) chirp:myhost.somewhere.edu:/> ls -la dir 4096 . Fri Sep 10 12:40:27 2004 dir 4096 .. Fri Sep 10 12:40:27 2004 file 147 hosts Fri Sep 10 12:39:54 2004 file 104857600 bigfile Fri Sep 10 12:53:21 2004 chirp:myhost.somewhere.edu:/> |
In scripts, you may find it easier to use the standalone commands chirp_get and chirp_put, which move single files to and from a Chirp server. These commands also allow for streaming data, which can be helpful in a shell pipeline. Also, the -f option to both commands allows you to follow a file, much like the Unix tail command:
% tar cvzf archive.tar.gz ~/mydata % chirp_put archive.tar.gz myhost.somewhere.edu archive.tar.gz % ... % chirp_get myhost.somewhere.edu archive.tar.gz - | tar xvzf % ... % chirp_get -f myhost.somewhere.edu logfile - |& less % |
You can also write programs that access the Chirp C interface directly. This interface is relatively self explanatory: programs written to use this library may perform explicit I/O operations in a manner very similar to Unix. For more information, see the HOWTO: Write Code that Uses Chirp
The storage catalog is highly dynamic. By default, each Chirp server makes itself known to the storage catalog every five minutes. The catalog server records and reports all Chirp servers that it knows about, but will discard servers that have not reported for fifteen minutes.
If you do not want your servers to report to a catalog, then run them with this option:
% chirp_server -u -
Alternatively, you may establish your own catalog server. This can be useful for keeping your systems logically distinct from the main storage pool, but can also help performance and availability if your catalog is close to your Chirp servers. The catalog server is installed in the same place as the Chirp server. Simply run it on any machine that you like and then direct your Chirp servers to update the new catalog with the -u option. The catalog will be published via HTTP on port 9097 of the catalog machine.
For example, suppose that you wish to run a catalog server on a machine named dopey and a Chirp server on a machine named sneezy:
dopey% catalog_server ... sneezy% chirp_server -u dopey [more options]Finally, point your web browser to:
http://dopey:9097Or, set an environment variable and use Parrot:
% setenv CATALOG_HOST dopey % parrot tcsh % ls /chirpAnd you will see something like this. You may easily run multiple catalogs for either scalability or fault tolerance. Simply give each Chirp server multiple -u options to name each running catalog.
(Hint: If you want to ensure that your chirp and catalog servers run continuously and are automatically restarted after an upgrade, consider using Watchdog.)
Security really has two aspects: authentication and authorization. Authentication deals with the question "Who are you?" Once your identity has been established, then authorization deals with the question "What are you allowed to do?" Let's deal with each in turn.
| Type | Summary | Regular User? | Root? |
| (non-root) | (root) | ||
| kerberos | Centralized private key system | no | yes (host cert) |
| globus | Distributed public key system | yes (user cert) | yes (host cert) |
| unix | Authenticate with local unix user ids. | yes | yes |
| hostname | Reverse DNS lookup | yes | yes |
| address | Identify by IP address | yes | yes |
The Chirp tools will attempt all of the authentication types that are known and available in the order above until one works. For example, if you have Kerberos installed in your system, Chirp will try that first. If not, Chirp attempts the others.
Once an authentication scheme has succeeded, Chirp assigns the incoming user a subject that describes both the authentication method and the user name within that method. For example, a user that authenticates via Kerberos might have the subject:
kerberos:dthain@nd.edu
A user authenticating with Globus credentials might be:
globus:/O=Cooperative_Computing_Lab/CN=Douglas_L_Thain
While another user authenticating by local unix ids might be:
unix:dthain
While a user authenticating by simple hostnames might be:
hostname:pigwidgeon.cse.nd.edu
Take note that Chirp considers all of the subjects as different
identities, although some of them might correspond to the same
person in varying circumstances.
Every directory in a Chirp server has an ACL, much like filesystems such as as AFS or NTFS. To see the ACL for a directory, use the Chirp tool and the getacl command:
chirp:host.mydomain.edu:/> getacl unix:dthain rwlda hostname:*.mydomain.edu rwlOr, if you are using Parrot, you can use parrot_getacl to examine ACLs in the same way:
% parrot tcsh % cd /chirp/host.mydomain.edu % parrot_getacl unix:dthain rwlda hostname:*.mydomain.edu rwlThis ACL indicates that the subject unix:dthain has five access rights, while the subject pattern hostname:*.mydomain.edu has only three access rights. The access rights are as follows:
| r | - The subject may read items in the directory. |
| w | - The subject may write items in the directory. |
| l | - The subject may list the directory contents. |
| d | - The subject may delete items in the directory. |
| p | - The subject may put new files into the directory. |
| a | - The subject may administer the directory, including changing the ACL. |
| x | - The subject may execute programs in the directory. |
| v | - The subject may reserve a directory. |
Access rights often come in combinations, so there are a few aliases for your convenience:
| read | - alias for rl |
| write | - alias for rwld |
| admin | - alias for rwlda |
| none | - delete the entry |
chirp:host.mydomain.edu:/> setacl / kerberos:dthain@nd.edu write chirp:host.mydomain.edu:/> getacl unix:dthain rwlda hostname:*.mydomain.edu rwl kerberos:dthain@nd.edu rwldNote that for subject names that contain spaces, you should simply substitute underscores. For example, if your subject name is /O=Univ of Somewhere/CN=Fred Flint, then you might issue a setacl command like this:
chirp:host.mydomain.edu:/> setacl / /O=Univ_of_Somewhere/CN=Fred_Flint rwldaOr, you can accomplish the same thing using parrot_setacl inside of Parrot:
% parrot tcsh % cd /chirp/host.mydomain.edu % parrot_setacl . /O=Univ_of_Somewhere/CN=Fred_Flint rwlda
The meaning of ACLs is fairly obvious, but there are few subtleties you should know:
Rights are generally inherited. When a new directory is created, it automatically gets the ACL of its parent. Exception: read about the reserve right below.
Rights are generally not hierarchical. In order to access a directory, you only need the appropriate permissions on that directory. For example, if you have permission to write to /data/x/y/z, you do not need any other permissions on /data, /data/x and so forth. Of course, it may be difficult to discover a deep directory without rights on the parents, but you can still access it.
The delete right is absolute. If you have permission to delete a directory, then you are able to delete the entire subtree that it contains, regardless of any other ACLs underneath.
Note that it is possible to use Chirp to export an existing directory tree without manually populating every directory with ACLs. Simply create an ACL in an external file, and then use the -A option to tell the Chirp server to use that file as the default ACL.
A shared-storage environment such as Chirp aims to allow many people to read and write common storage space. Of course, with many people reading and writing, we need some mechanism to make sure that everybody does not step on each other's toes.
The reserve right allows a user to create what is essentially a fresh workspace for their own use. When a user creates a new directory and has the v right (but not the w right), Chirp will create a new directory with a fresh ACL that gives the creating user restricted rights.
A good way to use the reserve right is with a wildcard at the top directory. Here's an example. Suppose that Fred creates a new Chirp server on the host bigwig. Initially, no-one except Fred can access the server. The first time it starts, the Chirp server initializes its root directory with the following ACL:
unix:fred rwlaNow, Fred wants other users in his organization to be able to use this storage, but doesn't want them messing up his existing data. So, Fred uses the Chirp tool to give the list (l) and reserve (v) rights to anyone calling from any machine in his organization:
chirp:bigwig:> setacl / hostname:*.somewhere.edu lv(rwlda) chirp:bigwig:> getacl / unix:fred rwlda hostname:*.somewhere.edu lv(rwlda)Now, any user calling from anywhere in somewhere.edu can access this server. But, all that any user can do is issue ls or mkdir in the root directory. For example, suppose that Betty logs into this server from ws1.somewhere.edu. She can not modify the root directory, but she can create her own directory:
chirp:bigwig:> mkdir /mydataAnd, in the new directory, ws1.somewhere.edu can do anything, including edit the access control. Here is the new ACL for /mydata:
chirp:bigwig:> getacl /mydata hostname:ws1.somewhere.edu rwldaIf Betty wants to authenticate with Globus credentials from here on, she can change the access control as follows:
chirp:bigwig:> setacl /mydata globus:/O=Univ_of_Somewhere/CN=Betty rwlaAnd, the new acl will look as follows:
chirp:bigwig:> getacl /mydata hostname:ws1.somewhere.edu rwlda globus:/O=Univ_of_Somewhere/CN=Betty rwla
Kerberos: The server will attempt to use the Kerberos identity of the host it is run on. (i.e. host/coral.cs.wisc.edu@CS.WISC.EDU) Thus, it must be run as the superuser in order to access its certificates. Once authentication is complete, there is no need for the server to keep its root access, so it will change to any unprivileged user that you like. Use the -i option to select the userid.
Globus: The server and client will attempt to perform client authentication using the Grid Security Infrastructure (GSI)> Both sides will load either user or host credentials, depending on what is available. If the server is running as an ordinary user, then you must give a it a proxy certificate with grid-proxy-init. Or, the server can be run as root and will use host certificates in the usual place.
Unix: This method makes use of a challenge-response in the local Unix filesystem to determine the client's Unix identity. It assumes that both machines share the same conception of the user database and have a common directory which they can read and write. By default, the server will pick a filename in /tmp, and challenge the client to create that file. If it can, than the server will examine the owner of the file to determine the client's username. Naturally, /tmp will only be available to clients on the same machine. However, if a shared filesystem directory is available, give that to the chirp server via the -c option. Then, any authorized client of the filesystem can authenticate to the server. For example, at Notre Dame, we use -c /afs/nd.edu/user37/ccl/software/rendezvous to authenticate via our AFS distributed file system.
Hostname: The server will rely on a reverse DNS lookup to establish the fully-qualified hostname of the calling client. The second field gives the hostname to be accepted. It may contain an asterisk as a wildcard. The third field is ignored. The fourth field is then used to select an appropriate local username.
Address: Like "hostname" authentication, except the server simply looks at the client's IP address.
By default, Chirp and/or Parrot will attempt every authentication type knows until one succeeds. If you wish to restrict or re-order the authentication types used, give one or more -a options to the client, naming the authentication types to be used, in order. For example, to attempt only hostname and kerberos authentication, in that order:
% chirp -a hostname -a kerberos
First, a Java visual display applet gives a graphical view of all servers in a cluster, as well as active network connections between each client and server. This tool can be used to quickly view whether storage is free or used, whether CPUs are idle or busy, and whether the network is idle or in use. Clicking on individual nodes shows the same detailed data as is avaliable in the catalog page.
Next, it can be helpful to give a single 'superuser' limited access to all servers and directories in a cluster, allowing them to fix broken access controls and solve other problems. To allow this, the -P user argument can be given to a Chirp server, and will implicitly give the named user the L and A rights on any directory on that server.
When managing a large system with many users, it is important to keep track of what users are employing the cluster, and how much space they have consumed. We refer to this as auditing the cluster. To audit a single node, use the audit command of the Chirp tool. This produces a listing of all users of a single host. (You must have the A right in the root directory of the server to run this command.) For example:
% chirp ccl01.cse.nd.edu audit
FILES DIRS DATA OWNER
82842 27 5.0 GB globus:/O=UnivNowhere/CN=Fred
6153 607 503.4 MB unix:fred
2 2 200.3 MB hostname:laptop.nowhere.edu
12 2 3.9 MB unix:betty
To audit an entire cluster, run the chirp_audit_cluster tool.
This will extract the current list of hosts from your catalog,
run an audit on all hosts in parallel, and then produce several
reports in text files: audit.users.txt, audit.hosts.txt,
audit.users.hosts.txt, and audit.hosts.users.txt.
Often, users of a cluster will wish to replicate commonly used data across all disks in the system, perhaps to provide fast access to relatively static data. The chirp_distribute tool can be used to rapidly move data from one node to all others. Given a source host and path, chirp_distribute will create a spanning tree and then move data directly from host to host in parallel. This is much faster than running cp or chirp put directly. For example, this will copy the /database directory from host server.nd.edu to all hosts in your cluster:
% chirp_distribute server.nd.edu /database `chirp_status -s`Another common problem is cleaning up data that has been copied this way. To delete, simply run chirp_distribute again with the -X option and the same arguments.
The simpler tool is just a free space limit.
If run with the -F option, a Chirp server will stop
consuming space when the free space on the disk falls below
this limit. External users will see a
The more complex tool is a user-level quota and allocation system.
If run with the -Q option, a Chirp server will establish
a software quota for all external users. That is, -Q 2GB
will limit external users to consuming a total of 2 GB of storage
within a single Chirp server.
This mechanism imposes some run-time performance penalty,
and also delays server startup somewhere: the Chirp server must
traverse its storage directory to count up the available space.
With the -Q option enabled, external users can
Note: Users employing Parrot can also use the parrot_mkalloc
and parrot_lsalloc commands in ordinary scripts to achieve
the same effect.
To examine an allocation, use the lsalloc command.
To destroy an allocation, simply delete the corresponding directory.
By default, a Chirp server allows only for data storage.
Active storage must be manually enabled through two
distinct steps:
To execute a program remotely, create a fresh directory,
ensure that you have the X right on the directory, and
transfer in a program. Use the job_run command from
the Chirp tool to run that program. By default, Chirp
will store the outputs in stdout.txt and stderr.txt
in the current working directory, and then fetch them when
the job completes.
As a very simple example, suppose that you want to sort
some data stored in /chirp/server.nd.edu/data/records.txt.
Using the Chirp tool, set the X right, send the sort
binary, and then execute it:
The Chirp security model allows for many flexible policies
on what programs may be run. The owner of a server can
allow a user to run and execute arbitrary code on a server
by granting the W and X rights on a directory. If the owner
wants to restrict the runnable programs, he/she may provide
a directory of programs with only the R and X bits set.
This allows users to execute only those programs provided.
By constructing ACLs appropriately, various classes of users
may be given different policies.
Note that the same security model is used for both
data access as remote execution. Internally, Chirp uses
Parrot to run the program in an identity box that
ensures it can only access files according to the ACLs
as described above. If you were to dispatch the program
whoami to the remote node, you would see the
same identity you used to authenticate to the Chirp server.
Thus, it is not necessary for any special accounts to
be created for visiting users.
The Chirp server is able to bind to backend filesystems besides the local
filesystem. In particular, it is able to act as a frontend for the Hadoop HDFS
filesystem. When used on top of Hadoop, Chirp gives you the benefit of a robust
system of ACLs, simple userspace access and POSIX semantics (with some
limitations, discussed below). Perhaps best of all, client jobs will no
longer have any Hadoop or Java (version) dependencies.
chirp_server takes two new options to support HDFS: -f hdfs
to switch to an HDFS backend instead of local storage and -x host:port
to set the Hadoop head node.
To run a Chirp server as a frontend to Hadoop use the
chirp_server_hdfs command. It acts as a wrapper around the
chirp_server command to setup environment variables for Hadoop on
behalf of the user. chirp_server_hdfs also adds the -f hdfs
for you when invoking chirp_server.
Just like when running Chirp on the local filesystem, you must pick a root
storage directory for the server. The root for the server in this case will
(naturally) be located in HDFS. For the chirp_server_hdfs command, you
need only give the location of the HDFS head node. You run the server like so:
Chirp allows you to setup a location to place temporary files such as those
used for Active Storage jobs. You can set this using the -y path
option. This allows for faster access, POSIX semantics, and less load on HDFS.
By default, Chirp assumes the current directory for temporary storage.
Chirp tries to preserve POSIX filesystem semantics where possible despite HDFS
violating certain assumptions. For example, random writes are not possible for
Chirp on HDFS. When the user requests to open a file for writing, Chirp assumes
an implicit O_APPEND flag was added. In addition, HDFS does not
maintain an execute permission bit for regular files. Chirp assumes all files
have the execute bit set.
Chirp also does not allow using the thirdput command or user space management
(-F) when using HDFS as a backend.
In all of the Chirp and Parrot tools, the -d option allows
you to turn on selected debugging messages. The simplest option
is -d all which will show every event that occurs in
the system.
To best debug a problem, we recommend that you turn on the debugging
options on both the client and server that you are operating.
For example, if you are having trouble getting Parrot to connect
to a Chirp server, then run both as follows:
There are a large number of debugging flags. Currently, the
choices are: syscall notice channel process resolve libcall
tcp dns auth local http ftp nest chirp dcap rfio cache
poll remote summary debug time pid all. When debugging problems
with Chirp and Parrot, we recommend selectively using
-d chirp, -d tcp, -d auth, and
-d libcall as needed.
Advanced Topic: Active Storage
Using Parrot and Chirp, it is easy to run programs
that access data remotely over the network. In essence,
the data mvoes to wherever the program is executed.
However, when the program to the run is smaller than
the data to be processed, it makes more sense to send
the program to the data. This concept is called active storage.
% chirp server.nd.edu
connected to server.nd.edu:9094 as unix:fred
chirp:server.nd.edu> setacl . unix:fred rwlax
chirp:server.nd.edu> put /bin/sort
chirp:server.nd.edu> job_run sort -n records.txt >sorted.txt
jobid 33 created
jobid 33 submitted.
jobid 33 completed with exit code 1
jobid 33 removed.
Now, the sorted records are found in sorted.txt.
You could extract the data for further manipulation,
or you could simply dispatch further programs
like grep to sort it:
chirp:server.nd.edu> put /bin/grep
chirp:server.nd.edu> job_run grep needle sorted.txt
If you have long-running jobs, you may disconnect from the Chirp
server, and use the job_list command to see the current
state of the job. Using the unique job number created when it
was submitted, you may use job_wait to wait for completion,
or job_kill to stop a long running job. job_remove
should be used to remove the record of a completed or killed job.
Advanced Topic: HDFS Backend Storage for Chirp
% chirp_server_hdfs -x headnode.hadoop.domain.edu -r /mydata
After setting up the environment variables, this command expands to:
% chirp_server -f hdfs -x headnode.hadoop.domain.edu -r /mydata
Temporary Local Storage
Limitations
Debugging Advice
Debugging a distributed system can be quite difficult because of the
sheer number of hosts involved and the mass of information to be
collected. If you are having difficulty with Chirp, we recommend
that you make good use of the debugging traces built into the tools.
% chirp_server -d all [more options] ...
% parrot -d all tcsh
Of course, this is likely to show way more information than you
will be able to process. Instead, turn on a debugging flags
selectively. For example, if you are having a problem with authentication,
just show those messages with -d auth on both sides.