Parrot User's Manual

24 August 2004

Parrot is Copyright (C) 2004 Douglas Thain and the University of Notre Dame. This software is distributed under a BSD-style license. See the file COPYING for details.

Overview

Parrot is a tool for attaching old programs to new storage systems. Parrot makes a remote storage system appear as a file system to a legacy application. Parrot does not require any special privileges, any recompiling, or any change whatsoever to existing programs. It can be used by normal users doing normal tasks. For example, an anonymous FTP service is made available to vi like so:
% parrot vi /anonftp/ftp.cs.wisc.edu/RoadMap

Parrot is useful to users of distributed systems, because it frees them from rewriting code to work with new systems and relying on remote administrators to trust and install new software. Parrot is also useful to developers of distributed systems, because it allows rapid deployment of new code to real applications and real users that do not have the time, inclination, or permissions to build a kernel-level filesystem.

Parrot currently supports a variety of remote I/O systems, all detailed below. We welcome contributions of new remote I/O drivers from others. However, if you are working on a protocol driver please drop us a note so that we can make sure work is not duplicated.

Almost any application - whether static or dynmically linked, standard or commercial, command-line or GUI - should work with Parrot. There are a few exceptions. Because Parrot relies on the Linux ptrace interface any program that relies on the ptrace interface cannot run under Parrot. This means Parrot cannot run a debugger, nor can it run itself recursively. In addition, Parrot cannot run setuid programs, as the operating system considers this a security risk.

Parrot currently runs only on the Linux operating system. It relies on some fairly low level details in order to implement system call trapping. Ports to other platforms that are similar to Linux may be possible in the future.

Like any software, Parrot is bound to have some bugs. Please check the known bugs page for the latest scoop.

Installation

Parrot is distributed as part of the Cooperative Computing Tools. To install, please read the cctools installation instructions.

Examples

To use Parrot, you simply use the parrot command followed by any other Unix program. For example, to run a Parrot-enabled vi, execute this command:
% parrot vi /anonftp/ftp.cs.wisc.edu/RoadMap
Of course, it can be clumsy to put parrot before every command you run, so try starting a shell with Parrot already loaded:
% parrot tcsh
Now, you should be able to run any standard command using Parrot filenames. Here are some examples to get you thinking:
% acroread /http/www.cs.wisc.edu/condor/doc/usenix_1.92.pdf
% grep Yahoo /http/www.yahoo.com
% set autolist
% cat /anonftp/ftp.cs.wisc.edu/[Press TAB here]
We have limited the examples so far to HTTP and anonymous FTP, as they are the only services we know that absolutely everyone is familiar with. There are a number of other more powerful and secure remote services that you may be less familiar with. PFS supports them in the same form: The filename begins with the service type, then the host name, then the file name. Here are all the currently supported services:
example pathremote servicemore info
/http/www.yahoo.com/index.htmlHypertext Transfer Protocolincluded
/ftp/ftp.cs.wisc.edu/RoadMapFile Transfer Protocolincluded
/anonftp/ftp.cs.wisc.edu/RoadMapAnonymous File Transfer Protocolincluded
/chirp/target.cs.wisc.edu/pathCondor Chirp I/Oincluded
/gsiftp/ftp.globus.org/pathGlobus Security + File Transfer Protocolmore info
/nest/nest.cs.wisc.edu/pathNetwork Storage Technologymore info
/rfio/host.cern.ch/pathCastor Remote File I/Omore info
/dcap/dcap.cs.wisc.edu/pnfs/cs.wisc.edu/pathDCache Access Protocolmode info

You will notice quite quickly that few remote I/O systems provide all of the functionality common to an ordinary file system. For example, HTTP is incapable of listing files. (This is a design limitation of HTTP, not a bug in Parrot.) If you attempt to perform a directory listing on an HTTP server, Parrot will attempt to keep ls happy by producing a bogus directory entry:
    % parrot ls -la /http/www.yahoo.com/
    -r--r--r--    1 thain    thain           0 Jul 16 11:50 /http/www.yahoo.com
A less-drastic example is found in FTP. If you attempt to perform a directory listing of an FTP server, Parrot fills in the available information -- the file names and their sizes -- but again inserts bogus information to fill the rest out:
    % parrot ls -la /anonftp/ftp.cs.wisc.edu
    total 0
    -rwxrwxrwx    1 thain    thain        2629 Jul 16 11:53 RoadMap
    -rwxrwxrwx    1 thain    thain     1622222 Jul 16 11:53 ls-lR
    -rwxrwxrwx    1 thain    thain      367507 Jul 16 11:53 ls-lR.Z
    -rwxrwxrwx    1 thain    thain      212125 Jul 16 11:53 ls-lR.gz
If you would like to get a better idea of the underlying behavior of Parrot, try running it with the -d remote option, which will display all of the remote I/O operations that it performs on a program's behalf:
    % parrot -d remote ls -la /anonftp/ftp.cs.wisc.edu
    ...
    ftp.cs.wisc.edu <-- TYPE I
    ftp.cs.wisc.edu --> 200 Type set to I.
    ftp.cs.wisc.edu <-- PASV
    ftp.cs.wisc.edu --> 227 Entering Passive Mode (128,105,2,28,194,103)
    ftp.cs.wisc.edu <-- NLST /
    ftp.cs.wisc.edu --> 150 Opening BINARY mode data connection for file list.
    ...
If your program is upset by the unusual semantics of such storage systems, then consider using the Chirp protocol and server, described in more detail below.

Name Resolution

In addition to accessing remote storage, Parrot allows you to create a custom namespace for any program. All file name activity passes through the Parrot name resolver, which can transform any given filename according to a series of rules that you specify.

The simplest name resolver is the mountlist, given by the -m mountfile option. This file corresponds closely to /etc/ftsab in Unix. A mountlist is simply a file with two columns. The first column gives a logical directory or file name, while the second gives the physical path that it must be connected to.

For example, if a database is stored at an FTP server under the path /anonftp/ftp.cs.wisc.edu/db, it may be spliced into the filesystem under /dbase with a mount list like this:

     /dbase       /anonftp/ftp.cs.wisc.edu/db
Instruct Parrot to use the mountlist as follows:
    % parrot -m mountfile tcsh
    % cd /dbase
    % ls -la
A single mount entry may be given on the command line with the -M option as follows:
    % parrot -M /dbase=/anonftp/ftp.cs.wisc.edu/db tcsh

The Chirp Protocol and Server

Most programs are quite satisifed by the Unix emulation performed by Parrot. However, some programs may require access to the full Unix interface in order to perform administrative tasks such as changing file permissions, owners, and so forth. For such programs, we recommend using the Chirp protocol and server, both included with the standard distribution of Parrot.

Chirp is a simple protocol that corresponds closely to the traditional Unix I/O interface, include open(), read(), stat(), readdir(), and so forth. A standalone Chirp server can offer your programs fine-grained file access from anywhere on the network. A Chirp server is started as follows:

    % chirp_server -d all -a my.authfile
The -d all option turns on debugging, which helps you to understand how it works initially. You may remove this option once everything is working. The -a my.authfile specifies a file which gives the authentication and authorization policy for the server.

Suppose the Chirp server is running on bird.cs.wisc.edu. Using Parrot, you may access all of the Unix features of that host from elsewhere:

    % parrot tcsh
    % cd /chirp/bird.cs.wisc.edu/tmp
    % ls -la
    % ...
In general, Parrot gives better performance and usability with Chirp than with other protocols. You can read extensively about the Chirp server and protocol in the Chirp manual.

Options and Environment

Parrot has several command line options and corresponding environment variables. Use these Chirp authentication methods.(PARROT_CHIRP_AUTH)
OptionPurposeEnvironment Variable
-a <list>
-b <bytes> Set the recommended remote I/O block size.PARROT_LOCAL_BLOCK_SIZE
-B <bytes> Set the recommended local I/O block size.PARROT_REMOTE_BLOCK_SIZE
-cConnect to the local Condor Chirp proxy.PARROT_RESOLVE_CHIRP
-C <MB>Set the size of the I/O channel.PARROT_CHANNEL_SIZE
-d <system>Enable debugging for this sub-system.PARROT_DEBUG_FLAGS
-hShow this screen.
-m <file> Use this file as a mountlist.PARROT_MOUNT_FILE
-M <local>=<remote>Mount this remote file on this local directory.
-o <file>Send debugging messages to this file.PARROT_DEBUG_FILE
-p <host:port>Use this proxy for HTTP requests.PARROT_HTTP_PROXY
-t <dir>Where to store temporary files.PARROT_TEMP_DIR
-vDisplay version number.

The flexible debugging flags can be a great help in both debugging and understanding Parrot. To turn on multiple debugging flags, you may either issue multiple -d options:

    % parrot -d ftp -d chirp tcsh
Or, you may give a space separated list in the corresponding environment variable:
    % setenv PARROT_DEBUG_FLAGS "ftp chirp"
    % parrot tcsh
Here is the meaning of each of the debug flags.

syscallThis shows all of the system calls attempted by each program, even those that Parrot does not trap or modify. (To see arguments and return values, try -d libcall instead.)
libcallThis shows only the I/O calls that are actually trapped and implemented by Parrot. The arguments and return codes are the logical values seen by the application, not the underlying operations. (To see the underlying operations try -d remote or -d local instead.)
cacheThis shows all of the shared segments that are loaded into the channel cache and shared by multiple programs. For most programs, this means all the shared libraries.
processThis shows all process creations, deletions, signals, and process state changes.
resolveThis shows every invocation of the name resolver. A plain file name indicates the name was not modified, while more detailed records show names that were changed or denied access.
localThis shows all local I/O calls from the perspective of Parrot. Notice that the file descriptors and file names shown are internal to Parrot. (To see fds and names from the perspective of the job, try -d libcall.)
remoteThis shows all non-local file activity.
httpThis shows only HTTP operations.
ftpThis shows only FTP operations.
nestThis shows only NeST operations.
chirpThis shows only Chirp operations.
rfioThis shows only RFIO operations.
pollThis shows all activity related to processes that block (explicitly or implicitly) waiting for I/O.
timeThis adds the current time to every debug message.
pidThis adds the calling process id to every debug message.
allThis shows all possible debugging messages.