Parrot is Copyright (C) 2003-2004 Douglas Thain and Copyright (C) 2005- The University of Notre Dame. All rights reserved. This software is distributed under the GNU General Public License. See the file COPYING for details.
The Glite module of Parrot is Copyright (c) Members of the EGEE Collaboration. 2004. See http://eu-egee.org/partners/ for details on the copyright holders. For license conditions see the license file or http://eu-egee.org/license.html
Please use the following citation to refer to Parrot:
vi like so:
% parrot_run vi /anonftp/ftp.gnu.org/pub/README
Parrot is useful to users of distributed systems, because it frees them from rewriting code to work with new systems and relying on remote administrators to trust and install new software. Parrot is also useful to developers of distributed systems, because it allows rapid deployment of new code to real applications and real users that do not have the time, inclination, or permissions to build a kernel-level filesystem.
Parrot currently supports a variety of remote I/O systems, all detailed below. We welcome contributions of new remote I/O drivers from others. However, if you are working on a protocol driver please drop us a note so that we can make sure work is not duplicated.
Almost any application - whether static or dynmically linked,
standard or commercial, command-line or GUI - should work with
Parrot. There are a few exceptions. Because Parrot relies on
the Linux ptrace interface
any program that relies on the ptrace interface cannot run under Parrot.
This means Parrot cannot run a debugger, nor can it run itself recursively.
In addition, Parrot cannot run setuid programs, as the operating system
system considers this a security risk.
Parrot also provide a new experimental features called identity boxing. This feature allows you to securely run a visiting application within a protection domain without become root or creating a new account. Read below for more information on identity boxing.
Parrot currently runs on the Linux operating system with either Intel compatible (i386) or AMD compatible (x86_64) processors. It relies on some fairly low level details in order to implement system call trapping. Ports to other platforms and processors Linux may be possible in the future.
Like any software, Parrot is bound to have some bugs. Please check the known bugs page for the latest scoop.
parrot command followed by any other Unix program. For example, to run a Parrot-enabled vi, execute this command:
% parrot_run vi /anonftp/ftp.gnu.org/pub/README
parrot_run before every command you run, so try starting a shell with Parrot already loaded:
% parrot_run bash
% cp /http/www.nd.edu/~dthain/papers/parrot-agm2003.pdf . % grep Yahoo /http/www.yahoo.com % set autolist % cat /anonftp/ftp.gnu.org/pub[Press TAB here]
Hint: You may find it useful to have some visual indication
of when Parrot is active, so we recommend that you modify
your shell startup scripts to change the prompt when Parrot is enabled.
If you use tcsh, you might add something like this to your .cshrc:
if ( $?PARROT_ENABLED ) then
set prompt = " (Parrot) %n@%m%~%# "
else
set prompt = " %n@%m%~%# "
endif
|
We have limited the examples so far to HTTP and anonymous FTP, as they are the only services we know that absolutely everyone is familiar with. There are a number of other more powerful and secure remote services that you may be less familiar with. Parrot supports them in the same form: The filename begins with the service type, then the host name, then the file name. Here are all the currently supported services:
| example path | remote service | more info |
| /http/www.somewhere.com/index.html | Hypertext Transfer Protocol | included |
| /grow/www.somewhere.com/index.html | GROW - Global Read-Only Web Filesystem | included |
| /ftp/ftp.cs.wisc.edu/RoadMap | File Transfer Protocol | included |
| /anonftp/ftp.cs.wisc.edu/RoadMap | Anonymous File Transfer Protocol | included |
| /gsiftp/ftp.globus.org/path | Globus Security + File Transfer Protocol | more info |
| /irods/host:port/zone/home/user/path | iRODS | more info |
| /hdfs/namenode:port/path | Hadoop Distributed File System (HDFS) | more info |
| /xrootd/host:port/path | XRootD/Scalla Distributed Storage System (xrootd) | more info |
| /chirp/target.cs.wisc.edu/path | Chirp Storage System | included + more info |
| /cvmfs/grid.cern.ch/path | CernVM-FS | more info |
The following protocols have been supported in the past, but are not currently in active use.
| /nest/nest.cs.wisc.edu/path | Network Storage Technology | more info |
| /rfio/host.cern.ch/path | Castor Remote File I/O | more info |
| /dcap/dcap.cs.wisc.edu/pnfs/cs.wisc.edu/path | DCache Access Protocol | more info |
| /lfn/logical/path | Logical File Name - Grid File Access Library | more info |
| /srm/server/path | Site File Name - Grid File Access Library | more info |
| /guid/abc123 | Globally Unique File Name - Grid File Access Library | more info |
| /gfal/protocol://host//path | Grid File Access Library | more info |
You will notice quite quickly that not all remote I/O systems
provide all of the functionality common to an ordinary file system.
For example, HTTP is incapable of listing files.
If you attempt to perform a directory listing on an HTTP
server, Parrot will attempt to keep ls happy
by producing a bogus directory entry:
% parrot_run ls -la /http/www.yahoo.com/
-r--r--r-- 1 dthain dthain 0 Jul 16 11:50 /http/www.yahoo.com
A less-drastic example is found in FTP. If you attempt
to perform a directory listing of an FTP server, Parrot
fills in the available information -- the file names and
their sizes -- but again inserts bogus information to fill the rest out:
% parrot_run ls -la /anonftp/ftp.gnu.org/pub
-rwxrwxrwx 1 dthain dip 580 Oct 5 16:00 CRYPTO.README
-rwxrwxrwx 1 dthain dip 166697 Oct 5 16:00 find.txt.gz
drwxrwxrwx 1 dthain dip 0 Oct 5 16:00 gnu
...
If you would like to get a better idea of the underlying behavior
of Parrot, try running it with the -d remote option,
which will display all of the remote I/O operations that it performs
on a program's behalf:
% parrot_run -d remote ls -la /anonftp/ftp.gnu.org
...
ftp.cs.wisc.edu <-- TYPE I
ftp.cs.wisc.edu --> 200 Type set to I.
ftp.cs.wisc.edu <-- PASV
ftp.cs.wisc.edu --> 227 Entering Passive Mode (128,105,2,28,194,103)
ftp.cs.wisc.edu <-- NLST /
ftp.cs.wisc.edu --> 150 Opening BINARY mode data connection for file list.
...
If your program is upset by the unusual semantics of such
storage systems, then consider using the Chirp protocol and
server:
To start a Chirp server, simply do the following:
% chirp_server -d all
The -d all option turns on debugging, which helps you to understand
how it works initially. You may remove this option once everything
is working.
Suppose the Chirp server is running on bird.cs.wisc.edu. Using Parrot, you may access all of the Unix features of that host from elsewhere:
% parrot_run tcsh
% cd /chirp/bird.cs.wisc.edu
% ls -la
% ...
In general, Parrot gives better performance and usability
with Chirp than with other protocols.
You can read extensively about the Chirp server and protocol
in the Chirp manual.
In addition, Parrot provides several custom command line tools (parrot_getacl, parrot_setacl, parrot_lsalloc, and parrot_mkalloc) that can be used to manage the access control and space allocation features of Chirp from the Unix command line.
The simplest name resolver is the mountlist, given by the -m mountfile option. This file corresponds closely to /etc/ftsab in Unix. A mountlist is simply a file with two columns. The first column gives a logical directory or file name, while the second gives the physical path that it must be connected to.
For example, if a database is stored at an FTP server under the path /anonftp/ftp.cs.wisc.edu/db, it may be spliced into the filesystem under /dbase with a mount list like this:
/dbase /anonftp/ftp.cs.wisc.edu/db
Instruct Parrot to use the mountlist as follows:
% parrot_run -m mountfile tcsh
% cd /dbase
% ls -la
A single mount entry may be given on the command line
with the -M option as follows:
% parrot_run -M /dbase=/anonftp/ftp.cs.wisc.edu/db tcsh
A more sophisticated way to perform name binding is with
an external resolver. This is a program executed
whenever Parrot needs to locate a file or directory.
The program accepts a logical file name and then returns
the physical location where it can be found.
Suppose that you have a database service that locates the nearest copy of a file for you. If you run the command locate_file, it will print out the nearest copy of a file. For example:
% locate_file /1523.data
/chirp/server.nd.edu/mix/1523.data
To connect the program locate_file to Parrot,
simply give a mount string that specifies the program
as a resolver:
% parrot_run -M /dbase=resolver:/path/to/locate_file tcsh
Now, if you attempt to access files under /dbase,
Parrot will execute locate_file and access
the data stored there:
% cat /dbase/1523.data
(see contents of /chirp/server.nd.edu/mix/1523.data)
To use parrot_cp, simply use your shell to alias calls to cp with calls to parrot_cp:
% parrot_run tcsh % alias cp parrot_cp % cp /tmp/mydata /chirp/server.nd.edu/joe/data % cp -rR /chirp/server.nd.edu/joe /tmp/joe
If run outside of Parrot, parrot_cp will operate as an ordinary cp without any performance gain or loss.
% setenv HTTP_PROXY "http://proxy.nd.edu:8080"
Multiple proxy servers can be given, separated by a semicolon.
This will cause Parrot to try each proxy in order until one succeeds.
If DIRECT is given as the last name in the list, then
Parrot will fall back on a direct connection to the target web server.
For example:
% setenv HTTP_PROXY "http://proxy.nd.edu:8080;http://proxy.wisc.edu:1000;DIRECT"
To set up an GROW filesystem, you must run make_growfs on the web server machine with the name of the local storage directory as the argument. For example, suppose that the web server my.server.com stores pages for the URL http://my.server.com/~fred in the local directory /home/fred/www. In this case, you should run the following command:
% make_growfs /home/fred/www
Now, others may perceive the web server as a file server under the /grow
hierarchy. For example:
% parrot_run tcsh
% cd /grow/my.server.com/~fred
% ls -la
In addition to providing precise directory metadata, GROW offers
two additional advantages over plain HTTP:
CVMFS is a read-only filesystem, which was initially based on GROW. It is used within CernVM-FS to provide access to software repositories. It may be used outside of CernVM-FS by mounting it as a FUSE module. Parrot makes it possible to access CVMFS in cases where mounting CVMFS via FUSE is not an option. Like GROW, CVMFS makes use of web proxies and local disk caching for scalability. For security, data integrity is verified with cryptographic checksums. For increased reliability and performance, CVMFS repositories may also be mirrored in multiple locations and accessed via groups of load-balanced web proxies, with fail-over between groups.
The CVMFS repositories hosted by the CernVM project are enabled by default. To access a different repository, it is necessary to configure parrot to know how to access the repository. This may be done with the -r option or with the PARROT_CVMFS_REPO environment variable. The repository configuration syntax is
repo_name:options repo_name2:option2 ...
The repository with repo_name is used when the parrot user attempts to access the matching path /cvmfs/repo_name/.... The configured repository name may begin with *, which acts as a wildcard, matching one or more characters in the requested repository name. This is useful when multiple repositories are hosted at the same site and all configuration details are the same except the beginnings of the repository names, as in atlas.cern.ch and cms.cern.ch. Any * appearing in the options is replaced by the characters in the requested path that were matched by the * in the configured repository name. If a cvmfs path matches more than one configured repository, the last one appearing in the configuration takes precedence.
The format of the repository options is
option1=value1,option2=value2,...
Literal spaces, tabs, newlines, asterisks, equal signs, or commas in the options must be proceeded by a backslash to avoid being interpreted as delimiters. If the same option is specified more than once, the last value takes precedence. The possible options are listed in the table below. The url option is required and has no default. The proxies option is required and defaults to the proxy used by parrot, if any.
| url=URL | The URL of the CernVM-FS server(s): 'url1;url2;...' |
| proxies=HTTP_PROXIES | Set the HTTP proxy list, such as 'proxy1|proxy2'; default is given by -P option (HTTP_PROXY) |
| Proxies separated by '|' are randomly chosen for load balancing. Groups of proxies separated by ';' may be specified for failover. If the first group fails, the second group is used, and so on down the chain. | |
| cachedir=DIR | Where to store disk cache; default is within parrot temp directory (-t option) |
| timeout=SECONDS | Timeout for network operations; default is given by -T option (PARROT_TIMEOUT) |
| timeout_direct=SECONDS | Timeout in for network operations without proxy; default is given by -T option (PARROT_TIMEOUT) |
| max_ttl=MINUTES | Maximum TTL for file catalogs; default: take from catalog |
| allow_unsigned | Accept unsigned catalogs (allows man-in-the-middle attacks) |
| whitelist=URL | HTTP location of trusted catalog certificates (defaults is /.cvmfswhitelist) |
| pubkey=PEMFILE | Public RSA key that is used to verify the whitelist signature. |
| rebuild_cachedb | Force rebuilding the quota cache db from cache directory |
| quota_limit=MB | Limit size of cache. -1 (the default) means unlimited. |
| If not -1, files larger than quota_limit-quota_threshold will not be readable. | |
| quota_threshold=MB | Cleanup cache until size is <= threshold |
| deep_mount=prefix | Path prefix if a repository is mounted on a nested catalog |
| repo_name=NAME | Unique name of the mounted repository; default is the name used for this configuration entry |
| mountpoint=PATH | Path to root of repository; default is /cvmfs/repo_name |
| blacklist=FILE | Local blacklist for invalid certificates. Has precedence over the whitelist. |
Setting the CVMFS configuration overrides the default
configuration. If it is desired to configure additional repositories
but still retain the default repositories, the configuration entry
parrot_run -r '<default-repositories> my.repo:url=http://cvmfs.server.edu/cvmfs/my.repo.edu' ...
The configuration of the default repositories may be modified by
specifying additional options using the syntax
Currently, there is a limitation of one active CVMFS repository per parrot session. Attempts to access a second CVMFS repository in the same parrot session will be rejected. Therefore, the utility of configuring multiple repositories is merely to provide the option of accessing any one of those repositories during a parrot session.
In particular, you must ensure that you define the following environmental variables:
JAVA_HOME Location of your Java installation. HADOOP_HOME Location of your Hadoop installation.Based on these environmental variables, parrot_run_hdfs will attempt to find the appropriate paths for libjvm.so and libhdfs.so. These paths are stored in the environmental variables LIBJVM_PATH and LIBHDFS_PATH, which are used by the HDFS Parrot module to load the necessary shared libraries at run-time. To avoid the startup overhead of searching for these libraries, you may set the paths manually in your environment before calling parrot_run_hdfs, or you may edit the script directly.
Note that while Parrot supports read access to HDFS, it only provides write-once support on HDFS. This is because the current implementations of HDFS do not provide reliable append operations. Likewise, files can only be opened in either read (O_RDONLY) or write mode (O_WRONLY), and not both (O_RDWR).
The simplest way to use Parrot with EGEE is through the Grid File Access Library (GFAL). This library is attached to Parrot under the paths /lfn, /guid, /srm, /rfio, /dcap, and /gfal. This manual does not document all of the details of GFAL itself, you should read that manual for more information.
For example, suppose that you have an RFIO server running on server.somewhere.edu. To access this server, try the following:
% parrot_run tcsh
% cd /rfio/server.somewhere.edu/
% ls -la
Or, to access a file by logical name or by GUID, note that you can either use
a pathname-like syntax, or specify a URL with colons. Both ways are acceptable:
% parrot_run tcsh
% cat /lfn/baud/testgfal15
% cat lfn:baud/testgfal15
% cat /guid/2cd59291-7ae7-4778-af6d-b1f423719441
% cat guid:2cd59291-7ae7-4778-af6d-b1f423719441
In these cases, Parrot pre-processes the paths so that they are formatted correctly
for use by the GFAL. However, you can also compose paths directly for you by GFAL,
by accessing files underneath /gfal. This syntax can be useful if you
are attempting to harness some new GFAL capability that Parrot is not aware of.
For example:
% parrot_run tcsh
% cd /gfal/rfio://server.somewhere.edu//
% ls -la
As noted above, GFAL requires some setup to work properly.
If you are having difficulties, make sure that you have
set all of the GFAL environment variables appropriately:
LCG_GFAL_INFOSYS
LCG_CATALOG_TYPE
LFC_HOST
LCG_RFIO_TYPE
LCG_GFAL_VO
And, if using RFIO, make sure that the appropriate
dynamic library libshift.so is in your
library search path:
export LD_LIBRARY_PATH=/opt/lhc/lib
Use of the GFAL involves many complex interacting pieces
of software. It is recommended that you run Parrot with the -d remote
flag in order to debug problems using GFAL.
% parrot_run cat /lfc/my/logical/name
In this case, Parrot itself performs the lookup against the LFC,
and uses its own access methods to retrieve the file directly.
This can be useful when Parrot supports a protocol (such as GSIFTP)
that is not directly supported by the GFAL library itself.
Use of the LFC involves many complex interacting pieces of software. It is recommended that you run Parrot with the -d remote flag in order to debug problems using LFC.
For example, suppose that you wish to allow a friend to log into your private workstation. Instead of creating a new account, simply use a script supplied with Parrot to create an identity box:
% whoami dthain % parrot_identity_box MyFriend % whoami MyFriend % touch ~dthain/private-data touch: creating ~dthain/private-data': Permission denied
Note that the shell running within the identity box cannot change or modify any of the supervising user's data. In fact, the contained user can only access items that are world-readable or world-writable.
You can give the contained user access to other parts of the filesystem by creating access control lists. (ACLs) An ACL is a list of users and the resources that they are allowed to access. Each directory has it's own ACL in the file .__acl. This file does not appear in a directory listing, but you can read and write it just the same.
For example, MyFriend above can see his initial ACL as follows:
% cat .__acl MyFriend rwlxaThis means that MyFriend can read, write, list, execute, and administer items in the current directory. Now, suppose that MyFriend wants to allow Freddy read access to the same directory. Simply edit the ACL file to read:
MyFriend rwlxa Freddy rlIdentity boxing and ACLs are particularly useful when using distributed storage. You can read more about ACLs and identity boxing in the Chirp manual.
Parrot supports 64 bit programs and processors in the following combinations:
| Program Type | ||
| 32-bit | 64-bit | CPU Type |
| YES | NO | Parrot for 32-bit X86 CPU Pentium, Xeon, Athlon, Sempron |
| YES | YES | Parrot for 64-bit X86_64 CPU Opteron, Athlon64, Turion64, Sempron64 |