Reading 05: Scraping The Web

Readings

The readings for Monday, February 20 are:

1. Scipy Lecture Notes: 1.2. The Python language

Optional Resources

1. A Byte of Python

2. Automate The Boring Stuff

3. Think Python

4. Learn Python The Hard Way

# CSH
setenv PATH ~pbui/pub/anaconda2-4.1.1/bin:$PATH

# BASH
export PATH=~pbui/pub/anaconda2-4.1.1/bin:$PATH

Questions

1. Given the following Python script, echo.py:

   #!/usr/bin/env python2.7
   
   import sys
   
   for arg in sys.argv[1:]:
       print arg,
   

   In your reading05/README.md file, answer the following questions:

   1. What is the purpose of import sys?

   2. Explain what for arg in sys.argv[1:]: does.

   3. Why is there a trailing , in print arg,?

   Use the echo.py as the basis for a Python script called exists.py, which accomplishes the same task as the modified version of exists.sh in Reading 02.

2. Given the following Python script, cat.py:

   #!/usr/bin/env python2.7
   
   import os
   import sys
   
   # Global Variables
   
   ENDING = ''
   
   # Usage function
   
   def usage(status=0):
       print '''Usage: {} files...
   
       -E  display $ at end of each line'''.format(os.path.basename(sys.argv[0]))
       sys.exit(status)
   
   # Parse command line options
   
   args = sys.argv[1:]
   while len(args) and args[0].startswith('-') and len(args[0]) > 1:
       arg = args.pop(0)
       if arg == '-E':
           ENDING = '$'
       elif arg == '-h':
           usage(0)
       else:
           usage(1)
   
   if len(args) == 0:
       args.append('-')
   
   # Main execution
   
   for path in args:
       if path == '-':
           stream = sys.stdin
       else:
           stream = open(path)
   
       for line in stream:
           line = line.rstrip()
           print line + ENDING
   
       stream.close()
   

   In your reading05/README.md file, answer the following questions:

   1. Explain how the command line arguments are parsed by discussing how this loop while len(args) and args[0].startswith('-') and len(args) > 1 works.

   2. What are the purposes of the following code blocks:

      if len(args) == 0:
          args.append('-')
      

      And

      if path == '-':
          stream = sys.stdin
      else:
          stream = open(path)
      

   3. What does line = line.rstrip() do? Is it necessary?

   Use the cat.py as the basis for a Python script called head.py, which is a simple implementation of the head command:

   # Print Usage
   $ ./head.py -h
   Usage: head.py files...
   
       -n  NUM      print the first NUM lines instead of the first 10
   
   # Read from STDIN
   $ seq 10 | ./head.py
   1
   2
   3
   4
   5
   6
   7
   8
   9
   10
   
   # Read from file specified in command line argument
   $ ./head.py README.md
   Reading 05 - README
   ===================
   
   # Limit number of lines
   $ seq 10 | ./head.py -n 1
   1
   
   # Limit number of lines and read from explicit STDIN (-)
   $ seq 10 | ./head.py -n 1 -
   1
   

Testing

To test your exists.py script, you can use the same process in Reading 02:

# Run script and check error code
$./exists.py * && echo Success
exists.py exists!
README.md exists!
Success

$ ./exists.py * ASDF || echo Success
# Run script and check error code
exists.py exists!
README.md exists!
ASDF does not exist!
Success

To test your head.py script, you can the provided test_head.sh script:

# Download script
$ curl -O http://www3.nd.edu/~pbui/teaching/cse.20289.sp17/static/sh/test_head.sh

# # Make script executable
$ chmod +x test_head.sh

# Run test script
$ ./test_head.sh
head.py test successful!

Feedback

If you have any questions, comments, or concerns regarding the course, please provide your feedback at the end of your response.

Submission

To submit your assignment, please commit your work to the reading05 folder in your assignments GitLab repository. Your reading05 folder should only contain the following files:

• README.md
• exists.py
• head.py