Reading 04: Scraping Data

Everyone:

Next week, we will transition from shell scripting and filters to creating more sophisticated scripts using the Python programming language. Using a more advanced scripting language such as Python will allow us to use sequence data structures such as lists and mapping data structures such as dictionaries, and thus perform more sophisticated data manipulation efficiently and effectively.

TL;DR

The focus of this reading is to introduce scripting in Python.

Readings

The readings for Monday, February 12 are:

  1. Scipy Lecture Notes: 1.2. The Python language

Optional Resources

Here are some additional resources:

  1. A Byte of Python

  2. Automate The Boring Stuff

Python 3

In this course, we will be using Python 3.6 rather than Python 2.x. Although Python 2 is still widely available, it is deprecated and will soon be unsupported. Therefore, most new projects and applications are moving to Python 3, so we will follow suit1.

Because Python 3 is not the default on most systems, you may need to install it yourself. On Windows or macOS, I recommend the Anaconda distribution, which includes Python and a bunch of useful libraries and tools.

On the student machines, you can use the instructor's copy of Anaconda by performing the following modifications to your PATH environmental variable:

# CSH
setenv PATH ~pbui/pub/anaconda3/bin:$PATH

# BASH
export PATH=~pbui/pub/anaconda3/bin:$PATH

This will give you access to a more current version of Python along with some of the libraries we will be using such as Requests.

Exam 01

Note, there will not be an homework assignment next week. Instead, we will have an in-class exam on Friday, February 16. To prepare for this exam, please take a look at Checklist 01.

Quiz

This week, your reading quiz is split into two sections: the first part is your normal dredd quiz, while the second part involves two Python scripts: exists.py and head.py.

To test these scripts, you will need to download the Makefile and test scripts:

$ git checkout master                 # Make sure we are in master branch
$ git pull --rebase                   # Make sure we are up-to-date with GitLab

$ git checkout -b reading04           # Create reading04 branch and check it out

$ cd reading04                        # Go into reading04 folder

# Download Reading 04 Makefile
$ curl -LO https://gitlab.com/nd-cse-20289-sp18/cse-20289-sp18-assignments/raw/master/reading04/Makefile

# Execute tests (and download them)
$ make

Questions

Record the answers to the following Reading 04 Quiz questions in your reading04 branch:

exists.py

Use the echo.py as the basis for a Python script called exists.py, which accomplishes the same task as the modified version of exists.sh in Reading 02:

# Run script and check error code
$./exists.py * && echo Success
exists.py exists!
README.md exists!
Success

$ ./exists.py * ASDF || echo Success
# Run script and check error code
exists.py exists!
README.md exists!
ASDF does not exist!
Success

head.py

Use the cat.py as the basis for a Python script called head.py, which is a simple implementation of the head command:

# Print Usage
$ ./head.py -h
Usage: head.py files...

    -n  NUM      print the first NUM lines instead of the first 10

# Read from STDIN
$ seq 10 | ./head.py
1
2
3
4
5
6
7
8
9
10

# Read from file specified in command line argument
$ ./head.py README.md
Reading 04 - README
===================

# Limit number of lines
$ seq 10 | ./head.py -n 1
1

# Limit number of lines and read from explicit STDIN (-)
$ seq 10 | ./head.py -n 1 -
1

Submission

To submit you work, follow the same process outlined in Reading 01:

$ git checkout master                 # Make sure we are in master branch
$ git pull --rebase                   # Make sure we are up-to-date with GitLab

$ git checkout -b reading04           # Create reading04 branch and check it out

$ cd reading04                        # Go into reading04 folder
$ $EDITOR answers.json                # Edit your answers.json file

$ ../.scripts/submit.py               # Check reading04 quiz
Submitting reading04 assignment ...
Submitting reading04 quiz ...
     Q1 0.20
     Q2 0.40
     Q3 0.40
  Score 1.00

$ git add answers.json                # Add answers.json to staging area
$ git commit -m "Reading 04: Quiz"    # Commit work

$ $EDITOR exists.py                   # Edit your exists.py file

$ ./test_exists.sh                    # Test your exists.py
Testing exists.py ...
 exists.py *                    ... Success
 exists.py * ASDF               ... Success
   Score .50

$ $EDITOR head.py                     # Edit your head.py file

$ ./test_head.sh                      # Test your head.py
Testing head.py ...
 Usage                          ... Success
 Head on /etc/hosts             ... Success
 Head on /etc/passwd            ... Success
 Head on stdin (implicit)       ... Success
 Head on stdin (explicit)       ... Success
 Head -n 1                      ... Success
 Head -n 2                      ... Success
 Head -n 3                      ... Success
 Head -n 4                      ... Success
 Head -n 5                      ... Success
 Head -n 6                      ... Success
 Head -n 7                      ... Success
 Head -n 8                      ... Success
 Head -n 9                      ... Success
 Head -n 10                     ... Success
   Score 2.50

$ git add Makefile                    # Add Makefile to staging area
$ git add exists.py                   # Add exists.py to staging area
$ git add head.py                     # Add head.py to staging area
$ git commit -m "Reading 04: Scripts" # Commit work

$ git push -u origin reading04        # Push branch to GitLab

Remember to create a merge request and assign the appropriate TA from the Reading 04 TA List.

  1. As a luddite, this breaks my heart, but there is no point in yelling at clouds. Evolve or die. ↩