Homework 03: Weathering with Zipcodes

The goal of this homework assignment is to allow you to practice using regular expressions and filters in shell scripts. In this assignment, you will build on your knowledge of the bourne shell language to write scripts that slice and dice data.

For this assignment, record your scripts and any responses to the following activities in the in the homework03 folder of your assignments GitHub repository and push your work by noon Saturday, February 27.

Activity 0: Preparation¶

Before starting this homework assignment, you should first perform a git pull to retrieve any changes in your remote GitHub repository:

$ cd path/to/repository                   # Go to assignments repository

$ git checkout master                     # Make sure we are in master branch

$ git pull --rebase                       # Get any remote changes not present locally

Next, create a new branch for this assignment:

$ git checkout -b homework03              # Create homework03 branch and check it out

Once this is done, download the Makefile and test scripts:

# Go to homework03 folder
$ cd homework03

# Download the Makefile
$ curl -LO https://raw.githubusercontent.com/nd-cse-20289-sp21/cse-20289-sp21-assignments/master/homework03/Makefile

# Add and commit Makefile
$ git add Makefile
$ git commit -m "homework03: add Makefile"

# Download the test scripts
$ make test-scripts

You are now ready to work on the activities below.

Activity 1: Weather (4 Points)¶

Michael is not amused by all the cold snow on campus and longs for the warmth and comfort of spring. Obsessed with the weather, he compulsively checks weather.gov, which is a website by the National Weather Service. While checking the weather for the fifth time on his phone during class, he realized that he could actually apply some of the things he is learning in class to his current obsession.

For example, he realized that he can fetch the weather information for any zip code, using the following URL:

$ curl -sL https://forecast.weather.gov/zipcity.php?inputstring=$ZIPCODE

For instance, the weather information for Notre Dame, Indiana can be retrieved using:

$ curl -sL https://forecast.weather.gov/zipcity.php?inputstring=46556

Looking at the data retrieved from the website, he notices that the forecast and temperature information is wrapped in HTML fields that contain the text "myforecast":

...
<p class="myforecast-current">Overcast</p>
<p class="myforecast-current-lrg">37&deg;F</p>
<p class="myforecast-current-sm">3&deg;C</p>
...

Michael wants to make a script weather.sh that will fetch the weather information for any specified zip code and extract the temperature and forecast. Unfortunately, despite taking AP Computer Science in high school, he is still a bit confused about pipelines and shell scripting.

You decide that you could use some practice wth shell scripting and regular expressions, so you help him complete the script.

Task 1: weather.sh¶

The weather.sh script takes three possible flags:

$ ./weather.sh -h
Usage: weather.sh [zipcode]

-c    Use Celsius degrees instead of Fahrenheit for temperature
-f    Display forecast text

If zipcode is not provided, then it defaults to 46556.

The -c flag will have the script output the temperature as Celsius rather than Fahrenheit (the default).

The -f flag will have the script output the forecast text before the current temperature.

The -h flag will display the usage message and exit with success.

Here are some examples of weather.sh in action:

# Show current temperature for Notre Dame, Indiana (46556)
$ ./weather.sh
Temperature: 17 degrees

# Show forecast and current temperature for Orange, California (92867)
$ ./weather.sh -f 92867
Forecast:    NA
Temperature: 59 degrees

# Show forecast and current temperature (celsius) for Eau Claire, Wisconsin (54701)
$ ./weather.sh -c -f 54701
Forecast:    Fair
Temperature: -21 degrees

Hints¶

Here is a skeleton you can use to start your weather.sh script:

#!/bin/sh

# Globals

URL="https://forecast.weather.gov/zipcity.php"
ZIPCODE=46556
FORECAST=0
CELSIUS=0

# Functions

usage() {
    cat 1>&2 <<EOF
Usage: $(basename $0) [zipcode]

-c    Use Celsius degrees instead of Fahrenheit for temperature
-f    Display forecast text

If zipcode is not provided, then it defaults to $ZIPCODE.
EOF
    exit $1
}

weather_information() {
    # Fetch weather information from URL based on ZIPCODE
}

temperature() {
    # Extract temperature information from weather source
    weather_information | ...
}

forecast() {
    # Extract forecast information from weather source
    weather_information | ...
}

# Parse Command Line Options

while [ $# -gt 0 ]; do
    case $1 in
        -h) usage 0;;
    esac
    shift
done

# Display Information

echo "Temperature: $(temperature) degrees"

• The general flow of your script should be to parse arguments and then to execute different pipelines.

• You will need to expand the case statement to parse command line arguments.

• You will need to use if statements to guard or control which could gets executed.

• You will probably want to use curl to fetch the weather information.

• You will probably want to use sed and grep and maybe awk.

• You will need to trim leading and trailing whitespace for the forecast.

Task 2: test_weather.sh¶

To aid you in testing the weather.sh script, we provided you with test_weather.sh, which you can use as follows:

$ ./test_weather.sh
Testing weather.sh ...
   Usage                                    ... Success
   Default                                  ... Success
   46556                                    ... Success
   46556 Celsius                            ... Success
   46556 Forecast                           ... Success
   46556 Celsius Forecast                   ... Success
   54701                                    ... Success
   54701 Celsius                            ... Success
   54701 Forecast                           ... Success
   54701 Celsius Forecast                   ... Success
   92867                                    ... Success
   92867 Celsius                            ... Success
   92867 Forecast                           ... Success
   92867 Celsius Forecast                   ... Success

   Score 4.00

Activity 2: Zipcode (4 Points)¶

Being able to lookup weather information based on zip codes is great... if you know the zip code of the place you are interested in. Like most people, however, Samantha is not really familiar with zip codes around the country.

Like the instructor, Samantha hails from Orange County¹, which is situated on the Best Coast and is the home of Mickey Mouse, John Wayne, No Doubt², and StarCraft³ (among other things). Having never really left paradise ⁴, Samantha doesn't really know much about other places in America. This is problematic as she has made many friends from all over the country, such as her roommate from Toledo, Ohio, or her other roommate from Jacksonville, Florida ⁵. Since she needs to figure out the zip codes to these unfamiliar places, she is creating a script called zipcode.sh, which scrapes the zip codes from the website Zip Codes To Go and allows her to list all the zip codes in a specific state or even a particular city.

For instance, using curl, she can view all the raw HTML for Indiana by doing the following:

$ curl -s https://www.zipcodestogo.com/Indiana/

Because you are pretty good with regular expressions now, you decide to help Samantha out with parsing this HTML and extracting the zip codes.

Task 1: zipcode.sh¶

The zipcode.sh script takes three possible flags:

$ ./zipcode.sh -h
Usage: zipcode.sh

  -c      CITY    Which city to search
  -s      STATE   Which state to search (Indiana)

If no CITY is specified, then all the zip codes for the STATE are displayed.

The -c flag takes a CITY argument, which specifies the city to search for within the STATE. If no CITY is specified, then the script should return all the zip codes in the STATE.

The -s flag takes a STATE argument, which specifies the STATE to search through. If no STATE is specified, then the script should assume the STATE is "Indiana".

The -h flag will display the usage message and exit with success.

Here are some examples of zipcode.sh in action:

# Show all Zip Codes from default state (Indiana)
$ ./zipcode.sh
46001
46011
46012
46013
...
47994
47995
47996
47997

# Show all Zip Codes in Notre Dame, Indiana
$ ./zipcode.sh -s Indiana -c "Notre Dame"
46556

Hints¶

Here is a skeleton you can use to start your zipcode.sh script:

#!/bin/sh

# Globals

URL=https://www.zipcodestogo.com/

# Functions

usage() {
}

# Parse Command Line Options

while [ $# -gt 0 ]; do
    case $1 in
    -h) usage 0;;
     *) usage 1;;
    esac
    shift
done

# Filter Pipeline(s)

curl -s $URL

• The general flow of your script should be to parse arguments and then to execute different pipelines.

• You will probably want to use sed and grep and maybe awk.

• You will probably want to create different functions and use them as stages in your pipelines.

Task 2: test_zipcode.sh¶

To aid you in testing the zipcode.sh script, we provided you with test_zipcode.sh, which you can use as follows:

$ ./test_zipcode.sh
Testing zipcode.sh ...
   Usage                                    ... Success
   Default                                  ... Success
   Indiana                                  ... Success
   Notre Dame, Indiana                      ... Success
   South Bend, Indiana                      ... Success
   California                               ... Success
   Orange, California                       ... Success
   Los Angeles, California                  ... Success
   New York                                 ... Success
   Buffalo, New York                        ... Success
   New York, New York                       ... Success

   Score 4.00

Activity 3: Quiz (2 Points)¶

Once you have completed all the activities above, you are to complete the following reflection quiz:

As with Reading 01, you will need to store your answers in a homework03/answers.json file. You can use the form above to generate the contents of this file, or you can write the JSON by hand.

To test your quiz, you can use the check.py script:

$ ../.scripts/check.py
Checking homework03 quiz ...
     Q01 0.25
     Q02 0.25
     Q03 0.25
     Q04 1.25
   Score 2.00 / 2.00
  Status Success

Guru Point (1 Extra Credit Point)¶

Professor Tim Weninger, Notre Dame's resident Reddit expert, lamblasted the instructor for failing to introduce students to cron in previous course sections. To rectify this, you are to create a cronjob that periodically uses your weather.sh script to output the weather for a zip code of your choice in a file in your HOME directory. You will then need to modify your ~/.bashrc file to output the contents of the resulting weather file every time you login.

To help you setup your cronjob, here are some resources regarding:

• Scheduling Tasks with Cron Jobs

• How To Add Jobs To cron Under Linux or UNIX

• How To Schedule Routine Tasks With Cron and Anacron on a VPS

Submission¶

To submit your assignment, please commit your work to the homework03 folder of your homework03 branch in your assignments GitHub repository:

#--------------------------------------------------
# BE SURE TO DO THE PREPARATION STEPS IN ACTIVITY 0
#--------------------------------------------------

$ cd homework03                                       # Go to Homework 03 directory

$ $EDITOR weather.sh                                  # Edit script
$ git add weather.sh                                  # Mark changes for commit
$ git commit -m "homework03: Activity 1 completed"    # Record changes
...
$ $EDITOR zipcode.sh                                  # Edit script
$ git add zipcode.sh                                  # Mark changes for commit
$ git commit -m "homework03: Activity 2 completed"    # Record changes
...
$ $EDITOR answers.json                                # Edit quiz
$ git add answers.json                                # Mark changes for commit
$ git commit -m "homework03: Activity 3 completed"    # Record changes
...
$ git push -u origin homework03                       # Push branch to GitHub