This course website is from a previous semester. If you are currently in the class, please make sure you are viewing the latest course website instead of this old one.
The goal of this homework assignment is to allow you to practice using regular expressions and filters in shell scripts. In this assignment, you will build on your knowledge of the bourne shell language to write scripts that slice and dice data.
For this assignment, record your scripts and any responses to the following
activities in the in the homework03 folder of your assignments GitLab
repository and push your work by 11:59 AM Saturday, February 8.
Before starting this homework assignment, you should first perform a git
pull to retrieve any changes in your remote GitLab repository:
$ cd path/to/repository # Go to assignments repository $ git checkout master # Make sure we are in master branch $ git pull --rebase # Get any remote changes not present locally
Next, create a new branch for this assignment:
$ git checkout -b homework03 # Create homework03 branch and check it out
Once this is done, download the Makefile and test scripts:
# Go to homework03 folder $ cd homework03 # Download the Makefile $ curl -LO https://gitlab.com/nd-cse-20289-sp20/cse-20289-sp20-assignments/raw/master/homework03/Makefile # Add and commit Makefile $ git add Makefile $ git commit -m "homework03: add Makefile" # Download the test scripts $ make test-scripts
Note, you do not need to add and commit the test scripts since the
Makefile will automatically download them again whenever you run make.
You are now ready to work on the activities below.
Note, the provided test scripts are used to verify the correctness of your
code (ie. does it do the right thing). Good code, however, requires more
than just correct behavior; it demands consistent formatting and
concise implementation. Because of this, your scripts will also be
graded on coding style. In general, we expect you to strive for clean code and
will look for the following:
Consistency
No dead code
Readability
Not too dense, not too sparse
Organization
Reasonably structured and ordered
For each of the activities below, 0.5 points are reserved for coding style.
Robert is not amused by all the cold snow on campus and longs for the warmth and comfort of spring. Obsessed with the weather, he compulsively checks weather.gov, which is a website by the National Weather Service. While checking the weather for the fifth time on his phone during class, he realized that he could actually apply some of the things he is learning in class to his current obsession.
For example, he realized that he can fetch the weather information for any zip code, using the following URL:
$ curl -sL https://forecast.weather.gov/zipcity.php?inputstring=$ZIPCODE
For instance, the weather information for Notre Dame, Indiana can be retrieved using:
$ curl -sL https://forecast.weather.gov/zipcity.php?inputstring=46556
Looking at the data retrieved from the website, he notices that the forecast and temperature information is wrapped in HTML fields that contain the text "myforecast":
... <p class="myforecast-current">Overcast</p> <p class="myforecast-current-lrg">37°F</p> <p class="myforecast-current-sm">3°C</p> ...
Robert wants to make a script weather.sh that will fetch the weather
information for any specified zip code and extract the temperature and
forecast. Unfortunately, his phone is buzzing, so he has to step out take
take the call...
You decide that you could use some practice wth shell scripting and regular expressions, so you help him complete the script.
weather.shThe weather.sh script takes three possible flags:
$ ./weather.sh -h Usage: weather.sh [zipcode] -c Use Celsius degrees instead of Fahrenheit for temperature -f Display forecast text If zipcode is not provided, then it defaults to 46556.
The -c flag will have the script output the temperature as Celsius rather
than Fahrenheit (the default).
The -f flag will have the script output the forecast text before the
current temperature.
The -h flag will display the usage message and exit with success.
Here are some examples of weather.sh in action:
# Show current temperature for Notre Dame, Indiana (46556) $ ./weather.sh Temperature: 34 degrees # Show forecast and current temperature for Orange, California (92867) $ ./weather.sh -f 92867 Forecast: NA Temperature: 50 degrees # Show forecast and current temperature (celsius) for Eau Claire, Wisconsin (54701) $ ./weather.sh -c -f 54701 Forecast: Fog/Mist Temperature: -2 degrees
Here is a skeleton you can use to start your weather.sh script:
#!/bin/sh # Globals URL="https://forecast.weather.gov/zipcity.php" ZIPCODE=46556 FORECAST=0 CELSIUS=0 # Functions usage() { cat 1>&2 <<EOF Usage: $(basename $0) [zipcode] -c Use Celsius degrees instead of Fahrenheit for temperature -f Display forecast text If zipcode is not provided, then it defaults to $ZIPCODE. EOF exit $1 } weather_information() { # Fetch weather information from URL based on ZIPCODE } temperature() { # Extract temperature information from weather source weather_information | ... } forecast() { # Extract forecast information from weather source weather_information | ... } # Parse Command Line Options while [ $# -gt 0 ]; do case $1 in -h) usage 0;; esac shift done # Display Information echo "Temperature: $(temperature) degrees"
The general flow of your script should be to parse arguments and then to execute different pipelines.
You will need to expand the case statement to parse command line
arguments.
You will need to use if statements to guard or control which could gets
executed.
You will probably want to use curl to fetch the weather information.
You will need to trim leading and trailing whitespace for the forecast.
test_weather.shTo aid you in testing the weather.sh script, we provided you with
test_weather.sh, which you can use as follows:
$ ./test_weather.sh Testing weather.sh ... Usage ... Success Default ... Success 46556 ... Success 46556 Celsius ... Success 46556 Forecast ... Success 46556 Celsius Forecast ... Success 92867 ... Success 92867 Celsius ... Success 92867 Forecast ... Success 92867 Celsius Forecast ... Success Score 4.00
Because the data is being pulled from a remote website, the tests might take a while (but no more than 30 seconds).
Sometimes the pipeline fails not because your code is wrong, but because of hiccups with the server (it is live data after all). If you suspect that your code is correct, you can always retry the pipeline to see if it will succeed without changing your code.
If it doesn't work after three retries... then your code is probably wrong.
Being able to lookup weather information based on zip codes is great... if you know the zip code of the place you are interested in. Like most people, however, Stella is not really familiar with zip codes around the country.
Like the instructor, Stella hails from Orange County1, which is situated
on the Best Coast and is the home of Mickey Mouse, John Wayne, No
Doubt2, and StarCraft3 (among other things). Having never really
left paradise 4, Stella doesn't really know much about other places in
America. This is problematic as she has made many friends from all over the
country, such as her roommate from Toledo, Ohio, or her other roommate from
Jacksonville, Florida 5. Since she needs to figure out the zip codes to
these unfamiliar places, she is creating a script called zipcode.sh, which
scrapes the zip codes from the website Zip Codes To
Go and allows her to list all the zip codes
in a specific state or even a particular city.
For instance, using curl, she can view all the raw HTML for Indiana by doing the following:
$ curl -s https://www.zipcodestogo.com/Indiana/
Because you are pretty good with regular expressions now, you decide to help Stella out with parsing this HTML and extracting the zip codes.
zipcode.shThe zipcode.sh script takes three possible flags:
$ ./zipcode.sh -h Usage: zipcode.sh -c CITY Which city to search -s STATE Which state to search (Indiana) If no CITY is specified, then all the zip codes for the STATE are displayed.
The -c flag takes a CITY argument, which specifies the city to search for
within the STATE. If no CITY is specified, then the script should return
all the zip codes in the STATE.
The -s flag takes a STATE argument, which specifies the STATE to search
through. If no STATE is specified, then the script should assume the
STATE is "Indiana".
The -h flag will display the usage message and exit with success.
Here are some examples of zipcode.sh in action:
# Show all Zip Codes from default state (Indiana) $ ./zipcode.sh 46001 46011 46012 46013 ... 47994 47995 47996 47997 # Show all Zip Codes in Notre Dame, Indiana $ ./zipcode.sh -s Indiana -c "Notre Dame" 46556
Here is a skeleton you can use to start your zipcode.sh script:
#!/bin/sh # Globals URL=https://www.zipcodestogo.com/ # Functions usage() { } # Parse Command Line Options while [ $# -gt 0 ]; do case $1 in -h) usage 0;; *) usage 1;; esac shift done # Filter Pipeline(s) curl -s $URL
The general flow of your script should be to parse arguments and then to execute different pipelines.
You will probably want to create different functions and use them as stages in your pipelines.
test_zipcode.shTo aid you in testing the zipcode.sh script, we provided you with
test_zipcode.sh, which you can use as follows:
$ ./test_zipcode.sh
Testing zipcode.sh ...
Usage ... Success
Default ... Success
Indiana ... Success
Notre Dame, Indiana ... Success
South Bend, Indiana ... Success
California ... Success
Orange, California ... Success
Los Angeles, California ... Success
New York ... Success
Buffalo, New York ... Success
New York, New York ... Success
Score 4.00
Because the data is being pulled from a remote website, the tests might take a while (but no more than 30 seconds).
Once you have completed all the activities above, you are to complete the following reflection quiz:
As with Reading 01, you will need to store your answers in a
homework03/answers.json file. You can use the form above to generate the
contents of this file, or you can write the JSON by hand.
To test your quiz, you can use the test_quiz.py script:
$ ./test_quiz.py
Submitting homework03 quiz ...
Q01 0.25
Q02 0.25
Q03 0.25
Q04 1.25
Score 2.00
Professor Tim Weninger, Notre Dame's resident Reddit expert, lamblasted
the instructor for failing to introduce students to cron in previous course
sections. To rectify this, you are to create a cronjob that periodically
uses your weather.sh script to output the weather for a zip code of your
choice in a file in your HOME directory. You will then need to modify
your ~/.bashrc file to output the contents of the resulting weather file
every time you login.
To help you setup your cronjob, here are some resources regarding:
To get credit for this Guru Point, show your cronjob in action to the instructor or a TA to verify. You have up until a week after this assignment is due to verify your Guru Point.
Remember that you can always forgo this Guru Point for two extra days to do the homework. That is, if you need an extension, you can simply skip the Guru Point and you will automatically have until Monday to complete the assignment for full credit.
Just leave a note on your Merge Request of your intensions.
To submit your assignment, please commit your work to the homework03 folder
of your homework03 branch in your assignments GitLab repository:
#-------------------------------------------------- # BE SURE TO DO THE PREPARATION STEPS IN ACTIVITY 0 #-------------------------------------------------- $ cd homework03 # Go to Homework 03 directory $ $EDITOR weather.sh # Edit script $ git add weather.sh # Mark changes for commit $ git commit -m "homework03: Activity 1 completed" # Record changes ... $ $EDITOR zipcode.sh # Edit script $ git add zipcode.sh # Mark changes for commit $ git commit -m "homework03: Activity 2 completed" # Record changes ... $ $EDITOR answers.json # Edit quiz $ git add answers.json # Mark changes for commit $ git commit -m "homework03: Activity 3 completed" # Record changes ... $ git push -u origin homework03 # Push branch to GitLab
Remember to create a merge request and assign the appropriate TA from the Reading 03 TA List.
DO NOT MERGE your own merge request. The TAs use open merge requests to keep track of which assignments to grade. Closing them yourself will cause a delay in grading and confuse the TAs.