Homework 03: Regex, Filters

The goal of this homework assignment is to allow you to practice using regular expressions and filters in shell scripts. In this assignment, you will build on your knowledge of the bourne shell language to write scripts that slice and dice data.

For this assignment, record your scripts and any responses to the following activities in the in the homework03 folder of your assignments GitLab repository and push your work by 11:59 AM Saturday, February 9.

Activity 0: Preparation

Before starting this homework assignment, you should first perform a git pull to retrieve any changes in your remote GitLab repository:

$ cd path/to/repository                   # Go to assignments repository

$ git checkout master                     # Make sure we are in master branch

$ git pull --rebase                       # Get any remote changes not present locally

Next, create a new branch for this assignment:

$ git checkout -b homework03              # Create homework03 branch and check it out

Once this is done, download the Makefile and test scripts:

# Go to homework03 folder
$ cd homework03

# Download the Makefile
$ curl -LOk https://gitlab.com/nd-cse-20289-sp19/cse-20289-sp19-assignments/raw/master/homework03/Makefile

# Add and commit Makefile
$ git add Makefile
$ git commit -m "homework03: add Makefile"

# Download the test scripts
$ make test-scripts

Note, you do not need to add and commit the test scripts since the Makefile will automaticaly download them again whenever you run make.

You are now ready to work on the activities below.

Activity 1: Caesar Cipher (3 Points)

In light of the Apple's letter to its customers a few years ago and the recent renewal of warrantless surveillance, Steve has decided it is time to take security and cryptography seriously. Fearful of government surveillance, he begins implementing a script, caesar.sh, that implements a state-of-the-art ¹ cipher technique know as the Caesar cipher. In this encryption technique letters are shifted by a fixed amount as shown below:

In the example above, letters are shifted 3 to the left (or rather shifted 23 to the right). So the string FARK THE NSA! becomes CXOH QEB KPX!. To reverse the string, you simply perform the same operation, except you shift in the opposite direction.

This encryption process can be described mathematically as:

Similarly, the decryption process is defined as:

Since you've already taken Discrete Math and are familiar with modular arithmetic, you decide to help Steve implement caesar.sh.

Task 1: caesar.sh

The caesar.sh script takes one possible argument and then reads input from STDIN:

$ ./caesar.sh -h
Usage: caesar.sh [rotation]

This program will read from stdin and rotate the text by the specified
rotation.  If none is specified, then the default value is 13.

The one possible argument is to be used as the key or the amount to rotate the letters (in this script, we will perform right shifts).

Here are some examples of caesar.sh in action:

$ echo "I came, I saw, I conquered." | ./caesar.sh
V pnzr, V fnj, V pbadhrerq.

$ echo "I came, I saw, I conquered." | ./caesar.sh | ./caesar.sh
I came, I saw, I conquered.

$ echo "Experience is the teacher of all things." | ./caesar.sh 10
Ohzobsoxmo sc dro dokmrob yp kvv drsxqc.

$ echo "Experience is the teacher of all things." | ./caesar.sh 10 | ./caesar.sh 16
Experience is the teacher of all things.

$ echo "It is better to create than to learn! Creating is the essence of life." | ./caesar.sh 40
Wh wg pshhsf hc qfsohs hvob hc zsofb! Qfsohwbu wg hvs sggsbqs ct zwts.

$ echo "It is better to create than to learn! Creating is the essence of life." | ./caesar.sh 40 | ./caesar.sh 12
It is better to create than to learn! Creating is the essence of life.

Hints

Here is a skeleton you can use to start your caesar.sh script:

#!/bin/sh

# Functions

usage() {
    cat <<EOF
Usage: $(basename $0) [rotation]

This program will read from stdin and rotate the text by the specified
rotation.  If none is specified, then the default value is 13.
EOF
    exit $1
}

source_set() {
    echo $LOWERCASE
}

target_set() {
    echo $LOWERCASE
}

# Parse command-line options

if [ "$1" = "-h" ]; then
    usage 0
fi

LOWERCASE=abcdefghijklmnopqrstuvwxyz

# Filter pipeline

tr $(source_set) $(target_set)

• The general flow of your script should be to construct a source set and a target set and then use the tr command on these sets.

• You may want to create functions to help you assemble the source and target sets.

• You may want to use cut and tr to construct the source and target sets.

• Be sure to handle upper and lower case letters.

• Be sure to handle rotations larger than 26.

Task 2: test_caesar.sh

To aid you in testing the caesar.sh script, we provided you with test_caesar.sh, which you can use as follows:

$ ./test_caesar.sh
Testing caesar.sh ...
   Usage                ... Success
   ROT13-E              ... Success
   ROT13-D              ... Success
   ROT10-E              ... Success
   ROT16-D              ... Success
   ROT40-E              ... Success
   ROT12-D              ... Success

   Score 2.50

Task 3: README.md

In your README.md, describe how you implemented the caesar.sh script. In particular, briefly discuss:

1. How you parsed the command line arguments.

2. How you constructed the source set (ie. SET1).

3. How you constructed the target set (ie. SET2).

4. How you used both of these sets to perform the encryption/decryption.

Activity 2: Broification (3 Points)

Mark is a bit of a brogrammer and attempts to show off his 1337 skills by making his life harder than necessary. One way this manifests itself is in his habit of removing comments from all his code and configuration files. After all, according to Mark, "comments are for noobs". Additionally, he finds blank lines wasteful and likes to strip any lines consisting of only whitespace out of his files because, you know, those 4 bytes really matter. Finally, Mark also removes any trailing whitespace in his files².

For instance, given the following file:

# Super useful comment describing a tricky configuration option
SETUP_THE_BOMB="no" # Hide the evidence

# Another super userful comment
BASE_OWNERSHIP="us" # All your base are belong to...

Mark would broify the file down to the following:

SETUP_THE_BOMB="no"
BASE_OWNERSHIP="us"

Unfortunately, Mark's bravado is just a facade and he is no better at programming than those Mendoza kids who keep asking you to join their social networking startup³. Although you are not supporter of brogrammer culture, you decide to help Mark out⁴ by writing the script broify.sh that removes comments and empty lines from files.

Task 1: broify.sh

The broify.sh script takes two possible flags and then reads input via STDIN:

$ ./broify.sh -h
Usage: broify.sh

  -d DELIM    Use this as the comment delimiter.
  -W          Don't strip empty lines.

The -d flag takes a DELIM argument which is used at the comment delimiter. By default this is #. The DELIM is used to indicate the start of the comment block (we are only concerned with line-based comments that begin with the DELIM and go to the end of the line).

When the -W flag is enabled, then the script should not strip empty lines. By default, the broify.sh script will remove any empty lines.

Here are some examples of broify.sh in action:

$ ./broify.sh < /etc/resolv.conf
domain cse.nd.edu
search cse.nd.edu
nameserver 66.205.160.99
nameserver 129.74.250.99

$ ./broify.sh -d '//' <<EOF
// C++ is so cool
int main() {
    // Totes
    return 0;
}
EOF
int main() {
    return 0;
}

$ ./broify.sh -d '//' -W <<EOF
// C++ is so cool
int main() {
    // Totes
    return 0;
}
EOF

int main() {

    return 0;
}

Hints

Here is a skeleton you can use to start your broify.sh script:

#!/bin/sh

# Functions

usage() {
}

# Parse command-line options

while [ $# -gt 0 ]; do
    case $1 in
    *) usage ;;
    esac
    shift
done

# Filter pipeline

sed

• The general flow of your script should be to parse arguments and then to execute a pipeline.

• You will probably want to use sed and possibly cat.

• You may wish to create a function that will allow you to choose which command to run based on the -W flag and then use this function in the pipeline.

• Remember to remove trailing whitespace.

• Remember that sed can use alternative separators.

Task 2: test_broify.sh

To aid you in testing the broify.sh script, we provided you with test_broify.sh, which you can use as follows:

$ ./test_broify.sh
Testing broify.sh ...
   Usage                ... Success
   BASE                 ... Success
   BASE -W              ... Success
   C++                  ... Success
   C++ -W               ... Success
   Resolv               ... Success
   Resolv -W            ... Success

   Score 2.50

Task 3: README.md

In your README.md, describe how you implemented the broify.sh script. In particular, briefly discuss:

1. How you parsed the command line arguments.

2. How you removed comments.

3. How you removed empty lines.

4. How the command line options affected your main text processing pipeline.

Activity 3: Zip Codes (4 Points)

Like the instructor, Stella hails from Orange County⁵, which is situated on the Best Coast and is the home of Mickey Mouse, John Wayne, No Doubt⁶, and StarCraft⁷ (among other things). Having never really left paradise ⁸, Stella doesn't really know much about other places in America. This is problematic as she has made many friends from all over the country, such as her roommate from Toledo, Ohio, or her other roommate from Jacksonville, Florida ⁹. Since she needs to figure out the zip codes to these unfamiliar places, she is creating a script called zipcode.sh, which scrapes the zip codes from the website Zip Codes To Go and allows her to list all the zip codes in a specific state or city.

For instance, using curl, she can view all the raw HTML for Indiana by doing the following:

$ curl -sk https://www.zipcodestogo.com/Indiana/

Because you are pretty good with regular expressions now, you decide to help Stella out with parsing this HTML and extracting the zip codes.

Task 1: zipcode.sh

The zipcode.sh script takes three possible flags:

$ ./zipcode.sh -h
Usage: zipcode.sh

  -c      CITY    Which city to search
  -s      STATE   Which state to search (Indiana)

If no CITY is specified, then all the zip codes for the STATE are displayed.

The -c flag takes a CITY argument, which specifies the city to search for within the STATE. If no CITY is specified, then the script should return all the zip codes in the STATE.

The -s flag takes a STATE argument, which specifies the STATE to search through. If no STATE is specified, then the script should assume the STATE is "Indiana".

Here are some examples of zipcode.sh in action:

# Show all Zip Codes from default state (Indiana)
$ ./zipcode.sh
46001
46011
46012
46013
...
47994
47995
47996
47997

# Show all Zip Codes in Notre Dame, Indiana
$ ./zipcode.sh -s Indiana -c "Notre Dame"
46556

Hints

Here is a skeleton you can use to start your zipcode.sh script:

#!/bin/sh

# Functions

usage() {
}

# Parse command-line options

URL=https://www.zipcodestogo.com/

while [ $# -gt 0 ]; do
    case $1 in
    *) usage ;;
    esac
    shift
done

# Filter pipeline(s)

curl -sk $URL

• The general flow of your script should be to parse arguments and then to execute different pipelines.

• You will probably want to use sed and grep and maybe awk.

• You will probably want to create different functions and use them as stages in your pipelines.

Task 2: test_zipcode.sh

To aid you in testing the zipcode.sh script, we provided you with test_zipcode.sh, which you can use as follows:

$ ./test_zipcode.sh
Testing zipcode.sh ...
   Usage                ... Success
   Default              ... Success
   IN                   ... Success
   ND, IN               ... Success
   SB, IN               ... Success
   CA                   ... Success
   OR, CA               ... Success
   LA, CA               ... Success
   NY                   ... Success
   BF, NY               ... Success
   NY, NY               ... Success

   Score 3.50

Task 3: README.md

In your README.md, describe how you implemented the zipcode.sh script. In particular, briefly discuss:

1. How you parsed the command line arguments.

2. How you extracted the zip codes.

3. How you filtered by STATE and CITY.

Guru Point (1 Point)

For extra credit, you are to setup SSH Keys and configure your GitLab account to utilize them. The following are resources on how to set this up:

• How to create your SSH Keys
• Generating an SSH Key
• How To Set Up SSH Keys

Basically, with SSH Keys properly configured, you can do things such as push and pull to GitLab without having to enter your password everytime and login to remote Unix machines without a password. This is handy for automating tasks where no human interaction is desirable.

Once you setup your SSH Keys, you will need to change your remote URL.

To get credit, you must show either a TA or the instructor a demonstration of either pushing to your repository or pulling from it using SSH.

Feedback

If you have any questions, comments, or concerns regarding the course, please provide your feedback at the end of your README.md.

Submission

To submit your assignment, please commit your work to the homework03 folder of your homework03 branch in your assignments GitLab repository:

#--------------------------------------------------
# BE SURE TO DO THE PREPARATION STEPS IN ACTIVITY 0
#--------------------------------------------------

$ cd homework03                           # Go to Homework 03 directory
...
$ $EDITOR caesar.sh                       # Edit script
$ git add caesar.sh                       # Mark changes for commit
$ git commit -m "homework03: activity 1"  # Record changes
...
$ $EDITOR broify.sh                       # Edit script
$ git add broify.sh                       # Mark changes for commit
$ git commit -m "homework03: activity 2"  # Record changes
...
$ $EDITOR zipcode.sh                      # Edit script
$ git add zipcode.sh                      # Mark changes for commit
$ git commit -m "homework03: activity 3"  # Record changes
...
$ $EDITOR README.md                       # Edit appropriate README.md
$ git add README.md                       # Mark changes for commit
$ git commit -m "homework03: README"      # Record changes
...
$ git push -u origin homework03           # Push branch to GitLab

Remember to create a merge request and assign the appropriate TA from the Reading 03 TA List.