Homework 03: Regex, Filters

The goal of this homework assignment is to allow you to practice using regular expressions and filters in shell scripts. In this assignment, you will build on your knowledge of the bourne shell language to write scripts that slice and dice data.

For this assignment, record your scripts and any responses to the following activities in the in the homework03 folder of your assignments GitLab repository and push your work by 11:59 PM Friday, February 10, 2017.

Activity 1: Caesar Cipher (5 Points)

In light of the Apple's letter to its customers last year and the recent announcement of Yahoo secretly scanning customer emails for US intelligence agencies, Steve has decided it is time to take security and cryptography seriously. Fearful of government surveillance, he begins implementing a script, caesar.sh, that implements a state-of-the-art 1 cipher technique know as the Caesar cipher. In this encryption technique letters are shifted by a fixed amount as shown below:

In the example above, letters are shifted 3 to the left (or rather shifted 23 to the right). So the string FARK THE NSA! becomes CXOH QEB KPX!. To reverse the string, you simply perform the same operation, except you shift in the opposite direction.

This encryption process can be described mathematically as:

Similarly, the decryption process is defined as:

Since you've already taken Discrete Math and are familiar with modular arithmetic, you decide to help Steve implement caesar.sh.

Task 1: caesar.sh

The caesar.sh script takes one possible argument and then reads input from STDIN:

$ ./caesar.sh -h
Usage: caesar.sh [rotation]

This program will read from stdin and rotate (shift right) the text by
the specified rotation.  If none is specified, then the default value is
13.

The one possible argument is to be used as the key or the amount to shift the letters (in this case, we will perform right shifts).

Here are some examples of caesar.sh in action:

$ echo "I came, I saw, I conquered." | ./caesar.sh
V pnzr, V fnj, V pbadhrerq.

$ echo "I came, I saw, I conquered." | ./caesar.sh | ./caesar.sh
I came, I saw, I conquered.

$ echo "Experience is the teacher of all things." | ./caesar.sh 10
Ohzobsoxmo sc dro dokmrob yp kvv drsxqc.

$ echo "Experience is the teacher of all things." | ./caesar.sh 10 | ./caesar.sh 16
Experience is the teacher of all things.

$ echo "It is better to create than to learn! Creating is the essence of life." | ./caesar.sh 40
Wh wg pshhsf hc qfsohs hvob hc zsofb! Qfsohwbu wg hvs sggsbqs ct zwts.

$ echo "It is better to create than to learn! Creating is the essence of life." | ./caesar.sh 40 | ./caesar.sh 12
It is better to create than to learn! Creating is the essence of life.

Hints

Task 2: test_caesar.sh

To aid you in testing the caesar.sh script, we are providing you with test_caesar.sh, which you can use as follows:

# Download script
$ curl -O http://www3.nd.edu/~pbui/teaching/cse.20289.sp17/static/sh/test_caesar.sh

# Make script executable
$ chmod +x test_caesar.sh

# Run test script
$ ./test_caesar.sh
caesar.sh test successful!

Task 3: README.md

In your README.md, describe how you implemented the caesar.sh script. In particular, briefly discuss:

  1. How you parsed the command line arguments.

  2. How you constructed the source set (ie. SET1).

  3. How you constructed the target set (ie. SET2).

  4. How you used both of these sets to perform the encryption.

Activity 2: Broification (5 Points)

Winston is a bit of a brogrammer and attempts to show off his 1337 skills by making his life harder than necessary. One way this manifests itself is in his habit of removing comments from all his code and configuration files. After all, according to Winston, "comments are for noobs". Additionally, he finds blank lines wasteful and likes to strip any lines consisting of only whitespace out of his files because, you know, those 4 bytes really matter. Finally, Winston also removes any trailing whitespace in this files2.

For instance, given the following file:

# Super useful comment describing a tricky configuration option
SETUP_THE_BOMB="no" # Hide the evidence

# Another super userful comment
BASE_OWNERSHIP="us" # All your base are belong to...

Winston would broify the file down to the following:

SETUP_THE_BOMB="no"
BASE_OWNERSHIP="us"

Unfortunately, Winston's bravado is just a facade and he is no better at programming than those Mendoza kids who keep asking you to join their social networking startup3. Although you are not supporter of brogrammer culture, you decide to help Winston out4 by writing the script broify.sh that removes comments and empty lines from files.

Task 1: broify.sh

The broify.sh script takes two possible flags and then reads input via STDIN:

$ ./broify.sh -h
Usage: broify.sh

  -d DELIM    Use this as the comment delimiter.
  -W          Don't strip empty lines.

The -d flag takes a DELIM argument which is used at the comment delimiter. By default this is #. The DELIM is used to indicate the start of the comment block (we are only concerned with line-based comments that begin with the DELIM and go to the end of the line).

When the -W flag is enabled, then the script should not strip empty lines. By default, the broify.sh script will remove any empty lines.

Here are some examples of broify.sh in action:

$ ./broify.sh < /etc/resolv.conf
domain cse.nd.edu
search cse.nd.edu
nameserver 66.205.160.99
nameserver 129.74.250.99

$ ./broify.sh -d '//' <<EOF
// C++ is so cool
int main() {
    // Totes
    return 0;
}
EOF
int main() {
    return 0;
}

$ ./broify.sh -d '//' -W <<EOF
// C++ is so cool
int main() {
    // Totes
    return 0;
}
EOF

int main() {

    return 0;
}

Hints

Task 2: test_broify.sh

To aid you in testing the broify.sh script, we are providing you with test_broify.sh, which you can use as follows:

# Download script
$ curl -O http://www3.nd.edu/~pbui/teaching/cse.20289.sp17/static/sh/test_broify.sh

# Make script executable
$ chmod +x test_broify.sh

# Run test script
$ ./test_broify.sh
broify.sh test successful!

Task 3: README.md

In your README.md, describe how you implemented the broify.sh script. In particular, briefly discuss:

  1. How you parsed the command line arguments.

  2. How you removed comments.

  3. How you removed empty lines.

  4. How the command line options affected your operations above.

Activity 3: Zip Codes (5 Points)

Like the instructor, Stella hails from Orange County5, which is situated on the Best Coast and is the home of Mickey Mouse, John Wayne, No Doubt6, and StarCraft7 (among other things). Having never really left paradise, Stella doesn't really know much about other places in America. This is problematic as she has made many friends from all over the country, such as her roommate from Toledo, Ohio, or her other roommate from Jacksonville, Florida 8. Since she needs to figure out the zip codes to these unfamiliar places, she is creating a script called zipcode.sh, which scrapes the zip codes from the website Zip Codes To Go and allows her to list all the zip codes in a specific state or city.

For instance, using curl, she can view all the raw HTML for Indiana by doing the following:

$ curl -s http://www.zipcodestogo.com/Indiana/

Because you are pretty good with regular expressions now, you decide to help Stella out with parsing this HTML and extracting the zip codes.

Task 1: zipcode.sh

The zipcode.sh script takes three possible flags:

$ ./zipcode.sh -h
Usage: zipcode.sh

  -c      CITY    Which city to search
  -f      FORMAT  Which format (text, csv)
  -s      STATE   Which state to search (Indiana)

If not CITY is specified, then all the zip codes for the STATE are displayed.

The -c flag takes a CITY argument, which specifies the city to search for within the STATE. If no CITY is specified, then the script should return all the zip codes in the STATE.

The -f flag takes a FORMAT argument which can be either text or csv. If the FORMAT is text, it should output the zip codes one line at a time. If the FORMAT is csv, it should output the zip codes as in CSV format.

The -s flag takes a STATE argument, which specifies the STATE to search through. If no STATE is specified, then the script should assume the STATE is "Indiana".

Here are some examples of zipcode.sh in action:

# Show all Zip Codes from default state (Indiana)
$ ./zipcode.sh
46001
46011
46012
46013
...
47994
47995
47996
47997

# Show all Zip Codes in Notre Dame, Indiana
$ ./zipcode.sh -s Indiana -c "Notre Dame"
46556

# Show all Zip Codes in South Bend, Indiana in CSV format
$ ./zipcode.sh -s Indiana -c "South Bend" -f csv
46601,46604,46613,46614,46615,46616,46617,46619,46620,46624,46626,46628,46634,46635,46637,46660,46680,46699

Hints

Task 2: test_zipcode.sh

To aid you in testing the zipcode.sh script, we are providing you with test_zipcode.sh, which you can use as follows:

# Download script
$ curl -O http://www3.nd.edu/~pbui/teaching/cse.20289.sp17/static/sh/test_zipcode.sh

# Make script executable
$ chmod +x test_zipcode.sh

# Run test script
$ ./test_zipcode.sh
zipcode.sh test successful!

We'll Do It Live

Because the data is being pulled from a remote website, the tests might take a while (but no more than 30 seconds).

Task 3: README.md

In your README.md, describe how you implemented the zipcode.sh script. In particular, briefly discuss:

  1. How you parsed the command line arguments.

  2. How you extracted the zip codes.

  3. How you filtered by STATE and CITY.

  4. How you handled the text and csv FORMAT options.

Guru Point (1 Point)

For extra credit, you are to setup SSH Keys and configure your GitLab account to utilize them. The following are resources on how to set this up:

Basically, with SSH Keys properly configured, you can do things such as push and pull to GitLab without having to enter your password everytime and login to remote Unix machines without a password. This is handy for automating tasks where no human interaction is desirable.

To get credit, you must show either a TA or the instructor a demonstration of either pushing to your repository or pulling from it using SSH.

Feedback

If you have any questions, comments, or concerns regarding the course, please provide your feedback at the end of your README.md.

Submission

To submit your assignment, please commit your work to the homework03 folder in your assignments GitLab repository. Your homework03 folder should only contain the following files:

  1. State-of-the-art circa 0 AD. ↩

  2. Okay. This is a legit pet peeve. ↩

  3. I get asked this too. Awkward. ↩

  4. This is mainly because Winston promised to use his flex points to get you some nachos and a smoothie at Reckers. ↩

  5. Don't you dare call it the OC. ↩

  6. Don't speak, I know what you're thinking; I don't need your reasons; Don't tell me cause it hurts. ↩

  7. 1-A to victory. Zerg OP. ↩

  8. This for you WenTao and Pat. 210 Dillon Hall 4 Lyfe. ↩