Homework 05: Regex, Filters

The goal of the fifth homework assignment is to allow you to practice using regular expressions and filters in shell scripts. In this assignment, you will build on your knowledge of the bourne shell language to write scripts that slice and dice data.

For this assignment, record your scripts and any responses to the following activities in the in the homework05 folder of your assignments Bitbucket repository and push your work by 11:59 PM Friday, February 26, 2016.

Activity 1: Caesar Cipher (5 Points)

In light of the Apple's recent letter to its customers, Steve has decided it is time to take security and cryptography seriously. Fearful of government backdoors, he begins implementing a script, caesar.sh, that implements a state-of-the-art ¹ cipher technique know as the Caesar cipher. In this encryption technique letters are shifted by a fixed amount as shown below:

In the example above, letters are shifted 3 to the left (or rather shifted 23 to the right). So the string FARK THE NSA! becomes CXOH QEB KPX!. To reverse the string, you simply perform the same operation, except you shift in the opposite direction.

This encryption process can be described mathematically as:

Similarly, the decryption process is defined as:

Since you've already taken Discrete Math and are familiar with modular arithmetic, you decide to help Steve implement caesar.sh.

Task 1: `caesar.sh`

The caesar.sh script takes one possible argument and then reads input from STDIN:

./caesar.sh -h
usage: caesar.sh [rotation]

This program will read from stdin and rotate (shift right) the text by
the specified rotation.  If none is specified, then the default value is
13.

The one possible argument is to be used as the key or the amount to shift the letters (in this case, we will perform right shifts).

Here are some examples of caesar.sh in action:

$ echo "I came, I saw, I conquered." | ./caesar.sh
V pnzr, V fnj, V pbadhrerq.

$ echo "I came, I saw, I conquered." | ./caesar.sh | ./caesar.sh
I came, I saw, I conquered.

$ echo "Experience is the teacher of all things." | ./caesar.sh 10
Ohzobsoxmo sc dro dokmrob yp kvv drsxqc.

$ echo "Experience is the teacher of all things." | ./caesar.sh 10 | ./caesar.sh 16
Experience is the teacher of all things.

$ echo "It is better to create than to learn! Creating is the essence of life." | ./caesar.sh 40
Wh wg pshhsf hc qfsohs hvob hc zsofb! Qfsohwbu wg hvs sggsbqs ct zwts.

$ echo "It is better to create than to learn! Creating is the essence of life." | ./caesar.sh 40 | ./caesar.sh 12
It is better to create than to learn! Creating is the essence of life.

Hints

The general flow of your script should be to construct a source set and a target set and then use a command on these sets.
You may want to use tr and cut.
You may want to create functions to help you assemble the source and target sets.
Be sure to handle upper and lower case letters.
Be sure to handle rotations larger than 26.

Task 2: `test_caesar.sh`

To aid you in testing the caesar.sh script, we are providing you with test_caesar.sh, which you can use as follows:

# Download script
$ curl -O http://www3.nd.edu/~pbui/teaching/cse.20189.sp16/static/sh/test_caesar.sh

# Make script executable
$ chmod +x test_caesar.sh

# Run test script
$ ./test_caesar.sh
caesar.sh test successful!

Task 3: `README.md`

In your README.md, describe how you implemented the caesar.sh script. In particular, briefly discuss:

How you constructed the source set (ie. SET1).
How you constructed the target set (ie. SET2).
How you used both of these sets to perform the encryption.

Activity 2: Reddit Links (5 Points)

Jill likes to sit in the back of the class. It has its perks:

She can beat the rush out the door when class ends.
She can see everyone browsing Facebook, playing video games, watching YouTube, or doing schoolwork.
She feels safe from being called upon by the instructor... except when he does that strange thing where he goes around the class and tries to talk to people. Totally weird.

That said, sitting in the back has its downsides:

She can never see what the instructor is writing because he has terrible handwriting and always writes too small.
She is prone to falling asleep because the instructor is really boring and the class is not as interesting as Discrete Math was last semester.

To combat her boredom, Jill typically just browses Reddit. Her favorite subreddits are AdviceAnimals, aww, todayilearned, and of course UnixPorn. Jill is tired of having to go to each subreddit and browsing for cool links, however, and decides she wants to create a script, reddit.sh, that will display all the links on a given subreddit.

Fortunately for Jill, Reddit provides a JSON feed for every subreddit. You simply need to append .json to the end of each subreddit. For instance, the JSON feed for todayilearned can be found here:

https://www.reddit.com/r/todayilearned/.json

To fetch that data and format it in a usuable form, Jill does the following:

$ curl -s http://www.reddit.com/r/todayilearned/.json | python -m json.tool

Looking through that stream of text, Jill sees that there are some lines that contain the field "url":. She wants to extract the values associated with this field and display anything that begins with http.

Since you are a pro at regular expressions, you decide to help Jill out in writing the reddit.sh script.

Task 1: `reddit.sh`

The reddit.sh script takes three possible flags followed by a list of subreddits.

usage: reddit.sh [options] subreddits...

    -r      Shuffle the links
    -s      Sort the links
    -n N    Number of links to display (default is 10)

The -r flag means that the script should shuffle the output of the script. The -s flags means that the script should sort the output instead. If neither of these are set, then the order of the output is however the input is received. If both flags are set, then the last flag set takes precedence.

The -n flag takes an argument that is used to indicate how many items to display.

Here are some examples of reddit.sh in action:

$ ./reddit.sh linux
http://blog.linuxmint.com/?p=3001?
http://blog.linuxmint.com/?p=2994&amp;_utm_source=1-2-2
http://www.shashlik.io/news/2016/02/18/shashlik-0-9-0-kubuntu-package/
https://torrentfreak.com/software-piracy-hurts-linux-adoption-research-finds-160221/
https://xairy.github.io/blog/2016/cve-2016-2384
http://blog.elementary.io/post/139731273161/re-geary
https://micahflee.com/2016/02/backdoored-linux-mint-and-the-perils-of-checksums/
https://kodi.tv/kodi-16-0-jarvis-mark-xvi/
http://freedompenguin.com/articles/news/lessons-linux-mint-hack/
https://www.reddit.com/r/linux/comments/46xvv8/im_seeing_ads_for_linux_tablets_on_ali_express/

$ ./reddit.sh -r linux
http://freedompenguin.com/articles/news/lessons-linux-mint-hack/
http://blog.linuxmint.com/?p=3001?
http://i.imgur.com/9OndU3T.png
https://www.youtube.com/channel/UC_itDlR1_mWB0AWvah5l2Rg
http://www.linuxandubuntu.com/home/encrypt-data-in-linux-with-veracrypt-an-alternative-to-truecrypt
https://insights.ubuntu.com/2016/02/18/canonical-and-samsung-demonstrate-ubuntu-core-on-samsung-artik/
http://www.shashlik.io/news/2016/02/18/shashlik-0-9-0-kubuntu-package/
https://xairy.github.io/blog/2016/cve-2016-2384
http://www.youtube.com/watch?v=ki6NhykJwxU
https://www.youtube.com/

$ ./reddit.sh -s linux
http://blog.elementary.io/post/139731273161/re-geary
http://blog.linuxmint.com/?p=2994&amp;_utm_source=1-2-2
http://blog.linuxmint.com/?p=3001?
http://fortune.com/2016/02/18/ted-linus-torvalds/
http://freedompenguin.com/articles/news/lessons-linux-mint-hack/
http://ideas.ted.com/why-im-teaching-prisoners-to-code/
http://i.imgur.com/9OndU3T.png
http://lists.gnu.org/archive/html/coreutils/2016-02/msg00019.html
https://distrowatch.com/weekly.php?issue=20160222#zorin
https://flexion.org/posts/2014-03-memory-consumption-of-linux-desktop-environments/

Since this script works on live data from a website, we cannot provide an automated test script.

Hints

The general flow of your script should be to parse arguments and then to execute a pipeline for each command line argument.
You may want to use curl, python -m json.tool, sort, cat, cut, shuf, or head.
You may wish to create some functions to simplify parts of your pipeline.

Task 2: `README.md`

In your README.md, describe how you implemented the reddit.sh script. In particular, briefly discuss:

How you filtered the URLs and extracted only the relevant portion of the data.
How you managed the different ordering (ie. sort, shuffle, as-is) operations.
How the command line options affected your operations above.

Activity 3: Broification (5 Points)

Winston is a bit of a brogrammer and attempts to show off his 1337 skills by making his life harder than necessary. One way this manifests itself is in his habit of removing comments from all his code and configuration files. After all, according to Winston, "comments are for noobs". Additionally, he finds blank lines wasteful and likes to strip any lines consisting of only whitespace out of his files because, you know, those 4 bytes really matter. Finally, Winston also removes any trailing whitespace in this files².

For instance, given the following file:

# Super useful comment describing a tricky configuration option
SETUP_THE_BOMB="no" # Hide the evidence

# Another super userful comment
BASE_OWNERSHIP="us" # All your base are belong to...

Winston would broify the file down to the following:

SETUP_THE_BOMB="no"
BASE_OWNERSHIP="us"

Unfortunately, Winston's bravado is just a facade and he is no better at programming than those Mendoza kids who keep asking you to join their social networking startup³. Although you are not supporter of brogrammer culture, you decide to help Winston out⁴ by writing the script broify.sh that removes comments and empty lines from files.

Task 1: `broify.sh`

The broify.sh script takes two possible flags and then reads input via STDIN:

$ ./broify.sh -h
usage: broify.sh

  -d DELIM    Use this as the comment delimiter.
  -W          Don't strip empty lines.

The -d flag takes a DELIM argument which is used at the comment delimiter. By default this is #. The DELIM is used to indicate the start of the comment block (we are only concerned with line-based comments that begin with the DELIM and go to the end of the line).

When the -W flag is enabled, then the script should not strip empty lines. By default, the broify.sh script will remove any empty lines.

Here are some examples of broify.sh in action:

$ ./broify.sh < /etc/resolv.conf
domain cse.nd.edu
search cse.nd.edu
nameserver 66.205.160.99
nameserver 129.74.250.99

$ ./broify.sh < /etc/csh.login
if ( $?PATH ) then
else
  if ( $uid == 0 ) then
    setenv PATH "/sbin:/usr/sbin:/usr/local/sbin:/bin:/usr/bin:/usr/local/bin"
  else
    setenv PATH "/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin"
  endif
endif
setenv HOSTNAME `/bin/hostname`
set history=1000
if ( -d /etc/profile.d ) then
        set nonomatch
        foreach i ( /etc/profile.d/*.csh )
                if ( -r "$i" ) then
                          if ($?prompt) then
                                source "$i"
                          else
                                source "$i" >& /dev/null
                          endif
                endif
        end
        unset i nonomatch
endif

$ ./broify.sh -W < /etc/csh.login




if ( $?PATH ) then

else
  if ( $uid == 0 ) then
    setenv PATH "/sbin:/usr/sbin:/usr/local/sbin:/bin:/usr/bin:/usr/local/bin"
  else
    setenv PATH "/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin"
  endif
endif

setenv HOSTNAME `/bin/hostname`
set history=1000

if ( -d /etc/profile.d ) then
        set nonomatch
        foreach i ( /etc/profile.d/*.csh )
                if ( -r "$i" ) then
                          if ($?prompt) then
                                source "$i"
                          else
                                source "$i" >& /dev/null
                          endif
                endif
        end
        unset i nonomatch
endif

$ ./broify.sh -d '//' <<EOF
// C++ is so cool
int main() {
    // Totes
    return 0;
}
EOF
int main() {
    return 0;
}

$ ./broify.sh -d '//' -W <<EOF
// C++ is so cool
int main() {
    // Totes
    return 0;
}
EOF

int main() {

    return 0;
}

Hints

The general flow of your script should be to parse arguments and then to execute a pipeline.
You will probably want to use sed and possibly cat.
You may wish to create a function that will allow you to choose which command to run based on the -W flag and then use this function in the pipeline.
Remember to remove trailing whitespace.

Task 2: `test_broify.sh`

To aid you in testing the broify.sh script, we are providing you with test_broify.sh, which you can use as follows:

# Download script
$ curl -O http://www3.nd.edu/~pbui/teaching/cse.20189.sp16/static/sh/test_broify.sh

# Make script executable
$ chmod +x test_broify.sh

# Run test script
$ ./test_broify.sh
broify.sh test successful!

Task 3: `README.md`

In your README.md, describe how you implemented the broify.sh script. In particular, briefly discuss:

How you removed comments.
How you removed empty lines.
How the command line options affected your operations above.

Guru Point (1 Point)

For extra credit, you are practice using Git branches and performing a pull request on Bitbucket. Instructions on what to do can be found at the Guru Point 05 repository:

https://bitbucket.org/CSE-20189-SP16/gurupoint_05

To get credit, you have your pull request accepted by the instructor.

Feedback

If you have any questions, comments, or concerns regarding the course, please provide your feedback at the end of your README.md.

Submission

To submit your assignment, please commit your work to the homework05 folder in your assignments Bitbucket repository by 11:59 PM Friday, February 26, 2016. Your homework05 folder should only contain the following files:

README.md
caesar.sh
reddit.sh
broify.sh

State-of-the-art circa 0 AD. ↩
Okay. This is a legit pet peeve. ↩
I get asked this too. Awkward. ↩
This is mainly because Winston promised to use his flex points to get you some nachos and a smoothie at Reckers. ↩

Homework 05: Regex, Filters

Activity 1: Caesar Cipher (5 Points)

Task 1: caesar.sh

Hints

Task 2: test_caesar.sh

Task 3: README.md

Activity 2: Reddit Links (5 Points)

Task 1: reddit.sh

Hints

Task 2: README.md

Activity 3: Broification (5 Points)

Task 1: broify.sh

Hints

Task 2: test_broify.sh

Task 3: README.md

Guru Point (1 Point)

Feedback

Submission

Task 1: `caesar.sh`

Task 2: `test_caesar.sh`

Task 3: `README.md`

Task 1: `reddit.sh`

Task 2: `README.md`

Task 1: `broify.sh`

Task 2: `test_broify.sh`

Task 3: `README.md`