Reading 03: Text Filtering

Everyone:

Next week, we will continue to explore shell scripting. This time, however, we will unlock the true power of the Unix philosophy by studying sophisticated pipelines for filtering text. Additionally, we will also learn to wield regular expressions to create nifty pattern matching tools.

Readings¶

The readings for this week are:

1. The Linux Command Line:

   • Chapter 19 - Regular Expressions
   • Chapter 20 - Text Processing

2. RegexOne

   Work through the Lessons in the Interactive Tutorial. You can use the Regexpr to visualize your regular expressions.

Optional Resources¶

• Interactive Regex Tester

• Linux Command Line Tutorial - Regular Expressions

• Regular Expressions - User Guide

• Digital Ocean Tutorials:

  • Using Grep & Regular Expressions to Search for Text Patterns in Linux

  • The Basics of Using the Sed Stream Editor to Manipulate Text in Linux

• Advanced Bash Scripting - Text Processing Commands

• The Unix School - Awk & Sed

• Sed - An Introduction and Tutorial

• Grep - An introduction to grep and egrep

• USEFUL ONE-LINE SCRIPTS FOR SED

Quiz¶

This week, your reading quiz is split into two sections: the first part is your normal dredd quiz, while the second part involves a series of pipelines.

Questions¶

Record the answers to the following Reading 03 Quiz questions in your reading03 branch:

Filters¶

For the second part, you are to complete the following pipelines:

1. Convert all the input text to upper case:

   $ echo "All your base are belong to us" | ...
   ALL YOUR BASE ARE BELONG TO US
   

2. Find and replace all instances of monkeys to gorillaz:

   $ echo "monkeys love bananas" | ...
   gorillaz love bananas
   

3. Remove any leading whitespace from a string of text:

   $ echo "     monkeys love bananas" | ...
   monkeys love bananas
   

4. Parse the /etc/passwd file for the shell of the root user:

   $ curl -sL https://yld.me/raw/yWh | ...
   /bin/bash
   

   Hint: You may need to read up on the format of /etc/passwd

5. Find and replace all instances of /bin/bash, /bin/csh, and /bin/tcsh to /usr/bin/python in /etc/passwd:

   $ curl -sL https://yld.me/raw/yWh | ... | grep python
   root:x:0:0:root:/root:/usr/bin/python
   mysql:x:27:27:MySQL Server:/var/lib/mysql:/usr/bin/python
   xguest:x:500:501:Guest:/home/xguest:/usr/bin/python
   condor:x:108172:40:Condor Batch System:/afs/nd.edu/user37/condor:/usr/bin/python
   lukew:x:522:40:Luke Westby temp access:/var/tmp/lukew:/usr/bin/python
   

6. Find all the records in /etc/passwd that have a number that begins with a 4 and ends with a 7:

   $ curl -sL https://yld.me/raw/yWh | ...
   rtkit:x:499:497:RealtimeKit:/proc:/sbin/nologin
   qpidd:x:497:495:Owner of Qpidd Daemons:/var/lib/qpidd:/sbin/nologin
   uuidd:x:495:487:UUID generator helper daemon:/var/lib/libuuid:/sbin/nologin
   mailnull:x:47:47::/var/spool/mqueue:/sbin/nologin
   

Template¶

To record your pipelines, you will need to write your answers to the filters.sh script:

# Download filters template
$ curl -LO https://github.com/nd-cse-20289-sp21/cse-20289-sp21-assignments/raw/master/reading03/filters.sh

Each question has a corresponding function; for instance, the solution for Question 1 should be recorded in q1_answer:

#!/bin/bash

q1_answer() {
    # TODO: Complete pipeline
    echo "All your base are belong to us" | ...
}

To construct your pipelines, you should try them interactively in your shell. Once you found a reasonable solution, you can record them in the filters.sh script under the appropriate function.

Makefile¶

To test your pipelines, you need to download the Reading 03 Makefile, which in turn will download the test_filters.sh test script and execute it:

# Download Makefile
$ curl -LO https://raw.githubusercontent.com/nd-cse-20289-sp21/cse-20289-sp21-assignments/master/reading03/Makefile

# Download test scripts and run test
$ make
...

# Test your filters
$ ./test_filters.sh
Testing filters.sh ...
     Q1 Success
     Q2 Success
     Q3 Success
     Q4 Success
     Q5 Success
     Q6 Success
  Score 3.00

Submission¶

To submit your work, follow the same process outlined in Reading 01:

$ git checkout master                 # Make sure we are in master branch
$ git pull --rebase                   # Make sure we are up-to-date with GitHub

$ git checkout -b reading03           # Create reading03 branch and check it out

$ cd reading03                        # Go into reading03 folder
$ $EDITOR answers.json                # Edit your answers.json file

$ ../.scripts/check.py                # Check reading03 quiz
Checking reading03 quiz ...
      Q1 1.00
   Score 1.00 / 1.00
  Status Success

$ git add answers.json                # Add answers.json to staging area
$ git commit -m "Reading 03: Quiz"    # Commit work

$ $EDITOR filters.sh                  # Edit your filters.sh file

$ ./test_filters.sh                   # Test your filters
Testing filters.sh ...
     Q1 Success
     Q2 Success
     Q3 Success
     Q4 Success
     Q5 Success
     Q6 Success
  Score 3.00

$ git add Makefile                    # Add Makefile to staging area
$ git add filters.sh                  # Add filters.sh to staging area
$ git commit -m "Reading 03: Filters" # Commit work

$ git push -u origin reading03        # Push branch to GitHub