Next week, we will continue to explore shell scripting. This time, however, we will unlock the true power of the Unix philosophy by studying sophisticated pipelines for filtering text. Additionally, we will also learn to wield regular expressions to create nifty pattern matching tools.


The focus of this reading is to introduce regular expressions and revisit filters and pipelines.


The readings for Monday, February 5 are:

  1. The Linux Command Line:

    • Chapter 19 - Regular Expressions
    • Chapter 20 - Text Processing
  2. RegexOne

    Work through the Lessons in the Interactive Tutorial. You can use the Regexpr to visualize your regular expressions.

Optional Resources


This week, your reading quiz is split into two sections: the first part is your normal dredd quiz, while the second part involves a series of pipelines.


Record the answers to the following Reading 03 Quiz questions in your reading03 branch:


For the second part, you are to complete the following pipelines:

  1. Convert all the input text to upper case:

    $ echo "All your base are belong to us" | ...
  2. Find and replace all instances of monkeys to gorillaz:

    $ echo "monkeys love bananas" | ...
    gorillaz love bananas
  3. Remove any leading whitespace from a string of text:

    $ echo "     monkeys love bananas" | ...
    monkeys love bananas
  4. Parse the /etc/passwd file for the shell of the root user:

    $ curl -sL https://yld.me/raw/yWh | ...

    Hint: You may need to read up on the format of /etc/passwd

  5. Find and replace all instances of /bin/bash, /bin/csh, and /bin/tcsh to /usr/bin/python in /etc/passwd:

    $ curl -sL https://yld.me/raw/yWh | ... | grep python
    mysql:x:27:27:MySQL Server:/var/lib/mysql:/usr/bin/python
    condor:x:108172:40:Condor Batch System:/afs/nd.edu/user37/condor:/usr/bin/python
    lukew:x:522:40:Luke Westby temp access:/var/tmp/lukew:/usr/bin/python
  6. Find all the records in /etc/passwd that have a number that begins with a 4 and ends with a 7:

    $ curl -sL https://yld.me/raw/yWh | ...
    qpidd:x:497:495:Owner of Qpidd Daemons:/var/lib/qpidd:/sbin/nologin
    uuidd:x:495:487:UUID generator helper daemon:/var/lib/libuuid:/sbin/nologin

To record your pipelines, you will need to write your answers to the filters.sh script:

# Download filters template
$ curl -LO https://www3.nd.edu/~pbui/teaching/cse.20289.sp18/static/sh/filters.sh

Each question has a corresponding function; for instance, the solution for Question 1 should be recorded in q1_answer:


q1_answer() {
    # TODO: Complete pipeline
    echo "All your base are belong to us" | ...

To construct your pipelines, you should try them interactively in your shell. Once you found a reasonable solution, you can record them in the filters.sh script under the appropriate function.

To test your pipelines, you need to download the test_filters.sh script and create a Makefile in your reading03 folder:

# Download test script and make it executable
$ curl -LO https://www3.nd.edu/~pbui/teaching/cse.20289.sp18/static/sh/test_filters.sh
$ chmod +x test_filters.sh

# Create Makefile (you can use your favorite $EDITOR instead)
$ cat > Makefile <<EOF

# Test your filters
$ ./test_filters.sh
Testing filters.sh ...
     Q1 Success
     Q2 Success
     Q3 Success
     Q4 Success
     Q5 Success
     Q6 Success
  Score 3.00


To submit you work, follow the same process outlined in Reading 01:

$ git checkout master                 # Make sure we are in master branch
$ git pull --rebase                   # Make sure we are up-to-date with GitLab

$ git checkout -b reading03           # Create reading03 branch and check it out

$ cd reading03                        # Go into reading03 folder
$ $EDITOR answers.json                # Edit your answers.json file

$ ../.scripts/submit.py               # Check reading03 quiz
Submitting reading03 assignment ...
Submitting reading03 quiz ...
  Score 1.00

$ git add answers.json                # Add answers.json to staging area
$ git commit -m "Reading 03: Quiz"    # Commit work

$ $EDITOR filters.sh                  # Edit your filters.sh file

$ ./test_filters.sh                   # Test your filters
Testing filters.sh ...
     Q1 Success
     Q2 Success
     Q3 Success
     Q4 Success
     Q5 Success
     Q6 Success
  Score 3.00

$ git add Makefile                    # Add Makefile to staging area
$ git add test_filters.sh             # Add test_filters.sh to staging area
$ git add filters.sh                  # Add filters.sh to staging area
$ git commit -m "Reading 03: Filters" # Commit work

$ git push -u origin reading03        # Push branch to GitLab

Remember to create a merge request and assign the appropriate TA from the Reading 03 TA List.