Reading 06: Processing Data

Readings

The readings for Wednesday, March 1 are:

  1. Map, Filter, Lambda, and List Comprehensions in Python

  2. Iterators & Generators

  3. MapReduce: Simplified Data Processing on Large Clusters

TL;DR

The focus of this reading is to introduce you to functional programming in Python and MapReduce.

Optional Resources

  1. A practical introduction to functional programming

  2. Comprehensions in Python the Jedi way

  3. Functional Programming HOWTO

  4. Iterators, generator expressions and generators

  5. Introduction to Python Generators

  6. Generator Tricks for Systems Programmers

Questions

  1. Given the following Python script, evens.py:

    #!/usr/bin/env python2.7

import sys

results = []
for number in sys.stdin:
    number = number.strip()
    if int(number) % 2 == 0:
        results.append(number)

print ' '.join(results)

    A. Create a second version of this script called evens_fp.py that re-implements the evens.py script in one line using map, filter, and lambda.

    B. Create a third version of this script called evens_lc.py that re-implements the evens.py script in one line using list comprehensions.

    Note, both scripts should look like this:

    #!/usr/bin/env python2.7

import sys

print ' '.join(...)

    Where the ... is where your one line should go.

    C. Create a fourth version of this script called evens_gr.py that re-implements the evens.py script by creating a generator function using the yield keyword.

    This script should look like this:

    #!/usr/bin/env python2.7

import sys

def evens(stream):
    ...

print ' '.join(evens(sys.stdin))

    Where the ... is where your generator code should go.

    To test your scripts, you should be able to reproduce the following output:

    $ seq 1 10 | ./evens_fp.py
2 4 6 8 10

$ seq 1 10 | ./evens_lc.py
2 4 6 8 10

$ seq 1 10 | ./evens_gr.py
2 4 6 8 10

  2. In your reading06/README.md file, answer the following questions:

    A. What problem is MapReduce trying to solve?

    B. Describe the three phases of a typical MapReduce workflow.

Feedback

If you have any questions, comments, or concerns regarding the course, please provide your feedback at the end of your response.

Submission

To submit your assignment, please commit your work to the reading06 folder in your assignments GitLab repository. Your reading06 folder should only contain the following files: