CSE 40822/60822
Cloud Computing - Fall 2018

MWF 10:30-11:20, Debartolo 319
Prof. Douglas Thain
Email: dthain at nd dot edu
Office: 384C Fitzpatrick Hall
Office Hours: Mon 2-4PM and Fri 3-5PM
TA: Haihui Tan
Email: htan2 at nd dot edu
Office: Cushing 212
Office Hours: T/R 1-3PM

Getting Help

  • Piazza Discussion Page - The best place for technical questions like "What does this error message mean?"
  • Office Hours Prof. Thain: Mon 2-4PM, Fri 3-5PM, Haihui: T/R 1-3PM
  • Email - Contact Prof. Thain for questions about grades, course policies, etc.
  • Grades - Are available on the course ND Sakai page.
  • Course Overview

    This class is an introduction to the theory and practice building large scale computer systems that harness hundreds or thousands for machines to attack problems of enormous scale. Such distributed systems are necessary to solve problems of such large size that they cannot complete in any reasonable time on a single machine. In recent years, these system have been known as clouds but have a much longer history known as distributed systems.

    Cloud computing encompasses a variety of modes of computing, including infrastructure and data center management, high throughput computing, distributed programming models, No-SQL storage, and more. We will take a tour of many of these topics by alternating with a high-level discussion of the principles, followed by case studies with a current technology.

    Each assignment will involve designing a program or system that scales up to a large number of machines, using a variety of technologies. This will be a highly practical class, and should be enjoyable to any student who likes to write lots of code and make real systems work. Many students who take this class end up using these tools in their daily work. The class is open to juniors, seniors, and graduate students.

    Course Documents

  • Syllabus
  • A0 - Warm Up Assignment
  • A1 - High Throughput Ray-Tracing with HTCondor
  • A2 - Parallel DNA Analysis with Work Queue
  • A3 - Web Data Processing with Map-Reduce
  • A4 - From Parallel to Serverless
  • Final Project
  • Progress Reports
  • Presentation
  • Tentative Schedule

    Week Lecture Topic Readings and Materials
    20 August The Cloud Landscape
    A View of Cloud Computing
    Exascale Computing and Big Data
    Going Serverless
    Edge Computing: Vision and Challenges
    Gartner 2017 Hype Cycle
    27 August Principles of Distributed Computing A Note on Distributed Computing
    A0: Due Friday
    3 September Case Study: HTCondor
    HTCondor Overview Paper
    HTCondor at Notre Dame
    HTCondor 8.0 Manual
    HTCondor Log Analyzer
    10 September Workflows and Makeflow
    (Workflows Lecture Online)
    Makeflow Web Page
    Makeflow Manual
    JX Tutorial
    JX Quick Reference
    A1: Due Friday
    17 September Distributed Programming with Work Queue
    (Work Queue Lecture Online)
    Work Queue Web Page
    Work Queue Manual
    Work Queue Example
    Bibliography Due Friday (Grads Only)
    24 September Map-Reduce and Hadoop
    (No Class Wed)
    Google Map-Reduce Paper
    Hadoop Paper
    Hadoop Project Web Page
    Hadoop at Notre Dame
    A2 Due Friday
    1 October The Hadoop Stack
    PigLatin Notes
    HBase Notes
    Pig Latin Paper (Apache Pig)
    BigTable Paper (Apache HBase)
    Project Proposals due Friday
    8 October The Hadoop Stack Continued
    A3 Due Friday
    15 October Fall Break
    22 October Spark Notes
    Amazon Notes
    Spark Paper (Apache Spark)
    Amazon AWS Docs
    Midterm Exam Friday
    29 October Amazon Notes Amazon AWS Docs
    Amazon Architecture Center
    5 November Scaling Up and Consistency
    Scaling Up Notes
    Consistency Notes
    Memcached Project
    Paper: CAP Theorem
    Paper: Eventually Consistent
    A4 Due Friday
    12 November Coordination and Configuration:
    Mon/Wed: No Class - Work on Projects
    Fri: Zookeeper Notes
    Zookeeper Paper(Apache Zookeeper)
    Progress Report due Friday
    19 November Coordination and Configuration:
    Mon: Puppet Notes
    Wed/Fri: Thanksgiving Break
    Puppet Docs
    26 November Project Presentations

    Monday:

    Wednesday: Friday:
    3 December Project Presentations

    Monday:

    Wednesday:
    Project Due Friday at Noon
    Wednesday
    12 December
    4:15-6:15PM
    Final Exam

    Sample Extra Credit Submissions