The third project is to build an event-driven pub/sub library (ie. libps_client.a) using POSIX threads and network sockets. In a pub/sub system there is usually one server and multiple clients:

As shown above, a typical pub/sub system has a server who maintains a collection of topics, which serve as endpoints for messages. Clients connect to this server and perform the following operations:

  1. IDENTIFY: This identifies and authenticates the client to the server.

  2. PUBLISH: This posts a message to a particular topic queue.

  3. SUBSCRIBE: This notifies the server that a client is interested in a particular topic.

  4. RETRIEVE: This fetches all the messages destined for the client.

  5. DISCONNECT: This notifies the server that the client is disconnecting.

In the diagram, Client A connects to the Server and first performs an IDENTIFY command with its name and nonce or key. It then publishes a message to the TOPIC_A endpoint and then disconnects.

Client B, on the other hand, also performs an IDENTIFY operation and the proceeds to SUBSCRIBE to TOPIC_A and then RETRIEVE any messages for it. If Client B had subscribed to TOPIC_A before Client A published its message, then Client B would receive the message during this RETRIEVE operation.

Working in groups of one or two people, you are to create a library that utilizes POSIX threads and concurrent data structures to implement a client for the described pub/sub system by midnight on Tuesday, October 10, 2017. Additionally, you are to utilize this library to build and demonstrate a distributed and parallel application of your choosing by Thursday, October 26, 2017.

More details about this project and your deliverables are described below.

Publication / Subscriber

As mentioned in class, pub/sub systems are used on many real-world platforms such as Google Cloud and Amazon Web Services. These systems allow developers to construct distributed and parallel applications that operate concurrently by utilizing both message passing and event-driven programming paradigms.


Due to the limited time frame, the server is provided to you and located in AFS:

# Usage
$ /afs/nd.edu/user15/pbui/pub/bin/ps_server -h
Usage: /afs/nd.edu/user15/pbui/pub/bin/ps_server [options]

General Options:
      -h         Print this help message
      -p PORT    Port to listen on (default: 9611)

# Start Server on port 9411
$ /afs/nd.edu/user15/pbui/pub/bin/ps_server -p 9411
[1507080093] INFO  Listening on port 9411


The communication between the client and server follows the simple text-based request and respond protocol described below:

Client Requests Server Responds



IDENTIFY pbui 123

  • If the user has identified previously and the $NONCE does not match the recorded entry, then the server will respond with:
    402 Nonce $NONCE does not match for $USER_ID
  • If the user has not been identified yet, or the $NONCE matches the recorded entry match, then the server will respond with:
    200 Identified as $USER_ID with $NONCE



SUBSCRIBE hot_topic

  • If the user is already subscribed to $TOPIC, then the server will respond with:
    403 Already subscribed to $TOPIC
  • Otherwise, the server will respond with:
    200 Subscribed to $TOPIC




  • If the user is not subsribed to $TOPIC, then the server will respond with:
    404 Already unsubscribed from $TOPIC
  • Otherwise, the server will respond with:
    200 Unsubscribed from $TOPIC




PUBLISH hot_topic 12

hello, world

  • The server will respond with:
    200 Published $LENGTH bytes to $TOPIC




  • The server will respond with one message from the client's queue:
    Note, there is no newline after the $BODY.




  • If the user has identified previously and the $NONCE does not match the recorded entry, then the server will respond with:
    402 Nonce $NONCE does not match for $USER_ID
  • If the user has been identified and the $NONCE matches the recorded entry match, then the server will respond with:
    200 Disconnected $USER_ID with $NONCE

Some notes about this protocol:

Furthermore, in this pub/sub system, the server will handle as many clients as system resources allow and will continuously process requests until the client disconnects (implicitly or explicitly).

Client Library

The main goal of this project is to create a client library (ie. lib/libps_client.a) that communicates to a pub/sub server as described above. This library will be used by a variety of test programs and an user application.

Application Programming Interface (API)

The client library must implement a Client class with the following methods:

  1. This constructs a Client object with the specified server host, server port, and client identify cid.

    Client::Client(const char *host, const char *port, const char *cid)
  2. This schedules a message with the specified topic, message, and length to be published to the server.

    void Client::publish(const char *topic, const char *message, size_t length)
  3. This schedules a subscription to the topic be sent to the server, and records a Callback for that particular topic.

    void Client::subscribe(const char *topic, Callback *callback)
  4. This schedules an unsubscribe operation from the topic to be sent to the server, and removes all Callbacks for that particular topic.

    void Client::unsubscribe(const char *topic)
  5. This schedules a disconnect operation and causes the client to shutdown.

    void Client::disconnect()
  6. This starts any required background messaging threads and then processes incoming messages by calling the appropriate Callback on any messages until it is time to shutdown.

    void Client::run()
  7. This returns whether or not the client should shutdown.

    bool Client::shutdown()

Additionally, the library must provide a Callback class with the following methods:

  1. This is a virtual method that all derived classes must implement. It is called whenever the Callback should be be used to process a Message.
    void Callback::run(Message &m);

For instance, the library should provide the following EchoCallback as used in the bin/echo_test program:

class EchoCallback : public Callback {
    void run(Message &m) {

Likewise, the library must provide a Message struct with the following fields:

std::string topic       // Message topic
std::string sender      // Message sender
size_t      nonce       // Sender's nonce
std::string body        // Message body

Finally, the library must provide a Thread class with the following methods:

  1. This starts the thread with the specified func and arg.

    void Thread::start(thread_func func, void *arg);
  2. This joins the thread and stores the return value in the result.

    void Thread::join(void **result);
  3. This detaches the thread so it can run without needing to be joined.

    void Thread::detach();


The client library should utilize multiple POSIX threads to enable concurrent publishing and retrieving messages. This means that at any one time, the library should be able to do the following things concurrently:

  1. Publishing: The library should be able to publish any messages that have been queued up.

  2. Retrieving: The library should be able to retrieve any messages that the server has available to the client.

  3. Callbacks: The library should able to process any retrieved messages by passing the retrieved message to any matching callback handlers.

Because of this, you should have at least 3 POSIX threads. To pass messages between these threads, you should utilize concurrent data structures.

Back to the Queue

Hide most of the complexity of multi-threaded programming by using concurrent data structures such as the queue we created in Lecture 11: Condition Variables. Note, you can use any combination of locks, condition variables, and semaphores to synchronize the threads in your client library.


To test the client library, we have provided the echo_test program, which is located in AFS:

# Run echo client test on pub/sub server on port 9611
$ /afs/nd.edu/user15/pbui/pub/bin/echo_test localhost 9611 `whoami`

To further test your client library, you will need to implement unit tests using the Google Test framework as described below.

User Application

Once you have implemented the client library and are confident in your testing, you must create a distributed and parallel application that utilizes the pub/sub system. Here are some possible applications:

  1. Real-time message service (ie. chat)

  2. System monitoring service (ie. nagios)

  3. Parallel data processing service (ie. filtering, aggregation)

  4. Internet-of-Things messaging (eg. send messages between some Raspberry Pi's)

  5. Web crawler.

The exact user application is up to you. However, the application must utilize your client library and must utilize multiple threads correctly.


As noted above, you are to work in groups of one or two (three is permitted, but discouraged) to implement pq. You must use C++ as the implementation language. Any test scripts or auxillary tools can be written in any reasonable scripting language.


Here is a timeline of events related to this project:

Date Event
Tuesday, October 03 Project description and repository are available.
Tuesday, October 10 Client library is due (pushed to GitLab).
Tuesday, October 24 User application is due (pushed to GitLab).
Thursday, October 26 Demonstrations of application must be completed.


To start this project, one group member must fork the Project 03 repository on GitLab:


Once this repository has been forked, follow the instructions from Reading 00 to:

  1. Make the repository private

  2. Configure access to the repository

    Make sure you add all the members of the team in addition to the instructional staff.

Source Code

As you can see, the base Project 03 repository contains a README.md file and the following folder hierarchy:

    \_  Makefile        # This is the project Makefile
    \_  bin             # This contains the binary executables
    \_  contrib
        \_ gtest        # This contains the Google Test library
    \_  include
        \_  ps_client   # This contains the ps_client header files
    \_  lib             # This contains the ps_client library
    \_  src
        \_  client      # This contains the ps_client source code
        \_  tests       # This contains any test source code / scripts
        \_  units

You must maintain this folder structure for your project and place files in their appropriate place.

To help you get started, we have provided you with a Makefile with all the necessary targets:

$ make                  # Builds lib/libps_client.a bin/echo_test bin/client_unit

$ make bin/echo_test    # Builds echo_test executable

$ make build/gtest      # Explicitly build Google Test framework

$ make clean            # Removes all targets and intermediate objects

You will need to modify this Makefile to support your user application.


While the exact organization of the project code is up to you, keep in mind that you will be graded in part on coding style, cleaniness, and organization. This means your code should be consistently formatted, not contain any dead code, have reasonable comments, and appropriate naming among other things:

Please refer to these Coding Style slides for some tips and guidelines on coding style expectations.


As part of your grade, you will need to present your user application to a TA where you will demonstrate the correctness of your pub/sub client library and the implementation of your user application.


While creating your user application, you should be creating tests in the form of functional tests (ie. bin/echo_test) and unit tests (ie. bin/client_unit).

The point of these tests is to ensure your application works correctly and to allow you to catch any possible bugs in your code. For instance, you may wish to test any concurrent data structures you created or to check how your client library handles message parsing and communication.

Ideally, each method or function you write would have a corresponding unit test or is covered by a functional test application.

To help with testing, we have included the Google Test framework in the Project 03 repository and have include some stubs you can expand to write your unit tests.

For this project, you must have both functional and unit tests and utilize them regularly during your application development process.

Test-Driven Development

Writing all the tests at the end of your development process defeats the purpose of the test-driven development. What you should do is try to implement one feature or function at a time and test it as you build it. Rather than trying to implement multiple pieces all at once, focus on one thing at a time and incrementally build your application.


As part of your demonstration, you must provide a presentation (between 5 - 10 slides) with the following content:

  1. Design: An overview of the design of your user application.

  2. Implementation: A summary of how you implemented the user application and utilized the pub/sub system.

  3. Testing: A discussion on how you tested your client library and your end user application.

  4. Paradigms: A discussion on how you used three different concurrency programming paradigms: message passing, threading, and events and what the challenge of each model is.

  5. Summary: A summary of what you learned.

Note, you should incorporate images, graphs, diagrams and other visual elements as part of your presentation where reasonable.

Please upload these slides to Google Drive and place a link in your README.md.

Be prepared to be asked about different aspects of your project, as the TA may ask you questions to probe your understanding of the material and your work.


As noted above, the Project 03 repository includes a README.md file with the following sections:

  1. Members: This should be a list of the project members.

  2. Design: This is a list of design questions that you should answer before you do any coding as they will guide you towards the resources you need.

  3. Demonstration: This is where you should provide a Google Drive link to your demonstration slides.

  4. Errata: This is a section where you can describe any deficiencies or known problems with your implementation.

You must complete this document report as part of your project.

Design and Implementation

You should look at the design questions in the README.md file of the Project 03 repository. The questions there will help you design and plan your process queue implementation.

Feel free to ask the TAs or the instructor for some guidance on these questions before attempting to implement this project.

Extra credit

Once you have completed your project, you may extend your implementation of libps_client by performing either (or both) of the following modifications:

  1. Graphical User Interface: Create a graphical application using a toolkit such as Qt that utilizes your libps_client.

  2. Scripting Interface: Create a language binding to your libps_client so that you can utilize the library from a scripting language such as [Python] or [Ruby] to replicate the echo_test program.

Each of these modifications is worth 1 Point of extra credit each.


Your project will be graded on the following metrics:

Metric Points
Source Code
  1. General
    • Builds and cleans without warnings or errors
    • Manages resources such as memory and files appropriately
    • Uses system calls appropriately
    • Is consistent, readable, and organized
  2. Client Library
    • Performs IDENTIFY properly
    • Performs PUBLISH properly
    • Performs SUBSCRIBE properly
    • Performs UNSUBSCRIBE properly
    • Performs RETRIEVE properly
    • Performs DISCONNECT properly
    • Implements Client run() method
    • Implements Client shutdown() method
    • Implements Callback class properly
    • Implements Thread class properly
    • Implements Message struct properly
    • Utilizes multiple threads for sending and receiving
    • Utilizes concurrent data structures
    • Free of concurrency bugs (race conditions, deadlocks, etc.)
  3. Testing
    • Echo Test performs correctly
    • Client Unit tests concurrent data structures
    • Client Unit tests Callback, Client, Thread, Message classes
  4. User Application
    • Utilizes multiple threads
    • Utilizes Client library for communication
    • Free of concurrency bugs (race conditions, deadlocks, etc.)
  1. 4.0
    • 1.0
    • 1.0
    • 1.0
    • 1.0
  2. 9.0
    • 0.5
    • 0.5
    • 0.5
    • 0.5
    • 0.5
    • 0.5
    • 0.75
    • 0.5
    • 0.25
    • 0.25
    • 0.25
    • 2.0
    • 1.0
    • 1.0
  3. 2.5
    • 1.0
    • 1.0
    • 0.5
  4. 2.5
    • 1.0
    • 0.5
    • 1.0
  1. Slides
  2. Design
  3. Implementation
  4. Testing
  5. Paradigms
  6. Summary
  1. 1.0
  2. 0.5
  3. 1.0
  4. 0.5
  5. 0.5
  6. 0.5
  1. Design
  1. 2.0