[SOLVED] CS4513 -  Distributed Systems - Project final

Description

Project final: Bring Your Own Project!

This is a group project.

Our last project is designed to provide you with more flexibility to explore. After all, this is likely to be one of your last projects done at WPI given that you are graduating! Even though this course is coming to an end, there are still so much more to learn about distributed systems. One of the key goals of this project is to convey this sentiment and to provide a high-level roadmap for your own adventure.

To help you get started, we provide the following three potential options for you to choose from. Each option includes recommended workflow and examples. Of course, you are welcome and encouraged to propose and come up with your own ultimate project—provided it is related to distributed systems.

Option 1: Implementation

If you are interested in gaining more experiences with Go beyond the Project MapReduce, you probably will like this option. At the high level, this option is about adding more features to the simple MapReduce framework you have implemented previously.

For this option, the recommended steps are following:

1. Read the original MapReduce paper from Google to gain deeper understanding about the design.
2. Pick one feature that interests you the most and sketch out what you need to do to incorporate the said feature into your existing MapReduce framework.
3. Design a few corresponding test cases.
4. Implement the feature based on the test cases.

Option 2: Case Study

If you are interested in seeing how classical distributed systems techniques are being used in modern distributed systems, then you might like this option. At the high level, you will pick an open-source distributed system and do a deep dive of its internal with the key goal of identifying which techniques are used and for what.

Here are a list of modern distributed systems that you might find interesting to explore:

1. Apache Spark is an in-memory data processing framework. Github link: https://github.com/ apache/spark
2. Alluxio which is a virtual distributed storage system. Github link: https://github.com/Alluxio/ alluxio
3. Pytorch Distributed which allows training deep learning models on different workers. Github link:

https://github.com/pytorch/pytorch/tree/master/torch/distributed

Option 3: Measurement

At the high level, you will pick a distributed systems related technology, learn how to use it, and measure the performance. For example, if you want to know more about grpc which is Google’s RPC system, the recommended steps are following:

1. spend some time reading the docs for grpc: https://www.grpc.io/docs/
2. make sure you understand how to measure event times with precision with functions clock gettime() and gettimeofday().
3. think about performance metrics that are important for RPC
4. come up with a list of factors such as data types and network conditions that might lead to different RPC performance.
5. vary each factor while keeping the other factors constant to perform the corresponding measurements.
6. (Optional) If you are really interested in RPC and are curious about how grpc compares to other systems such as Apache Thrift, then you can repeat the above steps and do a shoulder-to-shoulder comparison.^[1]

Here are a list of distributed systems technology that you might find interesting to measure:

1. Any cloud-based services, such as Virtual Machines, Containers, and serverless: as a first-time user, you can get one-year free access to most of AWS services.
2. Any open-source distributed systems: including the ones mentioned in the Option 2: Case Study.