This project has an individual component and one in teams of 2-3persons.
For this project, you will design, implement, and thoroughly test the runtime system for Google MapReduce.
The project would be done a part in the class workshops and a part outside of the class.
Your project grade will be based on the quality of your report, on the usefulness of the system you’ve built, on the extent to which your design is a good fit for the problem you’re solving, and the quality of the code submitted.
This project consists of the next modules:
- Homework – Individual
- Applications on top of MapReduce
- MapReduce Distributed Files System on top of Azure Services
- MapReduce Master
- Communication Patterns
- Worker Creation
- Map Reduce Workers
- Completed MapReduce Runtime
Libraries and tools that may be handy
Azure HDInsight, Azure Blob storage, ZMQ, Hadoop MapReduce, ssh, wget, scp, Google Protobuf