- Automated provisioning
- Elastic systems
Workshop: Map and Reduce functionality
Implement the functionality required for distributing the computation, running the handlers, and storing the results.
Intention: create the base code for the map and reduce functionality. The student should learn how to handle resources in the cloud.
Implement both the mapper and reducer code, using the DFS base code created in week 8.
Your MapReduce implementation should be able to:
- Have the master distribute the binaries for both the Map and Reduce phase.
- Be able to execute both the mapper and the reducer code in any worker.
- After executing the map phase, sort the mapper result in place, and store it locally. (Assume that the data fits in the RAM).
- Store the required information in the master to be able to fetch the required information from the mapper, to execute the Reduce phase.
- Store the final results into Azure Blob, you should be able to use this data as an input for a pipelined map/reduce computation.