Week 9: Resource Management

Topics

  • Automated provisioning
  • Balancing.
  • Scheduling
  • Elastic systems

References

Workshop: Map and Reduce functionality

Definition

Implement the functionality required for distributing the computation, running the handlers, and storing the results.

Intention: create the base code for the map and reduce functionality. The student should learn how to handle resources in the cloud.

Specification

Implement both the mapper and reducer code, using the DFS base code created in week 8.
 Your MapReduce implementation should be able to:
  • Have the master distribute the binaries for both the Map and Reduce phase.
  • Be able to execute both the mapper and the reducer code in any worker.
  • After executing the map phase, sort the mapper result in place, and store it locally. (Assume that the data fits in the RAM).
  • Store the required information in the master to be able to fetch the required information from the mapper, to execute the Reduce phase.
  • Store the final results into Azure Blob, you should be able to use this data as an input for a pipelined map/reduce computation.
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s