- Distributed FileSystems
- Azure Blob Storage (involve azure developer with a guest lecture)
- NoSQL Databases
- Cassandra – A Decentralized Structured Storage System
- Dynamo: Amazon’s Highly Available Key-value Store
- Finding a needle in Haystack: Facebook’s photo storage
- Bigtable: A Distributed Storage System for Structured Data
- NoSQL on Microsoft Azure
- Azure Blob Storage
- Scalable SQL and NoSQL Data Stores
Workshop: Master Functionality
Intention: Teach how to use the distributed filesystems in Azure, and make the student think about the requirements of the framework. Developed the base code for the Master implementation. Create the handlers, interfaces and scoreboard required for the Master.
- Design the base interface and functionalities for Map Reduce DFS (Distributed File System) for moving and copying data, between the map and reduce phase, and to the final result.
- Implement the function that is going to shard the data and distributed among the M available resources.
Using Azure Blob storage and HDInsight implement the required interface and functionalities for the MapReduce Runtime.
The system should be able to:
- Distribute and give access to the input files to the mappers.
- Distribute the <key,value> pairs to the corresponding reducer
Using the functionality implemented in the previous week, distribute the data from the master to the workers. Additionally test sending the temporal data between the two workers, similar to what is going to be done between the map and the reduce phase.
Additionally, you should measure the difference between having the information locally and accessing the information through the Azure blob interface using C++