Difference between revisions of "MapReduce Frameworks Lab"
Jump to navigation
Jump to search
(→mapAll) |
|||
| (4 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
| + | [[Bottleneck_MapReduce_Framework_Assignment|Bottleneck MapReduce Framework]] | ||
| + | |||
| + | [[Matrix_MapReduce_Framework_Assignment|Matrix MapReduce Framework]] | ||
| + | |||
| + | <!-- | ||
credit for this assignment: Finn Voichick and Dennis Cosgrove | credit for this assignment: Finn Voichick and Dennis Cosgrove | ||
=Motivation= | =Motivation= | ||
| Line 77: | Line 82: | ||
* [[#Optional_Warm_Up]] | * [[#Optional_Warm_Up]] | ||
* [[#Bottlenecked_MapReduce_Framework]] | * [[#Bottlenecked_MapReduce_Framework]] | ||
| − | |||
| − | |||
| − | |||
* [[Cholera_MapReduce_Application]] | * [[Cholera_MapReduce_Application]] | ||
* [[#Matrix_MapReduce_Framework]] | * [[#Matrix_MapReduce_Framework]] | ||
| Line 224: | Line 226: | ||
In this method, you will take in the array of lists you previously created and accumulate the key value pairs in the lists into a newly defined map. To help deal with this issue, you must make use of the Collector provided to you. More specifically, access the accumulator in the collector by calling the <code>accumulator()</code> method and accept the key/value pair when you add it to the map. You probably noticed that the method must return a map of <K, A>, which differs from the <K, V> generics fed into the method. The framework is designed this way as the data originally fed into the mapping stage can be collected into a mutable container before reaching the finish/reduce stage. In order to access the correct value for the map if the key has no associated value yet, use the supplier associated with the Collector with the <code>supplier()</code> method. | In this method, you will take in the array of lists you previously created and accumulate the key value pairs in the lists into a newly defined map. To help deal with this issue, you must make use of the Collector provided to you. More specifically, access the accumulator in the collector by calling the <code>accumulator()</code> method and accept the key/value pair when you add it to the map. You probably noticed that the method must return a map of <K, A>, which differs from the <K, V> generics fed into the method. The framework is designed this way as the data originally fed into the mapping stage can be collected into a mutable container before reaching the finish/reduce stage. In order to access the correct value for the map if the key has no associated value yet, use the supplier associated with the Collector with the <code>supplier()</code> method. | ||
| − | |||
| − | |||
| − | |||
| − | |||
[[File:Bottlenecked accumulate all.png|400px]] [https://docs.google.com/presentation/d/1Gtpj6_5_8imUMccpwxmfrOxKSED791OKl4lZR6G2Bm4/pub?start=false&loop=false&delayms=3000&slide=id.g7ea283730e_0_165 slide] | [[File:Bottlenecked accumulate all.png|400px]] [https://docs.google.com/presentation/d/1Gtpj6_5_8imUMccpwxmfrOxKSED791OKl4lZR6G2Bm4/pub?start=false&loop=false&delayms=3000&slide=id.g7ea283730e_0_165 slide] | ||
| Line 261: | Line 259: | ||
[[File:Matrix map accumulate art.png|400px]] [https://docs.google.com/presentation/d/1bSLKsI5u2e_tFc0d-RSb0BIDwA-kD75U6yXfxvpMf6Y/pub?start=false&loop=false&delayms=3000&slide=id.g7ebdc248f6_0_388 slide] | [[File:Matrix map accumulate art.png|400px]] [https://docs.google.com/presentation/d/1bSLKsI5u2e_tFc0d-RSb0BIDwA-kD75U6yXfxvpMf6Y/pub?start=false&loop=false&delayms=3000&slide=id.g7ebdc248f6_0_388 slide] | ||
| − | |||
===combineAndFinishAll=== | ===combineAndFinishAll=== | ||
| Line 311: | Line 308: | ||
* Correct mapAndAccumulateAll (30) | * Correct mapAndAccumulateAll (30) | ||
* Correct combineAndFinishAll (30) | * Correct combineAndFinishAll (30) | ||
| + | --> | ||
