Difference between revisions of "MapReduce Mapper Assignment"

From CSE231 Wiki
Jump to navigation Jump to search
Line 6: Line 6:
 
=Code To Use=
 
=Code To Use=
 
[https://docs.oracle.com/javase/8/docs/api/java/util/function/BiConsumer.html BiConsumer]
 
[https://docs.oracle.com/javase/8/docs/api/java/util/function/BiConsumer.html BiConsumer]
: [https://docs.oracle.com/javase/8/docs/api/java/util/function/BiConsumer.html#accept-T-U- accept(t,u)]
+
: [https://docs.oracle.com/javase/8/docs/api/java/util/function/BiConsumer.html#accept-T-U- accept(t,u)] <-- "emit"
 
[https://docs.oracle.com/javase/8/docs/api/java/util/stream/Collector.html Collector]
 
[https://docs.oracle.com/javase/8/docs/api/java/util/stream/Collector.html Collector]
: [https://docs.oracle.com/javase/8/docs/api/java/util/stream/Collector.html#finisher-- finisher()]
+
: [https://docs.oracle.com/javase/8/docs/api/java/util/stream/Collector.html#finisher-- finisher()] <-- "reduce"
  
 
=Code To Implement=
 
=Code To Implement=

Revision as of 17:18, 20 February 2018

Motivation

In previous semesters the MapReduce lab has proven to be the most challenging. We have split things up to allow you to get familiar with the how Mappers and Reducers work first. We will build a card mapper that matches the spec outlined in the prep video, a simple word counting mapper, an analogous k-mer counting mapper, and an integer sum reducer to wrap things up.

The k-mer counting mapper will prepare us for (and hopefully lessen the burden of) upcoming lab 6.

Code To Use

BiConsumer

accept(t,u) <-- "emit"

Collector

finisher() <-- "reduce"

Code To Implement

Card Mapper

The specification for this mapper is outlined in the prep video:

Non-numeric cards are considered to be bad data and ignored. Numeric cards should be emitted with their suit as the key and the numeric value as the value.

class: CardMapper.java Java.png
methods: map
package: mapreduce.apps.intsum.cards.studio
source folder: student/src/main/java

method: public void map(Deck deck, BiConsumer<Suit, Integer> keyValuePairConsumer) Sequential.svg (sequential implementation only)

Word Count Mapper

class: WordCountMapper.java Java.png
methods: map
package: mapreduce.apps.intsum.wordcount.studio
source folder: student/src/main/java

method: public void map(TextSection textSection, BiConsumer<String, Integer> keyValuePairConsumer) Sequential.svg (sequential implementation only)

K-mer Count Mapper

K-mer counting is a useful technique in bioinformatics: http://www.csbio.unc.edu/mcmillan/Comp555S17/Lecture02.pdf

Background information on k-mer counting can be found here: https://en.wikipedia.org/wiki/K-mer

This mapper is similar to the word count mapper except that the k-mers overlap with each other while words are separate.

class: KMerMapper.java Java.png
methods: map
package: mapreduce.apps.intsum.kmer.studio
source folder: student/src/main/java

method: public void map(byte[] sequence, BiConsumer<String, Integer> keyValuePairConsumer) Sequential.svg (sequential implementation only)

Int Sum Reducer

class: IntegerSumClassicReducer.java Java.png
methods: finisher
package: mapreduce.apps.intsum.studio
source folder: student/src/main/java

method: public Function<List<Integer>, Integer> finisher() Sequential.svg (sequential implementation only)

Testing Your Solution

Correctness

class: IntSumStudioTestSuite.java Junit.png
package: mapreduce
source folder: testing/src/test/java