Difference between revisions of "MapReduce Mapper Assignment"

From CSE231 Wiki
Jump to navigation Jump to search
 
(60 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
=Motivation=
 
=Motivation=
−
In previous semesters the MapReduce lab has proven to be the most challenging.  We have split things up to allow you to get familiar with the how Mappers and Reducers work first.  We will build a card mapper that matches the spec outlined in the prep video, a simple word counting mapper, an analogous k-mer counting mapper, and an integer sum reducer to wrap things up.
+
In previous semesters the MapReduce exercise has proven to be the most challenging.  We will start by building some Mappers on our way to the [[Matrix_MapReduce_Framework_Assignment|final boss]].   
  
−
The k-mer counting mapper will prepare us for (and hopefully lessen the burden of) upcoming lab 6.
+
Each of the Mappers built today can be pairs with an Int Summing AccumulatorCombinerReducer:
 +
* a card mapper that matches the spec outlined in the prep video,
 +
* a simple word counting mapper, and
 +
* an analogous k-mer counting mapper.
  
−
=Background=
+
Note: the k-mer counting mapper will prepare us for (and hopefully lessen the burden of) an [[K_Mer_Counting_Assignment|exercise]] later in the semester.
−
<youtube>mySyARBDwoI</youtube>
 
  
 
=Code To Use=
 
=Code To Use=
−
[https://docs.oracle.com/javase/8/docs/api/java/util/function/BiConsumer.html BiConsumer]
+
==Previous Exercise==
−
: [https://docs.oracle.com/javase/8/docs/api/java/util/function/BiConsumer.html#accept-T-U- accept(t,u)] (note: closest relative to "emit" from RiceX prep)
+
[[Not_Thread_Safe_Hash_Table_Assignment#DefaultEntry.3CK.2CV.3E|DefaultEntry<K,V>]]
−
[https://docs.oracle.com/javase/8/docs/api/java/util/stream/Collector.html Collector]
 
−
: [https://docs.oracle.com/javase/8/docs/api/java/util/stream/Collector.html#finisher-- finisher()] (note: closest relative to "reduce" from RiceX prep)
 
  
−
[https://www.cse.wustl.edu/~cosgroved/courses/cse231/current/apidocs/mapreduce/apps/cards/core/Deck.html Deck] (note: Iterable<Card>)
+
==Provided==
 +
===CardMapper Utilities===
 +
[https://www.cse.wustl.edu/~cosgroved/courses/cse231/current/apidocs/mapreduce/apps/cards/core/Deck.html Deck] '''implements Iterable<Card>'''
  
 
[https://www.cse.wustl.edu/~cosgroved/courses/cse231/current/apidocs/mapreduce/apps/cards/core/Card.html Card]
 
[https://www.cse.wustl.edu/~cosgroved/courses/cse231/current/apidocs/mapreduce/apps/cards/core/Card.html Card]
−
: [https://www.cse.wustl.edu/~cosgroved/courses/cse231/current/apidocs/mapreduce/apps/cards/core/Card.html#getRank-- getRank()]
+
: [https://www.cse.wustl.edu/~cosgroved/courses/cse231/current/apidocs/mapreduce/apps/cards/core/Card.html#rank-- card.rank()]
−
: [https://www.cse.wustl.edu/~cosgroved/courses/cse231/current/apidocs/mapreduce/apps/cards/core/Card.html#getSuit-- getSuit()]
+
: [https://www.cse.wustl.edu/~cosgroved/courses/cse231/current/apidocs/mapreduce/apps/cards/core/Card.html#suit-- card.suit()]
  
 
[https://www.cse.wustl.edu/~cosgroved/courses/cse231/current/apidocs/mapreduce/apps/cards/core/Rank.html Rank]
 
[https://www.cse.wustl.edu/~cosgroved/courses/cse231/current/apidocs/mapreduce/apps/cards/core/Rank.html Rank]
−
: [https://www.cse.wustl.edu/~cosgroved/courses/cse231/current/apidocs/mapreduce/apps/cards/core/Rank.html#isNumeric-- isNumeric()]
+
: [https://www.cse.wustl.edu/~cosgroved/courses/cse231/current/apidocs/mapreduce/apps/cards/core/Rank.html#numericValue-- rank.numericValue()] '''note: returns Optional<Integer>'''
−
: [https://www.cse.wustl.edu/~cosgroved/courses/cse231/current/apidocs/mapreduce/apps/cards/core/Rank.html#getNumericValue-- getNumericValue()]
 
  
 
[https://www.cse.wustl.edu/~cosgroved/courses/cse231/current/apidocs/mapreduce/apps/cards/core/Suit.html Suit]
 
[https://www.cse.wustl.edu/~cosgroved/courses/cse231/current/apidocs/mapreduce/apps/cards/core/Suit.html Suit]
  
−
[https://www.cse.wustl.edu/~cosgroved/courses/cse231/current/apidocs/mapreduce/apps/wordcount/core/TextSection.html TextSection]
+
===WordCount Mapper Utilities===
−
: [https://www.cse.wustl.edu/~cosgroved/courses/cse231/current/apidocs/mapreduce/apps/wordcount/core/TextSection.html#getWords-- getWords()]
+
[https://www.cse.wustl.edu/~dennis.cosgrove/courses/cse231/current/apidocs/edu/wustl/cse231s/text/core/TextSection.html TextSection]
 +
: [https://www.cse.wustl.edu/~dennis.cosgrove/courses/cse231/current/apidocs/edu/wustl/cse231s/text/core/TextSection.html#words-- textSection.words()]
 +
 
 +
===K-mer Mapper Utilities===
 +
{{CollapsibleCode|toStringKMer(sequence, offset, kMerLength)|
 +
<syntaxhighlight lang="java">
 +
private static String toStringKMer(byte[] sequence, int offset, int kMerLength) {
 +
return new String(sequence, offset, kMerLength, StandardCharsets.UTF_8);
 +
}
 +
</syntaxhighlight>}}
 +
 
 +
=Code To Invesitigate=
 +
Note: each of the clients print entries.  The entries produced by the map methods of the Mappers are instances of DefaultEntry.  The entries produced by the StreamFramework are instances of a different implementation of Entry.  Their toString() methods might be slightly different, but rest assured they are all Entries.
 +
==Card Mapping Clients==
 +
===CardMapperClient===
 +
{{Client|CardMapperClient|mapreduce.apps.cards.client|main}}
 +
 
 +
{{CollapsibleCode|CardMapperClient|
 +
<syntaxhighlight lang="java">
 +
Deck deck = Deck.createFull();
 +
CardMapper mapper = new CardMapper();
 +
List<Map.Entry<Suit, Integer>> keyValuePairs = mapper.map(deck);
 +
keyValuePairs.forEach(kv -> {
 +
System.out.println(kv);
 +
});
 +
</syntaxhighlight>}}
 +
 
 +
{{CollapsibleConsole|CardMapperClient Output|<pre style="border: 0px; background: #000; color:#fff;">SPADES=10
 +
SPADES=9
 +
SPADES=8
 +
SPADES=7
 +
SPADES=6
 +
SPADES=5
 +
SPADES=4
 +
SPADES=3
 +
SPADES=2
 +
HEARTS=10
 +
HEARTS=9
 +
HEARTS=8
 +
HEARTS=7
 +
HEARTS=6
 +
HEARTS=5
 +
HEARTS=4
 +
HEARTS=3
 +
HEARTS=2
 +
DIAMONDS=10
 +
DIAMONDS=9
 +
DIAMONDS=8
 +
DIAMONDS=7
 +
DIAMONDS=6
 +
DIAMONDS=5
 +
DIAMONDS=4
 +
DIAMONDS=3
 +
DIAMONDS=2
 +
CLUBS=10
 +
CLUBS=9
 +
CLUBS=8
 +
CLUBS=7
 +
CLUBS=6
 +
CLUBS=5
 +
CLUBS=4
 +
CLUBS=3
 +
CLUBS=2</pre>}}
 +
 
 +
===CardMapReduceClient===
 +
{{Client|CardMapReduceClient|mapreduce.apps.cards.client|main}}
 +
 
 +
{{CollapsibleCode|CardMapReduceClient|
 +
<syntaxhighlight lang="java">
 +
Deck[] decks = {
 +
Deck.createFull(),
 +
Deck.createFull(),
 +
Deck.createFull(),
 +
Deck.createFull(),
 +
};
 +
CardMapper mapper = new CardMapper();
 +
AccumulatorCombinerReducer<Integer, ?, Integer> accumulatorCombinerReducer = StreamUtils.summingIntAccumulatorCombinerReducer();
 +
MapReduceFramework<Deck, Suit, Integer, ?, Integer> framework = new StreamMapReduceFramework<>(mapper, accumulatorCombinerReducer);
 +
Map<Suit, Integer> map = framework.mapReduceAll(decks);
 +
map.entrySet().forEach(entry -> {
 +
System.out.println(entry);
 +
});
 +
</syntaxhighlight>}}
 +
 
 +
{{CollapsibleConsole|CardMapReduceClient Output|<pre style="border: 0px; background: #000; color:#fff;">HEARTS=216
 +
SPADES=216
 +
DIAMONDS=216
 +
CLUBS=216</pre>}}
 +
 
 +
==Word Count Mapping Clients==
 +
The word count mapping example clients use the beginning of [https://www.poetryfoundation.org/poems/46473/if--- If--- by Rudyard Kipling].
 +
===WordCountMapperClient===
 +
{{Client|WordCountMapperClient|mapreduce.apps.wordcount.client|main}}
 +
 
 +
Passing:
 +
 
 +
new TextSection("If you can keep your head when all about you")
 +
 
 +
to the WordCountMapper's map method will return a list of Entries containing:
 +
 
 +
[[File:WordCount MapResult.svg|1200px]]
 +
 
 +
{{CollapsibleCode|WordCountMapperClient|
 +
<syntaxhighlight lang="java">
 +
TextSection textSection = new TextSection("If you can keep your head when all about you");
 +
WordCountMapper mapper = new WordCountMapper();
 +
List<Map.Entry<String, Integer>> keyValuePairs = mapper.map(textSection);
 +
keyValuePairs.forEach(kv -> {
 +
    System.out.println(kv);
 +
});
 +
</syntaxhighlight>}}
 +
 
 +
{{CollapsibleConsole|WordCountMapperClient Output|<pre style="border: 0px; background: #000; color:#fff;">if=1
 +
you=1
 +
can=1
 +
keep=1
 +
your=1
 +
head=1
 +
when=1
 +
all=1
 +
about=1
 +
you=1</pre>}}
 +
 
 +
===WordCountMapReduceClient===
 +
{{Client|WordCountMapReduceClient|mapreduce.apps.wordcount.client|main}}
 +
 
 +
{{CollapsibleCode|WordCountMapReduceClient|
 +
<syntaxhighlight lang="java">
 +
TextSection[] textSections = {
 +
        new TextSection("If you can keep your head when all about you"),
 +
        new TextSection("  Are losing theirs and blaming it on you,"),
 +
};
 +
WordCountMapper mapper = new WordCountMapper();
 +
AccumulatorCombinerReducer<Integer, ?, Integer> accumulatorCombinerReducer = StreamUtils.summingIntAccumulatorCombinerReducer();
 +
MapReduceFramework<TextSection, String, Integer, ?, Integer> framework = new StreamMapReduceFramework<>(mapper, accumulatorCombinerReducer);
 +
Map<String, Integer> map = framework.mapReduceAll(textSections);
 +
map.entrySet().forEach(entry -> {
 +
    System.out.println(entry);
 +
});
 +
</syntaxhighlight>}}
 +
 
 +
{{CollapsibleConsole|WordCountMapReduceClient Output|<pre style="border: 0px; background: #000; color:#fff;">all=1
 +
theirs=1
 +
about=1
 +
it=1
 +
your=1
 +
when=1
 +
losing=1
 +
head=1
 +
can=1
 +
blaming=1
 +
are=1
 +
and=1
 +
keep=1
 +
if=1
 +
you=3
 +
on=1</pre>}}
 +
 
 +
==K-mer Mapping Clients==
 +
The word count mapping example clients use the beginning of [https://www.poetryfoundation.org/poems/46473/if--- If--- by Rudyard Kipling].
 +
===KMerMapperClient===
 +
{{Client|KMerMapperClient|mapreduce.apps.wordcount.client|main}}
 +
 
 +
{{CollapsibleCode|KMerMapperClient|
 +
<syntaxhighlight lang="java">
 +
byte[] sequence = "ACTCATGAG".getBytes(StandardCharsets.UTF_8);
 +
KMerMapper mapper = new KMerMapper(3);
 +
List<Map.Entry<String, Integer>> keyValuePairs = mapper.map(sequence);
 +
keyValuePairs.forEach(kv -> {
 +
    System.out.println(kv);
 +
});
 +
</syntaxhighlight>}}
 +
 
 +
{{CollapsibleConsole|KMerMapperClient Output|<pre style="border: 0px; background: #000; color:#fff;">ACT=1
 +
CTC=1
 +
TCA=1
 +
CAT=1
 +
ATG=1
 +
TGA=1
 +
GAG=1</pre>}}
 +
 
 +
===KMerMapReduceClient===
 +
{{Client|KMerMapReduceClient|mapreduce.apps.wordcount.client|main}}
 +
 
 +
{{CollapsibleCode|KMerMapReduceClient|
 +
<syntaxhighlight lang="java">
 +
byte[][] sequences = {
 +
        "ACTCATGAG".getBytes(StandardCharsets.UTF_8),
 +
        "CATGAAAAAA".getBytes(StandardCharsets.UTF_8),
 +
};
 +
KMerMapper mapper = new KMerMapper(3);
 +
AccumulatorCombinerReducer<Integer, ?, Integer> accumulatorCombinerReducer = StreamUtils.summingIntAccumulatorCombinerReducer();
 +
MapReduceFramework<byte[], String, Integer, ?, Integer> framework = new StreamMapReduceFramework<>(mapper, accumulatorCombinerReducer);
 +
Map<String, Integer> map = framework.mapReduceAll(sequences);
 +
map.entrySet().forEach(entry -> {
 +
    System.out.println(entry);
 +
});
 +
</syntaxhighlight>}}
 +
 
 +
{{CollapsibleConsole|KMerMapReduceClient Output|<pre style="border: 0px; background: #000; color:#fff;">AAA=4
 +
ACT=1
 +
TCA=1
 +
CTC=1
 +
ATG=2
 +
GAA=1
 +
CAT=2
 +
GAG=1
 +
TGA=2</pre>}}
  
 
=Code To Implement=
 
=Code To Implement=
−
==Card Mapper==
+
==CardMapper==
 
The specification for this mapper is outlined in the prep video:
 
The specification for this mapper is outlined in the prep video:
 +
{{CollapsibleYouTube|MapReduce Tutorial|<youtube>K8VPHHPS3BQ</youtube>}}
 +
{{CollapsibleYouTube|Learning MapReduce with Playing Cards|<youtube>bcjSe0xCHbE</youtube>}}
 +
 +
 +
Non-numeric cards are considered to be bad data and ignored.  Numeric cards should be emitted with their suit as the key and the numeric value as the value.  Emitted key-value pairs are returned in a list of Entries.
 +
 +
{{CodeToImplement|CardMapper|map|mapreduce.app.cards.exercise}}
 +
 +
{{Sequential|List<Map.Entry<Suit, Integer>> map(Deck deck)}}
 +
 +
Make sure you have implemented [[Not_Thread_Safe_Hash_Table_Assignment#DefaultEntry.3CK.2CV.3E|DefaultEntry<K,V>]] before starting this assignment.
 +
 +
==Word Count Mapper==
  
−
<youtube>bcjSe0xCHbE</youtube>
+
{{CodeToImplement|WordCountMapper|map|mapreduce.apps.wordcount.exercise}}
  
−
Non-numeric cards are considered to be bad data and ignored.  Numeric cards should be emitted with their suit as the key and the numeric value as the value.
+
{{Sequential|List<Map.Entry<String, Integer>> map(TextSection textSection)}}
  
−
{{CodeToImplement|CardMapper|map|mapreduce.app.cards.studio}}
+
Counting occurrences of words in text is a classic example of MapReduce. We will '''ignore any zero length words and convert the remaining words to lower-case''' so as to get a case insensitive count. Emitting each lower-cased word as the key with the value of 1 should do the trick here.
  
−
{{Sequential|public void map(Deck deck, BiConsumer<Suit, Integer> keyValuePairConsumer)}}
+
<!-- COMMENTED OUT count the number of times a word appears in a given text, using MapReduce. To accomplish this, you will need to create both the mapper and the reducer. Navigate to the <code>WordCountMapper.java</code> and <code>IntSumListAccumulatingReducer.java</code> classes. You will specifically define how the framework accomplishes the map and reduce methods.
  
−
==K-mer Count Mapper==
+
The only method you will need to alter is the map method.
  
−
K-mer counting is a useful technique in bioinformatics: http://www.csbio.unc.edu/mcmillan/Comp555S17/Lecture02.pdf
+
In this method, you need to record every instance of a given word and feed it into the keyValuePairConsumer. To do this, access all of the words in the TextSection and if the length of the word is greater than zero (meaning it is not just blank space), convert it into lower-case and accept it into the consumer. COMMENTED OUT-->
 +
Hint: Look at the methods in TextSection and the toLowerCase() method for strings for assistance.
  
−
Background information on k-mer counting can be found here: https://en.wikipedia.org/wiki/K-mer
+
==K-mer Count Mapper==
  
−
This mapper is similar to the word count mapper except that the k-mers overlap with each other while words are separate.
+
[http://www.csbio.unc.edu/mcmillan/Comp555S17/Lecture02.pdf K-mer counting is a useful technique in bioinformatics].
  
−
{{CodeToImplement|KMerMapper|map|mapreduce.apps.kmer.studio}}
+
Further background information on k-mer counting can be found [https://en.wikipedia.org/wiki/K-mer here].
  
−
{{Sequential|public void map(byte[] sequence, BiConsumer<String, Integer> keyValuePairConsumer)}}
+
The 3-mers in the chromosome data:
  
−
Be sure to use the provided toStringKMer(sequence, offset, kMerLength) method to generate your k-mers:
+
<nowiki>ACTCATGAG</nowiki>
  
−
<nowiki> private static String toStringKMer(byte[] sequence, int offset, int kMerLength) {
+
are:
−
return new String(sequence, offset, kMerLength, StandardCharsets.UTF_8);
 
−
}</nowiki>
 
  
−
==Word Count Mapper==
+
<nowiki>ACT
−
Counting occurrences of words in text is a classic example of mapreduce. We will ignore any zero length words and convert the remaining words to lower-case so as to get a case insensitive count.  Emitting each lower-cased word as the key with the value of 1 should do the trick here.
+
  CTC
 +
  TCA
 +
  CAT
 +
    ATG
 +
    TGA
 +
      GAG</nowiki>
  
−
{{CodeToImplement|WordCountMapper|map|mapreduce.apps.wordcount.studio}}
+
The 4-mers in that same chromosome data:
  
−
{{Sequential|public void map(TextSection textSection, BiConsumer<String, Integer> keyValuePairConsumer)}}
+
<nowiki>ACTCATGAG</nowiki>
  
−
The goal of this implementation is to count the number of times a word appears in a given text, using MapReduce. To accomplish this, you will need to create both the mapper and the reducer. Navigate to the <code>WordCountMapper.java</code> and <code>IntegerSumClassicReducer.java</code> classes. You will specifically define how the framework accomplishes the map and reduce methods.
+
are:
  
−
The only method you will need to alter is the map method. In this method, you need to record every instance of a given word and feed it into the keyValuePairConsumer. To do this, access all of the words in the TextSection and if the length of the word is greater than zero (meaning it is not just blank space), convert it into lower-case and accept it into the consumer.
+
<nowiki>ACTC
 +
CTCA
 +
  TCAT
 +
  CATG
 +
    ATGA
 +
    TGAG</nowiki>
  
−
Hint: Look at the methods in TextSection and the toLowerCase() method for strings for assistance.
+
{{CodeToImplement|KMerMapper|map|mapreduce.apps.kmer.studio}}
  
−
==Int Sum Reducer==
+
{{Sequential|List<Map.Entry<String, Integer>> map(byte[] sequence)}}
−
{{CodeToImplement|IntSumClassicReducer|finisher|mapreduce.apps.intsum.studio}}
 
  
−
{{Sequential|public Function<List<Integer>, Integer> finisher()}}
+
Take note of the private instance variable declared and initialized for you.
 +
Be sure to use the provided toStringKMer(sequence, offset, kMerLength) method to generate your k-mers:
  
−
The sea of code created by the anonymous inner class can admittedly be a bit intimidating at first:
+
<syntaxhighlight lang="java">private static String toStringKMer(byte[] sequence, int offset, int kMerLength) {
 +
return new String(sequence, offset, kMerLength, StandardCharsets.UTF_8);
 +
}</syntaxhighlight>
  
−
<nowiki> @Override
+
This mapper is similar to the [[#Word Count Mapper]] except that the k-mers overlap with each other while words are separate.
−
public Function<List<Integer>, Integer> finisher() {
 
−
return new Function<List<Integer>, Integer>() {
 
−
@Override
 
−
public Integer apply(List<Integer> list) {
 
−
throw new NotYetImplementedException();
 
−
}
 
−
};
 
−
}</nowiki>
 
  
−
but in the end, the apply method is passed a list of integers which it should simply sum up and return.  Feel free to replace the bulky anonymous inner class with a lambda if you would like.
+
As the emitted values for each key will be later summed up in the reduction phase, what value makes sense to emit with each key?
  
 
=Testing Your Solution=
 
=Testing Your Solution=
−
==Correctness: IntSumStudioTestSuite==
+
{{TestSuite|_MappersSuitableForPairingWithIntSummingReducerTestSuite|mapreduce.apps}}
−
{{TestSuite|IntSumStudioTestSuite|mapreduce.intsum.studio}}
+
 
−
===CardMapperTestSuite===
+
=Pledge, Acknowledgments, Citations=
−
{{TestSuite|CardMapperTestSuite|mapreduce.apps.cards.studio}}
+
{{Pledge|map-reduce-mapper}}
−
===KMerMapperTestSuite===
 
−
{{TestSuite|KMerMapperTestSuite|mapreduce.apps.kmer.studio}}
 
−
===WordCountStudioTestSuite===
 
−
{{TestSuite|WordCountStudioTestSuite|mapreduce.apps.wordcount.studio}}
 
−
===IntSumClassicReducerStudioTestSuite===
 
−
{{TestSuite|IntSumClassicReducerStudioTestSuite|mapreduce.apps.intsum.studio}}
 

Latest revision as of 18:49, 20 February 2024

Motivation

In previous semesters the MapReduce exercise has proven to be the most challenging. We will start by building some Mappers on our way to the final boss.

Each of the Mappers built today can be pairs with an Int Summing AccumulatorCombinerReducer:

  • a card mapper that matches the spec outlined in the prep video,
  • a simple word counting mapper, and
  • an analogous k-mer counting mapper.

Note: the k-mer counting mapper will prepare us for (and hopefully lessen the burden of) an exercise later in the semester.

Code To Use

Previous Exercise

DefaultEntry<K,V>

Provided

CardMapper Utilities

Deck implements Iterable<Card>

Card

card.rank()
card.suit()

Rank

rank.numericValue() note: returns Optional<Integer>

Suit

WordCount Mapper Utilities

TextSection

textSection.words()

K-mer Mapper Utilities

toStringKMer(sequence, offset, kMerLength)   
private static String toStringKMer(byte[] sequence, int offset, int kMerLength) {
	return new String(sequence, offset, kMerLength, StandardCharsets.UTF_8);
}

Code To Invesitigate

Note: each of the clients print entries. The entries produced by the map methods of the Mappers are instances of DefaultEntry. The entries produced by the StreamFramework are instances of a different implementation of Entry. Their toString() methods might be slightly different, but rest assured they are all Entries.

Card Mapping Clients

CardMapperClient

class: CardMapperClient.java CLIENT
package: mapreduce.apps.cards.client
source folder: student/src/main/java
CardMapperClient   
Deck deck = Deck.createFull();
CardMapper mapper = new CardMapper();
List<Map.Entry<Suit, Integer>> keyValuePairs = mapper.map(deck);
keyValuePairs.forEach(kv -> {
	System.out.println(kv);
});
CardMapperClient Output   
SPADES=10
SPADES=9
SPADES=8
SPADES=7
SPADES=6
SPADES=5
SPADES=4
SPADES=3
SPADES=2
HEARTS=10
HEARTS=9
HEARTS=8
HEARTS=7
HEARTS=6
HEARTS=5
HEARTS=4
HEARTS=3
HEARTS=2
DIAMONDS=10
DIAMONDS=9
DIAMONDS=8
DIAMONDS=7
DIAMONDS=6
DIAMONDS=5
DIAMONDS=4
DIAMONDS=3
DIAMONDS=2
CLUBS=10
CLUBS=9
CLUBS=8
CLUBS=7
CLUBS=6
CLUBS=5
CLUBS=4
CLUBS=3
CLUBS=2

CardMapReduceClient

class: CardMapReduceClient.java CLIENT
package: mapreduce.apps.cards.client
source folder: student/src/main/java
CardMapReduceClient   
Deck[] decks = {
	Deck.createFull(),
	Deck.createFull(),
	Deck.createFull(),
	Deck.createFull(),
};
CardMapper mapper = new CardMapper();
AccumulatorCombinerReducer<Integer, ?, Integer> accumulatorCombinerReducer = StreamUtils.summingIntAccumulatorCombinerReducer();
MapReduceFramework<Deck, Suit, Integer, ?, Integer> framework = new StreamMapReduceFramework<>(mapper, accumulatorCombinerReducer);
Map<Suit, Integer> map = framework.mapReduceAll(decks);
map.entrySet().forEach(entry -> {
	System.out.println(entry);
});
CardMapReduceClient Output   
HEARTS=216
SPADES=216
DIAMONDS=216
CLUBS=216

Word Count Mapping Clients

The word count mapping example clients use the beginning of If--- by Rudyard Kipling.

WordCountMapperClient

class: WordCountMapperClient.java CLIENT
package: mapreduce.apps.wordcount.client
source folder: student/src/main/java

Passing: 

new TextSection("If you can keep your head when all about you")

to the WordCountMapper's map method will return a list of Entries containing:

WordCount MapResult.svg

WordCountMapperClient   
TextSection textSection = new TextSection("If you can keep your head when all about you");
WordCountMapper mapper = new WordCountMapper();
List<Map.Entry<String, Integer>> keyValuePairs = mapper.map(textSection);
keyValuePairs.forEach(kv -> {
    System.out.println(kv);
});
WordCountMapperClient Output   
if=1
you=1
can=1
keep=1
your=1
head=1
when=1
all=1
about=1
you=1

WordCountMapReduceClient

class: WordCountMapReduceClient.java CLIENT
package: mapreduce.apps.wordcount.client
source folder: student/src/main/java
WordCountMapReduceClient   
TextSection[] textSections = {
        new TextSection("If you can keep your head when all about you"),
        new TextSection("   Are losing theirs and blaming it on you,"),
};
WordCountMapper mapper = new WordCountMapper();
AccumulatorCombinerReducer<Integer, ?, Integer> accumulatorCombinerReducer = StreamUtils.summingIntAccumulatorCombinerReducer();
MapReduceFramework<TextSection, String, Integer, ?, Integer> framework = new StreamMapReduceFramework<>(mapper, accumulatorCombinerReducer);
Map<String, Integer> map = framework.mapReduceAll(textSections);
map.entrySet().forEach(entry -> {
    System.out.println(entry);
});
WordCountMapReduceClient Output   
all=1
theirs=1
about=1
it=1
your=1
when=1
losing=1
head=1
can=1
blaming=1
are=1
and=1
keep=1
if=1
you=3
on=1

K-mer Mapping Clients

The word count mapping example clients use the beginning of If--- by Rudyard Kipling.

KMerMapperClient

class: KMerMapperClient.java CLIENT
package: mapreduce.apps.wordcount.client
source folder: student/src/main/java
KMerMapperClient   
byte[] sequence = "ACTCATGAG".getBytes(StandardCharsets.UTF_8);
KMerMapper mapper = new KMerMapper(3);
List<Map.Entry<String, Integer>> keyValuePairs = mapper.map(sequence);
keyValuePairs.forEach(kv -> {
    System.out.println(kv);
});
KMerMapperClient Output   
ACT=1
CTC=1
TCA=1
CAT=1
ATG=1
TGA=1
GAG=1

KMerMapReduceClient

class: KMerMapReduceClient.java CLIENT
package: mapreduce.apps.wordcount.client
source folder: student/src/main/java
KMerMapReduceClient   
byte[][] sequences = {
        "ACTCATGAG".getBytes(StandardCharsets.UTF_8),
        "CATGAAAAAA".getBytes(StandardCharsets.UTF_8),
};
KMerMapper mapper = new KMerMapper(3);
AccumulatorCombinerReducer<Integer, ?, Integer> accumulatorCombinerReducer = StreamUtils.summingIntAccumulatorCombinerReducer();
MapReduceFramework<byte[], String, Integer, ?, Integer> framework = new StreamMapReduceFramework<>(mapper, accumulatorCombinerReducer);
Map<String, Integer> map = framework.mapReduceAll(sequences);
map.entrySet().forEach(entry -> {
    System.out.println(entry);
});
KMerMapReduceClient Output   
AAA=4
ACT=1
TCA=1
CTC=1
ATG=2
GAA=1
CAT=2
GAG=1
TGA=2

Code To Implement

CardMapper

The specification for this mapper is outlined in the prep video:

Video: MapReduce Tutorial  
Video: Learning MapReduce with Playing Cards  


Non-numeric cards are considered to be bad data and ignored. Numeric cards should be emitted with their suit as the key and the numeric value as the value. Emitted key-value pairs are returned in a list of Entries.

class: CardMapper.java Java.png
methods: map
package: mapreduce.app.cards.exercise
source folder: student/src/main/java

method: List<Map.Entry<Suit, Integer>> map(Deck deck) Sequential.svg (sequential implementation only)

Make sure you have implemented DefaultEntry<K,V> before starting this assignment.

Word Count Mapper

class: WordCountMapper.java Java.png
methods: map
package: mapreduce.apps.wordcount.exercise
source folder: student/src/main/java

method: List<Map.Entry<String, Integer>> map(TextSection textSection) Sequential.svg (sequential implementation only)

Counting occurrences of words in text is a classic example of MapReduce. We will ignore any zero length words and convert the remaining words to lower-case so as to get a case insensitive count. Emitting each lower-cased word as the key with the value of 1 should do the trick here.

Hint: Look at the methods in TextSection and the toLowerCase() method for strings for assistance.

K-mer Count Mapper

K-mer counting is a useful technique in bioinformatics.

Further background information on k-mer counting can be found here.

The 3-mers in the chromosome data: 

ACTCATGAG

are: 

ACT
 CTC
  TCA
   CAT
    ATG
     TGA
      GAG

The 4-mers in that same chromosome data: 

ACTCATGAG

are: 

ACTC
 CTCA
  TCAT
   CATG
    ATGA
     TGAG
class: KMerMapper.java Java.png
methods: map
package: mapreduce.apps.kmer.studio
source folder: student/src/main/java

method: List<Map.Entry<String, Integer>> map(byte[] sequence) Sequential.svg (sequential implementation only)

Take note of the private instance variable declared and initialized for you. Be sure to use the provided toStringKMer(sequence, offset, kMerLength) method to generate your k-mers: 

private static String toStringKMer(byte[] sequence, int offset, int kMerLength) {
	return new String(sequence, offset, kMerLength, StandardCharsets.UTF_8);
}

This mapper is similar to the #Word Count Mapper except that the k-mers overlap with each other while words are separate.

As the emitted values for each key will be later summed up in the reduction phase, what value makes sense to emit with each key?

Testing Your Solution

class: _MappersSuitableForPairingWithIntSummingReducerTestSuite.java Junit.png
package: mapreduce.apps
source folder: testing/src/test/java

Pledge, Acknowledgments, Citations

file: map-reduce-mapper-pledge-acknowledgments-citations.txt

More info about the Honor Pledge

Retrieved from "http://classes.engineering.wustl.edu/cse231/core/index.php?title=MapReduce_Mapper_Assignment&oldid=5147"