MapReduce Mapper Assignment

Motivation

In previous semesters the MapReduce lab has proven to be the most challenging. We have split things up to allow you to get familiar with the how Mappers and Reducers work first. We will build a card mapper that matches the spec outlined in the prep video, a simple word counting mapper, an analogous k-mer counting mapper, and an integer sum reducer to wrap things up.

The k-mer counting mapper will prepare us for (and hopefully lessen the burden of) upcoming lab 6.

Code To Use

BiConsumer

accept(t,u) (note: closest relative to "emit" from RiceX prep)

Collector

finisher() (note: closest relative to "reduce" from RiceX prep)

Deck (note: Iterable<Card>)

Card

getRank()

getSuit()

Rank

isNumeric()

getNumericValue()

Suit

TextSection

getWords()

Code To Implement

Card Mapper

The specification for this mapper is outlined in the prep video:

Non-numeric cards are considered to be bad data and ignored. Numeric cards should be emitted with their suit as the key and the numeric value as the value.

class:	CardMapper.java
methods:	map
package:	mapreduce.app.cards.studio
source folder:	student/src/main/java

method: public void map(Deck deck, BiConsumer<Suit, Integer> keyValuePairConsumer) (sequential implementation only)

K-mer Count Mapper

K-mer counting is a useful technique in bioinformatics: http://www.csbio.unc.edu/mcmillan/Comp555S17/Lecture02.pdf

Background information on k-mer counting can be found here: https://en.wikipedia.org/wiki/K-mer

This mapper is similar to the word count mapper except that the k-mers overlap with each other while words are separate.

class:	KMerMapper.java
methods:	map
package:	mapreduce.apps.kmer.studio
source folder:	student/src/main/java

method: public void map(byte[] sequence, BiConsumer<String, Integer> keyValuePairConsumer) (sequential implementation only)

Be sure to use the provided toStringKMer(sequence, offset, kMerLength) method to generate your k-mers:

	private static String toStringKMer(byte[] sequence, int offset, int kMerLength) {
		return new String(sequence, offset, kMerLength, StandardCharsets.UTF_8);
	}

Word Count Mapper

Counting occurrences of words in text is a classic example of mapreduce. We will ignore any zero length words and convert the remaining words to lower-case so as to get a case insensitive count. Emitting each lower-cased word as the key with the value of 1 should do the trick here.

class:	WordCountMapper.java
methods:	map
package:	mapreduce.apps.wordcount.studio
source folder:	student/src/main/java

method: public void map(TextSection textSection, BiConsumer<String, Integer> keyValuePairConsumer) (sequential implementation only)

The goal of this implementation is to count the number of times a word appears in a given text, using MapReduce. To accomplish this, you will need to create both the mapper and the reducer. Navigate to the WordCountMapper.java and IntegerSumClassicReducer.java classes. You will specifically define how the framework accomplishes the map and reduce methods.

The only method you will need to alter is the map method. In this method, you need to record every instance of a given word and feed it into the keyValuePairConsumer. To do this, access all of the words in the TextSection and if the length of the word is greater than zero (meaning it is not just blank space), convert it into lower-case and accept it into the consumer.

Hint: Look at the methods in TextSection and the toLowerCase() method for strings for assistance.

Int Sum Reducer

class:	IntSumClassicReducer.java
methods:	finisher
package:	mapreduce.apps.intsum.studio
source folder:	student/src/main/java

method: public Function<List<Integer>, Integer> finisher() (sequential implementation only)

The sea of code created by the anonymous inner class can admittedly be a bit intimidating at first:

	@Override
	public Function<List<Integer>, Integer> finisher() {
		return new Function<List<Integer>, Integer>() {
			@Override
			public Integer apply(List<Integer> list) {
				throw new NotYetImplementedException();
			}
		};
	}

but in the end, the apply method is passed a list of integers which it should simply sum up and return. Feel free to replace the bulky anonymous inner class with a lambda if you would like.

Testing Your Solution

Correctness: IntSumStudioTestSuite

class:	IntSumStudioTestSuite.java
package:	mapreduce.intsum.studio
source folder:	testing/src/test/java

CardMapperTestSuite

class:	CardMapperTestSuite.java
package:	mapreduce.apps.cards.studio
source folder:	testing/src/test/java

KMerMapperTestSuite

class:	KMerMapperTestSuite.java
package:	mapreduce.apps.kmer.studio
source folder:	testing/src/test/java

WordCountStudioTestSuite

class:	WordCountStudioTestSuite.java
package:	mapreduce.apps.wordcount.studio
source folder:	testing/src/test/java

IntSumClassicReducerStudioTestSuite

class:	IntSumClassicReducerStudioTestSuite.java
package:	mapreduce.apps.intsum.studio
source folder:	testing/src/test/java