Difference between revisions of "Nucleobase Counting"

From CSE231 Wiki
Jump to navigation Jump to search
 
(58 intermediate revisions by 3 users not shown)
Line 20: Line 20:
  
 
==Parallel Programming==
 
==Parallel Programming==
 +
===Parallel Constructs===
 +
* [https://www.cse.wustl.edu/~cosgroved/courses/cse231/current/apidocs/edu/wustl/cse231s/v5/V5.html#finish(edu.wustl.cse231s.v5.api.CheckedRunnable) finish]
 +
* [https://www.cse.wustl.edu/~cosgroved/courses/cse231/current/apidocs/edu/wustl/cse231s/v5/V5.html#async(edu.wustl.cse231s.v5.api.CheckedRunnable) async]
 +
* Feel free to use the [[Reference_Page#Async|course reference page]] if you need a reminder on how to use asyncs and finishes
 +
 
===Related Videos===
 
===Related Videos===
LoopingAndFinal: https://wustl.box.com/s/wix18u57v7x1rdx1d1dgfedf6ibr0n92
+
<youtube>ShcnjoO89ZU</youtube>
 +
 
 +
<youtube>DjVep9485l4</youtube>
 +
 
 +
<youtube>1x-jGwcW4WU</youtube>
 +
 
 +
<youtube>0_JrsJrqgDg</youtube>
 +
 
 +
<youtube>8oq9CKgIOiE</youtube>
 +
 
 +
<youtube>--GfpYtL0lI</youtube>
  
SliceArrayDemo: https://wustl.box.com/s/p3ooceekguiaygg4hggw76rbpt28kf6o
+
<!--
 +
* [https://wustl.app.box.com/s/2w4c9xn5du3sqwmmji7v8w35ntpq1j7z Finish & Async: Lower Upper Split]
 +
* [https://wustl.app.box.com/s/dx0f8ycz82e3r894rnpk0cjw8tj8jqzu Dealing With Final: Using Array Slots]
 +
* [https://wustl.app.box.com/s/wix18u57v7x1rdx1d1dgfedf6ibr0n92 Dealing With Final: IntegerRange]
 +
* [https://wustl.app.box.com/s/p3ooceekguiaygg4hggw76rbpt28kf6o CSE]
 +
* [https://wustl.app.box.com/s/smckzu78cvg8seocgluuo1afydgd0gmt Finish & Async Coarsening: N-Way Split]
 +
* [https://wustl.app.box.com/s/wqrauxpsz8xskt897qyukm6rkx9u4zmn Divide and Conquer: Array Sum Example]
 +
-->
  
AddingParallelism: https://wustl.box.com/s/smckzu78cvg8seocgluuo1afydgd0gmt
+
=The Core Questions=
 +
*What are the tasks?
 +
*What is the data?
 +
*Is the data mutable?
 +
*If so, how is it shared?
  
 
=Mistakes to Avoid=
 
=Mistakes to Avoid=
 +
{{Warning | Be sure to remove each NotYetImplementedException as you implement your solutions.}}
 
{{Warning | Do NOT copy the data.  We are simply reading the chromosome data, so there is no reason to copy it.}}
 
{{Warning | Do NOT copy the data.  We are simply reading the chromosome data, so there is no reason to copy it.}}
 +
{{Warning | Do NOT share data beyond what is necessary.  Do NOT use static class fields when you could use local variables instead.}}
 +
{{Warning | Do NOT have the same for loop code duplicated throughout your solution.  Invoke countRangeSequential where appropriate.}}
 +
{{Warning | Do NOT not invoke countSequential to count the entire chromosome by mistake when you want to invoke countRangeSequential to count the range from [min,max).}}
 +
 +
=Getting Started=
 +
*[[Initial_Setup]]
  
 
=Code to Implement=
 
=Code to Implement=
{{CodeToImplement|MidpointUtils|caclulateMidpoint|midpoint.assignment}}
+
==Midpoint==
 
+
{{CodeToImplement|MidpointUtils|caclulateMidpoint|midpoint.lab}}
{{CodeToImplement|NucleobaseCounting|countRange<br>countSequential<br>countParallelLowerUpperSplit<br>countParallelNWaySplit<br>countParallelDivideAndConquerKernel|count.assignment}}
 
 
 
 
 
You will find the starting point for this assignment is in the count folder.  In the count.assignment package you will find [[http://www.cse.wustl.edu/~cosgroved/courses/cse231/f17/apidocs/count/assignment/NucleobaseCounting.html NucleobaseCounting.java]].
 
  
==Utility Methods==
+
{{Sequential|public static int calculateMidpoint(int a, int b)}}
==={{Sequential|calculateMidpoint}}===
 
  
 
In this method, you will need to calculate the midpoint between two numbers. This is as simple as finding the average between the two numbers. There is no need to worry about rounding correctly, just drop everything after the decimal point if the midpoint is not automatically an int.
 
In this method, you will need to calculate the midpoint between two numbers. This is as simple as finding the average between the two numbers. There is no need to worry about rounding correctly, just drop everything after the decimal point if the midpoint is not automatically an int.
Line 50: Line 78:
  
 
It is hard to find 1D references for midpoint.  [http://mathworld.wolfram.com/ Wolfram MathWorld] has a breakdown of [http://mathworld.wolfram.com/Midpoint.html Midpoint in 2D and 3D].  Pick any dimension you like.
 
It is hard to find 1D references for midpoint.  [http://mathworld.wolfram.com/ Wolfram MathWorld] has a breakdown of [http://mathworld.wolfram.com/Midpoint.html Midpoint in 2D and 3D].  Pick any dimension you like.
==={{Sequential|countRangeSequential}}===
+
 
 +
==Threshold==
 +
{{CodeToImplement|GreaterThanThresholdPredicate|constructor<br>test|threshold.lab}}
 +
 
 +
{{Sequential|public GreaterThanThresholdPredicate(int threshold)}}
 +
 
 +
{{Sequential|public boolean test(int value)}}
 +
 
 +
Class GreaterThanThresholdPredicate implements [https://docs.oracle.com/javase/8/docs/api/java/util/function/IntPredicate.html Java IntPredicate interface]. The test method should return a boolean value that indicates whether the input argument is '''greater than''' the threshold provided in the class constructor.
 +
 
 +
==Count Range==
 +
{{CodeToImplement|NucleobaseCounting|countRange<br>countSequential<br>countParallelLowerUpperSplit<br>countParallelNWaySplit<br>countParallelDivideAndConquerKernel|count.lab}}
 +
 
 +
{{Sequential|public static int countRangeSequential(byte[] chromosome, Nucleobase targetNucleobase, int min, int maxExclusive)}}
 +
 
 +
This utility method should count the number of times a particular nucleobase occurs in the array between [min, maxExclusive).
 
<spoiler show="spoiler" hide="spoiler">Use a for loop to iterate from min to maxExclusive.</spoiler>
 
<spoiler show="spoiler" hide="spoiler">Use a for loop to iterate from min to maxExclusive.</spoiler>
  
 
==Sequential Solution==
 
==Sequential Solution==
{{Sequential|countSequential}}
+
 
 +
{{Sequential|public static int countSequential(byte[] chromosome, Nucleobase targetNucleobase)}}
  
 
This solution should be achieved with a simple call to one of the utility methods you wrote.
 
This solution should be achieved with a simple call to one of the utility methods you wrote.
Line 64: Line 108:
 
   
 
   
 
===Lower/Upper Split===
 
===Lower/Upper Split===
{{Parallel|countParallelUpperLowerSplit}}
+
{{Parallel|public static int countParallelLowerUpperSplit(byte[] chromosome, Nucleobase targetNucleobase)}}
  
In order to start with then two equal halves solution, please modify the <code>countParallelUpperLowerSplit</code> method. When you’re ready to begin, delete the return statement and begin implementing your solution! Again, please refer to your notes from lecture for help, as we tackled a very similar problem in class.
+
In order to start with then two equal halves solution, please modify the <code>countParallelUpperLowerSplit</code> method.
 
Hint: don’t forget the finish block! The format will be very similar to the examples.
 
  
 
===Coarsening N-Way Split===
 
===Coarsening N-Way Split===
{{Parallel|countParallelNWaySplit}}
+
{{Parallel|public static int countParallelNWaySplit(byte[] chromosome, Nucleobase targetNucleobase, int numTasks)}}
 +
 
 +
After you’re finished with that, please modify the <code>countParallelNWaySplit</code> method for splitting the work up into n tasks solution.
 +
 
 +
If the chromosome was 1,000,000 long, and you were asked to split it up in 10 tasks, each task should obviously process 100,000.  Things get slightly tricky if the chromosome were 1,000,005 long.
  
After you’re finished with that, please modify the <code>countParallelNWaySplit</code> method for the n different pieces solution. As before, when you’re ready to begin, delete the return statement and begin implementing your solution! Again, please refer to your notes from lecture for help, as we tackled a very similar problem in class.
+
<youtube>hghmrLekmRs</youtube>
 
Hint: make an array to store the results of each task. Split the array into n different chunks, each of which contains 1/n elements. Once each task has summed up the results from its chunk, add the results from each chunk to get your final answer.
 
  
 
===Divide and Conquer===
 
===Divide and Conquer===
{{Parallel|countParallelDivideAndConquerKernel}}
+
In a pattern that we will use repeatedly in the course, we have a public method that is asked to count an entire chromosome.  This method just calls the kernel method to do all of the work:
  
=Testing Your Solution=
+
<nowiki> public static int countParallelDivideAndConquer(byte[] chromosome, Nucleobase targetNucleobase, IntPredicate isParallelRangeLengthPredicate)
==Correctness==
+
throws InterruptedException, ExecutionException {
{{TestSuite|CountTestSuite|count.assignment}}
+
return countParallelDivideAndConquerKernel(chromosome, targetNucleobase, isParallelRangeLengthPredicate, 0, chromosome.length);
 +
}</nowiki>
  
Launch CountTestSuite.java as a JUnit Test to run all of the tests.  CountTestSuite.java is located in src/test/java count.assignment package.  You can initiate this via right clicking on CountTestSuite.java and selecting "Run As..." -> "JUnit Test".
+
{{Parallel|static int countParallelDivideAndConquerKernel(byte[] chromosome, Nucleobase targetNucleobase, IntPredicate isParallelRangeLengthPredicate,
[[http://www.cse.wustl.edu/~cosgroved/courses/cse231/f17/hw1/run-junit-test-suite.png screenshot]]
+
int min, int maxExclusive)}}
  
==Performance==
+
The kernel method will recursively divide the work up in parallel until the range drops below the threshold specified by isParallelRangeLengthPredicate.  At that point it will simply use the sequential count range utility method.
{{Performance|NucleobaseCountTiming|count.assignment}}
 
  
=== Line Of Red Text On The Mac ===
+
=Testing Your Solution=
In computer science (and life in general) you should not make a habit of ignoring red text.
+
How to effectively use JUnit:
  
However, there is a benign bug on the Mac which reports:
+
<youtube>2pmgpWicfA0</youtube>
  
<span style="font-family:Courier;color:red">'''"objc[6477]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/bin/java and /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre/lib/libinstrument.dylib. One of the two will be used. Which one is undefined." '''</span>
+
==Correctness==
 +
{{TestSuite|CountTestSuite|count.lab}}
  
Reportedly, this has finally been fixed and will be released soon:
+
Launch CountTestSuite.java as a JUnit Test to run all of the tests.  CountTestSuite.java is located in src/test/java count.lab package.  You can initiate this via right clicking on CountTestSuite.java and selecting "Run As..." -> "JUnit Test".
 +
[[http://www.cse.wustl.edu/~cosgroved/courses/cse231/f17/hw1/run-junit-test-suite.png screenshot]]
  
https://stackoverflow.com/questions/43003012/class-javalaunchhelper-is-implemented-in-two-places
+
==Performance==
 +
{{Performance|NucleobaseCountTiming|count.lab}}
 +
After your JUnit tests are passing and you are confident that your solutions are correct, move on to checking out the timing performance of your different solutions with different task counts.
  
For now, you should ignore this.
+
=Pledge, Acknowledgments, Citations=
 +
{{Pledge|lab-count}}
  
 
=Rubric=
 
=Rubric=
Line 109: Line 158:
 
Total points: 100
 
Total points: 100
 
* Correct countRangeSequential (10)
 
* Correct countRangeSequential (10)
* Correct countSequential (5)
+
* Correct countSequential (10)
 
* Correct calculateMidpoint (5)
 
* Correct calculateMidpoint (5)
* Correct and Parallel countParallelLowerUpperSplit (20)
+
* Correct countParallelLowerUpperSplit (10)
* Correct and Parallel countParallelNWaySplit (25)
+
* Parallel countParallelLowerUpperSplit (10)
* Correct and Parallel countParallelDivideAndConquerKernel (25)
+
* Correct countParallelNWaySplit (15)
* Clarity and efficiency (10)
+
* Parallel countParallelNWaySplit (10)
 +
* Correct GreaterThanThresholdPredicate (5)
 +
* Correct countParallelDivideAndConquerKernel (15)
 +
* Parallel countParallelDivideAndConquerKernel (10)
 +
 
 +
Penalties may be assessed for clarity and efficiency.

Latest revision as of 04:36, 22 January 2022

Motivation

  • get our feet wet with parallel programming
  • gain experience with the async and finish constructs
  • take different approaches to splitting up the work

We will solve the problem sequentially, then split the work up into 2 tasks, then coarsen the work n-ways, and finally split up the work in a divide-and-conquer recursive style.

Background

Bioinformatics

For this assignment, you will be writing sequential and parallel code to count nucleobases in a human X chromosome.

DNA is made up of four nucleobases: cytosine, guanine, adenine, and thymine. A strand of DNA can thus be represented as a string of letters representing these nucleobases, for example: “ACCGCATAAAGTCC.” However, DNA sequencing is typically not 100% accurate, so some of the nucleobases are not read with high certainty. These bases can be represented as an “N.” A sequence then might look something like “NCCGCATNAAGTCC.” Your goal is to write code that counts the number of occurrences a particular nucleobase or uncertain reads.

We will be using actual data pulled from the US National Library of Medicine, a database maintained by the National Institute of Health. We have already provided you the code that you need to access the chromosome from the database and check your work. You must implement a sequential solution and three parallel solutions to count the given bases in these sequences.

For some more optional background on DNA and nucleotide bases, please check out

Parallel Programming

Parallel Constructs

Related Videos


The Core Questions

  • What are the tasks?
  • What is the data?
  • Is the data mutable?
  • If so, how is it shared?

Mistakes to Avoid

Attention niels epting.svg Warning: Be sure to remove each NotYetImplementedException as you implement your solutions.
Attention niels epting.svg Warning: Do NOT copy the data. We are simply reading the chromosome data, so there is no reason to copy it.
Attention niels epting.svg Warning: Do NOT share data beyond what is necessary. Do NOT use static class fields when you could use local variables instead.
Attention niels epting.svg Warning: Do NOT have the same for loop code duplicated throughout your solution. Invoke countRangeSequential where appropriate.
Attention niels epting.svg Warning: Do NOT not invoke countSequential to count the entire chromosome by mistake when you want to invoke countRangeSequential to count the range from [min,max).

Getting Started

Code to Implement

Midpoint

class: MidpointUtils.java Java.png
methods: caclulateMidpoint
package: midpoint.lab
source folder: student/src/main/java

method: public static int calculateMidpoint(int a, int b) Sequential.svg (sequential implementation only)

In this method, you will need to calculate the midpoint between two numbers. This is as simple as finding the average between the two numbers. There is no need to worry about rounding correctly, just drop everything after the decimal point if the midpoint is not automatically an int.

There are at least two correct ways to implement the midpoint:

Option A:
Option B:

It is hard to find 1D references for midpoint. Wolfram MathWorld has a breakdown of Midpoint in 2D and 3D. Pick any dimension you like.

Threshold

class: GreaterThanThresholdPredicate.java Java.png
methods: constructor
test
package: threshold.lab
source folder: student/src/main/java

method: public GreaterThanThresholdPredicate(int threshold) Sequential.svg (sequential implementation only)

method: public boolean test(int value) Sequential.svg (sequential implementation only)

Class GreaterThanThresholdPredicate implements Java IntPredicate interface. The test method should return a boolean value that indicates whether the input argument is greater than the threshold provided in the class constructor.

Count Range

class: NucleobaseCounting.java Java.png
methods: countRange
countSequential
countParallelLowerUpperSplit
countParallelNWaySplit
countParallelDivideAndConquerKernel
package: count.lab
source folder: student/src/main/java

method: public static int countRangeSequential(byte[] chromosome, Nucleobase targetNucleobase, int min, int maxExclusive) Sequential.svg (sequential implementation only)

This utility method should count the number of times a particular nucleobase occurs in the array between [min, maxExclusive).

Use a for loop to iterate from min to maxExclusive.

Sequential Solution

method: public static int countSequential(byte[] chromosome, Nucleobase targetNucleobase) Sequential.svg (sequential implementation only)

This solution should be achieved with a simple call to one of the utility methods you wrote.

Invoke countRangeSequential from 0 to the chromosome array length.

Parallel Solutions

You need to implement three different parallel solutions to this problem. The first will involve splitting the array into two equal halves, then going through each half of the array in parallel. The second will involve splitting the array into n different pieces, then going through each of those pieces of the array in parallel. The third and final implementation will recursively divide the array until a threshold is reached.

Lower/Upper Split

method: public static int countParallelLowerUpperSplit(byte[] chromosome, Nucleobase targetNucleobase) Parallel.svg (parallel implementation required)

In order to start with then two equal halves solution, please modify the countParallelUpperLowerSplit method.

Coarsening N-Way Split

method: public static int countParallelNWaySplit(byte[] chromosome, Nucleobase targetNucleobase, int numTasks) Parallel.svg (parallel implementation required)

After you’re finished with that, please modify the countParallelNWaySplit method for splitting the work up into n tasks solution.

If the chromosome was 1,000,000 long, and you were asked to split it up in 10 tasks, each task should obviously process 100,000. Things get slightly tricky if the chromosome were 1,000,005 long.

Divide and Conquer

In a pattern that we will use repeatedly in the course, we have a public method that is asked to count an entire chromosome. This method just calls the kernel method to do all of the work:

	public static int countParallelDivideAndConquer(byte[] chromosome, Nucleobase targetNucleobase, IntPredicate isParallelRangeLengthPredicate)
			throws InterruptedException, ExecutionException {
		return countParallelDivideAndConquerKernel(chromosome, targetNucleobase, isParallelRangeLengthPredicate, 0, chromosome.length);
	}

method: static int countParallelDivideAndConquerKernel(byte[] chromosome, Nucleobase targetNucleobase, IntPredicate isParallelRangeLengthPredicate, int min, int maxExclusive) Parallel.svg (parallel implementation required)

The kernel method will recursively divide the work up in parallel until the range drops below the threshold specified by isParallelRangeLengthPredicate. At that point it will simply use the sequential count range utility method.

Testing Your Solution

How to effectively use JUnit:

Correctness

class: CountTestSuite.java Junit.png
package: count.lab
source folder: testing/src/test/java

Launch CountTestSuite.java as a JUnit Test to run all of the tests. CountTestSuite.java is located in src/test/java count.lab package. You can initiate this via right clicking on CountTestSuite.java and selecting "Run As..." -> "JUnit Test". [screenshot]

Performance

class: NucleobaseCountTiming.java Noun Project stopwatch icon 386232 cc.svg
package: count.lab
source folder: src/main/java

After your JUnit tests are passing and you are confident that your solutions are correct, move on to checking out the timing performance of your different solutions with different task counts.

Pledge, Acknowledgments, Citations

file: lab-count-pledge-acknowledgments-citations.txt

More info about the Honor Pledge

Rubric

As always, please make sure to cite your work appropriately.

Total points: 100

  • Correct countRangeSequential (10)
  • Correct countSequential (10)
  • Correct calculateMidpoint (5)
  • Correct countParallelLowerUpperSplit (10)
  • Parallel countParallelLowerUpperSplit (10)
  • Correct countParallelNWaySplit (15)
  • Parallel countParallelNWaySplit (10)
  • Correct GreaterThanThresholdPredicate (5)
  • Correct countParallelDivideAndConquerKernel (15)
  • Parallel countParallelDivideAndConquerKernel (10)

Penalties may be assessed for clarity and efficiency.