Sharded BAM reader and a sample read counting pipeline

[iliat]

@wbrockman@jean-philippe-martin@dionloy - here is the first version of the sharded BAM reader. Works in the cloud with a workaround for credentials factory.

• Most of the common files are under readers/bam.
• The CountReads pipeline is there as an example of using the reader. It can read from the API or BAM file.
• I added a GCSOptions class to capture what is needed to access GCS.
• Some files added under htsjdk/samtools because I needed to deal with visibility restrictions in HTSJDK classes (not pretty but fwiw GATK code base does the same).
  Also added dependency on HTSJDK in pom.xml

[coveralls]

Coverage decreased (-31.58%) to 0.0% when pulling d7ba30a on iliat:dev-broad into 408ed35 on googlegenomics:master.

[pgrosu]

Congrats Ilia! This is a huge step forward!

~p

[wbrockman]

Typo - GATK

[iliat]

@dionloy@deflaux I fill fix the comment typo but LMK if there are any other comments, concerns before I merge.

[iliat]

@dionloy@deflaux Ok, I simplified credentials passing, using OflineAuth - no need for factory and side inputs. Will work on adding more tests a bit later.

[pgrosu]

Hey Ilia,

Travis is not happy :(

[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /home/travis/build/googlegenomics/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/pipelines/CountReads.java:[39,48] cannot find symbol
symbol: class WorkerCredentialFactoryWorkaround
location: package com.google.cloud.genomics.dataflow.utils
[ERROR] /home/travis/build/googlegenomics/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/pipelines/CountReads.java:[116,5] cannot find symbol
symbol: variable WorkerCredentialFactoryWorkaround
location: class com.google.cloud.genomics.dataflow.pipelines.CountReads
[ERROR] /home/travis/build/googlegenomics/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/pipelines/CountReads.java:[149,17] no suitable constructor found for ReadReader(com.google.cloud.genomics.utils.GenomicsFactory.OfflineAuth)
constructor com.google.cloud.genomics.dataflow.readers.ReadReader.ReadReader(com.google.cloud.genomics.utils.GenomicsFactory.OfflineAuth,com.google.cloud.genomics.utils.Paginator.ShardBoundary) is not applicable
(actual and formal argument lists differ in length)
constructor com.google.cloud.genomics.dataflow.readers.ReadReader.ReadReader(com.google.cloud.genomics.utils.GenomicsFactory.OfflineAuth,com.google.cloud.genomics.utils.Paginator.ShardBoundary,java.lang.String) is not applicable
(actual and formal argument lists differ in length)
[ERROR] /home/travis/build/googlegenomics/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/readers/bam/BAMShard.java:[70,11] cannot assign a value to final variable end
[ERROR] /home/travis/build/googlegenomics/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/readers/bam/BAMShard.java:[82,13] cannot assign a value to final variable end
[ERROR] /home/travis/build/googlegenomics/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/readers/bam/Sharder.java:[140,13] cannot assign a value to final variable end
[INFO] 6 errors
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 6.717 s
[INFO] Finished at: 2015-03-07T19:33:50+00:00
[INFO] Final Memory: 19M/136M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on project google-genomics-dataflow: Compilation failure: Compilation failure:
[ERROR] /home/travis/build/googlegenomics/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/pipelines/CountReads.java:[39,48] cannot find symbol
[ERROR] symbol: class WorkerCredentialFactoryWorkaround
[ERROR] location: package com.google.cloud.genomics.dataflow.utils
[ERROR] /home/travis/build/googlegenomics/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/pipelines/CountReads.java:[116,5] cannot find symbol
[ERROR] symbol: variable WorkerCredentialFactoryWorkaround
[ERROR] location: class com.google.cloud.genomics.dataflow.pipelines.CountReads
[ERROR] /home/travis/build/googlegenomics/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/pipelines/CountReads.java:[149,17] no suitable constructor found for ReadReader(com.google.cloud.genomics.utils.GenomicsFactory.OfflineAuth)
[ERROR] constructor com.google.cloud.genomics.dataflow.readers.ReadReader.ReadReader(com.google.cloud.genomics.utils.GenomicsFactory.OfflineAuth,com.google.cloud.genomics.utils.Paginator.ShardBoundary) is not applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] constructor com.google.cloud.genomics.dataflow.readers.ReadReader.ReadReader(com.google.cloud.genomics.utils.GenomicsFactory.OfflineAuth,com.google.cloud.genomics.utils.Paginator.ShardBoundary,java.lang.String) is not applicable
[ERROR] (actual and formal argument lists differ in length)
[ERROR] /home/travis/build/googlegenomics/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/readers/bam/BAMShard.java:[70,11] cannot assign a value to final variable end
[ERROR] /home/travis/build/googlegenomics/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/readers/bam/BAMShard.java:[82,13] cannot assign a value to final variable end
[ERROR] /home/travis/build/googlegenomics/dataflow-java/src/main/java/com/google/cloud/genomics/dataflow/readers/bam/Sharder.java:[140,13] cannot assign a value to final variable end
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
The command "eval mvn install -DskipTests=true -Dmaven.javadoc.skip=true -B -V" failed 3 times.
The command "mvn install -DskipTests=true -Dmaven.javadoc.skip=true -B -V" failed and exited with 1 during .
Your build has been stopped.

Paul

[iliat]

@pgrosu Yeah, I see that. Looking

[pgrosu]

@iliat, Cool - yeah they're minor fixes - thanks man :)

[iliat]

@pgrosu@deflaux@dionloy Now compiles :)

[pgrosu]

Awesome! Thanks Ilia :)

[wbrockman]

Is there really no builtin operation for this? Rats?

[iliat]

I could not find anything here: https://cloud.google.com/dataflow/java-sdk/JavaDoc/com/google/cloud/dataflow/sdk/transforms/package-summary
The only other alternative is to use a Coders and TextIO.Write.withCoder
but unfortunately the only prepackaged coder that can help is
TextualIntegerCoder and its not suitable for Longs.

[wbrockman]

Yup, thanks for the context - I concur there's no ToString transform that I can find at present. I've checked in with Jeff G from the Dataflow team to make sure I'm not missing anything.

[pgrosu]

So why not just take this out and put it into a common directory for all the extended functionality for Dataflow until it is implemented to make the code more readable? I get the feeling it will be reused.

[coveralls]

Coverage decreased (-7.79%) to 23.67% when pulling 653ae81 on iliat:dev-broad into 4b57fc1 on googlegenomics:master.

[lbergelson]

typo in storgae

[wbrockman]

LGTM - let's get the unit tests in a separate commit, as you say. Thanks for updating comments on the subtle question of why this code does find all the reads. Many thanks, and nice work - it's outstanding to have a full BAM reader available! Looking forward to experimenting with this further and perhaps iterating on the code design a bit once there are tests available.

[pgrosu]

Definitely +1 :)