readr-spark

Natural language processing library for Apache Spark. It offers the following features:

• Wraps Stanford CoreNLP, CJ Parser, Allenai PolyParser, and much more
• All pluggable through a common data model
• Easy preview of outputs
• Easy scaling through Spark
• Annotator outputs stored as columns
• Incremental processing of annotations
• Efficient serialization through Kryo

In addition, it offers easy connectivity with the Readr cloud tool:

• Indices needed by Readr cloud computed in spark, and bulk loaded into Readr cloud
• Also includes computation for Readr interface features, such as text similarity
• Can be used in combination with readr-connect

Usage

We assume you have sbt 0.13 or higher installed. Start by creating an assembly for readr-spark and directories for our inputs and outputs.

mkdir $HOME/readr
cd $HOME/readr
git clone https://github.com/readr-code/readr-spark.git
cd readr-spark
sbt assembly
mkdir in
mkdir out

This will create a file main/target/scala-2.10/main-assembly-1.1-SNAPSHOT.jar that contains the readr-spark with all dependent jars.

Next, you convert your data into a format readable by readr-spark. Create a new sbt project and add the following to build.sbt.

libraryDependencies ++=Seq(
"com.readr"%"model"%"1.1-SNAPSHOT",
"com.readr"%"client"%"1.1-SNAPSHOT",
)
resolvers ++=Seq(
"Readr snapshots" at "http://snapshots.mvn-repo.readr.com",
"Readr releases" at "http://releases.mvn-repo.readr.com"
)

You can now write your text documents as follows:

importorg.apache.hadoop.conf.Configurationimportcom.readr.model.annotation._importcom.readr.client.util.AnnotationSequenceFileWriterobjectWriteInput {
defmain(args:Array[String]) {
valinDir=System.getenv("HOME") +"/readr/in"valconf=newConfiguration()
valw=newAnnotationSequenceFileWriter(conf, inDir +"/data.col0.TextAnn")
for (clazz <-Annotations.annWithDependentClazzes) w.register(clazz)
w.write(0, TextAnn("This is the first document."))
w.write(1, TextAnn("This is the second document."))
w.close
}
}

Make sure you have Spark installed. From the Spark directory, can now start the spark-shell and run a series of processors on our data.

SPARK_MEM=4G bin/spark-shell --master local[2] --jars "$HOME/readr/readr-spark/main/target/scala-2.10/main-assembly-1.1-SNAPSHOT.jar" --driver-java-options "-Dspark.serializer=org.apache.spark.serializer.KryoSerializer -Dspark.kryo.registrator=com.readr.spark.MyRegistrator -Dspark.kryoserializer.buffer.mb=16"

Copy and paste the following commands into spark shell:

valinDir=System.getenv("HOME") +"/readr/in"valoutDir=System.getenv("HOME") +"/readr/out"implicitvalisc= sc
importcom.readr.spark._importcom.readr.spark.rr._importcom.readr.spark.stanford34._importcom.readr.spark.other._importcom.readr.spark.frame._implicitvalse=newSchemavala= read(inDir, se).repartition(2)
valb= annotate(a, newStanfordTokenizer, se)
valc= annotate(b, newStanfordSentenceSplitter, se)
vald= annotate(c, newStanfordSRParser, se)
vale= annotate(d, newStanfordDependencyExtractor, se)
valf= annotate(e, newStanfordPOSTagger, se)
valg= annotate(f, newStanfordLemmatizer, se)
valh= annotate(g, newStanfordNERTagger, se)
vali= annotate(h, newSimpleConstituentExtractor, se)
//val j = annotate(i, new SimpleNERSegmenter, se)//val k = annotate(j, new StanfordCorefResolver, se)//val l = annotate(k, new SimpleMentionExtractor, se)//val m = annotate(l, new MintzLikeFeatureExtractor, se)valn= i
n.persist
write(n, outDir, se)

You can also view annotations in the spark-shell; for example, f.first._1 contains the document id for the first document and f.first._2 an array of all annotations up to the PolyParser annotations. f.first._2(7) returns the sentence dependency annotations.

After the files have been written, you can read them in code as follows:

importorg.apache.hadoop.conf.Configurationimportcom.readr.model.annotation._importcom.readr.client.util.AnnotationSequenceFileReaderobjectReadOutput {
defmain(args:scala.Array[String]):Unit= {
valoutDir=System.getenv("HOME") +"/readr/out"valconf=newConfiguration()
valr=newAnnotationSequenceFileReader(conf,
Array(classOf[TextAnn], classOf[SentenceDependencyAnn]),
outDir +"/data.col0.TextAnn",
outDir +"/data.col7.SentenceDependencyAnn")
for (clazz <-Annotations.annWithDependentClazzes) r.register(clazz)
vart:scala.Tuple2[Long,scala.Array[Any]] =nullwhile ({ t = r.read; t !=null} ) {
valid= t._1
valta= t._2(0).asInstanceOf[TextAnn]
valsda= t._2(1).asInstanceOf[SentenceDependencyAnn]
println(ta)
println(sda)
}
r.close
}
}

Appendix

For more information on how to connect with Readr cloud, see these examples. You can build the indexes for Readr cloud as follows

importcom.readr.spark.index._DocumentIndexer.run(outDir, n)
SourceIndexer.run(outDir, n)
TextIndexer.run(outDir, n)
TokenIndexer.run(outDir, n)
DependencyIndexer.run(outDir, n)
POSIndexer.run(outDir, n)
LemmaIndexer.run(outDir, n)

For an alternative pipeline using the Allenai Tools, you can use

importcom.readr.spark.allenai._valb= annotate(a, newFactorieSegmenter, se)
valc= annotate(b, newFactorieTokenizer, se)
vald= annotate(c, newFactoriePOSTagger, se)
vale= annotate(d, newMorphaStemmer, se)
valf= annotate(e, newPolyParser, se)
valg= annotate(f, new com.readr.spark.allenai.SimpleMentionExtractor, se)

Note that you will need to build the readr-spark assembly with allenai enabled in this case. Seeproject/Build.scala.