diff --git a/phoenix-spark/README.md b/phoenix-spark/README.md index 1e53c98e2ad..907109102d7 100644 --- a/phoenix-spark/README.md +++ b/phoenix-spark/README.md @@ -1,7 +1,31 @@ -phoenix-spark extends Phoenix's MapReduce support to allow Spark to load Phoenix tables as RDDs or -DataFrames, and enables persisting RDDs of Tuples back to Phoenix. +# Apache Spark Plugin -## Reading Phoenix Tables +The phoenix-spark plugin extends Phoenix's MapReduce support to allow Spark to load Phoenix tables +as RDDs or DataFrames, and enables persisting them back to Phoenix. + +#### Prerequisites + +* Phoenix 4.4.0-SNAPSHOT + +* Spark 1.3.0 + + +#### Spark setup + +1. Ensure that all requisite Phoenix / HBase platform dependencies are available on the classpath for the Spark executors and drivers +2. One method is to add the phoenix-4.4.0-SNAPSHOT-client.jar to 'SPARK_CLASSPATH' in spark-env.sh, +or setting both 'spark.executor.extraClassPath' and 'spark.driver.extraClassPath' in +spark-defaults.conf +3. To help your IDE, you may want to add the following 'provided' dependency: + +``` + + org.apache.phoenix + phoenix-spark + ${phoenix.version} + provided + +``` + +### Reading Phoenix Tables Given a Phoenix table with the following DDL @@ -11,7 +35,7 @@ UPSERT INTO TABLE1 (ID, COL1) VALUES (1, 'test_row_1'); UPSERT INTO TABLE1 (ID, COL1) VALUES (2, 'test_row_2'); ``` -### Load as a DataFrame using the Data Source API +#### Load as a DataFrame using the Data Source API ```scala import org.apache.spark.SparkContext import org.apache.spark.sql.SQLContext @@ -31,7 +55,7 @@ df .show ``` -### Load as a DataFrame directly using a Configuration object +#### Load as a DataFrame directly using a Configuration object ```scala import org.apache.hadoop.conf.Configuration import org.apache.spark.SparkContext @@ -52,7 +76,7 @@ val df = sqlContext.phoenixTableAsDataFrame( df.show ``` -### Load as an RDD, using a Zookeeper URL +#### Load as an RDD, using a Zookeeper URL ```scala import org.apache.spark.SparkContext import org.apache.spark.sql.SQLContext @@ -71,10 +95,7 @@ val firstId = rdd1.first()("ID").asInstanceOf[Long] val firstCol = rdd1.first()("COL1").asInstanceOf[String] ``` -## Saving RDDs to Phoenix - -`saveToPhoenix` is an implicit method on RDD[Product], or an RDD of Tuples. The data types must -correspond to the Java types Phoenix supports (http://phoenix.apache.org/language/datatypes.html) +### Saving Phoenix Given a Phoenix table with the following DDL @@ -82,6 +103,12 @@ Given a Phoenix table with the following DDL CREATE TABLE OUTPUT_TEST_TABLE (id BIGINT NOT NULL PRIMARY KEY, col1 VARCHAR, col2 INTEGER); ``` +#### Saving RDDs + +`saveToPhoenix` is an implicit method on RDD[Product], or an RDD of Tuples. The data types must +correspond to the Java types Phoenix supports (http://phoenix.apache.org/language/datatypes.html) + + ```scala import org.apache.spark.SparkContext import org.apache.phoenix.spark._ @@ -98,7 +125,7 @@ sc ) ``` -## Saving DataFrames to Phoenix +#### Saving DataFrames The `save` is method on DataFrame allows passing in a data source type. You can use `org.apache.phoenix.spark`, and must also pass in a `table` and `zkUrl` parameter to @@ -130,7 +157,7 @@ df.save("org.apache.phoenix.spark", SaveMode.Overwrite, Map("table" -> "OUTPUT_T "zkUrl" -> hbaseConnectionString)) ``` -## Notes +### Notes The functions `phoenixTableAsDataFrame`, `phoenixTableAsRDD` and `saveToPhoenix` all support optionally specifying a `conf` Hadoop configuration parameter with custom Phoenix client settings, @@ -139,7 +166,7 @@ as well as an optional `zkUrl` parameter for the Phoenix connection URL. If `zkUrl` isn't specified, it's assumed that the "hbase.zookeeper.quorum" property has been set in the `conf` parameter. Similarly, if no configuration is passed in, `zkUrl` must be specified. -## Limitations +### Limitations - Basic support for column and predicate pushdown using the Data Source API - The Data Source API does not support passing custom Phoenix settings in configuration, you must