Skip to content

PHOENIX-1816 Update phoenix-spark README.md for website documentation#64

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jmahonin wants to merge 1 commit into from
Closed

PHOENIX-1816 Update phoenix-spark README.md for website documentation #64

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 40 additions & 13 deletions phoenix-spark/README.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,31 @@
phoenix-spark extends Phoenix's MapReduce support to allow Spark to load Phoenix tables as RDDs or
DataFrames, and enables persisting RDDs of Tuples back to Phoenix.
# Apache Spark Plugin

## Reading Phoenix Tables
The phoenix-spark plugin extends Phoenix's MapReduce support to allow Spark to load Phoenix tables
as RDDs or DataFrames, and enables persisting them back to Phoenix.

#### Prerequisites

* Phoenix 4.4.0-SNAPSHOT +
* Spark 1.3.0 +

#### Spark setup

1. Ensure that all requisite Phoenix / HBase platform dependencies are available on the classpath for the Spark executors and drivers
2. One method is to add the phoenix-4.4.0-SNAPSHOT-client.jar to 'SPARK_CLASSPATH' in spark-env.sh,
or setting both 'spark.executor.extraClassPath' and 'spark.driver.extraClassPath' in
spark-defaults.conf
3. To help your IDE, you may want to add the following 'provided' dependency:

```
<dependency>
<groupId>org.apache.phoenix</groupId>
<artifactId>phoenix-spark</artifactId>
<version>${phoenix.version}</version>
<scope>provided</scope>
</dependency>
```

### Reading Phoenix Tables

Given a Phoenix table with the following DDL

Expand All@@ -11,7 +35,7 @@ UPSERT INTO TABLE1 (ID, COL1) VALUES (1, 'test_row_1');
UPSERT INTO TABLE1 (ID, COL1) VALUES (2, 'test_row_2');
```

### Load as a DataFrame using the Data Source API
#### Load as a DataFrame using the Data Source API
```scala
import org.apache.spark.SparkContext
import org.apache.spark.sql.SQLContext
Expand All@@ -31,7 +55,7 @@ df
.show
```

### Load as a DataFrame directly using a Configuration object
#### Load as a DataFrame directly using a Configuration object
```scala
import org.apache.hadoop.conf.Configuration
import org.apache.spark.SparkContext
Expand All@@ -52,7 +76,7 @@ val df = sqlContext.phoenixTableAsDataFrame(
df.show
```

### Load as an RDD, using a Zookeeper URL
#### Load as an RDD, using a Zookeeper URL
```scala
import org.apache.spark.SparkContext
import org.apache.spark.sql.SQLContext
Expand All@@ -71,17 +95,20 @@ val firstId = rdd1.first()("ID").asInstanceOf[Long]
val firstCol = rdd1.first()("COL1").asInstanceOf[String]
```

## Saving RDDs to Phoenix

`saveToPhoenix` is an implicit method on RDD[Product], or an RDD of Tuples. The data types must
correspond to the Java types Phoenix supports (http://phoenix.apache.org/language/datatypes.html)
### Saving Phoenix

Given a Phoenix table with the following DDL

```sql
CREATE TABLE OUTPUT_TEST_TABLE (id BIGINT NOT NULL PRIMARY KEY, col1 VARCHAR, col2 INTEGER);
```

#### Saving RDDs

`saveToPhoenix` is an implicit method on RDD[Product], or an RDD of Tuples. The data types must
correspond to the Java types Phoenix supports (http://phoenix.apache.org/language/datatypes.html)


```scala
import org.apache.spark.SparkContext
import org.apache.phoenix.spark._
Expand All@@ -98,7 +125,7 @@ sc
)
```

## Saving DataFrames to Phoenix
#### Saving DataFrames

The `save` is method on DataFrame allows passing in a data source type. You can use
`org.apache.phoenix.spark`, and must also pass in a `table` and `zkUrl` parameter to
Expand DownExpand Up@@ -130,7 +157,7 @@ df.save("org.apache.phoenix.spark", SaveMode.Overwrite, Map("table" -> "OUTPUT_T
"zkUrl" -> hbaseConnectionString))
```

## Notes
### Notes

The functions `phoenixTableAsDataFrame`, `phoenixTableAsRDD` and `saveToPhoenix` all support
optionally specifying a `conf` Hadoop configuration parameter with custom Phoenix client settings,
Expand All@@ -139,7 +166,7 @@ as well as an optional `zkUrl` parameter for the Phoenix connection URL.
If `zkUrl` isn't specified, it's assumed that the "hbase.zookeeper.quorum" property has been set
in the `conf` parameter. Similarly, if no configuration is passed in, `zkUrl` must be specified.

## Limitations
### Limitations

- Basic support for column and predicate pushdown using the Data Source API
- The Data Source API does not support passing custom Phoenix settings in configuration, you must
Expand Down