[SPARK-8532][SQL] In Python's DataFrameWriter, save/saveAsTable/json/parquet/jdbc always override mode by yhuai · Pull Request #6937 · apache/spark

yhuai · 2015-06-22T17:44:09Z

https://issues.apache.org/jira/browse/SPARK-8532

This PR has two changes. First, it fixes the bug that save actions (i.e. save/saveAsTable/json/parquet/jdbc) always override mode. Second, it adds input argument partitionBy to save/saveAsTable/parquet.

yhuai · 2015-06-22T17:44:33Z

@davies @marmbrus @mengxr

SparkQA · 2015-06-22T17:50:26Z

Test build #35462 has finished for PR 6937 at commit 88eb6c4.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

davies · 2015-06-22T18:11:26Z

we should use "==" here

SparkQA · 2015-06-22T18:47:41Z

Test build #35465 has finished for PR 6937 at commit d696dff.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…de already uses "error" as the default value.

davies · 2015-06-22T19:19:20Z

new line here

davies · 2015-06-22T19:21:45Z

LGTM

SparkQA · 2015-06-22T19:39:18Z

Test build #35475 has finished for PR 6937 at commit d37abd2.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

davies · 2015-06-22T19:48:53Z

self.mode(mode)._jwrite.json(path)

SparkQA · 2015-06-22T19:48:58Z

Test build #35470 has finished for PR 6937 at commit 7fbc24b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

yhuai · 2015-06-22T19:54:31Z

Oh, this one hits

Archiving artifacts
WARN: No artifacts found that match the file pattern "**/target/unit-tests.log". Configuration error?
WARN: java.lang.InterruptedException: no matches found within 10000
Recording test results
ERROR: Publisher 'Publish JUnit test result report' failed: No test report files were found. Configuration error?
Finished: FAILURE

yhuai · 2015-06-22T20:36:50Z

I will merge it into master and branch-1.4 once SparkQA is happy.

SparkQA · 2015-06-22T20:47:12Z

Test build #35477 has finished for PR 6937 at commit f972d5d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class PCAModel(JavaVectorTransformer):
- class PCA(object):

AmplabJenkins · 2015-06-22T20:47:18Z

Merged build finished. Test FAILed.

…/parquet/jdbc always override mode https://issues.apache.org/jira/browse/SPARK-8532 This PR has two changes. First, it fixes the bug that save actions (i.e. `save/saveAsTable/json/parquet/jdbc`) always override mode. Second, it adds input argument `partitionBy` to `save/saveAsTable/parquet`. Author: Yin Huai <yhuai@databricks.com> Closes#6937 from yhuai/SPARK-8532 and squashes the following commits: f972d5d [Yin Huai] davies's comment. d37abd2 [Yin Huai] style. d21290a [Yin Huai] Python doc. 889eb25 [Yin Huai] Minor refactoring and add partitionBy to save, saveAsTable, and parquet. 7fbc24b [Yin Huai] Use None instead of "error" as the default value of mode since JVM-side already uses "error" as the default value. d696dff [Yin Huai] Python style. 88eb6c4 [Yin Huai] If mode is "error", do not call mode method. c40c461 [Yin Huai] Regression test. (cherry picked from commit 5ab9fcf) Signed-off-by: Yin Huai <yhuai@databricks.com>

…/parquet/jdbc always override mode https://issues.apache.org/jira/browse/SPARK-8532 This PR has two changes. First, it fixes the bug that save actions (i.e. `save/saveAsTable/json/parquet/jdbc`) always override mode. Second, it adds input argument `partitionBy` to `save/saveAsTable/parquet`. Author: Yin Huai <yhuai@databricks.com> Closesapache#6937 from yhuai/SPARK-8532 and squashes the following commits: f972d5d [Yin Huai] davies's comment. d37abd2 [Yin Huai] style. d21290a [Yin Huai] Python doc. 889eb25 [Yin Huai] Minor refactoring and add partitionBy to save, saveAsTable, and parquet. 7fbc24b [Yin Huai] Use None instead of "error" as the default value of mode since JVM-side already uses "error" as the default value. d696dff [Yin Huai] Python style. 88eb6c4 [Yin Huai] If mode is "error", do not call mode method. c40c461 [Yin Huai] Regression test. (cherry picked from commit 5ab9fcf) Signed-off-by: Yin Huai <yhuai@databricks.com>

…/parquet/jdbc always override mode https://issues.apache.org/jira/browse/SPARK-8532 This PR has two changes. First, it fixes the bug that save actions (i.e. `save/saveAsTable/json/parquet/jdbc`) always override mode. Second, it adds input argument `partitionBy` to `save/saveAsTable/parquet`. Author: Yin Huai <yhuai@databricks.com> Closesapache#6937 from yhuai/SPARK-8532 and squashes the following commits: f972d5d [Yin Huai] davies's comment. d37abd2 [Yin Huai] style. d21290a [Yin Huai] Python doc. 889eb25 [Yin Huai] Minor refactoring and add partitionBy to save, saveAsTable, and parquet. 7fbc24b [Yin Huai] Use None instead of "error" as the default value of mode since JVM-side already uses "error" as the default value. d696dff [Yin Huai] Python style. 88eb6c4 [Yin Huai] If mode is "error", do not call mode method. c40c461 [Yin Huai] Regression test.

yhuai added 2 commits June 22, 2015 10:40

Regression test.
c40c461

If mode is "error", do not call mode method.
88eb6c4

Python style.
d696dff

davies reviewed Jun 22, 2015
View reviewed changes

Comment threadpython/pyspark/sql/readwriter.py Outdated
daviesJun 22, 2015
Copy link
Copy Markdown
Contributor
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we should use "==" here

yhuai added 3 commits June 22, 2015 11:51

Use None instead of "error" as the default value of mode since JVM-si…
7fbc24b
…de already uses "error" as the default value.

Minor refactoring and add partitionBy to save, saveAsTable, and parquet.
889eb25

Python doc.
d21290a

davies reviewed Jun 22, 2015
View reviewed changes

Comment threadpython/pyspark/sql/readwriter.py Outdated
daviesJun 22, 2015
Copy link
Copy Markdown
Contributor
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
new line here

style.
d37abd2

davies reviewed Jun 22, 2015
View reviewed changes

Comment threadpython/pyspark/sql/readwriter.py Outdated
daviesJun 22, 2015
Copy link
Copy Markdown
Contributor
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
self.mode(mode)._jwrite.json(path)

davies's comment.
f972d5d

asfgit closed this in 5ab9fcfJun 22, 2015

Navigation Menu

Uh oh!

[SPARK-8532][SQL] In Python's DataFrameWriter, save/saveAsTable/json/parquet/jdbc always override mode - #6937

[SPARK-8532][SQL] In Python's DataFrameWriter, save/saveAsTable/json/parquet/jdbc always override mode#6937
yhuai wants to merge 8 commits into
apache:masterfrom
yhuai:SPARK-8532

yhuai commented Jun 22, 2015

Uh oh!

yhuai commented Jun 22, 2015

Uh oh!

SparkQA commented Jun 22, 2015

Uh oh!

daviesJun 22, 2015

Uh oh!

SparkQA commented Jun 22, 2015

Uh oh!

daviesJun 22, 2015

Uh oh!

davies commented Jun 22, 2015

Uh oh!

SparkQA commented Jun 22, 2015

Uh oh!

daviesJun 22, 2015

Uh oh!

SparkQA commented Jun 22, 2015

Uh oh!

yhuai commented Jun 22, 2015

Uh oh!

yhuai commented Jun 22, 2015

Uh oh!

SparkQA commented Jun 22, 2015

Uh oh!

AmplabJenkins commented Jun 22, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

yhuai commented Jun 22, 2015

Uh oh!

yhuai commented Jun 22, 2015

Uh oh!

SparkQA commented Jun 22, 2015

Uh oh!

daviesJun 22, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 22, 2015

Uh oh!

daviesJun 22, 2015

Choose a reason for hiding this comment

Uh oh!

davies commented Jun 22, 2015

Uh oh!

SparkQA commented Jun 22, 2015

Uh oh!

daviesJun 22, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 22, 2015

Uh oh!

yhuai commented Jun 22, 2015

Uh oh!

yhuai commented Jun 22, 2015

Uh oh!

SparkQA commented Jun 22, 2015

Uh oh!

AmplabJenkins commented Jun 22, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants