[SPARK-11691][SQL] Support setting hadoop compression codecs in DataFrameWriter#option by maropu · Pull Request #11324 · apache/spark

maropu · 2016-02-23T16:52:07Z

What changes were proposed in this pull request?

This pr is to support hadoop compression codecs when saving DataFrame to disk.
This is rework from #9657 because it gets stale.

closes#9657

SparkQA · 2016-02-23T20:37:30Z

Test build #51775 has finished for PR 11324 at commit b9efe53.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2016-02-24T00:37:56Z

@rxin Could you check this?

rxin · 2016-02-24T08:39:57Z

codec -> compressionCodec to make it more clear

rxin · 2016-02-24T08:48:41Z

@maropu can you add "closes #9657" to your pull request description?

maropu · 2016-02-24T08:50:17Z

okay, I'm fixing now.

SparkQA · 2016-02-24T12:13:33Z

Test build #51867 has finished for PR 11324 at commit a31e562.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-02-24T12:30:48Z

Test build #51872 has finished for PR 11324 at commit 7292aa7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2016-02-24T12:41:09Z

Jenkins, retest this please.

SparkQA · 2016-02-24T15:59:46Z

Test build #51875 has finished for PR 11324 at commit 7292aa7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2016-02-24T16:11:24Z

ISTM the fail is not related to this...

[info] HiveCompatibilitySuite:
[info] - lateral_view *** FAILED *** (933 milliseconds)
[info] Failed to execute query using catalyst:
[info] Error: Job aborted due to stage failure: Task 0 in stage 8347.0 failed 1 times, most recent failure: Lost task 0.0 in stage 8347.0 (TID 120049, localhost): java.lang.ClassCastException: org.apache.spark.sql.catalyst.expressions.JoinedRow cannot be cast to org.apache.spark.sql.catalyst.expressions.UnsafeRow

rxin · 2016-02-24T21:26:53Z

Take a look at the end of ParquetRelation. It would be great if we can consolidate the two and put in a single place where we specify short compression codec names.

rxin · 2016-02-24T21:27:01Z

Yea I reverted a patch that broke the test.

…tion when saving

maropu · 2016-02-25T03:12:47Z

Okay, I'll try.

SparkQA · 2016-02-25T05:57:40Z

Test build #51934 has finished for PR 11324 at commit 14ef39a.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2016-02-25T06:24:28Z

Jenkins, retest this please.

SparkQA · 2016-02-25T07:58:40Z

Test build #51935 has finished for PR 11324 at commit 86c8c0c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-02-25T08:30:38Z

Test build #51941 has finished for PR 11324 at commit 86c8c0c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2016-02-25T08:31:54Z

Jenkins, retest this please.

SparkQA · 2016-02-25T11:14:38Z

Test build #51949 has finished for PR 11324 at commit 8cfbaea.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2016-02-25T11:26:35Z

@rxin okay, ready to review again.

HyukjinKwon · 2016-02-26T12:26:49Z

@maropu @zjffdu @rxin I apologize that I carelessly open the same issue and submitted a PR. This is fixed in #11384.

maropu · 2016-02-26T14:58:51Z

@rxin @HyukjinKwon No problem ;) Is it okay to apply my diffs based on the @HyukjinKwon commit?

rxin · 2016-02-26T19:05:24Z

Yes - please.

maropu · 2016-02-27T01:48:09Z

@HyukjinKwon Is it okay to include document descriptions for codec options in this pr?

maropu · 2016-02-27T04:43:21Z

Close this pr, and move to a new pr.

…ent in ParquetRelation ## What changes were proposed in this pull request? This pr to make the short names of compression codecs in `ParquetRelation` consistent against other ones. This pr comes from #11324. ## How was this patch tested? Add more tests in `TextSuite`. Author: Takeshi YAMAMURO <linguin.m.s@gmail.com> Closes#11408 from maropu/SPARK-13528.

…ent in ParquetRelation ## What changes were proposed in this pull request? This pr to make the short names of compression codecs in `ParquetRelation` consistent against other ones. This pr comes from apache#11324. ## How was this patch tested? Add more tests in `TextSuite`. Author: Takeshi YAMAMURO <linguin.m.s@gmail.com> Closesapache#11408 from maropu/SPARK-13528.

rxin reviewed Feb 24, 2016
View reviewed changes

maropu force-pushed the pr9657 branch from d70844d to 7292aa7Compare February 24, 2016 10:38

zjffduand others added 11 commits February 25, 2016 12:08

[SPARK-11691][SQL] Allow to specify compression codec in HadoopFsRela…
6e07cc7
…tion when saving

fix compilation issue
e67c3b9

address comments
905e7fa

minor change
9d7c38b

change back to 1.7.0
691a8f4

fix review comments
04bcd93

fix code style
bf84998

minor fix
80eb810

Apply comments
6eb4191

Change variable names
7614240

Add shortcut names for compression codecs
9364b53

maropu force-pushed the pr9657 branch 2 times, most recently from 14ef39a to 86c8c0cCompare February 25, 2016 05:53

Add an utility class to map short codec names to qualified ones
8cfbaea

maropu force-pushed the pr9657 branch from 86c8c0c to 8cfbaeaCompare February 25, 2016 08:31

maropu mentioned this pull request Feb 27, 2016
[SPARK-13528][SQL] Make the short names of compression codecs consistent in ParquetRelation #11408
Closed

maropu closed this Feb 27, 2016

maropu deleted the pr9657 branch July 5, 2017 11:43

Uh oh!

Conversation

maropu commented Feb 23, 2016

What changes were proposed in this pull request?

Uh oh!

SparkQA commented Feb 23, 2016

Uh oh!

maropu commented Feb 24, 2016

Uh oh!

rxinFeb 24, 2016

Choose a reason for hiding this comment

Uh oh!

rxin commented Feb 24, 2016

Uh oh!

maropu commented Feb 24, 2016

Uh oh!

SparkQA commented Feb 24, 2016

Uh oh!

SparkQA commented Feb 24, 2016

Uh oh!

maropu commented Feb 24, 2016

Uh oh!

SparkQA commented Feb 24, 2016

Uh oh!

maropu commented Feb 24, 2016

Uh oh!

rxin commented Feb 24, 2016

Uh oh!

rxin commented Feb 24, 2016

Uh oh!

maropu commented Feb 25, 2016

Uh oh!

SparkQA commented Feb 25, 2016

Uh oh!

maropu commented Feb 25, 2016

Uh oh!

SparkQA commented Feb 25, 2016

Uh oh!

SparkQA commented Feb 25, 2016

Uh oh!

maropu commented Feb 25, 2016

Uh oh!

SparkQA commented Feb 25, 2016

Uh oh!

maropu commented Feb 25, 2016

Uh oh!

HyukjinKwon commented Feb 26, 2016

Uh oh!

maropu commented Feb 26, 2016

Uh oh!

rxin commented Feb 26, 2016

Uh oh!

maropu commented Feb 27, 2016

Uh oh!

maropu commented Feb 27, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants