[SPARK-20728][SQL] Make ORCFileFormat configurable between sql/hive and sql/core by dongjoon-hyun · Pull Request #17980 · apache/spark

dongjoon-hyun · 2017-05-15T02:44:57Z

What changes were proposed in this pull request?

SPARK-20682 is trying to improve Apache Spark to have a new ORCFileFormat based on Apache ORC for many reasons.

On top of that, this PR depends on SPARK-20682 and aims to provide a configuration to choose the default ORCFileFormat from legacy sql/hive module or new sql/core module.

For example, this configuration will affects the following operations.

spark.read.orc(...)

CREATETABLEt
USING ORC
...

Since SPARK-20682 (#17924 and #17943) are still under review, I'm inevitably including the dependent code. I'll update this and previous PR according to the review result.

How was this patch tested?

Pass the Jenkins with new test suites.

SparkQA · 2017-05-15T05:02:48Z

Test build #76924 has finished for PR 17980 at commit 7716234.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-05-15T05:42:35Z

Test build #76931 has started for PR 17980 at commit 73d56f2.

dongjoon-hyun · 2017-05-15T05:54:01Z

We can remove this test case when we remove sql/hive ORCFileFormat.

dongjoon-hyun · 2017-05-15T07:07:50Z

Retest this please.

SparkQA · 2017-05-15T10:04:03Z

Test build #76935 has finished for PR 17980 at commit 73d56f2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-05-17T06:15:09Z

Test build #77011 has started for PR 17980 at commit 87dd161.

dongjoon-hyun · 2017-05-17T15:33:00Z

Retest this please

SparkQA · 2017-05-17T19:57:22Z

Test build #77019 has finished for PR 17980 at commit 87dd161.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-05-24T00:31:58Z

Test build #77264 has finished for PR 17980 at commit 05716c8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-06-10T21:53:11Z

Hi, @sameeragarwal , @cloud-fan , @gatorsmile .
Could you give me some advice on this? I think this is the start of Feature Parity for ORC.

dongjoon-hyun · 2017-06-16T21:35:12Z

Retest this please.

SparkQA · 2017-06-17T00:35:30Z

Test build #78192 has finished for PR 17980 at commit 05716c8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-07-06T07:04:56Z

Test build #79264 has finished for PR 17980 at commit 39c3851.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-07-06T07:31:28Z

Retest this please.

SparkQA · 2017-07-06T10:28:58Z

Test build #79269 has finished for PR 17980 at commit 39c3851.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-07-06T22:46:46Z

Hi, @rxin, @sameeragarwal , @cloud-fan , @gatorsmile , @viirya .
Since Apache Spark 2.2.0 release is going to coming soon, it's a good time to restart this.
Could you give me some advice how to make ORC support up-to-date in Apache Spark 2.3?

SparkQA · 2017-07-09T12:02:35Z

Test build #79412 has finished for PR 17980 at commit cbc5bc2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-07-12T23:08:49Z

Test build #79570 has finished for PR 17980 at commit 5db6acf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

kiszk · 2017-07-14T18:51:45Z

In general, this PR seems to have the large number of lines.
Are there any ways to split this PR into multiple smaller PRs?

dongjoon-hyun · 2017-07-14T21:31:03Z

Thank you for review, @kiszk .

The first one about adding new ORC source is more smaller than this .
Also, there is more smaller version without vectorized part.

If we need to split more, we can split the followings.

Adding Apache ORC dependency (pom and dependency changes)
OrcReadBenchmark (388 lines)

Is that helpful for review?

In general, 50% lines are new ORC test codes which is duplicated from old ORC test codes. Could you recommend some better way?

dongjoon-hyun · 2017-07-14T21:55:23Z

Hi, @kiszk . I will start with Adding Apache ORC dependency (pom and dependency changes) in
SPARK-21422 first.

kiszk · 2017-07-15T03:59:58Z

If new test cases work for the existing orc component, how about updating test cases at first?

dongjoon-hyun · 2017-07-15T15:03:33Z

I developed new ORC component and tests in sql/core module without sql/hive module. Do you mean add a sql/hive module test-dependency into sql module?

kiszk · 2017-07-17T14:37:00Z

I see. I thought that new tests are for old and new ORC components.

dongjoon-hyun · 2017-07-17T15:46:42Z

I'm trying to split the PR as you recommended before. Thank you always.

dongjoon-hyun · 2017-08-03T23:18:06Z

Retest this please.

SparkQA · 2017-08-04T02:25:01Z

Test build #80224 has finished for PR 17980 at commit 5db6acf.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-08-04T16:21:51Z

Retest this please

SparkQA · 2017-08-04T19:34:03Z

Test build #80254 has finished for PR 17980 at commit 5db6acf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-08-14T05:48:23Z

Test build #80603 has finished for PR 17980 at commit 5db6acf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

## What changes were proposed in this pull request? Like Parquet, this PR aims to depend on the latest Apache ORC 1.4 for Apache Spark 2.3. There are key benefits for Apache ORC 1.4. - Stability: Apache ORC 1.4.0 has many fixes and we can depend on ORC community more. - Maintainability: Reduce the Hive dependency and can remove old legacy code later. Later, we can get the following two key benefits by adding new ORCFileFormat in SPARK-20728 (apache#17980), too. - Usability: User can use ORC data sources without hive module, i.e, -Phive. - Speed: Use both Spark ColumnarBatch and ORC RowBatch together. This will be faster than the current implementation in Spark. ## How was this patch tested? Pass the jenkins. Author: Dongjoon Hyun <dongjoon@apache.org> Closesapache#18640 from dongjoon-hyun/SPARK-21422.

SparkQA · 2017-08-16T07:59:21Z

Test build #80727 has finished for PR 17980 at commit 5a9958d.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-08-16T10:09:22Z

Test build #80729 has finished for PR 17980 at commit 61b27e5.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-08-16T15:40:56Z

Retest this please.

SparkQA · 2017-08-16T18:19:58Z

Test build #80741 has finished for PR 17980 at commit 61b27e5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2017-08-24T15:26:07Z

Retest this please.

SparkQA · 2017-08-24T15:33:01Z

Test build #81090 has finished for PR 17980 at commit 61b27e5.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-08-24T17:03:12Z

Test build #81095 has finished for PR 17980 at commit 02371c6.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-08-24T20:33:06Z

Test build #81096 has finished for PR 17980 at commit 04b8494.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-08-28T23:30:31Z

Test build #81190 has finished for PR 17980 at commit 257e5a6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-08-31T06:50:59Z

Test build #81274 has finished for PR 17980 at commit edead5d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-09-04T10:24:54Z

Test build #81379 has finished for PR 17980 at commit c50e997.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…nd sql/core

SparkQA · 2017-09-11T00:17:39Z

Test build #81612 has finished for PR 17980 at commit 01868ab.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

jiangxb1987 · 2017-11-06T09:06:07Z

@dongjoon-hyun Do we still need this PR?

dongjoon-hyun · 2017-11-06T21:59:29Z

Thank you for review, @jiangxb1987 . Yes. This is still required. I'll rebase this PR after #19651 .

dongjoon-hyun · 2017-12-21T17:12:05Z

Almost codes inside this PR are merged now except #19943 .

dongjoon-hyun commented May 15, 2017
View reviewed changes

dongjoon-hyun mentioned this pull request Jul 14, 2017
[SPARK-21422][BUILD] Depend on Apache ORC 1.4.0 #18640
Closed

dongjoon-hyun mentioned this pull request Aug 16, 2017
[SPARK-20682][SQL] Update ORC data source based on Apache ORC library #18953
Closed

dongjoon-hyun mentioned this pull request Sep 3, 2017
[SPARK-20682][SQL] Support a new faster ORC data source based on Apache ORC #17924
Closed

This was referenced Sep 8, 2017
[SPARK-20682][SQL] Implement new ORC data source based on Apache ORC #17943
Closed
[SPARK-21791][SQL] ORC should support column names with dot #19004
Closed

[SPARK-20728][SQL] Make ORCFileFormat configurable between sql/hive a…
01868ab
…nd sql/core

dongjoon-hyun mentioned this pull request Oct 26, 2017
[SPARK-15474][SQL] Write and read back non-emtpy schema with empty dataframe #19571
Closed

dongjoon-hyun mentioned this pull request Nov 3, 2017
[SPARK-20682][SPARK-15474][SPARK-21791] Add new ORCFileFormat based on ORC 1.4.1 #19651
Closed

dongjoon-hyun mentioned this pull request Dec 4, 2017
[SPARK-20728][SQL] Make OrcFileFormat configurable between sql/hive and sql/core #19871
Closed

dongjoon-hyun closed this Dec 21, 2017

dongjoon-hyun deleted the SPARK-20728 branch December 21, 2017 17:12

Uh oh!

Conversation

dongjoon-hyun commented May 15, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented May 15, 2017

Uh oh!

SparkQA commented May 15, 2017

Uh oh!

dongjoon-hyunMay 15, 2017

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented May 15, 2017

Uh oh!

SparkQA commented May 15, 2017

Uh oh!

SparkQA commented May 17, 2017

Uh oh!

dongjoon-hyun commented May 17, 2017

Uh oh!

SparkQA commented May 17, 2017

Uh oh!

SparkQA commented May 24, 2017

Uh oh!

dongjoon-hyun commented Jun 10, 2017

Uh oh!

dongjoon-hyun commented Jun 16, 2017

Uh oh!

SparkQA commented Jun 17, 2017

Uh oh!

SparkQA commented Jul 6, 2017

Uh oh!

dongjoon-hyun commented Jul 6, 2017

Uh oh!

SparkQA commented Jul 6, 2017

Uh oh!

dongjoon-hyun commented Jul 6, 2017

Uh oh!

SparkQA commented Jul 9, 2017

Uh oh!

SparkQA commented Jul 12, 2017

Uh oh!

kiszk commented Jul 14, 2017

Uh oh!

dongjoon-hyun commented Jul 14, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Jul 14, 2017

Uh oh!

kiszk commented Jul 15, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Jul 15, 2017

Uh oh!

kiszk commented Jul 17, 2017

Uh oh!

dongjoon-hyun commented Jul 17, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Aug 3, 2017

Uh oh!

SparkQA commented Aug 4, 2017

Uh oh!

dongjoon-hyun commented Aug 4, 2017

Uh oh!

SparkQA commented Aug 4, 2017

Uh oh!

SparkQA commented Aug 14, 2017

Uh oh!

SparkQA commented Aug 16, 2017

Uh oh!

SparkQA commented Aug 16, 2017

Uh oh!

dongjoon-hyun commented Aug 16, 2017

Uh oh!

SparkQA commented Aug 16, 2017

Uh oh!

dongjoon-hyun commented Aug 24, 2017

dongjoon-hyun commented May 15, 2017•
edited
Loading

dongjoon-hyun commented Jul 14, 2017•
edited
Loading

kiszk commented Jul 15, 2017•
edited
Loading

dongjoon-hyun commented Jul 17, 2017•
edited
Loading