[WIP][SPARK-2883][SQL]initial support ORC in spark sql

[scwf]

Provide a initial support for orc file formate in spark sql, support for both reading and writing ORC files . user can use ORC file just like parquet as follow:

... ...
rdd.registerTempTable("records")
rdd.saveAsOrcFile("pair.orc")
val orcFile = sqlContext.orcFile("pair.orc")
orcFile.registerTempTable("orcFile")
sql("SELECT * FROM orcFile").collect().foreach(println)

[AmplabJenkins]

Can one of the admins verify this patch?

[cloud-fan]

Have you considered cooperate with #2475?

[scwf]

@cloud-fan, no since #2475 has not merged to master. @marmbrus can you take a look at this?

[yhuai]

I guess we should revisit it after #2475 is in.

[pwendell]

Is there a way we can depend only on ORC file and not on the entire hive-exec package? It think we'll need to move this functionality into the hive package if it requires all of the hive dependencies.

[pwendell]

Yeah it appears ORC has deep dependencies on Hive types. So I think this will need to be moved into the Hive project.

[scwf]

Actually i also want that, but unfortunately have not found the solution

[scwf]

hive project have supported ORC, this PR is to enable sql project to support it, just as it support parquet

[marmbrus]

The whole point of having hive as a separate project is to avoid pulling in all of the dependencies of hive for all spark users. Thus, as Patrick said, this will have to go in the hive sub project. That said, I think this patch is still quite valuable. We generally recommend that all users use the HiveContext if they can tolerate hive's dependencies, so it'll still get use. Also, this patch provides better programatic API support in addition to the existing SQL support.

[marmbrus]

Hey @scwf, thanks for working on this! This will be a pretty awesome feature that people have been asking for. I did a quick pass and made some comments.

One higher level comment, we'll want to change this to use an API similar to #2475 as we are going to stop adding new data source methods directly to SQLContext (and we'll probably try and deprecate parquet and json from there as well eventually). I haven't had enough time to finish that PR yet (it doesn't yet support inserting data), but will try and get that merged in soon.

[yhuai]

Once #2616 is in, can we reuse stuff in hiveWriterContainers.scala and InsertIntoHiveTable.scala to avoid duplicating code?

[marmbrus]

ok to test

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21149/

[SparkQA]

QA tests have started for PR 2576 at commit f928657.

• This patch merges cleanly.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21288/Test FAILed.

[SparkQA]

QA tests have started for PR 2576 at commit 1505af4.

• This patch merges cleanly.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21290/Test FAILed.

[scwf]

Added support orc with new datasource API. Since now there is no sink interface in data source api, not remove the old version.
TODO:
to fix issue of partitioned table

[SparkQA]

Test build #23756 has started for PR 2576 at commit 1e0c1d9.

• This patch merges cleanly.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23756/
Test FAILed.

[zhzhan]

Orc files, instead of parquet files

[zhzhan]

@scwf I send a pull request to your orc1 branch for the predictor push down support. Please take a look at it, and merge them. Let me know if you have any concerns. scwf#13

[SparkQA]

Test build #24014 has started for PR 2576 at commit 601d242.

• This patch merges cleanly.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/24014/
Test FAILed.

[scwf]

@marmbrus, i am fixing the test failure and refactoring the code based on datasource api, and one question here is, should i keep the sink part(write interface) here? Or just provide the ability to read orc file based on datasource api?

[marmbrus]

We are adding support for writing data in the next version of the API so probably better to wait until that is available.

[scwf]

OK, so i make #3753 to support orc based on the new datasource api. @zhzhan, in that PR it's support to read partitioned orc files but no ppd(ppd also need refactory based on the new datasource api), you can make a follow up PR for ppd after it merged or you can send the ppd support to my branch, anyway is ok.

[marmbrus]

We can close this issue right?

[scwf]

yes, closed!