[SPARK-5212][SQL] Add support of schema-less, custom field delimiter and SerDe for HiveQL transform by viirya · Pull Request #4014 · apache/spark

viirya · 2015-01-13T07:40:15Z

This pr adds the support of schema-less syntax, custom field delimiter and SerDe for HiveQL's transform.

SparkQA · 2015-01-13T07:43:24Z

Test build #25462 has finished for PR 4014 at commit ccee49e.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-01-13T08:55:57Z

Test build #25463 has finished for PR 4014 at commit b1729d9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-01-14T14:38:29Z

Test build #25549 has finished for PR 4014 at commit 7a48e42.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-01-14T15:50:11Z

Test build #25550 has finished for PR 4014 at commit ab22f7b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-01-16T17:48:32Z

Test build #25669 has finished for PR 4014 at commit 5e0b864.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- val trimed_class = outputSerdeClass.split("'")(1)
- val trimed_class = inputSerdeClass.split("'")(1)

SparkQA · 2015-01-16T18:43:08Z

Test build #25670 has finished for PR 4014 at commit 4d21956.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- val trimed_class = outputSerdeClass.split("'")(1)
- val trimed_class = inputSerdeClass.split("'")(1)

SparkQA · 2015-01-17T07:59:08Z

Test build #25699 has finished for PR 4014 at commit a711657.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- val trimed_class = outputSerdeClass.split("'")(1)
- val trimed_class = inputSerdeClass.split("'")(1)

SparkQA · 2015-01-17T16:14:45Z

Test build #25703 has finished for PR 4014 at commit ab22f7b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-01-18T15:08:35Z

Test build #25723 has finished for PR 4014 at commit 32d3046.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- val trimed_class = outputSerdeClass.split("'")(1)
- val trimed_class = inputSerdeClass.split("'")(1)

SparkQA · 2015-01-18T16:35:26Z

Test build #25724 has finished for PR 4014 at commit be2c3fc.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- val trimed_class = outputSerdeClass.split("'")(1)
- val trimed_class = inputSerdeClass.split("'")(1)

SparkQA · 2015-01-18T20:23:25Z

Test build #25729 has finished for PR 4014 at commit 799b5e1.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- val trimed_class = outputSerdeClass.split("'")(1)
- val trimed_class = inputSerdeClass.split("'")(1)

SparkQA · 2015-01-19T10:06:18Z

Test build #25756 has finished for PR 4014 at commit 7a14f31.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- val trimed_class = outputSerdeClass.split("'")(1)
- val trimed_class = inputSerdeClass.split("'")(1)

… it.

SparkQA · 2015-01-19T11:46:53Z

Test build #25758 has finished for PR 4014 at commit 9a6dc04.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- val trimed_class = outputSerdeClass.split("'")(1)
- val trimed_class = inputSerdeClass.split("'")(1)

chenghao-intel · 2015-01-23T00:46:59Z

I think a better place to extract the schema (the output) is in Analyzer, HiveContext should be able to create its own rules for that, instead of doing this in Strategy. Otherwise it probably fails in resolving the attributes.
e.g.:
SELECT transform(key + 1, value) USING '/bin/cat' FROM src ORDER BY key, value`
sorry, I didn't test that, let me know if I am wrong.

Good point. I didn't notice that. New commit will fix it. Thanks.

SparkQA · 2015-01-29T02:04:56Z

Test build #26273 has finished for PR 4014 at commit aa10fbd.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class HiveScriptIOSchema (
- val trimed_class = serdeClassName.split("'")(1)
- case class ShimWritable(writable: Writable)
- case class ShimWritable(writable: Writable)

viirya · 2015-01-29T02:06:23Z

@rxin Would you like to take a look at this too and see if it is ready to merge? Thanks.

rxin · 2015-01-29T02:09:09Z

Can you explain in the PR what is schema-less delimiter?

viirya · 2015-01-29T02:20:39Z

Schema-less Map-reduce Scripts is a feature of Hive transform syntax. That is there is no AS clause after USING my_script. Hive assumes that the script output contains two columns: key and value. The example SQL looks like:

SELECT TRANSFORM (key, value) USING 'cat' FROM src

Custom delimiter is defined by ROW FORMAT clause such as:

SELECT TRANSFORM (key, value) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\002' USING 'cat' AS (tKey, tValue) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\002' FROM src

So you can use field delimiters other than default \t.

viirya · 2015-01-31T12:14:08Z

@rxin I have added the explanation for this feature. Would you have time to review this pr and see if it is ok to merge? Thanks!

marmbrus · 2015-02-02T01:56:00Z

Remove this extra line.

marmbrus · 2015-02-02T03:41:06Z

Thanks for working on this! It would be great if this could be updated soon so we can include it in 1.3.

SparkQA · 2015-02-02T14:22:04Z

Test build #26516 has finished for PR 4014 at commit ac2d1fe.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class HiveScriptIOSchema (
- val trimed_class = serdeClassName.split("'")(1)

viirya · 2015-02-02T14:29:06Z

@marmbrus I did some refactoring for the comments. It should be better now.

marmbrus · 2015-02-02T21:54:25Z

Thanks! Merged to master.

chenghao-intel · 2015-04-24T08:56:50Z

I just file a jira issue, https://issues.apache.org/jira/browse/SPARK-7119. @viirya can you help on investigate this?

viirya · 2015-04-24T09:59:32Z

@chenghao-intel ok.

viirya added 2 commits January 13, 2015 02:08

Add support of schema-less script transformation.
f561c37

Add unit test.
ccee49e

Fix style.
b1729d9

Add support of custom field delimiter.
7a48e42

viirya changed the title ~~[SPARK-5212][SQL] Add support of schema-less transformation~~[SPARK-5212][SQL] Add support of schema-less and custom field delimiter for HiveQL transformJan 14, 2015

Fix style.
ab22f7b

viirya changed the title ~~[SPARK-5212][SQL] Add support of schema-less and custom field delimiter for HiveQL transform~~[SPARK-5212][SQL] Add support of schema-less, custom field delimiter and SerDe for HiveQL transformJan 16, 2015

viirya force-pushed the schema_less_trans branch from a711657 to ab22f7bCompare January 17, 2015 15:04

viirya changed the title ~~[SPARK-5212][SQL] Add support of schema-less, custom field delimiter and SerDe for HiveQL transform~~[SPARK-5212][SQL] Add support of schema-less, custom field delimiter for HiveQL transformJan 17, 2015

Add SerDe support.
32d3046

Fix style.
be2c3fc

Call getSerializedClass instead of using Text.
799b5e1

viirya changed the title ~~[SPARK-5212][SQL] Add support of schema-less, custom field delimiter for HiveQL transform~~[SPARK-5212][SQL] Add support of schema-less, custom field delimiter and SerDe for HiveQL transformJan 18, 2015

Fix bug.
7a14f31

setRecordReaderID is introduced in 0.13.1, use reflection API to call…
9a6dc04
… it.

viirya mentioned this pull request Jan 22, 2015
[SPARK-5364] [SQL] HiveQL transform doesn't support the non output clause #4158
Closed

chenghao-intel reviewed Jan 23, 2015
View reviewed changes

marmbrus reviewed Feb 2, 2015
View reviewed changes

viirya added 2 commits February 2, 2015 20:53

Merge remote-tracking branch 'upstream/master' into schema_less_trans
a137933

Refactor codes for comments.
ac2d1fe

asfgit closed this in 683e938Feb 2, 2015

chenghao-intel mentioned this pull request Apr 23, 2015
[SPARK-7044] [SQL] Fix the deadlock in script transformation #5625
Closed

viirya deleted the schema_less_trans branch December 27, 2023 18:30

Uh oh!

Conversation

viirya commented Jan 13, 2015

Uh oh!

SparkQA commented Jan 13, 2015

Uh oh!

SparkQA commented Jan 13, 2015

Uh oh!

SparkQA commented Jan 14, 2015

Uh oh!

SparkQA commented Jan 14, 2015

Uh oh!

SparkQA commented Jan 16, 2015

Uh oh!

SparkQA commented Jan 16, 2015

Uh oh!

SparkQA commented Jan 17, 2015

Uh oh!

SparkQA commented Jan 17, 2015

Uh oh!

SparkQA commented Jan 18, 2015

Uh oh!

SparkQA commented Jan 18, 2015

Uh oh!

SparkQA commented Jan 18, 2015

Uh oh!

SparkQA commented Jan 19, 2015

Uh oh!

SparkQA commented Jan 19, 2015

Uh oh!

chenghao-intelJan 23, 2015

Choose a reason for hiding this comment

Uh oh!

viiryaJan 23, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jan 29, 2015

Uh oh!

viirya commented Jan 29, 2015

Uh oh!

rxin commented Jan 29, 2015

Uh oh!

viirya commented Jan 29, 2015

Uh oh!

viirya commented Jan 31, 2015

Uh oh!

marmbrusFeb 2, 2015

Choose a reason for hiding this comment

Uh oh!

marmbrus commented Feb 2, 2015

Uh oh!

SparkQA commented Feb 2, 2015

Uh oh!

viirya commented Feb 2, 2015

Uh oh!

marmbrus commented Feb 2, 2015

Uh oh!

chenghao-intel commented Apr 24, 2015

Uh oh!

viirya commented Apr 24, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants