Skip to content

[MINOR][DOC] Update Partition Discovery section to enumerate all available file sources - #19139

Closed
dongjoon-hyun wants to merge 4 commits into
apache:masterfrom
dongjoon-hyun:partitiondiscovery
Closed

[MINOR][DOC] Update Partition Discovery section to enumerate all available file sources#19139
dongjoon-hyun wants to merge 4 commits into
apache:masterfrom
dongjoon-hyun:partitiondiscovery

Conversation

@dongjoon-hyun

@dongjoon-hyundongjoon-hyun commented Sep 5, 2017•
edited
Loading

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

All built-in data sources support Partition Discovery. We had better update the document to give the users more benefit clearly.

AFTER

1

How was this patch tested?

SKIP_API=1 jekyll serve --watch

@gatorsmile

gatorsmile commented Sep 5, 2017

Copy link
Copy Markdown
Member

Are the partition discovery applicable to all the built-in data sources?

@SparkQA

SparkQA commented Sep 5, 2017

Copy link
Copy Markdown

Test build #81421 has finished for PR 19139 at commit 462fe27.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun

dongjoon-hyun commented Sep 5, 2017

Copy link
Copy Markdown
MemberAuthor

As you pointed out, I checked it again. Right, HadoopFsRelationTest have it.
Should we merge ParquetPartitionDiscoverySuite and OrcPartitionDiscoverySuite into HadoopFsRelationTest, too?

@gatorsmile

gatorsmile commented Sep 5, 2017

Copy link
Copy Markdown
Member

abstract class HadoopFsRelationTest is extended by four data sources. You can move some test cases into it. However, I am not sure whether you do it for all of them.

@dongjoon-hyun

dongjoon-hyun commented Sep 5, 2017

Copy link
Copy Markdown
MemberAuthor

@gatorsmile .
For this PR, I just update the doc only. Thank you so much!

@SparkQA

SparkQA commented Sep 5, 2017

Copy link
Copy Markdown

Test build #81423 has finished for PR 19139 at commit fd00fbd.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

gatorsmile
gatorsmile reviewed Sep 5, 2017
View reviewed changes
Comment threaddocs/sql-programming-guide.md Outdated
Table partitioning is a common optimization approach used in systems like Hive. In a partitioned
table, data are usually stored in different directories, with partitioning column values encoded in
the path of each partition directory. The Parquet data source is now able to discover and infer
the path of each partition directory. All built-in data sources are able to discover and infer

@gatorsmilegatorsmileSep 5, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-> All built-in file sources (including ...)

You know, it is not applicable to JDBC

@dongjoon-hyundongjoon-hyunSep 5, 2017

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops. Yep. I'll enumerate it.

@gatorsmile

gatorsmile commented Sep 5, 2017

Copy link
Copy Markdown
Member

BTW, please update the PR title.

@dongjoon-hyundongjoon-hyun changed the title [MINOR][DOC] Add ORC in Partition Discovery section.[MINOR][DOC] Update Partition Discovery section to enumerate all available data sourcesSep 5, 2017
@dongjoon-hyun

dongjoon-hyun commented Sep 5, 2017

Copy link
Copy Markdown
MemberAuthor

I added (including TEXT/CSV/JSON/ORC/Parquet) and updated the PR title, too.

@SparkQA

SparkQA commented Sep 5, 2017

Copy link
Copy Markdown

Test build #81428 has finished for PR 19139 at commit 128c779.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

gatorsmile
gatorsmile reviewed Sep 5, 2017
View reviewed changes
Comment threaddocs/sql-programming-guide.md Outdated
table, data are usually stored in different directories, with partitioning column values encoded in
the path of each partition directory. The Parquet data source is now able to discover and infer
partitioning information automatically. For example, we can store all our previously used
the path of each partition directory. All built-in data sources (including TEXT/CSV/JSON/ORC/Parquet)

@gatorsmilegatorsmileSep 5, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

data sources -> file sources
TEXT -> Text

@dongjoon-hyundongjoon-hyunSep 5, 2017

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. It's updated.

@gatorsmile

gatorsmile commented Sep 5, 2017

Copy link
Copy Markdown
Member

LGTM except a minor comment

@SparkQA

SparkQA commented Sep 5, 2017

Copy link
Copy Markdown

Test build #81429 has finished for PR 19139 at commit 018fdb3.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyundongjoon-hyun changed the title [MINOR][DOC] Update Partition Discovery section to enumerate all available data sources[MINOR][DOC] Update Partition Discovery section to enumerate all available file sourcesSep 5, 2017
@gatorsmile

gatorsmile commented Sep 5, 2017

Copy link
Copy Markdown
Member

Thanks! Merged to master/2.2

asfgit pushed a commit that referenced this pull request Sep 5, 2017
@dongjoon-hyun@gatorsmile
[MINOR][DOC] Update Partition Discovery section to enumerate all av…
7da8fbf
…ailable file sources
## What changes were proposed in this pull request?
All built-in data sources support `Partition Discovery`. We had better update the document to give the users more benefit clearly.
**AFTER**
<img width="906" alt="1" src="https://user-images.githubusercontent.com/9700541/30083628-14278908-9244-11e7-98dc-9ad45fe233a9.png">
## How was this patch tested?
```
SKIP_API=1 jekyll serve --watch
```
Author: Dongjoon Hyun <dongjoon@apache.org>
Closes#19139 from dongjoon-hyun/partitiondiscovery.
(cherry picked from commit 9e451bc)
Signed-off-by: gatorsmile <gatorsmile@gmail.com>
@asfgitasfgit closed this in 9e451bcSep 5, 2017
@dongjoon-hyun

dongjoon-hyun commented Sep 5, 2017

Copy link
Copy Markdown
MemberAuthor

Thank you, @gatorsmile !

@dongjoon-hyun
dongjoon-hyun deleted the partitiondiscovery branch September 5, 2017 22:15
MatthewRBruce pushed a commit to Shopify/spark that referenced this pull request Jul 31, 2018
@dongjoon-hyun@MatthewRBruce
[MINOR][DOC] Update Partition Discovery section to enumerate all av…
ad45b84
…ailable file sources
## What changes were proposed in this pull request?
All built-in data sources support `Partition Discovery`. We had better update the document to give the users more benefit clearly.
**AFTER**
<img width="906" alt="1" src="https://user-images.githubusercontent.com/9700541/30083628-14278908-9244-11e7-98dc-9ad45fe233a9.png">
## How was this patch tested?
```
SKIP_API=1 jekyll serve --watch
```
Author: Dongjoon Hyun <dongjoon@apache.org>
Closesapache#19139 from dongjoon-hyun/partitiondiscovery.
(cherry picked from commit 9e451bc)
Signed-off-by: gatorsmile <gatorsmile@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gatorsmilegatorsmilegatorsmile left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@dongjoon-hyun@gatorsmile@SparkQA