[SPARK-30505][DOCS] Deprecate Avro option `ignoreExtension` in sql-data-sources-avro.md by MaxGekk · Pull Request #27194 · apache/spark

MaxGekk · 2020-01-13T20:23:16Z

What changes were proposed in this pull request?

Updated docs/sql-data-sources-avro.md, and added a few sentences about already deprecated in code Avro option ignoreExtension.

Closes#27174

Why are the changes needed?

To make users doc consistent to the code where ignoreExtension has been already deprecated, see

spark/external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala

Lines 46 to 47 in 3663dbe

 logWarning(s"Option ${AvroOptions.ignoreExtensionKey} is deprecated. Please use the "+
"general data source option pathGlobFilter for filtering file names.") 

Does this PR introduce any user-facing change?

No

How was this patch tested?

by building docs

MaxGekk · 2020-01-13T20:24:49Z

This follows up #27174 (comment)

gengliangwang · 2020-01-13T20:26:29Z

 <td><code>ignoreExtension</code></td>
 <td>true</td>
- <td>The option controls ignoring of files without <code>.avro</code> extensions in read.<br> If the option is enabled, all files (with and without <code>.avro</code> extension) are loaded.</td>
+ <td>The option controls ignoring of files without <code>.avro</code> extensions in read.<br> If the option is enabled, all files (with and without <code>.avro</code> extension) are loaded.<br> The option has been already deprecated, and it will be removed in the future releases. Please use the general data source option <code>pathGlobFilter</code> for filtering file names.</td>


Thanks for the work.
Could you also create a doc entry for the option pathGlobFilter and link to it here in this PR?

I added it here

SparkQA · 2020-01-13T20:38:22Z

Test build #116659 has finished for PR 27194 at commit 5016556.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-01-13T21:02:23Z

Test build #116660 has finished for PR 27194 at commit 91a1dee.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang · 2020-01-14T01:04:59Z

Hi @MaxGekk

Sorry, I meant a new document for the option pathGlobFilter, which is not for Avro only.

I have come up with a good idea: https://issues.apache.org/jira/browse/SPARK-30506
With the new documentation, we can link the option pathGlobFilter in the Avro data source doc.

Are you interested in writing the new doc? It should be very helpful for users.

HyukjinKwon · 2020-01-14T01:05:35Z

Yeah, discussed with @gengliangwang offline. I think that's a better option.

MaxGekk · 2020-01-14T06:29:03Z

I described the global option because it has been already described in each load method - json, text, parquet ... in DataFrameReader, for example

spark/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala

Line 410 in f8d5957

 * <li>`pathGlobFilter`: an optional glob pattern to only include files with paths matching

Are you interested in writing the new doc?

No, I am not.

If you think, description of the option pathGlobFilter is useless in sql-data-sources-avro.md. Let me know, I will revert it from the PR but I would leave it as in this PR.

gengliangwang · 2020-01-14T07:05:32Z

@MaxGekk I see. I didn't know that the option pathGlobFilter is documented in DataFrameReader.scala.

I think reverting the description of the option pathGlobFilter in sql-data-sources-avro.md makes sense.

Still, we need to document the options in SPARK-30506. I will find someone else or do it myself.

This reverts commit 91a1dee.

SparkQA · 2020-01-14T07:25:24Z

Test build #116687 has finished for PR 27194 at commit bfbaf2b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun

Please update the PR description. The following looks incorrect. It's ORC related PR which is already closed.

Closes #27179

MaxGekk · 2020-01-14T18:28:15Z

@dongjoon-hyun I have fixed the PR number related to Avro

dongjoon-hyun · 2020-01-14T22:53:21Z

 <td><code>ignoreExtension</code></td>
 <td>true</td>
- <td>The option controls ignoring of files without <code>.avro</code> extensions in read.<br> If the option is enabled, all files (with and without <code>.avro</code> extension) are loaded.</td>
+ <td>The option controls ignoring of files without <code>.avro</code> extensions in read.<br> If the option is enabled, all files (with and without <code>.avro</code> extension) are loaded.<br> The option has been already deprecated, and it will be removed in the future releases. Please use the general data source option <code>pathGlobFilter</code> for filtering file names.</td>


been already deprecated -> been deprecated.

dongjoon-hyun

Also, please provide the new screenshot of the updated page at PR description.

MaxGekk · 2020-01-15T07:26:58Z

I have attached the screenshot.

SparkQA · 2020-01-15T07:28:57Z

Test build #116757 has finished for PR 27194 at commit 7561342.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon

LGTM

I am just going to merge this - for other file-based source options, we could do it separately.

dongjoon-hyun · 2020-01-15T20:06:49Z

+1, late LGTM.

Update sql-data-sources-avro.md
5016556

MaxGekk mentioned this pull request Jan 13, 2020
[SPARK-30490][SQL] Eliminate compiler warnings in Avro datasource #27174
Closed

gengliangwang reviewed Jan 13, 2020
View reviewed changes

Describe pathGlobFilter
91a1dee

HyukjinKwon approved these changes Jan 14, 2020
View reviewed changes

Revert "Describe pathGlobFilter"
bfbaf2b
This reverts commit 91a1dee.

dongjoon-hyun requested changes Jan 14, 2020
View reviewed changes

dongjoon-hyun added the DOCUMENTATION label Jan 14, 2020

dongjoon-hyun reviewed Jan 14, 2020
View reviewed changes

dongjoon-hyun requested changes Jan 14, 2020
View reviewed changes

been already deprecated -> been deprecated
7561342

HyukjinKwon approved these changes Jan 15, 2020
View reviewed changes

HyukjinKwon closed this in 51d2917Jan 15, 2020

MaxGekk deleted the avro-doc-deprecation-ignoreExtension branch June 5, 2020 19:42

	logWarning(s"Option ${AvroOptions.ignoreExtensionKey} is deprecated. Please use the "+
	"general data source option pathGlobFilter for filtering file names.")

Uh oh!

Conversation

MaxGekk commented Jan 13, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

MaxGekk commented Jan 13, 2020

Uh oh!

gengliangwangJan 13, 2020

Choose a reason for hiding this comment

Uh oh!

MaxGekkJan 13, 2020

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jan 13, 2020

Uh oh!

SparkQA commented Jan 13, 2020

Uh oh!

gengliangwang commented Jan 14, 2020

Uh oh!

HyukjinKwon commented Jan 14, 2020

Uh oh!

MaxGekk commented Jan 14, 2020

Uh oh!

gengliangwang commented Jan 14, 2020

Uh oh!

SparkQA commented Jan 14, 2020

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

MaxGekk commented Jan 14, 2020

Uh oh!

dongjoon-hyunJan 14, 2020

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaxGekk commented Jan 15, 2020

Uh oh!

SparkQA commented Jan 15, 2020

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jan 15, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

MaxGekk commented Jan 13, 2020•
edited
Loading

dongjoon-hyun left a comment •
edited
Loading