Skip to content

[SPARK-14596][SQL] Remove not used SqlNewHadoopRDD and some more unused imports - #12354

Closed
HyukjinKwon wants to merge 5 commits into
apache:masterfrom
HyukjinKwon:SPARK-14596
Closed

[SPARK-14596][SQL] Remove not used SqlNewHadoopRDD and some more unused imports#12354
HyukjinKwon wants to merge 5 commits into
apache:masterfrom
HyukjinKwon:SPARK-14596

Conversation

@HyukjinKwon

@HyukjinKwonHyukjinKwon commented Apr 13, 2016

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Old HadoopFsRelation API includes buildInternalScan() which uses SqlNewHadoopRDD in ParquetRelation.
Because now the old API is removed, SqlNewHadoopRDD is not used anymore.

So, this PR removes SqlNewHadoopRDD and several unused imports.

This was discussed in #12326.

How was this patch tested?

Several related existing unit tests and sbt scalastyle.

@SparkQA

SparkQA commented Apr 13, 2016

Copy link
Copy Markdown

Test build #55707 has finished for PR 12354 at commit 8b8e961.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon

HyukjinKwon commented Apr 13, 2016

Copy link
Copy Markdown
MemberAuthor

cc @cloud-fan

@cloud-fan

cloud-fan commented Apr 13, 2016

Copy link
Copy Markdown
Contributor

There is a SqlNewHadoopRDDState, we should rename it to FileScanRDDState

@HyukjinKwon

HyukjinKwon commented Apr 13, 2016

Copy link
Copy Markdown
MemberAuthor

Thanks! I just renamed.

@SparkQA

SparkQA commented Apr 13, 2016

Copy link
Copy Markdown

Test build #55708 has finished for PR 12354 at commit a291332.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Apr 13, 2016

Copy link
Copy Markdown

Test build #55709 has finished for PR 12354 at commit 3f8e878.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

cloud-fan
cloud-fan reviewed Apr 13, 2016
View reviewed changes
Comment threadcore/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala Outdated
split.inputSplit.value match {
case fs: FileSplit => SqlNewHadoopRDDState.setInputFileName(fs.getPath.toString)
case _ => SqlNewHadoopRDDState.unsetInputFileName()
case fs: FileSplit => FileScanRDDState.setInputFileName(fs.getPath.toString)

@cloud-fancloud-fanApr 13, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@liancheng@yhuai Do you still use HadoopRDD to read data source relation? If not, I think we don't need to update file name here anymore.

@yhuaiyhuaiApr 13, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is used by HiveTableScan

@HyukjinKwonHyukjinKwonApr 14, 2016

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cloud-fan I see. This is used in https://github.com/apache/spark/blob/d6dc12ef0146ae409834c78737c116050961f350/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/InputFileName.scala during code generation.

Should I remove the change of file name?

@cloud-fancloud-fanApr 14, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i see, how about renaming it to InputFileNameHolder?

@HyukjinKwonHyukjinKwonApr 14, 2016

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. Thanks.

cloud-fan
cloud-fan reviewed Apr 14, 2016
View reviewed changes
Comment threadcore/src/main/scala/org/apache/spark/rdd/SqlNewHadoopRDDState.scala Outdated
* TODO: Move/Combine this with org.apache.spark.sql.datasources.SqlNewHadoopRDD
* State for FileScanRDD objects. This is split this way because of the package splits.
*/
private[spark] object SqlNewHadoopRDDState {

@cloud-fancloud-fanApr 14, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SqlNewHadoopRDDState is definitely not a good name here, how about InputFileNameHolder?

@HyukjinKwonHyukjinKwonApr 14, 2016

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. Thanks!

@HyukjinKwon

HyukjinKwon commented Apr 14, 2016

Copy link
Copy Markdown
MemberAuthor

@cloud-fan The commits I just submitted include the changes for MiMa tests and some comments.

@SparkQA

SparkQA commented Apr 14, 2016

Copy link
Copy Markdown

Test build #55772 has finished for PR 12354 at commit 6cb3547.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Apr 14, 2016

Copy link
Copy Markdown

Test build #55769 has finished for PR 12354 at commit 9c5893d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon

HyukjinKwon commented Apr 14, 2016

Copy link
Copy Markdown
MemberAuthor

retest this please

@SparkQA

SparkQA commented Apr 14, 2016

Copy link
Copy Markdown

Test build #55784 has finished for PR 12354 at commit 6cb3547.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@cloud-fan

cloud-fan commented Apr 14, 2016

Copy link
Copy Markdown
Contributor

thanks! merging to master!

@HyukjinKwon
HyukjinKwon deleted the SPARK-14596 branch January 2, 2018 03:40
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@HyukjinKwon@SparkQA@cloud-fan@yhuai