Skip to content

[SPARK-27563][FOLLOWUP] Fix to download new release from dist.apache.org - #24544

Closed
wangyum wants to merge 3 commits into
apache:masterfrom
wangyum:Unable-to-download-Spark-2.4.3
Closed

[SPARK-27563][FOLLOWUP] Fix to download new release from dist.apache.org#24544
wangyum wants to merge 3 commits into
apache:masterfrom
wangyum:Unable-to-download-Spark-2.4.3

Conversation

@wangyum

@wangyumwangyum commented May 7, 2019•
edited
Loading

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

https://archive.apache.org/dist/spark/ does not have latest Spark 2.4.3:

This pr add https://dist.apache.org/repos/dist/release/spark/ to mirrors list to download latest Spark.

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/105202/testReport/org.apache.spark.sql.hive/HiveExternalCatalogVersionsSuite/_It_is_not_a_test_it_is_a_sbt_testing_SuiteSelector_/

How was this patch tested?

manual tests:

build/sbt "hive/testOnly *.HiveExternalCatalogVersionsSuite" -Phive

@SparkQA

SparkQA commented May 7, 2019

Copy link
Copy Markdown

Test build #105209 has finished for PR 24544 at commit bfb251b.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@beliefer

beliefer commented May 7, 2019

Copy link
Copy Markdown
Contributor

I want to know how the previous version of spark was downloaded to https://archive.apache.org/dist/spark/.

beliefer
beliefer reviewed May 7, 2019
View reviewed changes
Comment threadsql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
import scala.io.Source
try {
Source.fromURL("https://dist.apache.org/repos/dist/release/spark/").mkString
Source.fromURL(s"${releaseMirror}/spark").mkString

@belieferbelieferMay 7, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe $releaseMirror is better.

@dongjoon-hyundongjoon-hyunMay 7, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@srowensrowenMay 7, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're picking nits and you change it again, it's not really a mirror. It's the primary dist server. asfDist or something.

@SparkQA

SparkQA commented May 7, 2019

Copy link
Copy Markdown

Test build #105210 has finished for PR 24544 at commit 5e09e5b.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon

HyukjinKwon commented May 7, 2019

Copy link
Copy Markdown
Member

Wouldn't it be uploaded soon?

@SparkQA

SparkQA commented May 7, 2019

Copy link
Copy Markdown

Test build #105211 has finished for PR 24544 at commit cf946e7.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@wangyum

wangyum commented May 7, 2019

Copy link
Copy Markdown
MemberAuthor

Wouldn't it be uploaded soon?

Maybe @cloud-fan knows, but we can avoid this error in the next release.

@cloud-fan

cloud-fan commented May 7, 2019

Copy link
Copy Markdown
Contributor

After uploading, it takes a while for the resources to be available in the Mirror Network. This means the PR builder will be broken for a while.

This fix LGTM

@dongjoon-hyundongjoon-hyun changed the title [HOT-FIX] Unable to download Spark 2.4.3[SPARK-27563][FOLLOWUP][HOTFIX] Fix to download new release from dist.apache.orgMay 7, 2019
srowen
srowen approved these changes May 7, 2019
View reviewed changes
Comment threadsql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
import scala.io.Source
try {
Source.fromURL("https://dist.apache.org/repos/dist/release/spark/").mkString
Source.fromURL(s"${releaseMirror}/spark").mkString

@srowensrowenMay 7, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're picking nits and you change it again, it's not really a mirror. It's the primary dist server. asfDist or something.

@dongjoon-hyundongjoon-hyun changed the title [SPARK-27563][FOLLOWUP][HOTFIX] Fix to download new release from dist.apache.org[SPARK-27563][FOLLOWUP] Fix to download new release from dist.apache.orgMay 7, 2019
@dongjoon-hyun

dongjoon-hyun commented May 7, 2019

Copy link
Copy Markdown
Member

I removed [HOTFIX] from the title because it starts to pass. (#24516)

@srowen

srowen commented May 9, 2019

Copy link
Copy Markdown
Member

Merged to master

@srowensrowen closed this in b5ffec1May 9, 2019
@wangyum
wangyum deleted the Unable-to-download-Spark-2.4.3 branch June 10, 2019 09:23
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@belieferbelieferbeliefer left review comments
@dongjoon-hyundongjoon-hyundongjoon-hyun left review comments
@srowensrowensrowen approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@wangyum@SparkQA@beliefer@HyukjinKwon@cloud-fan@dongjoon-hyun@srowen