[SPARK-33105][INFRA] Change default R arch from i386 to x64 and parametrize BINPREF by zero323 · Pull Request #29991 · apache/spark

zero323 · 2020-10-09T18:37:35Z

What changes were proposed in this pull request?

Change default R arch from i386 to x64, to match Rtools version.
Parameterize BINPREF with WIN (https://stackoverflow.com/a/44035904)

Reported on dev:

http://apache-spark-developers-list.1001551.n3.nabble.com/Broken-rlang-installation-on-AppVeyor-td30294.html

Why are the changes needed?

It seems like update from rlang 0.4.7 to 0.4.8 exposed an issue, where build fails because of incompatible ddl

c:/Rtools40/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe:
skipping incompatible C:/R/bin/i386/R.dll when searching for -lR
[00:01:52]
c:/Rtools40/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe:
skipping incompatible C:/R/bin/i386/R.dll when searching for -lR
[00:01:52]
c:/Rtools40/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe:
cannot find -lR
[00:01:52] collect2.exe: error: ld returned 1 exit status

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing tests.

SparkQA · 2020-10-09T19:27:11Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34196/

SparkQA · 2020-10-09T19:46:56Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34196/

SparkQA · 2020-10-09T20:18:53Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34197/

SparkQA · 2020-10-09T20:35:30Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34197/

Source: https://stackoverflow.com/a/44035904

SparkQA · 2020-10-09T21:05:57Z

Test build #129593 has finished for PR 29991 at commit cd8b621.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-10-09T21:15:37Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34199/

zero323 · 2020-10-09T21:20:33Z

Notes:

I am not sure if we should put MinGW here. Ultimately we want to use one provided by Rtools, right?

spark/dev/appveyor-install-dependencies.ps1

Line 45 in 2e07ed3

$env:PATH=$RDrive+'\R\bin\'+$arch+';'+'C:\MinGW\msys\1.0\bin;'+$env:PATH

(thought we might have to add $RtoolsDrive + '\Rtools40\usr\bin to PATH, unless make is provided otherwise)

Here

spark/dev/appveyor-install-dependencies.ps1

Line 70 in 2e07ed3

$env:PATH=$RtoolsDrive+'\Rtools40\bin;'+$RtoolsDrive+'\Rtools40\mingw64\bin;'+$RtoolsDrive+'\Rtools40\'+$gccPath+'\bin;'+$env:PATH

$RtoolsDrive + '\Rtools40\' + $gccPath + '\bin;' doesn't seem to point to existing directory and it seem like it shouldn't be added in the first place. Shall we remove it?

zero323 · 2020-10-09T21:25:44Z

cc @HyukjinKwon

SparkQA · 2020-10-09T21:26:25Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34201/

SparkQA · 2020-10-09T21:38:00Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34199/

SparkQA · 2020-10-09T21:43:13Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34201/

SparkQA · 2020-10-09T22:12:04Z

Test build #129594 has finished for PR 29991 at commit 0d8f110.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-10-09T22:16:52Z

Should be good to go as long as AppVeyor build pass.

SparkQA · 2020-10-09T22:19:26Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34204/

SparkQA · 2020-10-09T22:26:44Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34205/

SparkQA · 2020-10-09T22:43:23Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34204/

SparkQA · 2020-10-09T22:46:42Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/34205/

SparkQA · 2020-10-09T23:05:31Z

Test build #129596 has finished for PR 29991 at commit 042ab55.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-10-09T23:36:35Z

Test build #129598 has finished for PR 29991 at commit 0d8f110.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-10-10T00:02:19Z

Test build #129600 has finished for PR 29991 at commit b6475be.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-10-10T00:28:33Z

Test build #129602 has finished for PR 29991 at commit cf4af04.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen

I don't know this well but seems plausible.

HyukjinKwon · 2020-10-10T04:49:06Z

Merged to master.

zero323 · 2020-10-10T07:03:28Z

Thanks!

…etrize BINPREF ### What changes were proposed in this pull request? - Change default R `arch` from `i386` to `x64`, to match Rtools version. - Parameterize `BINPREF` with `WIN` (https://stackoverflow.com/a/44035904) Reported on dev: http://apache-spark-developers-list.1001551.n3.nabble.com/Broken-rlang-installation-on-AppVeyor-td30294.html ### Why are the changes needed? It seems like update from rlang 0.4.7 to 0.4.8 exposed an issue, where build fails because of incompatible ddl ``` c:/Rtools40/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: skipping incompatible C:/R/bin/i386/R.dll when searching for -lR [00:01:52] c:/Rtools40/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: skipping incompatible C:/R/bin/i386/R.dll when searching for -lR [00:01:52] c:/Rtools40/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lR [00:01:52] collect2.exe: error: ld returned 1 exit status ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Existing tests. Closesapache#29991 from zero323/APPVEYOR-DEAFAULT-ARCH. Authored-by: zero323 <mszymkiewicz@gmail.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org>

zero323 force-pushed the APPVEYOR-DEAFAULT-ARCH branch from cd8b621 to 0d8f110Compare October 9, 2020 19:32

zero323 force-pushed the APPVEYOR-DEAFAULT-ARCH branch from 042ab55 to 0d8f110Compare October 9, 2020 20:40

zero323 added 2 commits October 9, 2020 22:57

Default arch to x64
b3dc85a

Parametrize BINPREF with WIN
cf4af04
Source: https://stackoverflow.com/a/44035904

zero323 force-pushed the APPVEYOR-DEAFAULT-ARCH branch 2 times, most recently from b6475be to cf4af04Compare October 9, 2020 21:12

zero323 changed the title ~~[WIP][INFRA] Change default R arch from i386 to x64~~[WIP][INFRA] Change default R arch from i386 to x64 and parametrize BINPREFOct 9, 2020

zero323 changed the title ~~[WIP][INFRA] Change default R arch from i386 to x64 and parametrize BINPREF~~[SPARK-33105][INFRA] Change default R arch from i386 to x64 and parametrize BINPREFOct 9, 2020

zero323 marked this pull request as ready for review October 9, 2020 22:31

HyukjinKwon approved these changes Oct 10, 2020
View reviewed changes

srowen reviewed Oct 10, 2020
View reviewed changes

HyukjinKwon closed this in 018811fOct 10, 2020

zero323 deleted the APPVEYOR-DEAFAULT-ARCH branch October 10, 2020 07:03

Uh oh!

Conversation

zero323 commented Oct 9, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Oct 9, 2020

Uh oh!

SparkQA commented Oct 9, 2020

Uh oh!

SparkQA commented Oct 9, 2020

Uh oh!

SparkQA commented Oct 9, 2020

Uh oh!

SparkQA commented Oct 9, 2020

Uh oh!

SparkQA commented Oct 9, 2020

Uh oh!

zero323 commented Oct 9, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zero323 commented Oct 9, 2020

Uh oh!

SparkQA commented Oct 9, 2020

Uh oh!

SparkQA commented Oct 9, 2020

Uh oh!

SparkQA commented Oct 9, 2020

Uh oh!

SparkQA commented Oct 9, 2020

Uh oh!

HyukjinKwon commented Oct 9, 2020

Uh oh!

SparkQA commented Oct 9, 2020

Uh oh!

SparkQA commented Oct 9, 2020

Uh oh!

SparkQA commented Oct 9, 2020

Uh oh!

SparkQA commented Oct 9, 2020

Uh oh!

SparkQA commented Oct 9, 2020

Uh oh!

SparkQA commented Oct 9, 2020

Uh oh!

SparkQA commented Oct 10, 2020

Uh oh!

SparkQA commented Oct 10, 2020

Uh oh!

srowen left a comment

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Oct 10, 2020

Uh oh!

zero323 commented Oct 10, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zero323 commented Oct 9, 2020•
edited
Loading

zero323 commented Oct 9, 2020•
edited
Loading