[SPARK-29403][INFRA][R] Uses Arrow R 0.14.1 in AppVeyor for now by HyukjinKwon · Pull Request #26041 · apache/spark

HyukjinKwon · 2019-10-07T08:41:39Z

What changes were proposed in this pull request?

This PR proposes to use Arrow R 0.14.1 for now in AppVeyor to make tests passed.

Why are the changes needed?

To make build passed with Arrow. It doesn't work with setting ARROW_PRE_0_15_IPC_FORMAT to 1 to allow Arrow R 0.15 compatibility.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

AppVeyor

HyukjinKwon · 2019-10-07T08:47:54Z

cc @BryanCutler FYI

HyukjinKwon · 2019-10-07T08:57:35Z

@BryanCutler did Arrow change the format itself? bunch of PySpark tests with Arrow fail too.

SparkQA · 2019-10-07T09:32:53Z

Test build #111837 has finished for PR 26041 at commit f30f4de.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-10-07T09:42:20Z

Seems Arrow isn't released yet in Windows. The AppVeyor build here used Areow 0.14.1.1. I'll trigger the build again when Areow 0.15 is available on Windows.

harpaj · 2019-10-07T13:32:50Z

Might have the same cause as Pandas UDFs not working with Arrow 0.15, see https://issues.apache.org/jira/projects/SPARK/issues/SPARK-29367

BryanCutler · 2019-10-07T17:42:26Z

@BryanCutler did Arrow change the format itself? bunch of PySpark tests with Arrow fail too.

Yes, there was a change but I have a patch ready that will be backwards compatible. I'll open a JIRA for the upgrade and we can go over the details.

BryanCutler · 2019-10-07T18:36:38Z

I opened a PR documenting the change in #26045

SparkQA · 2019-10-08T03:52:58Z

Test build #111868 has finished for PR 26041 at commit 34da71d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-10-08T05:34:59Z

Hm, setting ARROW_PRE_0_15_IPC_FORMAT to 1 doesn't seem working in SparkR case. I tested this both in Windows (via AppVeyor) and Ubuntu

SparkQA · 2019-10-08T06:52:48Z

Test build #111872 has finished for PR 26041 at commit bea530f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-10-08T07:05:02Z

Test build #111871 has finished for PR 26041 at commit f969c17.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-10-09T01:35:47Z

I am going to set the Arrow R version to 0.14.1 for now explicitly to avoid to break build. The build is borken in the master due to this.

SparkQA · 2019-10-09T01:52:22Z

Test build #111927 has started for PR 26041 at commit e1a6807.

BryanCutler · 2019-10-09T21:27:21Z

 environment:
 NOT_CRAN: true
+ # See SPARK-29378.
+ ARROW_PRE_0_15_IPC_FORMAT: 1


Does setting this here carry it through to the worker?

I removed it in the latest commit :-).

HyukjinKwon · 2019-10-10T00:01:14Z

I am merging this to keep our AppVeyor build alive for now ..

### What changes were proposed in this pull request? This PR proposes to use Arrow R 0.14.1 for now in AppVeyor to make tests passed. ### Why are the changes needed? To make build passed with Arrow. It doesn't work with setting `ARROW_PRE_0_15_IPC_FORMAT` to `1` to allow Arrow R 0.15 compatibility. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? AppVeyor Closesapache#26041 from HyukjinKwon/investigate. Authored-by: HyukjinKwon <gurwls223@apache.org> Signed-off-by: HyukjinKwon <gurwls223@apache.org>

dongjoon-hyun · 2019-11-16T16:58:22Z

Hi, All.
After we upgrade Arrow 0.15.1, AppVeyor is dead again~ I'm revisiting this issue in the following PR.

[SPARK-29378][R] Upgrade SparkR to use Arrow 0.15 API #26555

Investiagte Arrow 0.15 with R Arrow optiomization
2c98594

This comment has been minimized.
Sign in to view

Changes in R
f30f4de

Test
34da71d

HyukjinKwon changed the title ~~[DO-NOT-MERGE] Investigate Arrow 0.15 with R Arrow optimization~~[SPARK-29378][INFRA][R] Make AppVeyor's SparkR with Arrow tests compatible with Arrow R 0.15Oct 8, 2019

HyukjinKwon force-pushed the investigate branch from f969c17 to 07a618aCompare October 8, 2019 04:10

ARROW_PRE_0_15_IPC_FORMAT=1
bea530f

HyukjinKwon force-pushed the investigate branch from 07a618a to bea530fCompare October 8, 2019 04:11

HyukjinKwon changed the title ~~[SPARK-29378][INFRA][R] Make AppVeyor's SparkR with Arrow tests compatible with Arrow R 0.15~~[WIP][SPARK-29378][INFRA][R] Make AppVeyor's SparkR with Arrow tests compatible with Arrow R 0.15Oct 8, 2019

HyukjinKwon changed the title ~~[WIP][SPARK-29378][INFRA][R] Make AppVeyor's SparkR with Arrow tests compatible with Arrow R 0.15~~[SPARK-29403][INFRA][R] Uses Arrow R 0.14.1 in AppVeyor for nowOct 9, 2019

Use 0.14.1 for now
e1a6807

HyukjinKwon force-pushed the investigate branch from 2c24b49 to e1a6807Compare October 9, 2019 01:42

dependencies
564ec38

BryanCutler reviewed Oct 9, 2019
View reviewed changes

HyukjinKwon closed this in 7ba16ffOct 10, 2019

imback82 mentioned this pull request Oct 30, 2019
[SPARK-29643][SQL] ALTER TABLE/VIEW (DROP PARTITION) should look up catalog/table like v2 commands #26303
Closed

BryanCutler mentioned this pull request Nov 16, 2019
[SPARK-29378][R] Upgrade SparkR to use Arrow 0.15 API #26555
Closed

HyukjinKwon deleted the investigate branch March 3, 2020 01:17

Uh oh!

Conversation

HyukjinKwon commented Oct 7, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

This comment has been minimized.

HyukjinKwon commented Oct 7, 2019

Uh oh!

HyukjinKwon commented Oct 7, 2019

Uh oh!

SparkQA commented Oct 7, 2019

Uh oh!

HyukjinKwon commented Oct 7, 2019

Uh oh!

harpaj commented Oct 7, 2019

Uh oh!

BryanCutler commented Oct 7, 2019

Uh oh!

BryanCutler commented Oct 7, 2019

Uh oh!

SparkQA commented Oct 8, 2019

Uh oh!

HyukjinKwon commented Oct 8, 2019

Uh oh!

SparkQA commented Oct 8, 2019

Uh oh!

SparkQA commented Oct 8, 2019

Uh oh!

HyukjinKwon commented Oct 9, 2019

Uh oh!

SparkQA commented Oct 9, 2019

Uh oh!

BryanCutlerOct 9, 2019

Choose a reason for hiding this comment

Uh oh!

HyukjinKwonOct 10, 2019

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Oct 10, 2019

Uh oh!

dongjoon-hyun commented Nov 16, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

HyukjinKwon commented Oct 7, 2019•
edited
Loading