Uh oh!
There was an error while loading. Please reload this page.
ARROW-6437: [R] Add AWS SDK to system dependencies for macOS and Windows - #7928
Closed
nealrichardson wants to merge 13 commits into
Closed
ARROW-6437: [R] Add AWS SDK to system dependencies for macOS and Windows#7928nealrichardson wants to merge 13 commits into
nealrichardson wants to merge 13 commits into
Conversation
nealrichardson
commented
Aug 10, 2020
MemberAuthor
@github-actions crossbow submit homebrew-r-autobrew |
Revision: 65001f14bf78efafad024907d7c91dd404d40c09 Submitted crossbow builds: ursa-labs/crossbow @ actions-471
|
nealrichardson
commented
Aug 10, 2020
MemberAuthor
@github-actions crossbow submit homebrew-r-autobrew |
Revision: d8cc705f2e0f42a56109a49024f2a122be668ced Submitted crossbow builds: ursa-labs/crossbow @ actions-473
|
nealrichardson
commented
Aug 10, 2020
MemberAuthor
@github-actions crossbow submit homebrew-r-autobrew |
Revision: 9a8364dd3e89052f2925f34fc591f5e17563ab90 Submitted crossbow builds: ursa-labs/crossbow @ actions-474
|
nealrichardson
commented
Aug 10, 2020
MemberAuthor
Rtools40 is passing. Rtools35 needs a different library build, which Jeroen is working on: https://github.com/r-windows/rtools-backports/compare/aws-sdk autobrew build fails like this: |
pitrou
commented
Aug 11, 2020
Member
No idea. Can we have the entire output from |
nealrichardson
commented
Aug 11, 2020
MemberAuthor
@github-actions crossbow submit homebrew-cpp-autobrew |
Revision: 9a8364dd3e89052f2925f34fc591f5e17563ab90 Submitted crossbow builds: ursa-labs/crossbow @ actions-475
|
jeroen
commented
Aug 11, 2020
Contributor
Maybe there is a problem with the unity builds of aws-sdk-cpp or arrow |
nealrichardson
commented
Aug 11, 2020
MemberAuthor
@github-actions crossbow submit homebrew-cpp-autobrew |
Revision: 3e72b7ad35355b66ae9b34d2d26dbcb879d19f82 Submitted crossbow builds: ursa-labs/crossbow @ actions-476
|
jeroen
commented
Aug 11, 2020
Contributor
If I disable the arrow unity build, then it succeeds in linking aws-sdk, but then it fails building the vendored jemalloc. If I disable both jemalloc and unity-build, then everything works for me. |
nealrichardson
commented
Aug 11, 2020
MemberAuthor
In https://travis-ci.org/github/ursa-labs/crossbow/builds/716930465 it seems that cmake can't find aws-c-common, though I see it is downloaded, so I wonder if we're hitting that cmake issue I referenced yesterday @jeroen. |
nealrichardson
commented
Aug 11, 2020
MemberAuthor
Where do you see that? |
nealrichardson
commented
Aug 13, 2020
MemberAuthor
Ok I've conditionally enabled |
nealrichardson
commented
Aug 14, 2020
MemberAuthor
I ran the autobrew locally and think I've got the full error here. It looks like it's linking some parquet lib and maybe needs to include the aws-sdk-cpp libs but isn't? I don't understand what it's trying to do or why though, particularly since we're building with Does this provide any insight @pitrou@jeroen? |
nealrichardson
commented
Aug 14, 2020
MemberAuthor
@github-actions crossbow submit homebrew-r-autobrew |
Revision: f990f73a94a392306fdbdb5ce7735c3b4387193d Submitted crossbow builds: ursa-labs/crossbow @ actions-483
|
jeroen
commented
Aug 14, 2020
Contributor
@nealrichardson this works for me: autobrew/homebrew-core#17 |
nealrichardson
commented
Aug 14, 2020
MemberAuthor
If the choice is between jemalloc and S3 support, I think we have to go with jemalloc. But hopefully we don't have to choose. I got something working locally, just trying now to see if it works on Travis. |
nealrichardson
commented
Aug 14, 2020
MemberAuthor
Hey this seemed to work: turning off the unity build (but jemalloc is still on), plus a few tweaks to the |
nealrichardson
commented
Aug 14, 2020
MemberAuthor
@github-actions crossbow submit homebrew-r-autobrew |
Revision: f4afc43981c2cdae9f1758d521ec0ac0e28921fc Submitted crossbow builds: ursa-labs/crossbow @ actions-484
|
jeroen
commented
Aug 14, 2020
Contributor
For me that didn't work on at least one version of macos. Can you try to submit a draft pr to autobrew-core to confirm ? |
nealrichardson
commented
Aug 14, 2020
MemberAuthor
So on retry that failed like: Which I also saw once locally but not subsequently. I don't understand what would cause |
pitrou
commented
Aug 17, 2020
Member
Just retry the build? This could be a sporadic issue. |
pitrou
commented
Aug 18, 2020
Member
No idea. Perhaps you can find someone with CMake and macOS expertise... :-S |
MemberAuthor
Googling the error pointed me back to our own JIRA. Looks like #456 fixed this issue previously, but #2779 undid the fix. Unclear why we're seeing this race condition only here, but I'll re-fix it (in a separate PR, to help with the next time we do |
MemberAuthor
There was a problem hiding this comment.
@jeroen have you updated the CRAN repo with aws-sdk-cpp yet?
Contributor
There was a problem hiding this comment.
No not yet. Will do soon if we confirm that it works (and you are sure you don't want to add more modules again :))
MemberAuthor
There was a problem hiding this comment.
I believe the macOS jemalloc install issue has been fixed now but I don't think I can easily confirm that on autobrew/homebrew-core#19 because you'd need to brew install --HEAD to pull the fix, and the brew test-bot command doesn't support head.
From the arrow perspective, builds are passing, so if you're satisfied that this is working, we can merge. And once we merge, no modules can get added to the cmake config without breaking the build, so there shouldn't be any more surprises.
Contributor
There was a problem hiding this comment.
You can try to cherry-pick the patch in the homebrew formula like so: https://github.com/autobrew/homebrew-core/blob/adebb0d668a1625c2c28d01df145ab7376ef5aef/Formula/mariadb-connector-c.rb#L9-L12
MemberAuthor
There was a problem hiding this comment.
Cool, that worked: https://travis-ci.org/github/autobrew/homebrew-core/builds/719332651
MemberAuthor
There was a problem hiding this comment.
Ok @jeroen, LMK when the CRAN rtools-packages is updated and I'll switch off of the bintray. No problem if that can't be done easily/quickly, I can merge this and swap the repos later.
Contributor
There was a problem hiding this comment.
I have uploaded it to my server, it should sync to the mirrors in a few hours.
kou pushed a commit
that referenced
this pull request
Aug 19, 2020
On ARROW-6437 (#7928) we saw occasional "File exists" errors on `jemalloc_ep` on macOS. Googling the error message led back to ARROW-739 (#456), which fixed this before by forcing install with `-j1`. ARROW-3492 later made it so jemalloc would build (but not install) in parallel. Then ARROW-3539 (#2779) was addressing a problem with that change and, along with fixing the build parallelization issue, unfortunately reverted the original `make -j1 install` fix. This patch restores the fix from ARROW-739. Closes#7995 from nealrichardson/jemalloc-install Authored-by: Neal Richardson <neal.p.richardson@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
nealrichardson
commented
Aug 20, 2020
MemberAuthor
@github-actions crossbow submit homebrew-r-autobrew |
Revision: 97517a9 Submitted crossbow builds: ursa-labs/crossbow @ actions-494
|
This was referenced Aug 20, 2020
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds S3 support to the binary macOS and Windows (Rtools40 only, i.e. R >= 4.0) packages.