Uh oh!
There was an error while loading. Please reload this page.
ARROW-10224: [Python] Add support for Python 3.9 except macOS wheel and Windows wheel - #8386
ARROW-10224: [Python] Add support for Python 3.9 except macOS wheel and Windows wheel#8386terencehonles wants to merge 1 commit into
Conversation
0747e3d to
da52256CompareThere was a problem hiding this comment.
I'm assuming homebrew should be on the latest Python? It does look like it's available now https://formulae.brew.sh/formula/python@3.9
There was a problem hiding this comment.
I don't know if this still applies, but I figured it probably would
There was a problem hiding this comment.
Sounds good to me, thanks for kicking things off. I had also create voltrondata-labs/ursabot#205 to adjust buildbot since I realized 3.9 wasn't being built just by me creating this PR.
Uh oh!
There was an error while loading. Please reload this page.
terencehonles
commented
Oct 7, 2020
It looks like I "missed" the github workflows, but that might need to be done after the wheels are built and published as I haven't looked at what exactly are doing, and I kinda doubt they are building from source. I can open that up as a separate PR if that is helpful. |
kou
commented
Oct 8, 2020
@github-actions crossbow submit -g conda -g wheel |
Revision: da52256400bb1cdde72fb6b0cdcf92b9142a5eeb Submitted crossbow builds: ursa-labs/crossbow @ actions-616
|
da52256 to
59496ccCompareterencehonles
commented
Oct 8, 2020
It looks like cmake was already part of the quay.io/pypa/manylinux2014_x86_64:2020-10-06-b2ca7a1 and |
kou
commented
Oct 8, 2020
@github-actions crossbow submit -g conda -g wheel |
Revision: 59496cc5698070f4ddf7fb071d73239eff833a4c Submitted crossbow builds: ursa-labs/crossbow @ actions-617
|
Revision: 59496cc5698070f4ddf7fb071d73239eff833a4c Submitted crossbow builds: ursa-labs/crossbow @ actions-618
|
kou
commented
Oct 8, 2020
(We need to wait a while until our comments are processed.) |
There was a problem hiding this comment.
I think it's possible Travis doesn't have Python 3.9 in it's homebrew cache, but I would expect a different error than it succeeds and then /usr/local/opt/python@3.8/libexec/bin/pip is just not found...
Nevermind! Apparently it's just a "warning" 😒 :
Warning: 'python@3.9' formula is unreadable: No available formula with the name "python@3.9"
https://travis-ci.org/github/ursa-labs/crossbow/builds/733873746#L145
59496cc to
9ee37dfCompareterencehonles
commented
Oct 8, 2020
I reverted some of the OS X changes. I'm not sure I actually needed to change Python 3.8 there since it looks like that travis file is templated out to all OS X wheels and it is still using Python 3.8 in all of them. That's either a bug or that there's something else that's using the |
pitrou
commented
Oct 8, 2020
Note that Numpy doesn't provide pre-compiled wheels for 3.9 currently. |
pitrou
commented
Oct 8, 2020
Also, conda-forge doesn't seem to be ready either. |
Yeah, that's fine. I'm not really in a rush, I just wanted to help to get things in place that we could move forward as soon as all of the dependencies are met (and to identify where things are and what is still missing) It does build, and I was able to install everything in the project I am working on and run the tests (and discover other things that need to be updated), so that's really mainly what I was looking for. I figured I'd do it in public on a PR and use the built wheels instead of just building/testing locally and not sharing my work 🙂 |
xhochy
commented
Oct 18, 2020
Should be ready now. |
xhochy
commented
Oct 18, 2020
@github-actions crossbow submit -g conda |
Revision: 9ee37df987105f9546914f034b4d23c08b491a4d Submitted crossbow builds: ursa-labs/crossbow @ actions-651
|
9ee37df to
32582a2Compare
This comment has been minimized.
This comment has been minimized.
32582a2 to
5f695d4Compareterencehonles
commented
Oct 18, 2020
I pinned semver to 2.10.2 since it looks like there is a breaking change for |
This comment has been minimized.
This comment has been minimized.
Revision: 173d45e Submitted crossbow builds: ursa-labs/crossbow @ actions-783
|
kou
left a comment
There was a problem hiding this comment.
+1
Thanks! I'll merge this.
Could you open JIRA issues for macOS wheel support and Windows wheel support?
idanw206
commented
Dec 14, 2020
@kou When will the binaries be available in pypi? Thanks! |
kou
commented
Dec 15, 2020
We'll release them in 2021-01. |
n8henrie
commented
Dec 18, 2020
@kou You're likely well aware, but if looking for e.g. $ curl -s 'https://api.github.com/repos/ursa-labs/crossbow/releases'| jq -r '.[]["assets"] | .[]["browser_download_url"]'| grep manylinux2014-cp39https://github.com/ursa-labs/crossbow/releases/download/nightly-2020-12-18-0-azure-wheel-manylinux2014-cp39/pyarrow-2.1.0.dev412-cp39-cp39-manylinux2014_x86_64.whlhttps://github.com/ursa-labs/crossbow/releases/download/nightly-2020-12-18-0-azure-wheel-manylinux2014-cp39/pyarrow-2.1.0.dev412.tar.gzLooks like the api has a limit of 100 results per call, which doesn't go back far enough. Instead of dealing with pagination, it seems like $ gh release list --repo=ursa-labs/crossbow --limit=1000 | grep manylinux2014-cp39nightly-2020-12-18-0-azure-wheel-manylinux2014-cp39 nightly-2020-12-18-0-azure-wheel-manylinux2014-cp39 2020-12-18T07:26:20Znightly-2020-12-17-0-azure-wheel-manylinux2014-cp39 nightly-2020-12-17-0-azure-wheel-manylinux2014-cp39 2020-12-17T07:33:01Zactions-807-azure-wheel-manylinux2014-cp39 actions-807-azure-wheel-manylinux2014-cp39 2020-12-17T01:23:13Zactions-804-azure-wheel-manylinux2014-cp39 actions-804-azure-wheel-manylinux2014-cp39 2020-12-16T23:00:37Zactions-799-azure-wheel-manylinux2014-cp39 actions-799-azure-wheel-manylinux2014-cp39 2020-12-16T08:54:37Znightly-2020-12-16-0-azure-wheel-manylinux2014-cp39 nightly-2020-12-16-0-azure-wheel-manylinux2014-cp39 2020-12-16T06:58:23Zactions-794-azure-wheel-manylinux2014-cp39 actions-794-azure-wheel-manylinux2014-cp39 2020-12-15T21:35:42Znightly-2020-12-15-0-azure-wheel-manylinux2014-cp39 nightly-2020-12-15-0-azure-wheel-manylinux2014-cp39 2020-12-15T06:22:33Znightly-2020-12-14-0-azure-wheel-manylinux2014-cp39 nightly-2020-12-14-0-azure-wheel-manylinux2014-cp39 2020-12-14T07:19:20Zactions-783-azure-wheel-manylinux2014-cp39 actions-783-azure-wheel-manylinux2014-cp39 2020-12-12T22:57:41Zactions-772-azure-wheel-manylinux2014-cp39 actions-772-azure-wheel-manylinux2014-cp39 2020-12-11T07:19:59Zactions-767-azure-wheel-manylinux2014-cp39 actions-767-azure-wheel-manylinux2014-cp39 2020-12-10T20:02:41Zactions-759-azure-wheel-manylinux2014-cp39 actions-759-azure-wheel-manylinux2014-cp39 2020-12-10T04:27:52Zactions-758-azure-wheel-manylinux2014-cp39 actions-758-azure-wheel-manylinux2014-cp39 2020-12-09T23:30:31ZFrom the name of the release you can construct the URL: And from there you can find: From here you can install directly with pip: $ python -m pip install https://github.com/ursa-labs/crossbow/releases/download/actions-783-azure-wheel-manylinux2014-cp39/pyarrow-2.1.0.dev363-cp39-cp39-manylinux2014_x86_64.whl |
terencehonles
commented
Dec 18, 2020
Please be aware that's like running a nightly release, but if you are fine with that you can install a crossbow wheel. However, you probably want to just follow the instructions here https://arrow.apache.org/docs/python/install.html#installing-nightly-packages TL;DR:
|
n8henrie
commented
Dec 18, 2020
via email
Thanks for the suggestion!
Unfortunately my distro has already moved to 3.9 (including all my cuda and DL libraries etc) so I need a 39 compatible wheel or I can't use it. Nightly it is for now! This way I can make sure the release I'm installing is at least passing CI. |
liyucheng09
commented
Dec 30, 2020
Using the command mentioned by @terencehonles doesn't work for me. This line still looking for pyarrow from the default indexes. |
kou
commented
Dec 30, 2020
Adding |
liyucheng09
commented
Dec 30, 2020
@kou Thanks for your help. I think it may because the link |
kou
commented
Dec 30, 2020
We're still working on it at #8915. |
terencehonles
commented
Jan 4, 2021
@liyucheng09 yes that's correct, if you're impatient and OK with using a crossbow build as @n8henrie was attempting you can use https://github.com/ursa-labs/crossbow/branches/all?query=actions-686 which built both mavericks and high sierra as you can see in #8386 (comment) |
liyucheng09
commented
Jan 5, 2021
@terencehonles Thanks a lot. I installed pyarrorw sucessfully. I was just curious that when I type I thought that figure it out may help me to understand deeper about python package installing process. |
terencehonles
commented
Jan 5, 2021
I'm assuming you mean trying to install the most recently released pyarrow before you installed via crossbow? That's because pyarrow has a source distribution and you're trying to build from source. Pip will not have installed numpy at install time and is working to make isolated installs the default, if you have numpy already installed the version you have may be an issue, but I think it's more likely it's too old not too new. You can try using the flag pyarrow is probably not the easiest to learn about Python's build process on, but you should read their documentation if you would like to actually be able to build it from source. |
… wheels Picking back up from #8386 in case I can handle this or someone else wants to work off of it. Closes#8915 from terencehonles/arrow-10904 Authored-by: Terence D. Honles <terence@honles.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
Adds support and testing for Python 3.9. I am looking for review as this change may have touched too many things, but I'm also looking to get the CI to test all the different environments.
H/T: @kou, the documentation and #5685 for helping me get this off the ground.