ARROW-10658: [Python][Packaging] Wheel builds for Apple Silicon - #10659

Closed
kszucs wants to merge 72 commits into
apache:masterfrom
kszucs:m1
Closed

ARROW-10658: [Python][Packaging] Wheel builds for Apple Silicon#10659
kszucs wants to merge 72 commits into
apache:masterfrom
kszucs:m1

Conversation

@kszucs

@kszucskszucs commented Jul 6, 2021•
edited
Loading

Copy link
Copy Markdown
Member

The following compiles the 3.9 wheel on apple silicon though I'm getting a segfault on import despite that all binaries have the right arch and linkage/bundling seems correct. I wasn't able to produce a coredump yet.

@github-actions

github-actionsBot commented Jul 6, 2021

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-12122

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/scripts/python_wheel_macos_build.sh Outdated

@kszucskszucsJul 6, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need to use the CMAKE_SYSTEM_* variables.

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/scripts/python_wheel_macos_build.sh Outdated

@kszucskszucsJul 6, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gRPC is not supported on apple silicon at the moment

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/vcpkg/arm64-osx-static-release.cmake~ Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadcpp/cmake_modules/FindSnappyAlt.cmake Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadwheel.sh Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/scripts/python_wheel_macos_build.sh Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadwheel.sh Outdated
@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip. M1 mac [WIP]ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]Jul 6, 2021
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadcpp/thirdparty/versions.txt Outdated

@kszucskszucsJul 6, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to check that it won't cause any performance regressions.

@kszucskszucsJul 19, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lidavidm how can I verify that #10024 doesn't apply here?

@lidavidmlidavidmJul 19, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I can see that should only apply to 2.0.0 and 1.7.x should be safe (the commenter in microsoft/mimalloc#363 claims 1.7.0 doesn't affect them at least)

@lidavidmlidavidmJul 19, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And the problematic test is still there - so assuming tests pass we should be OK

@kszucskszucsJul 19, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadwheel.sh Outdated
@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]ARROW-12122: [Python] Cannot install via pip M1 mac Jul 12, 2021
@kszucs

kszucs commented Jul 12, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-macos-*

@github-actions

github-actionsBot commented Jul 12, 2021

Copy link
Copy Markdown

Revision: 0f094730b1e9f3e49077a8d746668251e2bb8ddf

Submitted crossbow builds: ursacomputing/crossbow @ actions-585

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

@kszucs

kszucs commented Jul 12, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-macos-*

@github-actions

github-actionsBot commented Jul 12, 2021

Copy link
Copy Markdown

Revision: e1ab42b861b169217a02102c969c939f47695808

Submitted crossbow builds: ursacomputing/crossbow @ actions-586

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

kszucs
kszucs commented Jul 12, 2021
View reviewed changes
Comment threadci/scripts/install_python.sh Outdated

@kszucskszucsJul 12, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like there is no universal2 installers for python < 3.9

@kszucs

kszucs commented Jul 12, 2021

Copy link
Copy Markdown
MemberAuthor

@kou could you please give it a preliminary review?

kou
kou approved these changes Jul 13, 2021
View reviewed changes

@koukou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure!

Comment threadci/scripts/install_python.sh Outdated
Comment threadci/scripts/install_python.sh Outdated
Comment threadci/scripts/python_wheel_macos_build.sh Outdated
Comment threadcpp/cmake_modules/FindSnappyAlt.cmake Outdated
Comment threaddev/tasks/python-wheels/github.osx.arm64.yml Outdated

@koukouJul 13, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this?
I think that our CMake configuration detects ccache automatically.

@kszucskszucsJul 19, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need this to use ccache when compiling the vcpkg binaries.

Comment threadpython/requirements-wheel-build.txt Outdated
Comment threadpython/requirements-wheel-build.txt Outdated
Comment threadpython/requirements-wheel-build.txt Outdated
Comment threaddev/tasks/python-wheels/github.osx.amd64.yml Outdated

@koukouJul 13, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you keep this list in alphabetical order?

@kszucs

kszucs commented Jul 13, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g wheel

@github-actions

github-actionsBot commented Jul 13, 2021

Copy link
Copy Markdown

Revision: af0f71743fce3a9b0e4ce648b1c861a4cbc3d079

Submitted crossbow builds: ursacomputing/crossbow @ actions-587

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp36Github Actions
wheel-windows-cp37Github Actions
wheel-windows-cp38Github Actions
wheel-windows-cp39Github Actions

@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip M1 mac ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]Jul 13, 2021
@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]ARROW-12122: [Python] Cannot install via pip M1 mac Jul 13, 2021
@kszucs

kszucs commented Jul 13, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g wheel

@github-actions

github-actionsBot commented Jul 13, 2021

Copy link
Copy Markdown

Revision: 736739d28dcacc2256671660be0a6bbe8ed0008d

Submitted crossbow builds: ursacomputing/crossbow @ actions-588

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp36Github Actions
wheel-windows-cp37Github Actions
wheel-windows-cp38Github Actions
wheel-windows-cp39Github Actions

@kszucs

kszucs commented Jul 19, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g nightly

@github-actions

github-actionsBot commented Jul 19, 2021

Copy link
Copy Markdown

Revision: d29f104

Submitted crossbow builds: ursacomputing/crossbow @ actions-612

TaskStatus
centos-7-amd64Github Actions
centos-8-amd64Github Actions
centos-8-arm64TravisCI
conda-cleanAzure
conda-linux-gcc-py36-arm64Azure
conda-linux-gcc-py36-cpu-r40Azure
conda-linux-gcc-py36-cudaAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py36-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2017-py36-r40Azure
conda-win-vs2017-py37-r41Azure
conda-win-vs2017-py38Azure
conda-win-vs2017-py39Azure
debian-bullseye-amd64Github Actions
debian-bullseye-arm64TravisCI
debian-buster-amd64Github Actions
debian-buster-arm64TravisCI
example-cpp-minimal-build-staticGithub Actions
example-cpp-minimal-build-static-system-dependencyGithub Actions
homebrew-cppGithub Actions
homebrew-r-autobrewGithub Actions
java-jarsGithub Actions
nugetGithub Actions
python-sdistGithub Actions
test-build-vcpkg-winGithub Actions
test-conda-cppGithub Actions
test-conda-cpp-valgrindGithub Actions
test-conda-python-3.6Github Actions
test-conda-python-3.6-pandas-0.23Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-dask-latestGithub Actions
test-conda-python-3.7-dask-masterGithub Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-kartothek-latestGithub Actions
test-conda-python-3.7-kartothek-masterGithub Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-branch-3.0Github Actions
test-conda-python-3.7-turbodbc-latestGithub Actions
test-conda-python-3.7-turbodbc-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-masterGithub Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-masterGithub Actions
test-debian-10-cppGithub Actions
test-debian-10-go-1.15Azure
test-debian-10-python-3Azure
test-debian-c-glibGithub Actions
test-debian-rubyGithub Actions
test-fedora-33-cppGithub Actions
test-fedora-33-python-3Azure
test-r-devdocsGithub Actions
test-r-gcc-11Github Actions
test-r-install-localGithub Actions
test-r-linux-as-cranGithub Actions
test-r-linux-rchkGithub Actions
test-r-linux-valgrindAzure
test-r-minimal-buildAzure
test-r-rhub-ubuntu-gcc-release-latestAzure
test-r-rocker-r-base-latestAzure
test-r-rstudio-r-base-3.6-bionicAzure
test-r-rstudio-r-base-3.6-centos7-devtoolset-8Azure
test-r-rstudio-r-base-3.6-centos8Azure
test-r-rstudio-r-base-3.6-opensuse15Azure
test-r-rstudio-r-base-3.6-opensuse42Azure
test-r-rtools-35Github Actions
test-r-ubuntu-21.04Github Actions
test-r-version-compatibilityGithub Actions
test-r-versionsGithub Actions
test-r-without-arrowAzure
test-ubuntu-18.04-cppGithub Actions
test-ubuntu-18.04-cpp-releaseGithub Actions
test-ubuntu-18.04-cpp-staticGithub Actions
test-ubuntu-18.04-python-3Azure
test-ubuntu-18.04-r-sanitizerAzure
test-ubuntu-20.04-cppGithub Actions
test-ubuntu-20.04-cpp-14Github Actions
test-ubuntu-20.04-cpp-17Github Actions
test-ubuntu-20.04-cpp-thread-sanitizerGithub Actions
test-ubuntu-20.10-docsAzure
test-ubuntu-c-glibGithub Actions
test-ubuntu-default-docsAzure
test-ubuntu-rubyGithub Actions
ubuntu-bionic-amd64Github Actions
ubuntu-bionic-arm64TravisCI
ubuntu-focal-amd64Github Actions
ubuntu-focal-arm64TravisCI
ubuntu-groovy-amd64Github Actions
ubuntu-groovy-arm64TravisCI
ubuntu-hirsute-amd64Github Actions
ubuntu-hirsute-arm64TravisCI
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp36-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions

@kszucs

kszucs commented Jul 19, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-macos-*

@github-actions

github-actionsBot commented Jul 19, 2021

Copy link
Copy Markdown

Revision: bb3b170

Submitted crossbow builds: ursacomputing/crossbow @ actions-614

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

This was referenced Jul 19, 2021
Closed
Closed
kszucs
kszucs commented Jul 19, 2021
View reviewed changes

@kszucskszucs left a comment

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, all the relevant builds are passing

@mdagost

mdagost commented Aug 7, 2021

Copy link
Copy Markdown

It's great to see wheel builds for Apple Silicon. Any chance you may back port them to older versions? The snowflake python connector still relies on versions of pyarrow>=3.0.0,<3.1.0, and I've gone through quite a few backflips but haven't been able to get pyarrow installed from source.

@kszucs

kszucs commented Aug 7, 2021

Copy link
Copy Markdown
MemberAuthor

I'm not sure since:

  • we haven't received many requests to build M1 wheels for older arrow releases yet
  • it would probably require more effort than it looks like at first glance

Either way, could you please create a JIRA ticket so we can track this?

@mdagost

mdagost commented Aug 7, 2021

Copy link
Copy Markdown

Cool. I crated a JIRA: https://issues.apache.org/jira/browse/ARROW-13583

Snowflake is pretty popular, so I'm surprised nobody has asked yet. They're saying they can't/won't upgrade their pyarrow until the September timeframe.

@asfimportasfimport mentioned this pull request Aug 7, 2021
Closed
@jorisvandenbosschejorisvandenbossche mentioned this pull request Aug 9, 2024
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm left review comments
@koukoukou approved these changes

Assignees

No one assigned

Labels

Component: C++Component: Python

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@kszucs@kou@xhochy@mdagost@lidavidm
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all \u003cpre\u003e\u003ccode\u003e blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks"); } } catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); } })(); (function(){ try { var __m = "github.com"; var __re = new RegExp('^' + "github\\.com" + '
Skip to content

ARROW-10658: [Python][Packaging] Wheel builds for Apple Silicon - #10659

Closed
kszucs wants to merge 72 commits into
apache:masterfrom
kszucs:m1
Closed

ARROW-10658: [Python][Packaging] Wheel builds for Apple Silicon#10659
kszucs wants to merge 72 commits into
apache:masterfrom
kszucs:m1

Conversation

@kszucs

@kszucskszucs commented Jul 6, 2021•
edited
Loading

Copy link
Copy Markdown
Member

The following compiles the 3.9 wheel on apple silicon though I'm getting a segfault on import despite that all binaries have the right arch and linkage/bundling seems correct. I wasn't able to produce a coredump yet.

@github-actions

github-actionsBot commented Jul 6, 2021

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-12122

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/scripts/python_wheel_macos_build.sh Outdated

@kszucskszucsJul 6, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need to use the CMAKE_SYSTEM_* variables.

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/scripts/python_wheel_macos_build.sh Outdated

@kszucskszucsJul 6, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gRPC is not supported on apple silicon at the moment

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/vcpkg/arm64-osx-static-release.cmake~ Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadcpp/cmake_modules/FindSnappyAlt.cmake Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadwheel.sh Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/scripts/python_wheel_macos_build.sh Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadwheel.sh Outdated
@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip. M1 mac [WIP]ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]Jul 6, 2021
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadcpp/thirdparty/versions.txt Outdated

@kszucskszucsJul 6, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to check that it won't cause any performance regressions.

@kszucskszucsJul 19, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lidavidm how can I verify that #10024 doesn't apply here?

@lidavidmlidavidmJul 19, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I can see that should only apply to 2.0.0 and 1.7.x should be safe (the commenter in microsoft/mimalloc#363 claims 1.7.0 doesn't affect them at least)

@lidavidmlidavidmJul 19, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And the problematic test is still there - so assuming tests pass we should be OK

@kszucskszucsJul 19, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadwheel.sh Outdated
@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]ARROW-12122: [Python] Cannot install via pip M1 mac Jul 12, 2021
@kszucs

kszucs commented Jul 12, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-macos-*

@github-actions

github-actionsBot commented Jul 12, 2021

Copy link
Copy Markdown

Revision: 0f094730b1e9f3e49077a8d746668251e2bb8ddf

Submitted crossbow builds: ursacomputing/crossbow @ actions-585

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

@kszucs

kszucs commented Jul 12, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-macos-*

@github-actions

github-actionsBot commented Jul 12, 2021

Copy link
Copy Markdown

Revision: e1ab42b861b169217a02102c969c939f47695808

Submitted crossbow builds: ursacomputing/crossbow @ actions-586

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

kszucs
kszucs commented Jul 12, 2021
View reviewed changes
Comment threadci/scripts/install_python.sh Outdated

@kszucskszucsJul 12, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like there is no universal2 installers for python < 3.9

@kszucs

kszucs commented Jul 12, 2021

Copy link
Copy Markdown
MemberAuthor

@kou could you please give it a preliminary review?

kou
kou approved these changes Jul 13, 2021
View reviewed changes

@koukou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure!

Comment threadci/scripts/install_python.sh Outdated
Comment threadci/scripts/install_python.sh Outdated
Comment threadci/scripts/python_wheel_macos_build.sh Outdated
Comment threadcpp/cmake_modules/FindSnappyAlt.cmake Outdated
Comment threaddev/tasks/python-wheels/github.osx.arm64.yml Outdated

@koukouJul 13, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this?
I think that our CMake configuration detects ccache automatically.

@kszucskszucsJul 19, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need this to use ccache when compiling the vcpkg binaries.

Comment threadpython/requirements-wheel-build.txt Outdated
Comment threadpython/requirements-wheel-build.txt Outdated
Comment threadpython/requirements-wheel-build.txt Outdated
Comment threaddev/tasks/python-wheels/github.osx.amd64.yml Outdated

@koukouJul 13, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you keep this list in alphabetical order?

@kszucs

kszucs commented Jul 13, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g wheel

@github-actions

github-actionsBot commented Jul 13, 2021

Copy link
Copy Markdown

Revision: af0f71743fce3a9b0e4ce648b1c861a4cbc3d079

Submitted crossbow builds: ursacomputing/crossbow @ actions-587

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp36Github Actions
wheel-windows-cp37Github Actions
wheel-windows-cp38Github Actions
wheel-windows-cp39Github Actions

@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip M1 mac ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]Jul 13, 2021
@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]ARROW-12122: [Python] Cannot install via pip M1 mac Jul 13, 2021
@kszucs

kszucs commented Jul 13, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g wheel

@github-actions

github-actionsBot commented Jul 13, 2021

Copy link
Copy Markdown

Revision: 736739d28dcacc2256671660be0a6bbe8ed0008d

Submitted crossbow builds: ursacomputing/crossbow @ actions-588

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp36Github Actions
wheel-windows-cp37Github Actions
wheel-windows-cp38Github Actions
wheel-windows-cp39Github Actions

@kszucs

kszucs commented Jul 19, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g nightly

@github-actions

github-actionsBot commented Jul 19, 2021

Copy link
Copy Markdown

Revision: d29f104

Submitted crossbow builds: ursacomputing/crossbow @ actions-612

TaskStatus
centos-7-amd64Github Actions
centos-8-amd64Github Actions
centos-8-arm64TravisCI
conda-cleanAzure
conda-linux-gcc-py36-arm64Azure
conda-linux-gcc-py36-cpu-r40Azure
conda-linux-gcc-py36-cudaAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py36-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2017-py36-r40Azure
conda-win-vs2017-py37-r41Azure
conda-win-vs2017-py38Azure
conda-win-vs2017-py39Azure
debian-bullseye-amd64Github Actions
debian-bullseye-arm64TravisCI
debian-buster-amd64Github Actions
debian-buster-arm64TravisCI
example-cpp-minimal-build-staticGithub Actions
example-cpp-minimal-build-static-system-dependencyGithub Actions
homebrew-cppGithub Actions
homebrew-r-autobrewGithub Actions
java-jarsGithub Actions
nugetGithub Actions
python-sdistGithub Actions
test-build-vcpkg-winGithub Actions
test-conda-cppGithub Actions
test-conda-cpp-valgrindGithub Actions
test-conda-python-3.6Github Actions
test-conda-python-3.6-pandas-0.23Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-dask-latestGithub Actions
test-conda-python-3.7-dask-masterGithub Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-kartothek-latestGithub Actions
test-conda-python-3.7-kartothek-masterGithub Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-branch-3.0Github Actions
test-conda-python-3.7-turbodbc-latestGithub Actions
test-conda-python-3.7-turbodbc-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-masterGithub Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-masterGithub Actions
test-debian-10-cppGithub Actions
test-debian-10-go-1.15Azure
test-debian-10-python-3Azure
test-debian-c-glibGithub Actions
test-debian-rubyGithub Actions
test-fedora-33-cppGithub Actions
test-fedora-33-python-3Azure
test-r-devdocsGithub Actions
test-r-gcc-11Github Actions
test-r-install-localGithub Actions
test-r-linux-as-cranGithub Actions
test-r-linux-rchkGithub Actions
test-r-linux-valgrindAzure
test-r-minimal-buildAzure
test-r-rhub-ubuntu-gcc-release-latestAzure
test-r-rocker-r-base-latestAzure
test-r-rstudio-r-base-3.6-bionicAzure
test-r-rstudio-r-base-3.6-centos7-devtoolset-8Azure
test-r-rstudio-r-base-3.6-centos8Azure
test-r-rstudio-r-base-3.6-opensuse15Azure
test-r-rstudio-r-base-3.6-opensuse42Azure
test-r-rtools-35Github Actions
test-r-ubuntu-21.04Github Actions
test-r-version-compatibilityGithub Actions
test-r-versionsGithub Actions
test-r-without-arrowAzure
test-ubuntu-18.04-cppGithub Actions
test-ubuntu-18.04-cpp-releaseGithub Actions
test-ubuntu-18.04-cpp-staticGithub Actions
test-ubuntu-18.04-python-3Azure
test-ubuntu-18.04-r-sanitizerAzure
test-ubuntu-20.04-cppGithub Actions
test-ubuntu-20.04-cpp-14Github Actions
test-ubuntu-20.04-cpp-17Github Actions
test-ubuntu-20.04-cpp-thread-sanitizerGithub Actions
test-ubuntu-20.10-docsAzure
test-ubuntu-c-glibGithub Actions
test-ubuntu-default-docsAzure
test-ubuntu-rubyGithub Actions
ubuntu-bionic-amd64Github Actions
ubuntu-bionic-arm64TravisCI
ubuntu-focal-amd64Github Actions
ubuntu-focal-arm64TravisCI
ubuntu-groovy-amd64Github Actions
ubuntu-groovy-arm64TravisCI
ubuntu-hirsute-amd64Github Actions
ubuntu-hirsute-arm64TravisCI
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp36-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions

@kszucs

kszucs commented Jul 19, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-macos-*

@github-actions

github-actionsBot commented Jul 19, 2021

Copy link
Copy Markdown

Revision: bb3b170

Submitted crossbow builds: ursacomputing/crossbow @ actions-614

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

This was referenced Jul 19, 2021
Closed
Closed
kszucs
kszucs commented Jul 19, 2021
View reviewed changes

@kszucskszucs left a comment

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, all the relevant builds are passing

@mdagost

mdagost commented Aug 7, 2021

Copy link
Copy Markdown

It's great to see wheel builds for Apple Silicon. Any chance you may back port them to older versions? The snowflake python connector still relies on versions of pyarrow>=3.0.0,<3.1.0, and I've gone through quite a few backflips but haven't been able to get pyarrow installed from source.

@kszucs

kszucs commented Aug 7, 2021

Copy link
Copy Markdown
MemberAuthor

I'm not sure since:

  • we haven't received many requests to build M1 wheels for older arrow releases yet
  • it would probably require more effort than it looks like at first glance

Either way, could you please create a JIRA ticket so we can track this?

@mdagost

mdagost commented Aug 7, 2021

Copy link
Copy Markdown

Cool. I crated a JIRA: https://issues.apache.org/jira/browse/ARROW-13583

Snowflake is pretty popular, so I'm surprised nobody has asked yet. They're saying they can't/won't upgrade their pyarrow until the September timeframe.

@asfimportasfimport mentioned this pull request Aug 7, 2021
Closed
@jorisvandenbosschejorisvandenbossche mentioned this pull request Aug 9, 2024
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm left review comments
@koukoukou approved these changes

Assignees

No one assigned

Labels

Component: C++Component: Python

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@kszucs@kou@xhochy@mdagost@lidavidm
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

ARROW-10658: [Python][Packaging] Wheel builds for Apple Silicon - #10659

Closed
kszucs wants to merge 72 commits into
apache:masterfrom
kszucs:m1
Closed

ARROW-10658: [Python][Packaging] Wheel builds for Apple Silicon#10659
kszucs wants to merge 72 commits into
apache:masterfrom
kszucs:m1

Conversation

@kszucs

@kszucskszucs commented Jul 6, 2021•
edited
Loading

Copy link
Copy Markdown
Member

The following compiles the 3.9 wheel on apple silicon though I'm getting a segfault on import despite that all binaries have the right arch and linkage/bundling seems correct. I wasn't able to produce a coredump yet.

@github-actions

github-actionsBot commented Jul 6, 2021

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-12122

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/scripts/python_wheel_macos_build.sh Outdated

@kszucskszucsJul 6, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need to use the CMAKE_SYSTEM_* variables.

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/scripts/python_wheel_macos_build.sh Outdated

@kszucskszucsJul 6, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gRPC is not supported on apple silicon at the moment

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/vcpkg/arm64-osx-static-release.cmake~ Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadcpp/cmake_modules/FindSnappyAlt.cmake Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadwheel.sh Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/scripts/python_wheel_macos_build.sh Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadwheel.sh Outdated
@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip. M1 mac [WIP]ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]Jul 6, 2021
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadcpp/thirdparty/versions.txt Outdated

@kszucskszucsJul 6, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to check that it won't cause any performance regressions.

@kszucskszucsJul 19, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lidavidm how can I verify that #10024 doesn't apply here?

@lidavidmlidavidmJul 19, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I can see that should only apply to 2.0.0 and 1.7.x should be safe (the commenter in microsoft/mimalloc#363 claims 1.7.0 doesn't affect them at least)

@lidavidmlidavidmJul 19, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And the problematic test is still there - so assuming tests pass we should be OK

@kszucskszucsJul 19, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadwheel.sh Outdated
@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]ARROW-12122: [Python] Cannot install via pip M1 mac Jul 12, 2021
@kszucs

kszucs commented Jul 12, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-macos-*

@github-actions

github-actionsBot commented Jul 12, 2021

Copy link
Copy Markdown

Revision: 0f094730b1e9f3e49077a8d746668251e2bb8ddf

Submitted crossbow builds: ursacomputing/crossbow @ actions-585

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

@kszucs

kszucs commented Jul 12, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-macos-*

@github-actions

github-actionsBot commented Jul 12, 2021

Copy link
Copy Markdown

Revision: e1ab42b861b169217a02102c969c939f47695808

Submitted crossbow builds: ursacomputing/crossbow @ actions-586

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

kszucs
kszucs commented Jul 12, 2021
View reviewed changes
Comment threadci/scripts/install_python.sh Outdated

@kszucskszucsJul 12, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like there is no universal2 installers for python < 3.9

@kszucs

kszucs commented Jul 12, 2021

Copy link
Copy Markdown
MemberAuthor

@kou could you please give it a preliminary review?

kou
kou approved these changes Jul 13, 2021
View reviewed changes

@koukou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure!

Comment threadci/scripts/install_python.sh Outdated
Comment threadci/scripts/install_python.sh Outdated
Comment threadci/scripts/python_wheel_macos_build.sh Outdated
Comment threadcpp/cmake_modules/FindSnappyAlt.cmake Outdated
Comment threaddev/tasks/python-wheels/github.osx.arm64.yml Outdated

@koukouJul 13, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this?
I think that our CMake configuration detects ccache automatically.

@kszucskszucsJul 19, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need this to use ccache when compiling the vcpkg binaries.

Comment threadpython/requirements-wheel-build.txt Outdated
Comment threadpython/requirements-wheel-build.txt Outdated
Comment threadpython/requirements-wheel-build.txt Outdated
Comment threaddev/tasks/python-wheels/github.osx.amd64.yml Outdated

@koukouJul 13, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you keep this list in alphabetical order?

@kszucs

kszucs commented Jul 13, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g wheel

@github-actions

github-actionsBot commented Jul 13, 2021

Copy link
Copy Markdown

Revision: af0f71743fce3a9b0e4ce648b1c861a4cbc3d079

Submitted crossbow builds: ursacomputing/crossbow @ actions-587

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp36Github Actions
wheel-windows-cp37Github Actions
wheel-windows-cp38Github Actions
wheel-windows-cp39Github Actions

@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip M1 mac ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]Jul 13, 2021
@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]ARROW-12122: [Python] Cannot install via pip M1 mac Jul 13, 2021
@kszucs

kszucs commented Jul 13, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g wheel

@github-actions

github-actionsBot commented Jul 13, 2021

Copy link
Copy Markdown

Revision: 736739d28dcacc2256671660be0a6bbe8ed0008d

Submitted crossbow builds: ursacomputing/crossbow @ actions-588

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp36Github Actions
wheel-windows-cp37Github Actions
wheel-windows-cp38Github Actions
wheel-windows-cp39Github Actions

@kszucs

kszucs commented Jul 19, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g nightly

@github-actions

github-actionsBot commented Jul 19, 2021

Copy link
Copy Markdown

Revision: d29f104

Submitted crossbow builds: ursacomputing/crossbow @ actions-612

TaskStatus
centos-7-amd64Github Actions
centos-8-amd64Github Actions
centos-8-arm64TravisCI
conda-cleanAzure
conda-linux-gcc-py36-arm64Azure
conda-linux-gcc-py36-cpu-r40Azure
conda-linux-gcc-py36-cudaAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py36-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2017-py36-r40Azure
conda-win-vs2017-py37-r41Azure
conda-win-vs2017-py38Azure
conda-win-vs2017-py39Azure
debian-bullseye-amd64Github Actions
debian-bullseye-arm64TravisCI
debian-buster-amd64Github Actions
debian-buster-arm64TravisCI
example-cpp-minimal-build-staticGithub Actions
example-cpp-minimal-build-static-system-dependencyGithub Actions
homebrew-cppGithub Actions
homebrew-r-autobrewGithub Actions
java-jarsGithub Actions
nugetGithub Actions
python-sdistGithub Actions
test-build-vcpkg-winGithub Actions
test-conda-cppGithub Actions
test-conda-cpp-valgrindGithub Actions
test-conda-python-3.6Github Actions
test-conda-python-3.6-pandas-0.23Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-dask-latestGithub Actions
test-conda-python-3.7-dask-masterGithub Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-kartothek-latestGithub Actions
test-conda-python-3.7-kartothek-masterGithub Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-branch-3.0Github Actions
test-conda-python-3.7-turbodbc-latestGithub Actions
test-conda-python-3.7-turbodbc-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-masterGithub Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-masterGithub Actions
test-debian-10-cppGithub Actions
test-debian-10-go-1.15Azure
test-debian-10-python-3Azure
test-debian-c-glibGithub Actions
test-debian-rubyGithub Actions
test-fedora-33-cppGithub Actions
test-fedora-33-python-3Azure
test-r-devdocsGithub Actions
test-r-gcc-11Github Actions
test-r-install-localGithub Actions
test-r-linux-as-cranGithub Actions
test-r-linux-rchkGithub Actions
test-r-linux-valgrindAzure
test-r-minimal-buildAzure
test-r-rhub-ubuntu-gcc-release-latestAzure
test-r-rocker-r-base-latestAzure
test-r-rstudio-r-base-3.6-bionicAzure
test-r-rstudio-r-base-3.6-centos7-devtoolset-8Azure
test-r-rstudio-r-base-3.6-centos8Azure
test-r-rstudio-r-base-3.6-opensuse15Azure
test-r-rstudio-r-base-3.6-opensuse42Azure
test-r-rtools-35Github Actions
test-r-ubuntu-21.04Github Actions
test-r-version-compatibilityGithub Actions
test-r-versionsGithub Actions
test-r-without-arrowAzure
test-ubuntu-18.04-cppGithub Actions
test-ubuntu-18.04-cpp-releaseGithub Actions
test-ubuntu-18.04-cpp-staticGithub Actions
test-ubuntu-18.04-python-3Azure
test-ubuntu-18.04-r-sanitizerAzure
test-ubuntu-20.04-cppGithub Actions
test-ubuntu-20.04-cpp-14Github Actions
test-ubuntu-20.04-cpp-17Github Actions
test-ubuntu-20.04-cpp-thread-sanitizerGithub Actions
test-ubuntu-20.10-docsAzure
test-ubuntu-c-glibGithub Actions
test-ubuntu-default-docsAzure
test-ubuntu-rubyGithub Actions
ubuntu-bionic-amd64Github Actions
ubuntu-bionic-arm64TravisCI
ubuntu-focal-amd64Github Actions
ubuntu-focal-arm64TravisCI
ubuntu-groovy-amd64Github Actions
ubuntu-groovy-arm64TravisCI
ubuntu-hirsute-amd64Github Actions
ubuntu-hirsute-arm64TravisCI
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp36-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions

@kszucs

kszucs commented Jul 19, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-macos-*

@github-actions

github-actionsBot commented Jul 19, 2021

Copy link
Copy Markdown

Revision: bb3b170

Submitted crossbow builds: ursacomputing/crossbow @ actions-614

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

This was referenced Jul 19, 2021
Closed
Closed
kszucs
kszucs commented Jul 19, 2021
View reviewed changes

@kszucskszucs left a comment

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, all the relevant builds are passing

@mdagost

mdagost commented Aug 7, 2021

Copy link
Copy Markdown

It's great to see wheel builds for Apple Silicon. Any chance you may back port them to older versions? The snowflake python connector still relies on versions of pyarrow>=3.0.0,<3.1.0, and I've gone through quite a few backflips but haven't been able to get pyarrow installed from source.

@kszucs

kszucs commented Aug 7, 2021

Copy link
Copy Markdown
MemberAuthor

I'm not sure since:

  • we haven't received many requests to build M1 wheels for older arrow releases yet
  • it would probably require more effort than it looks like at first glance

Either way, could you please create a JIRA ticket so we can track this?

@mdagost

mdagost commented Aug 7, 2021

Copy link
Copy Markdown

Cool. I crated a JIRA: https://issues.apache.org/jira/browse/ARROW-13583

Snowflake is pretty popular, so I'm surprised nobody has asked yet. They're saying they can't/won't upgrade their pyarrow until the September timeframe.

@asfimportasfimport mentioned this pull request Aug 7, 2021
Closed
@jorisvandenbosschejorisvandenbossche mentioned this pull request Aug 9, 2024
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm left review comments
@koukoukou approved these changes

Assignees

No one assigned

Labels

Component: C++Component: Python

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@kszucs@kou@xhochy@mdagost@lidavidm
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length \u003e 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

ARROW-10658: [Python][Packaging] Wheel builds for Apple Silicon - #10659

Closed
kszucs wants to merge 72 commits into
apache:masterfrom
kszucs:m1
Closed

ARROW-10658: [Python][Packaging] Wheel builds for Apple Silicon#10659
kszucs wants to merge 72 commits into
apache:masterfrom
kszucs:m1

Conversation

@kszucs

@kszucskszucs commented Jul 6, 2021•
edited
Loading

Copy link
Copy Markdown
Member

The following compiles the 3.9 wheel on apple silicon though I'm getting a segfault on import despite that all binaries have the right arch and linkage/bundling seems correct. I wasn't able to produce a coredump yet.

@github-actions

github-actionsBot commented Jul 6, 2021

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-12122

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/scripts/python_wheel_macos_build.sh Outdated

@kszucskszucsJul 6, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need to use the CMAKE_SYSTEM_* variables.

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/scripts/python_wheel_macos_build.sh Outdated

@kszucskszucsJul 6, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gRPC is not supported on apple silicon at the moment

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/vcpkg/arm64-osx-static-release.cmake~ Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadcpp/cmake_modules/FindSnappyAlt.cmake Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadwheel.sh Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/scripts/python_wheel_macos_build.sh Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadwheel.sh Outdated
@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip. M1 mac [WIP]ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]Jul 6, 2021
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadcpp/thirdparty/versions.txt Outdated

@kszucskszucsJul 6, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to check that it won't cause any performance regressions.

@kszucskszucsJul 19, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lidavidm how can I verify that #10024 doesn't apply here?

@lidavidmlidavidmJul 19, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I can see that should only apply to 2.0.0 and 1.7.x should be safe (the commenter in microsoft/mimalloc#363 claims 1.7.0 doesn't affect them at least)

@lidavidmlidavidmJul 19, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And the problematic test is still there - so assuming tests pass we should be OK

@kszucskszucsJul 19, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadwheel.sh Outdated
@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]ARROW-12122: [Python] Cannot install via pip M1 mac Jul 12, 2021
@kszucs

kszucs commented Jul 12, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-macos-*

@github-actions

github-actionsBot commented Jul 12, 2021

Copy link
Copy Markdown

Revision: 0f094730b1e9f3e49077a8d746668251e2bb8ddf

Submitted crossbow builds: ursacomputing/crossbow @ actions-585

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

@kszucs

kszucs commented Jul 12, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-macos-*

@github-actions

github-actionsBot commented Jul 12, 2021

Copy link
Copy Markdown

Revision: e1ab42b861b169217a02102c969c939f47695808

Submitted crossbow builds: ursacomputing/crossbow @ actions-586

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

kszucs
kszucs commented Jul 12, 2021
View reviewed changes
Comment threadci/scripts/install_python.sh Outdated

@kszucskszucsJul 12, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like there is no universal2 installers for python < 3.9

@kszucs

kszucs commented Jul 12, 2021

Copy link
Copy Markdown
MemberAuthor

@kou could you please give it a preliminary review?

kou
kou approved these changes Jul 13, 2021
View reviewed changes

@koukou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure!

Comment threadci/scripts/install_python.sh Outdated
Comment threadci/scripts/install_python.sh Outdated
Comment threadci/scripts/python_wheel_macos_build.sh Outdated
Comment threadcpp/cmake_modules/FindSnappyAlt.cmake Outdated
Comment threaddev/tasks/python-wheels/github.osx.arm64.yml Outdated

@koukouJul 13, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this?
I think that our CMake configuration detects ccache automatically.

@kszucskszucsJul 19, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need this to use ccache when compiling the vcpkg binaries.

Comment threadpython/requirements-wheel-build.txt Outdated
Comment threadpython/requirements-wheel-build.txt Outdated
Comment threadpython/requirements-wheel-build.txt Outdated
Comment threaddev/tasks/python-wheels/github.osx.amd64.yml Outdated

@koukouJul 13, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you keep this list in alphabetical order?

@kszucs

kszucs commented Jul 13, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g wheel

@github-actions

github-actionsBot commented Jul 13, 2021

Copy link
Copy Markdown

Revision: af0f71743fce3a9b0e4ce648b1c861a4cbc3d079

Submitted crossbow builds: ursacomputing/crossbow @ actions-587

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp36Github Actions
wheel-windows-cp37Github Actions
wheel-windows-cp38Github Actions
wheel-windows-cp39Github Actions

@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip M1 mac ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]Jul 13, 2021
@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]ARROW-12122: [Python] Cannot install via pip M1 mac Jul 13, 2021
@kszucs

kszucs commented Jul 13, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g wheel

@github-actions

github-actionsBot commented Jul 13, 2021

Copy link
Copy Markdown

Revision: 736739d28dcacc2256671660be0a6bbe8ed0008d

Submitted crossbow builds: ursacomputing/crossbow @ actions-588

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp36Github Actions
wheel-windows-cp37Github Actions
wheel-windows-cp38Github Actions
wheel-windows-cp39Github Actions

@kszucs

kszucs commented Jul 19, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g nightly

@github-actions

github-actionsBot commented Jul 19, 2021

Copy link
Copy Markdown

Revision: d29f104

Submitted crossbow builds: ursacomputing/crossbow @ actions-612

TaskStatus
centos-7-amd64Github Actions
centos-8-amd64Github Actions
centos-8-arm64TravisCI
conda-cleanAzure
conda-linux-gcc-py36-arm64Azure
conda-linux-gcc-py36-cpu-r40Azure
conda-linux-gcc-py36-cudaAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py36-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2017-py36-r40Azure
conda-win-vs2017-py37-r41Azure
conda-win-vs2017-py38Azure
conda-win-vs2017-py39Azure
debian-bullseye-amd64Github Actions
debian-bullseye-arm64TravisCI
debian-buster-amd64Github Actions
debian-buster-arm64TravisCI
example-cpp-minimal-build-staticGithub Actions
example-cpp-minimal-build-static-system-dependencyGithub Actions
homebrew-cppGithub Actions
homebrew-r-autobrewGithub Actions
java-jarsGithub Actions
nugetGithub Actions
python-sdistGithub Actions
test-build-vcpkg-winGithub Actions
test-conda-cppGithub Actions
test-conda-cpp-valgrindGithub Actions
test-conda-python-3.6Github Actions
test-conda-python-3.6-pandas-0.23Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-dask-latestGithub Actions
test-conda-python-3.7-dask-masterGithub Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-kartothek-latestGithub Actions
test-conda-python-3.7-kartothek-masterGithub Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-branch-3.0Github Actions
test-conda-python-3.7-turbodbc-latestGithub Actions
test-conda-python-3.7-turbodbc-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-masterGithub Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-masterGithub Actions
test-debian-10-cppGithub Actions
test-debian-10-go-1.15Azure
test-debian-10-python-3Azure
test-debian-c-glibGithub Actions
test-debian-rubyGithub Actions
test-fedora-33-cppGithub Actions
test-fedora-33-python-3Azure
test-r-devdocsGithub Actions
test-r-gcc-11Github Actions
test-r-install-localGithub Actions
test-r-linux-as-cranGithub Actions
test-r-linux-rchkGithub Actions
test-r-linux-valgrindAzure
test-r-minimal-buildAzure
test-r-rhub-ubuntu-gcc-release-latestAzure
test-r-rocker-r-base-latestAzure
test-r-rstudio-r-base-3.6-bionicAzure
test-r-rstudio-r-base-3.6-centos7-devtoolset-8Azure
test-r-rstudio-r-base-3.6-centos8Azure
test-r-rstudio-r-base-3.6-opensuse15Azure
test-r-rstudio-r-base-3.6-opensuse42Azure
test-r-rtools-35Github Actions
test-r-ubuntu-21.04Github Actions
test-r-version-compatibilityGithub Actions
test-r-versionsGithub Actions
test-r-without-arrowAzure
test-ubuntu-18.04-cppGithub Actions
test-ubuntu-18.04-cpp-releaseGithub Actions
test-ubuntu-18.04-cpp-staticGithub Actions
test-ubuntu-18.04-python-3Azure
test-ubuntu-18.04-r-sanitizerAzure
test-ubuntu-20.04-cppGithub Actions
test-ubuntu-20.04-cpp-14Github Actions
test-ubuntu-20.04-cpp-17Github Actions
test-ubuntu-20.04-cpp-thread-sanitizerGithub Actions
test-ubuntu-20.10-docsAzure
test-ubuntu-c-glibGithub Actions
test-ubuntu-default-docsAzure
test-ubuntu-rubyGithub Actions
ubuntu-bionic-amd64Github Actions
ubuntu-bionic-arm64TravisCI
ubuntu-focal-amd64Github Actions
ubuntu-focal-arm64TravisCI
ubuntu-groovy-amd64Github Actions
ubuntu-groovy-arm64TravisCI
ubuntu-hirsute-amd64Github Actions
ubuntu-hirsute-arm64TravisCI
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp36-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions

@kszucs

kszucs commented Jul 19, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-macos-*

@github-actions

github-actionsBot commented Jul 19, 2021

Copy link
Copy Markdown

Revision: bb3b170

Submitted crossbow builds: ursacomputing/crossbow @ actions-614

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

This was referenced Jul 19, 2021
Closed
Closed
kszucs
kszucs commented Jul 19, 2021
View reviewed changes

@kszucskszucs left a comment

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, all the relevant builds are passing

@mdagost

mdagost commented Aug 7, 2021

Copy link
Copy Markdown

It's great to see wheel builds for Apple Silicon. Any chance you may back port them to older versions? The snowflake python connector still relies on versions of pyarrow>=3.0.0,<3.1.0, and I've gone through quite a few backflips but haven't been able to get pyarrow installed from source.

@kszucs

kszucs commented Aug 7, 2021

Copy link
Copy Markdown
MemberAuthor

I'm not sure since:

  • we haven't received many requests to build M1 wheels for older arrow releases yet
  • it would probably require more effort than it looks like at first glance

Either way, could you please create a JIRA ticket so we can track this?

@mdagost

mdagost commented Aug 7, 2021

Copy link
Copy Markdown

Cool. I crated a JIRA: https://issues.apache.org/jira/browse/ARROW-13583

Snowflake is pretty popular, so I'm surprised nobody has asked yet. They're saying they can't/won't upgrade their pyarrow until the September timeframe.

@asfimportasfimport mentioned this pull request Aug 7, 2021
Closed
@jorisvandenbosschejorisvandenbossche mentioned this pull request Aug 9, 2024
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm left review comments
@koukoukou approved these changes

Assignees

No one assigned

Labels

Component: C++Component: Python

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@kszucs@kou@xhochy@mdagost@lidavidm
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

ARROW-10658: [Python][Packaging] Wheel builds for Apple Silicon - #10659

Closed
kszucs wants to merge 72 commits into
apache:masterfrom
kszucs:m1
Closed

ARROW-10658: [Python][Packaging] Wheel builds for Apple Silicon#10659
kszucs wants to merge 72 commits into
apache:masterfrom
kszucs:m1

Conversation

@kszucs

@kszucskszucs commented Jul 6, 2021•
edited
Loading

Copy link
Copy Markdown
Member

The following compiles the 3.9 wheel on apple silicon though I'm getting a segfault on import despite that all binaries have the right arch and linkage/bundling seems correct. I wasn't able to produce a coredump yet.

@github-actions

github-actionsBot commented Jul 6, 2021

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-12122

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/scripts/python_wheel_macos_build.sh Outdated

@kszucskszucsJul 6, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need to use the CMAKE_SYSTEM_* variables.

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/scripts/python_wheel_macos_build.sh Outdated

@kszucskszucsJul 6, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gRPC is not supported on apple silicon at the moment

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/vcpkg/arm64-osx-static-release.cmake~ Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadcpp/cmake_modules/FindSnappyAlt.cmake Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadwheel.sh Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/scripts/python_wheel_macos_build.sh Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadwheel.sh Outdated
@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip. M1 mac [WIP]ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]Jul 6, 2021
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadcpp/thirdparty/versions.txt Outdated

@kszucskszucsJul 6, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to check that it won't cause any performance regressions.

@kszucskszucsJul 19, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lidavidm how can I verify that #10024 doesn't apply here?

@lidavidmlidavidmJul 19, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I can see that should only apply to 2.0.0 and 1.7.x should be safe (the commenter in microsoft/mimalloc#363 claims 1.7.0 doesn't affect them at least)

@lidavidmlidavidmJul 19, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And the problematic test is still there - so assuming tests pass we should be OK

@kszucskszucsJul 19, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadwheel.sh Outdated
@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]ARROW-12122: [Python] Cannot install via pip M1 mac Jul 12, 2021
@kszucs

kszucs commented Jul 12, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-macos-*

@github-actions

github-actionsBot commented Jul 12, 2021

Copy link
Copy Markdown

Revision: 0f094730b1e9f3e49077a8d746668251e2bb8ddf

Submitted crossbow builds: ursacomputing/crossbow @ actions-585

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

@kszucs

kszucs commented Jul 12, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-macos-*

@github-actions

github-actionsBot commented Jul 12, 2021

Copy link
Copy Markdown

Revision: e1ab42b861b169217a02102c969c939f47695808

Submitted crossbow builds: ursacomputing/crossbow @ actions-586

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

kszucs
kszucs commented Jul 12, 2021
View reviewed changes
Comment threadci/scripts/install_python.sh Outdated

@kszucskszucsJul 12, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like there is no universal2 installers for python < 3.9

@kszucs

kszucs commented Jul 12, 2021

Copy link
Copy Markdown
MemberAuthor

@kou could you please give it a preliminary review?

kou
kou approved these changes Jul 13, 2021
View reviewed changes

@koukou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure!

Comment threadci/scripts/install_python.sh Outdated
Comment threadci/scripts/install_python.sh Outdated
Comment threadci/scripts/python_wheel_macos_build.sh Outdated
Comment threadcpp/cmake_modules/FindSnappyAlt.cmake Outdated
Comment threaddev/tasks/python-wheels/github.osx.arm64.yml Outdated

@koukouJul 13, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this?
I think that our CMake configuration detects ccache automatically.

@kszucskszucsJul 19, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need this to use ccache when compiling the vcpkg binaries.

Comment threadpython/requirements-wheel-build.txt Outdated
Comment threadpython/requirements-wheel-build.txt Outdated
Comment threadpython/requirements-wheel-build.txt Outdated
Comment threaddev/tasks/python-wheels/github.osx.amd64.yml Outdated

@koukouJul 13, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you keep this list in alphabetical order?

@kszucs

kszucs commented Jul 13, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g wheel

@github-actions

github-actionsBot commented Jul 13, 2021

Copy link
Copy Markdown

Revision: af0f71743fce3a9b0e4ce648b1c861a4cbc3d079

Submitted crossbow builds: ursacomputing/crossbow @ actions-587

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp36Github Actions
wheel-windows-cp37Github Actions
wheel-windows-cp38Github Actions
wheel-windows-cp39Github Actions

@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip M1 mac ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]Jul 13, 2021
@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]ARROW-12122: [Python] Cannot install via pip M1 mac Jul 13, 2021
@kszucs

kszucs commented Jul 13, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g wheel

@github-actions

github-actionsBot commented Jul 13, 2021

Copy link
Copy Markdown

Revision: 736739d28dcacc2256671660be0a6bbe8ed0008d

Submitted crossbow builds: ursacomputing/crossbow @ actions-588

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp36Github Actions
wheel-windows-cp37Github Actions
wheel-windows-cp38Github Actions
wheel-windows-cp39Github Actions

@kszucs

kszucs commented Jul 19, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g nightly

@github-actions

github-actionsBot commented Jul 19, 2021

Copy link
Copy Markdown

Revision: d29f104

Submitted crossbow builds: ursacomputing/crossbow @ actions-612

TaskStatus
centos-7-amd64Github Actions
centos-8-amd64Github Actions
centos-8-arm64TravisCI
conda-cleanAzure
conda-linux-gcc-py36-arm64Azure
conda-linux-gcc-py36-cpu-r40Azure
conda-linux-gcc-py36-cudaAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py36-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2017-py36-r40Azure
conda-win-vs2017-py37-r41Azure
conda-win-vs2017-py38Azure
conda-win-vs2017-py39Azure
debian-bullseye-amd64Github Actions
debian-bullseye-arm64TravisCI
debian-buster-amd64Github Actions
debian-buster-arm64TravisCI
example-cpp-minimal-build-staticGithub Actions
example-cpp-minimal-build-static-system-dependencyGithub Actions
homebrew-cppGithub Actions
homebrew-r-autobrewGithub Actions
java-jarsGithub Actions
nugetGithub Actions
python-sdistGithub Actions
test-build-vcpkg-winGithub Actions
test-conda-cppGithub Actions
test-conda-cpp-valgrindGithub Actions
test-conda-python-3.6Github Actions
test-conda-python-3.6-pandas-0.23Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-dask-latestGithub Actions
test-conda-python-3.7-dask-masterGithub Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-kartothek-latestGithub Actions
test-conda-python-3.7-kartothek-masterGithub Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-branch-3.0Github Actions
test-conda-python-3.7-turbodbc-latestGithub Actions
test-conda-python-3.7-turbodbc-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-masterGithub Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-masterGithub Actions
test-debian-10-cppGithub Actions
test-debian-10-go-1.15Azure
test-debian-10-python-3Azure
test-debian-c-glibGithub Actions
test-debian-rubyGithub Actions
test-fedora-33-cppGithub Actions
test-fedora-33-python-3Azure
test-r-devdocsGithub Actions
test-r-gcc-11Github Actions
test-r-install-localGithub Actions
test-r-linux-as-cranGithub Actions
test-r-linux-rchkGithub Actions
test-r-linux-valgrindAzure
test-r-minimal-buildAzure
test-r-rhub-ubuntu-gcc-release-latestAzure
test-r-rocker-r-base-latestAzure
test-r-rstudio-r-base-3.6-bionicAzure
test-r-rstudio-r-base-3.6-centos7-devtoolset-8Azure
test-r-rstudio-r-base-3.6-centos8Azure
test-r-rstudio-r-base-3.6-opensuse15Azure
test-r-rstudio-r-base-3.6-opensuse42Azure
test-r-rtools-35Github Actions
test-r-ubuntu-21.04Github Actions
test-r-version-compatibilityGithub Actions
test-r-versionsGithub Actions
test-r-without-arrowAzure
test-ubuntu-18.04-cppGithub Actions
test-ubuntu-18.04-cpp-releaseGithub Actions
test-ubuntu-18.04-cpp-staticGithub Actions
test-ubuntu-18.04-python-3Azure
test-ubuntu-18.04-r-sanitizerAzure
test-ubuntu-20.04-cppGithub Actions
test-ubuntu-20.04-cpp-14Github Actions
test-ubuntu-20.04-cpp-17Github Actions
test-ubuntu-20.04-cpp-thread-sanitizerGithub Actions
test-ubuntu-20.10-docsAzure
test-ubuntu-c-glibGithub Actions
test-ubuntu-default-docsAzure
test-ubuntu-rubyGithub Actions
ubuntu-bionic-amd64Github Actions
ubuntu-bionic-arm64TravisCI
ubuntu-focal-amd64Github Actions
ubuntu-focal-arm64TravisCI
ubuntu-groovy-amd64Github Actions
ubuntu-groovy-arm64TravisCI
ubuntu-hirsute-amd64Github Actions
ubuntu-hirsute-arm64TravisCI
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp36-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions

@kszucs

kszucs commented Jul 19, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-macos-*

@github-actions

github-actionsBot commented Jul 19, 2021

Copy link
Copy Markdown

Revision: bb3b170

Submitted crossbow builds: ursacomputing/crossbow @ actions-614

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

This was referenced Jul 19, 2021
Closed
Closed
kszucs
kszucs commented Jul 19, 2021
View reviewed changes

@kszucskszucs left a comment

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, all the relevant builds are passing

@mdagost

mdagost commented Aug 7, 2021

Copy link
Copy Markdown

It's great to see wheel builds for Apple Silicon. Any chance you may back port them to older versions? The snowflake python connector still relies on versions of pyarrow>=3.0.0,<3.1.0, and I've gone through quite a few backflips but haven't been able to get pyarrow installed from source.

@kszucs

kszucs commented Aug 7, 2021

Copy link
Copy Markdown
MemberAuthor

I'm not sure since:

  • we haven't received many requests to build M1 wheels for older arrow releases yet
  • it would probably require more effort than it looks like at first glance

Either way, could you please create a JIRA ticket so we can track this?

@mdagost

mdagost commented Aug 7, 2021

Copy link
Copy Markdown

Cool. I crated a JIRA: https://issues.apache.org/jira/browse/ARROW-13583

Snowflake is pretty popular, so I'm surprised nobody has asked yet. They're saying they can't/won't upgrade their pyarrow until the September timeframe.

@asfimportasfimport mentioned this pull request Aug 7, 2021
Closed
@jorisvandenbosschejorisvandenbossche mentioned this pull request Aug 9, 2024
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm left review comments
@koukoukou approved these changes

Assignees

No one assigned

Labels

Component: C++Component: Python

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@kszucs@kou@xhochy@mdagost@lidavidm
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

ARROW-10658: [Python][Packaging] Wheel builds for Apple Silicon - #10659

Closed
kszucs wants to merge 72 commits into
apache:masterfrom
kszucs:m1
Closed

ARROW-10658: [Python][Packaging] Wheel builds for Apple Silicon#10659
kszucs wants to merge 72 commits into
apache:masterfrom
kszucs:m1

Conversation

@kszucs

@kszucskszucs commented Jul 6, 2021•
edited
Loading

Copy link
Copy Markdown
Member

The following compiles the 3.9 wheel on apple silicon though I'm getting a segfault on import despite that all binaries have the right arch and linkage/bundling seems correct. I wasn't able to produce a coredump yet.

@github-actions

github-actionsBot commented Jul 6, 2021

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-12122

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/scripts/python_wheel_macos_build.sh Outdated

@kszucskszucsJul 6, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need to use the CMAKE_SYSTEM_* variables.

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/scripts/python_wheel_macos_build.sh Outdated

@kszucskszucsJul 6, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gRPC is not supported on apple silicon at the moment

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/vcpkg/arm64-osx-static-release.cmake~ Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadcpp/cmake_modules/FindSnappyAlt.cmake Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadwheel.sh Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/scripts/python_wheel_macos_build.sh Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadwheel.sh Outdated
@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip. M1 mac [WIP]ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]Jul 6, 2021
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadcpp/thirdparty/versions.txt Outdated

@kszucskszucsJul 6, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to check that it won't cause any performance regressions.

@kszucskszucsJul 19, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lidavidm how can I verify that #10024 doesn't apply here?

@lidavidmlidavidmJul 19, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I can see that should only apply to 2.0.0 and 1.7.x should be safe (the commenter in microsoft/mimalloc#363 claims 1.7.0 doesn't affect them at least)

@lidavidmlidavidmJul 19, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And the problematic test is still there - so assuming tests pass we should be OK

@kszucskszucsJul 19, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadwheel.sh Outdated
@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]ARROW-12122: [Python] Cannot install via pip M1 mac Jul 12, 2021
@kszucs

kszucs commented Jul 12, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-macos-*

@github-actions

github-actionsBot commented Jul 12, 2021

Copy link
Copy Markdown

Revision: 0f094730b1e9f3e49077a8d746668251e2bb8ddf

Submitted crossbow builds: ursacomputing/crossbow @ actions-585

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

@kszucs

kszucs commented Jul 12, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-macos-*

@github-actions

github-actionsBot commented Jul 12, 2021

Copy link
Copy Markdown

Revision: e1ab42b861b169217a02102c969c939f47695808

Submitted crossbow builds: ursacomputing/crossbow @ actions-586

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

kszucs
kszucs commented Jul 12, 2021
View reviewed changes
Comment threadci/scripts/install_python.sh Outdated

@kszucskszucsJul 12, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like there is no universal2 installers for python < 3.9

@kszucs

kszucs commented Jul 12, 2021

Copy link
Copy Markdown
MemberAuthor

@kou could you please give it a preliminary review?

kou
kou approved these changes Jul 13, 2021
View reviewed changes

@koukou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure!

Comment threadci/scripts/install_python.sh Outdated
Comment threadci/scripts/install_python.sh Outdated
Comment threadci/scripts/python_wheel_macos_build.sh Outdated
Comment threadcpp/cmake_modules/FindSnappyAlt.cmake Outdated
Comment threaddev/tasks/python-wheels/github.osx.arm64.yml Outdated

@koukouJul 13, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this?
I think that our CMake configuration detects ccache automatically.

@kszucskszucsJul 19, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need this to use ccache when compiling the vcpkg binaries.

Comment threadpython/requirements-wheel-build.txt Outdated
Comment threadpython/requirements-wheel-build.txt Outdated
Comment threadpython/requirements-wheel-build.txt Outdated
Comment threaddev/tasks/python-wheels/github.osx.amd64.yml Outdated

@koukouJul 13, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you keep this list in alphabetical order?

@kszucs

kszucs commented Jul 13, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g wheel

@github-actions

github-actionsBot commented Jul 13, 2021

Copy link
Copy Markdown

Revision: af0f71743fce3a9b0e4ce648b1c861a4cbc3d079

Submitted crossbow builds: ursacomputing/crossbow @ actions-587

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp36Github Actions
wheel-windows-cp37Github Actions
wheel-windows-cp38Github Actions
wheel-windows-cp39Github Actions

@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip M1 mac ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]Jul 13, 2021
@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]ARROW-12122: [Python] Cannot install via pip M1 mac Jul 13, 2021
@kszucs

kszucs commented Jul 13, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g wheel

@github-actions

github-actionsBot commented Jul 13, 2021

Copy link
Copy Markdown

Revision: 736739d28dcacc2256671660be0a6bbe8ed0008d

Submitted crossbow builds: ursacomputing/crossbow @ actions-588

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp36Github Actions
wheel-windows-cp37Github Actions
wheel-windows-cp38Github Actions
wheel-windows-cp39Github Actions

@kszucs

kszucs commented Jul 19, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g nightly

@github-actions

github-actionsBot commented Jul 19, 2021

Copy link
Copy Markdown

Revision: d29f104

Submitted crossbow builds: ursacomputing/crossbow @ actions-612

TaskStatus
centos-7-amd64Github Actions
centos-8-amd64Github Actions
centos-8-arm64TravisCI
conda-cleanAzure
conda-linux-gcc-py36-arm64Azure
conda-linux-gcc-py36-cpu-r40Azure
conda-linux-gcc-py36-cudaAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py36-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2017-py36-r40Azure
conda-win-vs2017-py37-r41Azure
conda-win-vs2017-py38Azure
conda-win-vs2017-py39Azure
debian-bullseye-amd64Github Actions
debian-bullseye-arm64TravisCI
debian-buster-amd64Github Actions
debian-buster-arm64TravisCI
example-cpp-minimal-build-staticGithub Actions
example-cpp-minimal-build-static-system-dependencyGithub Actions
homebrew-cppGithub Actions
homebrew-r-autobrewGithub Actions
java-jarsGithub Actions
nugetGithub Actions
python-sdistGithub Actions
test-build-vcpkg-winGithub Actions
test-conda-cppGithub Actions
test-conda-cpp-valgrindGithub Actions
test-conda-python-3.6Github Actions
test-conda-python-3.6-pandas-0.23Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-dask-latestGithub Actions
test-conda-python-3.7-dask-masterGithub Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-kartothek-latestGithub Actions
test-conda-python-3.7-kartothek-masterGithub Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-branch-3.0Github Actions
test-conda-python-3.7-turbodbc-latestGithub Actions
test-conda-python-3.7-turbodbc-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-masterGithub Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-masterGithub Actions
test-debian-10-cppGithub Actions
test-debian-10-go-1.15Azure
test-debian-10-python-3Azure
test-debian-c-glibGithub Actions
test-debian-rubyGithub Actions
test-fedora-33-cppGithub Actions
test-fedora-33-python-3Azure
test-r-devdocsGithub Actions
test-r-gcc-11Github Actions
test-r-install-localGithub Actions
test-r-linux-as-cranGithub Actions
test-r-linux-rchkGithub Actions
test-r-linux-valgrindAzure
test-r-minimal-buildAzure
test-r-rhub-ubuntu-gcc-release-latestAzure
test-r-rocker-r-base-latestAzure
test-r-rstudio-r-base-3.6-bionicAzure
test-r-rstudio-r-base-3.6-centos7-devtoolset-8Azure
test-r-rstudio-r-base-3.6-centos8Azure
test-r-rstudio-r-base-3.6-opensuse15Azure
test-r-rstudio-r-base-3.6-opensuse42Azure
test-r-rtools-35Github Actions
test-r-ubuntu-21.04Github Actions
test-r-version-compatibilityGithub Actions
test-r-versionsGithub Actions
test-r-without-arrowAzure
test-ubuntu-18.04-cppGithub Actions
test-ubuntu-18.04-cpp-releaseGithub Actions
test-ubuntu-18.04-cpp-staticGithub Actions
test-ubuntu-18.04-python-3Azure
test-ubuntu-18.04-r-sanitizerAzure
test-ubuntu-20.04-cppGithub Actions
test-ubuntu-20.04-cpp-14Github Actions
test-ubuntu-20.04-cpp-17Github Actions
test-ubuntu-20.04-cpp-thread-sanitizerGithub Actions
test-ubuntu-20.10-docsAzure
test-ubuntu-c-glibGithub Actions
test-ubuntu-default-docsAzure
test-ubuntu-rubyGithub Actions
ubuntu-bionic-amd64Github Actions
ubuntu-bionic-arm64TravisCI
ubuntu-focal-amd64Github Actions
ubuntu-focal-arm64TravisCI
ubuntu-groovy-amd64Github Actions
ubuntu-groovy-arm64TravisCI
ubuntu-hirsute-amd64Github Actions
ubuntu-hirsute-arm64TravisCI
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp36-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions

@kszucs

kszucs commented Jul 19, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-macos-*

@github-actions

github-actionsBot commented Jul 19, 2021

Copy link
Copy Markdown

Revision: bb3b170

Submitted crossbow builds: ursacomputing/crossbow @ actions-614

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

This was referenced Jul 19, 2021
Closed
Closed
kszucs
kszucs commented Jul 19, 2021
View reviewed changes

@kszucskszucs left a comment

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, all the relevant builds are passing

@mdagost

mdagost commented Aug 7, 2021

Copy link
Copy Markdown

It's great to see wheel builds for Apple Silicon. Any chance you may back port them to older versions? The snowflake python connector still relies on versions of pyarrow>=3.0.0,<3.1.0, and I've gone through quite a few backflips but haven't been able to get pyarrow installed from source.

@kszucs

kszucs commented Aug 7, 2021

Copy link
Copy Markdown
MemberAuthor

I'm not sure since:

  • we haven't received many requests to build M1 wheels for older arrow releases yet
  • it would probably require more effort than it looks like at first glance

Either way, could you please create a JIRA ticket so we can track this?

@mdagost

mdagost commented Aug 7, 2021

Copy link
Copy Markdown

Cool. I crated a JIRA: https://issues.apache.org/jira/browse/ARROW-13583

Snowflake is pretty popular, so I'm surprised nobody has asked yet. They're saying they can't/won't upgrade their pyarrow until the September timeframe.

@asfimportasfimport mentioned this pull request Aug 7, 2021
Closed
@jorisvandenbosschejorisvandenbossche mentioned this pull request Aug 9, 2024
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm left review comments
@koukoukou approved these changes

Assignees

No one assigned

Labels

Component: C++Component: Python

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@kszucs@kou@xhochy@mdagost@lidavidm
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

ARROW-10658: [Python][Packaging] Wheel builds for Apple Silicon - #10659

Closed
kszucs wants to merge 72 commits into
apache:masterfrom
kszucs:m1
Closed

ARROW-10658: [Python][Packaging] Wheel builds for Apple Silicon#10659
kszucs wants to merge 72 commits into
apache:masterfrom
kszucs:m1

Conversation

@kszucs

@kszucskszucs commented Jul 6, 2021•
edited
Loading

Copy link
Copy Markdown
Member

The following compiles the 3.9 wheel on apple silicon though I'm getting a segfault on import despite that all binaries have the right arch and linkage/bundling seems correct. I wasn't able to produce a coredump yet.

@github-actions

github-actionsBot commented Jul 6, 2021

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-12122

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/scripts/python_wheel_macos_build.sh Outdated

@kszucskszucsJul 6, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need to use the CMAKE_SYSTEM_* variables.

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/scripts/python_wheel_macos_build.sh Outdated

@kszucskszucsJul 6, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gRPC is not supported on apple silicon at the moment

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/vcpkg/arm64-osx-static-release.cmake~ Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadcpp/cmake_modules/FindSnappyAlt.cmake Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadwheel.sh Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/scripts/python_wheel_macos_build.sh Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadwheel.sh Outdated
@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip. M1 mac [WIP]ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]Jul 6, 2021
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadcpp/thirdparty/versions.txt Outdated

@kszucskszucsJul 6, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to check that it won't cause any performance regressions.

@kszucskszucsJul 19, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lidavidm how can I verify that #10024 doesn't apply here?

@lidavidmlidavidmJul 19, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I can see that should only apply to 2.0.0 and 1.7.x should be safe (the commenter in microsoft/mimalloc#363 claims 1.7.0 doesn't affect them at least)

@lidavidmlidavidmJul 19, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And the problematic test is still there - so assuming tests pass we should be OK

@kszucskszucsJul 19, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadwheel.sh Outdated
@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]ARROW-12122: [Python] Cannot install via pip M1 mac Jul 12, 2021
@kszucs

kszucs commented Jul 12, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-macos-*

@github-actions

github-actionsBot commented Jul 12, 2021

Copy link
Copy Markdown

Revision: 0f094730b1e9f3e49077a8d746668251e2bb8ddf

Submitted crossbow builds: ursacomputing/crossbow @ actions-585

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

@kszucs

kszucs commented Jul 12, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-macos-*

@github-actions

github-actionsBot commented Jul 12, 2021

Copy link
Copy Markdown

Revision: e1ab42b861b169217a02102c969c939f47695808

Submitted crossbow builds: ursacomputing/crossbow @ actions-586

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

kszucs
kszucs commented Jul 12, 2021
View reviewed changes
Comment threadci/scripts/install_python.sh Outdated

@kszucskszucsJul 12, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like there is no universal2 installers for python < 3.9

@kszucs

kszucs commented Jul 12, 2021

Copy link
Copy Markdown
MemberAuthor

@kou could you please give it a preliminary review?

kou
kou approved these changes Jul 13, 2021
View reviewed changes

@koukou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure!

Comment threadci/scripts/install_python.sh Outdated
Comment threadci/scripts/install_python.sh Outdated
Comment threadci/scripts/python_wheel_macos_build.sh Outdated
Comment threadcpp/cmake_modules/FindSnappyAlt.cmake Outdated
Comment threaddev/tasks/python-wheels/github.osx.arm64.yml Outdated

@koukouJul 13, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this?
I think that our CMake configuration detects ccache automatically.

@kszucskszucsJul 19, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need this to use ccache when compiling the vcpkg binaries.

Comment threadpython/requirements-wheel-build.txt Outdated
Comment threadpython/requirements-wheel-build.txt Outdated
Comment threadpython/requirements-wheel-build.txt Outdated
Comment threaddev/tasks/python-wheels/github.osx.amd64.yml Outdated

@koukouJul 13, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you keep this list in alphabetical order?

@kszucs

kszucs commented Jul 13, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g wheel

@github-actions

github-actionsBot commented Jul 13, 2021

Copy link
Copy Markdown

Revision: af0f71743fce3a9b0e4ce648b1c861a4cbc3d079

Submitted crossbow builds: ursacomputing/crossbow @ actions-587

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp36Github Actions
wheel-windows-cp37Github Actions
wheel-windows-cp38Github Actions
wheel-windows-cp39Github Actions

@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip M1 mac ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]Jul 13, 2021
@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]ARROW-12122: [Python] Cannot install via pip M1 mac Jul 13, 2021
@kszucs

kszucs commented Jul 13, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g wheel

@github-actions

github-actionsBot commented Jul 13, 2021

Copy link
Copy Markdown

Revision: 736739d28dcacc2256671660be0a6bbe8ed0008d

Submitted crossbow builds: ursacomputing/crossbow @ actions-588

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp36Github Actions
wheel-windows-cp37Github Actions
wheel-windows-cp38Github Actions
wheel-windows-cp39Github Actions

@kszucs

kszucs commented Jul 19, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g nightly

@github-actions

github-actionsBot commented Jul 19, 2021

Copy link
Copy Markdown

Revision: d29f104

Submitted crossbow builds: ursacomputing/crossbow @ actions-612

TaskStatus
centos-7-amd64Github Actions
centos-8-amd64Github Actions
centos-8-arm64TravisCI
conda-cleanAzure
conda-linux-gcc-py36-arm64Azure
conda-linux-gcc-py36-cpu-r40Azure
conda-linux-gcc-py36-cudaAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py36-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2017-py36-r40Azure
conda-win-vs2017-py37-r41Azure
conda-win-vs2017-py38Azure
conda-win-vs2017-py39Azure
debian-bullseye-amd64Github Actions
debian-bullseye-arm64TravisCI
debian-buster-amd64Github Actions
debian-buster-arm64TravisCI
example-cpp-minimal-build-staticGithub Actions
example-cpp-minimal-build-static-system-dependencyGithub Actions
homebrew-cppGithub Actions
homebrew-r-autobrewGithub Actions
java-jarsGithub Actions
nugetGithub Actions
python-sdistGithub Actions
test-build-vcpkg-winGithub Actions
test-conda-cppGithub Actions
test-conda-cpp-valgrindGithub Actions
test-conda-python-3.6Github Actions
test-conda-python-3.6-pandas-0.23Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-dask-latestGithub Actions
test-conda-python-3.7-dask-masterGithub Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-kartothek-latestGithub Actions
test-conda-python-3.7-kartothek-masterGithub Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-branch-3.0Github Actions
test-conda-python-3.7-turbodbc-latestGithub Actions
test-conda-python-3.7-turbodbc-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-masterGithub Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-masterGithub Actions
test-debian-10-cppGithub Actions
test-debian-10-go-1.15Azure
test-debian-10-python-3Azure
test-debian-c-glibGithub Actions
test-debian-rubyGithub Actions
test-fedora-33-cppGithub Actions
test-fedora-33-python-3Azure
test-r-devdocsGithub Actions
test-r-gcc-11Github Actions
test-r-install-localGithub Actions
test-r-linux-as-cranGithub Actions
test-r-linux-rchkGithub Actions
test-r-linux-valgrindAzure
test-r-minimal-buildAzure
test-r-rhub-ubuntu-gcc-release-latestAzure
test-r-rocker-r-base-latestAzure
test-r-rstudio-r-base-3.6-bionicAzure
test-r-rstudio-r-base-3.6-centos7-devtoolset-8Azure
test-r-rstudio-r-base-3.6-centos8Azure
test-r-rstudio-r-base-3.6-opensuse15Azure
test-r-rstudio-r-base-3.6-opensuse42Azure
test-r-rtools-35Github Actions
test-r-ubuntu-21.04Github Actions
test-r-version-compatibilityGithub Actions
test-r-versionsGithub Actions
test-r-without-arrowAzure
test-ubuntu-18.04-cppGithub Actions
test-ubuntu-18.04-cpp-releaseGithub Actions
test-ubuntu-18.04-cpp-staticGithub Actions
test-ubuntu-18.04-python-3Azure
test-ubuntu-18.04-r-sanitizerAzure
test-ubuntu-20.04-cppGithub Actions
test-ubuntu-20.04-cpp-14Github Actions
test-ubuntu-20.04-cpp-17Github Actions
test-ubuntu-20.04-cpp-thread-sanitizerGithub Actions
test-ubuntu-20.10-docsAzure
test-ubuntu-c-glibGithub Actions
test-ubuntu-default-docsAzure
test-ubuntu-rubyGithub Actions
ubuntu-bionic-amd64Github Actions
ubuntu-bionic-arm64TravisCI
ubuntu-focal-amd64Github Actions
ubuntu-focal-arm64TravisCI
ubuntu-groovy-amd64Github Actions
ubuntu-groovy-arm64TravisCI
ubuntu-hirsute-amd64Github Actions
ubuntu-hirsute-arm64TravisCI
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp36-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions

@kszucs

kszucs commented Jul 19, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-macos-*

@github-actions

github-actionsBot commented Jul 19, 2021

Copy link
Copy Markdown

Revision: bb3b170

Submitted crossbow builds: ursacomputing/crossbow @ actions-614

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

This was referenced Jul 19, 2021
Closed
Closed
kszucs
kszucs commented Jul 19, 2021
View reviewed changes

@kszucskszucs left a comment

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, all the relevant builds are passing

@mdagost

mdagost commented Aug 7, 2021

Copy link
Copy Markdown

It's great to see wheel builds for Apple Silicon. Any chance you may back port them to older versions? The snowflake python connector still relies on versions of pyarrow>=3.0.0,<3.1.0, and I've gone through quite a few backflips but haven't been able to get pyarrow installed from source.

@kszucs

kszucs commented Aug 7, 2021

Copy link
Copy Markdown
MemberAuthor

I'm not sure since:

  • we haven't received many requests to build M1 wheels for older arrow releases yet
  • it would probably require more effort than it looks like at first glance

Either way, could you please create a JIRA ticket so we can track this?

@mdagost

mdagost commented Aug 7, 2021

Copy link
Copy Markdown

Cool. I crated a JIRA: https://issues.apache.org/jira/browse/ARROW-13583

Snowflake is pretty popular, so I'm surprised nobody has asked yet. They're saying they can't/won't upgrade their pyarrow until the September timeframe.

@asfimportasfimport mentioned this pull request Aug 7, 2021
Closed
@jorisvandenbosschejorisvandenbossche mentioned this pull request Aug 9, 2024
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm left review comments
@koukoukou approved these changes

Assignees

No one assigned

Labels

Component: C++Component: Python

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@kszucs@kou@xhochy@mdagost@lidavidm
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

ARROW-10658: [Python][Packaging] Wheel builds for Apple Silicon - #10659

Closed
kszucs wants to merge 72 commits into
apache:masterfrom
kszucs:m1
Closed

ARROW-10658: [Python][Packaging] Wheel builds for Apple Silicon#10659
kszucs wants to merge 72 commits into
apache:masterfrom
kszucs:m1

Conversation

@kszucs

@kszucskszucs commented Jul 6, 2021•
edited
Loading

Copy link
Copy Markdown
Member

The following compiles the 3.9 wheel on apple silicon though I'm getting a segfault on import despite that all binaries have the right arch and linkage/bundling seems correct. I wasn't able to produce a coredump yet.

@github-actions

github-actionsBot commented Jul 6, 2021

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-12122

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/scripts/python_wheel_macos_build.sh Outdated

@kszucskszucsJul 6, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need to use the CMAKE_SYSTEM_* variables.

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/scripts/python_wheel_macos_build.sh Outdated

@kszucskszucsJul 6, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gRPC is not supported on apple silicon at the moment

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/vcpkg/arm64-osx-static-release.cmake~ Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadcpp/cmake_modules/FindSnappyAlt.cmake Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadwheel.sh Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadci/scripts/python_wheel_macos_build.sh Outdated
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadwheel.sh Outdated
@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip. M1 mac [WIP]ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]Jul 6, 2021
kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadcpp/thirdparty/versions.txt Outdated

@kszucskszucsJul 6, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to check that it won't cause any performance regressions.

@kszucskszucsJul 19, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lidavidm how can I verify that #10024 doesn't apply here?

@lidavidmlidavidmJul 19, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I can see that should only apply to 2.0.0 and 1.7.x should be safe (the commenter in microsoft/mimalloc#363 claims 1.7.0 doesn't affect them at least)

@lidavidmlidavidmJul 19, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And the problematic test is still there - so assuming tests pass we should be OK

@kszucskszucsJul 19, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

kszucs
kszucs commented Jul 6, 2021
View reviewed changes
Comment threadwheel.sh Outdated
@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]ARROW-12122: [Python] Cannot install via pip M1 mac Jul 12, 2021
@kszucs

kszucs commented Jul 12, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-macos-*

@github-actions

github-actionsBot commented Jul 12, 2021

Copy link
Copy Markdown

Revision: 0f094730b1e9f3e49077a8d746668251e2bb8ddf

Submitted crossbow builds: ursacomputing/crossbow @ actions-585

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

@kszucs

kszucs commented Jul 12, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-macos-*

@github-actions

github-actionsBot commented Jul 12, 2021

Copy link
Copy Markdown

Revision: e1ab42b861b169217a02102c969c939f47695808

Submitted crossbow builds: ursacomputing/crossbow @ actions-586

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

kszucs
kszucs commented Jul 12, 2021
View reviewed changes
Comment threadci/scripts/install_python.sh Outdated

@kszucskszucsJul 12, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like there is no universal2 installers for python < 3.9

@kszucs

kszucs commented Jul 12, 2021

Copy link
Copy Markdown
MemberAuthor

@kou could you please give it a preliminary review?

kou
kou approved these changes Jul 13, 2021
View reviewed changes

@koukou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure!

Comment threadci/scripts/install_python.sh Outdated
Comment threadci/scripts/install_python.sh Outdated
Comment threadci/scripts/python_wheel_macos_build.sh Outdated
Comment threadcpp/cmake_modules/FindSnappyAlt.cmake Outdated
Comment threaddev/tasks/python-wheels/github.osx.arm64.yml Outdated

@koukouJul 13, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this?
I think that our CMake configuration detects ccache automatically.

@kszucskszucsJul 19, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need this to use ccache when compiling the vcpkg binaries.

Comment threadpython/requirements-wheel-build.txt Outdated
Comment threadpython/requirements-wheel-build.txt Outdated
Comment threadpython/requirements-wheel-build.txt Outdated
Comment threaddev/tasks/python-wheels/github.osx.amd64.yml Outdated

@koukouJul 13, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you keep this list in alphabetical order?

@kszucs

kszucs commented Jul 13, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g wheel

@github-actions

github-actionsBot commented Jul 13, 2021

Copy link
Copy Markdown

Revision: af0f71743fce3a9b0e4ce648b1c861a4cbc3d079

Submitted crossbow builds: ursacomputing/crossbow @ actions-587

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp36Github Actions
wheel-windows-cp37Github Actions
wheel-windows-cp38Github Actions
wheel-windows-cp39Github Actions

@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip M1 mac ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]Jul 13, 2021
@kszucskszucs changed the title ARROW-12122: [Python] Cannot install via pip M1 mac [WIP]ARROW-12122: [Python] Cannot install via pip M1 mac Jul 13, 2021
@kszucs

kszucs commented Jul 13, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g wheel

@github-actions

github-actionsBot commented Jul 13, 2021

Copy link
Copy Markdown

Revision: 736739d28dcacc2256671660be0a6bbe8ed0008d

Submitted crossbow builds: ursacomputing/crossbow @ actions-588

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp36Github Actions
wheel-windows-cp37Github Actions
wheel-windows-cp38Github Actions
wheel-windows-cp39Github Actions

@kszucs

kszucs commented Jul 19, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g nightly

@github-actions

github-actionsBot commented Jul 19, 2021

Copy link
Copy Markdown

Revision: d29f104

Submitted crossbow builds: ursacomputing/crossbow @ actions-612

TaskStatus
centos-7-amd64Github Actions
centos-8-amd64Github Actions
centos-8-arm64TravisCI
conda-cleanAzure
conda-linux-gcc-py36-arm64Azure
conda-linux-gcc-py36-cpu-r40Azure
conda-linux-gcc-py36-cudaAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py36-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2017-py36-r40Azure
conda-win-vs2017-py37-r41Azure
conda-win-vs2017-py38Azure
conda-win-vs2017-py39Azure
debian-bullseye-amd64Github Actions
debian-bullseye-arm64TravisCI
debian-buster-amd64Github Actions
debian-buster-arm64TravisCI
example-cpp-minimal-build-staticGithub Actions
example-cpp-minimal-build-static-system-dependencyGithub Actions
homebrew-cppGithub Actions
homebrew-r-autobrewGithub Actions
java-jarsGithub Actions
nugetGithub Actions
python-sdistGithub Actions
test-build-vcpkg-winGithub Actions
test-conda-cppGithub Actions
test-conda-cpp-valgrindGithub Actions
test-conda-python-3.6Github Actions
test-conda-python-3.6-pandas-0.23Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-dask-latestGithub Actions
test-conda-python-3.7-dask-masterGithub Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-kartothek-latestGithub Actions
test-conda-python-3.7-kartothek-masterGithub Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-branch-3.0Github Actions
test-conda-python-3.7-turbodbc-latestGithub Actions
test-conda-python-3.7-turbodbc-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-masterGithub Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-pandas-masterGithub Actions
test-debian-10-cppGithub Actions
test-debian-10-go-1.15Azure
test-debian-10-python-3Azure
test-debian-c-glibGithub Actions
test-debian-rubyGithub Actions
test-fedora-33-cppGithub Actions
test-fedora-33-python-3Azure
test-r-devdocsGithub Actions
test-r-gcc-11Github Actions
test-r-install-localGithub Actions
test-r-linux-as-cranGithub Actions
test-r-linux-rchkGithub Actions
test-r-linux-valgrindAzure
test-r-minimal-buildAzure
test-r-rhub-ubuntu-gcc-release-latestAzure
test-r-rocker-r-base-latestAzure
test-r-rstudio-r-base-3.6-bionicAzure
test-r-rstudio-r-base-3.6-centos7-devtoolset-8Azure
test-r-rstudio-r-base-3.6-centos8Azure
test-r-rstudio-r-base-3.6-opensuse15Azure
test-r-rstudio-r-base-3.6-opensuse42Azure
test-r-rtools-35Github Actions
test-r-ubuntu-21.04Github Actions
test-r-version-compatibilityGithub Actions
test-r-versionsGithub Actions
test-r-without-arrowAzure
test-ubuntu-18.04-cppGithub Actions
test-ubuntu-18.04-cpp-releaseGithub Actions
test-ubuntu-18.04-cpp-staticGithub Actions
test-ubuntu-18.04-python-3Azure
test-ubuntu-18.04-r-sanitizerAzure
test-ubuntu-20.04-cppGithub Actions
test-ubuntu-20.04-cpp-14Github Actions
test-ubuntu-20.04-cpp-17Github Actions
test-ubuntu-20.04-cpp-thread-sanitizerGithub Actions
test-ubuntu-20.10-docsAzure
test-ubuntu-c-glibGithub Actions
test-ubuntu-default-docsAzure
test-ubuntu-rubyGithub Actions
ubuntu-bionic-amd64Github Actions
ubuntu-bionic-arm64TravisCI
ubuntu-focal-amd64Github Actions
ubuntu-focal-arm64TravisCI
ubuntu-groovy-amd64Github Actions
ubuntu-groovy-arm64TravisCI
ubuntu-hirsute-amd64Github Actions
ubuntu-hirsute-arm64TravisCI
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp36-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions

@kszucs

kszucs commented Jul 19, 2021

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-macos-*

@github-actions

github-actionsBot commented Jul 19, 2021

Copy link
Copy Markdown

Revision: bb3b170

Submitted crossbow builds: ursacomputing/crossbow @ actions-614

TaskStatus
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp36-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp36-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

This was referenced Jul 19, 2021
Closed
Closed
kszucs
kszucs commented Jul 19, 2021
View reviewed changes

@kszucskszucs left a comment

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, all the relevant builds are passing

@mdagost

mdagost commented Aug 7, 2021

Copy link
Copy Markdown

It's great to see wheel builds for Apple Silicon. Any chance you may back port them to older versions? The snowflake python connector still relies on versions of pyarrow>=3.0.0,<3.1.0, and I've gone through quite a few backflips but haven't been able to get pyarrow installed from source.

@kszucs

kszucs commented Aug 7, 2021

Copy link
Copy Markdown
MemberAuthor

I'm not sure since:

  • we haven't received many requests to build M1 wheels for older arrow releases yet
  • it would probably require more effort than it looks like at first glance

Either way, could you please create a JIRA ticket so we can track this?

@mdagost

mdagost commented Aug 7, 2021

Copy link
Copy Markdown

Cool. I crated a JIRA: https://issues.apache.org/jira/browse/ARROW-13583

Snowflake is pretty popular, so I'm surprised nobody has asked yet. They're saying they can't/won't upgrade their pyarrow until the September timeframe.

@asfimportasfimport mentioned this pull request Aug 7, 2021
Closed
@jorisvandenbosschejorisvandenbossche mentioned this pull request Aug 9, 2024
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm left review comments
@koukoukou approved these changes

Assignees

No one assigned

Labels

Component: C++Component: Python

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@kszucs@kou@xhochy@mdagost@lidavidm