Skip to content

ARROW-12626: [C++] Support toolchain xsimd, update toolchain version to version 8.1.0 - #13244

Merged
kou merged 43 commits into
apache:masterfrom
wesm:ARROW-12626-toolchain-xsimd
Jun 14, 2022
Merged

ARROW-12626: [C++] Support toolchain xsimd, update toolchain version to version 8.1.0#13244
kou merged 43 commits into
apache:masterfrom
wesm:ARROW-12626-toolchain-xsimd

Conversation

@wesm

@wesmwesm commented May 26, 2022•
edited by kou
Loading

Copy link
Copy Markdown
Member

This also updates pinned vcpkg to use xsimd 8.1.0.

This also implements auto python-wheel-windows-vs2017 image update mechanism.
We have a problem of "docker build" on Windows. "docker build" doesn't reuse pulled
image as caches. "docker build" always rebuilds an image. This implements manual
reuse mechanism like the following:

if ! docker pull; then
docker build # build only when built images don't exist
fi
docker run

But this doesn't work when ci/docker/python-wheel-windows-vs2017.dockerfile is updated
but pinned vcpkg revision isn't changed. In the case, "docker build" isn't run because
"docker pull" is succeeded.

To work this mechanism, this introduces "PYTHON_WHEEL_WINDOWS_IMAGE_REVISION".
We must bump it manually when we update ci/docker/python-wheel-windows-vs2017.dockerfile.
"PYTHON_WHEEL_WINDOWS_IMAGE_REVISION" is used in tag name. So "docker pull"
is failed with new "PYTHON_WHEEL_WINDOWS_IMAGE_REVISION" and then "docker build"
is used.

@github-actions

github-actionsBot commented May 26, 2022

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-12626

@cyb70289
cyb70289force-pushed the ARROW-12626-toolchain-xsimd branch from 6db3a2e to d7d3101CompareMay 27, 2022 07:47
@cyb70289

cyb70289 commented May 30, 2022

Copy link
Copy Markdown
Contributor

@kou, is there convenient way to verify if this pr may break packaging and release in github UI?

@kou

kou commented May 30, 2022

Copy link
Copy Markdown
Member

We can use @github-actions crossbow submit -g nightly-packaging -g nightly-release for it.

@cyb70289

cyb70289 commented May 30, 2022

Copy link
Copy Markdown
Contributor

@github-actions crossbow submit -g nightly-packaging -g nightly-release

@github-actions

This comment was marked as outdated.

Sign in to view
kou
kou reviewed May 30, 2022
View reviewed changes
Comment threadcpp/cmake_modules/ThirdpartyToolchain.cmake Outdated
@cyb70289
cyb70289force-pushed the ARROW-12626-toolchain-xsimd branch 2 times, most recently from 9af5ea8 to 4a006adCompareMay 30, 2022 07:38
@cyb70289

cyb70289 commented May 30, 2022

Copy link
Copy Markdown
Contributor

@github-actions crossbow submit -g nightly-packaging -g nightly-release

@github-actions

github-actionsBot commented May 30, 2022

Copy link
Copy Markdown

Revision: 4a006add27daf95fbad6393ff82528279ee3378f

Submitted crossbow builds: ursacomputing/crossbow @ actions-8103008ca7

TaskStatus
almalinux-8-amd64Github Actions
almalinux-8-arm64TravisCI
amazon-linux-2-amd64Github Actions
centos-7-amd64Github Actions
centos-8-stream-amd64Github Actions
centos-8-stream-arm64TravisCI
conan-maximumGithub Actions
conan-minimumGithub Actions
conda-cleanAzure
conda-linux-gcc-py310-arm64Azure
conda-linux-gcc-py310-cpuAzure
conda-linux-gcc-py310-cudaAzure
conda-linux-gcc-py310-ppc64leAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py37-ppc64leAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py38-ppc64leAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-linux-gcc-py39-ppc64leAzure
conda-osx-arm64-clang-py310Azure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py310Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2017-py310Azure
conda-win-vs2017-py37-r40Azure
conda-win-vs2017-py37-r41Azure
conda-win-vs2017-py38Azure
conda-win-vs2017-py39Azure
debian-bookworm-amd64Github Actions
debian-bookworm-arm64TravisCI
debian-bullseye-amd64Github Actions
debian-bullseye-arm64TravisCI
debian-buster-amd64Github Actions
debian-buster-arm64TravisCI
homebrew-cppGithub Actions
homebrew-r-autobrewGithub Actions
homebrew-r-brewGithub Actions
java-jarsGithub Actions
nugetGithub Actions
python-sdistGithub Actions
r-nightly-packagesGithub Actions
ubuntu-bionic-amd64Github Actions
ubuntu-bionic-arm64TravisCI
ubuntu-focal-amd64Github Actions
ubuntu-focal-arm64TravisCI
ubuntu-impish-amd64Github Actions
ubuntu-impish-arm64TravisCI
ubuntu-jammy-amd64Github Actions
ubuntu-jammy-arm64TravisCI
verify-rc-source-cpp-linux-almalinux-8-amd64Github Actions
verify-rc-source-cpp-linux-conda-latest-amd64Github Actions
verify-rc-source-cpp-linux-ubuntu-18.04-amd64Github Actions
verify-rc-source-cpp-linux-ubuntu-20.04-amd64Github Actions
verify-rc-source-cpp-linux-ubuntu-22.04-amd64Github Actions
verify-rc-source-cpp-macos-amd64Github Actions
verify-rc-source-cpp-macos-arm64Github Actions
verify-rc-source-cpp-macos-conda-amd64Github Actions
verify-rc-source-csharp-linux-almalinux-8-amd64Github Actions
verify-rc-source-csharp-linux-conda-latest-amd64Github Actions
verify-rc-source-csharp-linux-ubuntu-18.04-amd64Github Actions
verify-rc-source-csharp-linux-ubuntu-20.04-amd64Github Actions
verify-rc-source-csharp-linux-ubuntu-22.04-amd64Github Actions
verify-rc-source-csharp-macos-amd64Github Actions
verify-rc-source-csharp-macos-arm64Github Actions
verify-rc-source-go-linux-almalinux-8-amd64Github Actions
verify-rc-source-go-linux-conda-latest-amd64Github Actions
verify-rc-source-go-linux-ubuntu-18.04-amd64Github Actions
verify-rc-source-go-linux-ubuntu-20.04-amd64Github Actions
verify-rc-source-go-linux-ubuntu-22.04-amd64Github Actions
verify-rc-source-go-macos-amd64Github Actions
verify-rc-source-go-macos-arm64Github Actions
verify-rc-source-integration-linux-almalinux-8-amd64Github Actions
verify-rc-source-integration-linux-conda-latest-amd64Github Actions
verify-rc-source-integration-linux-ubuntu-18.04-amd64Github Actions
verify-rc-source-integration-linux-ubuntu-20.04-amd64Github Actions
verify-rc-source-integration-linux-ubuntu-22.04-amd64Github Actions
verify-rc-source-integration-macos-amd64Github Actions
verify-rc-source-integration-macos-arm64Github Actions
verify-rc-source-integration-macos-conda-amd64Github Actions
verify-rc-source-java-linux-almalinux-8-amd64Github Actions
verify-rc-source-java-linux-conda-latest-amd64Github Actions
verify-rc-source-java-linux-ubuntu-18.04-amd64Github Actions
verify-rc-source-java-linux-ubuntu-20.04-amd64Github Actions
verify-rc-source-java-linux-ubuntu-22.04-amd64Github Actions
verify-rc-source-java-macos-amd64Github Actions
verify-rc-source-js-linux-almalinux-8-amd64Github Actions
verify-rc-source-js-linux-conda-latest-amd64Github Actions
verify-rc-source-js-linux-ubuntu-18.04-amd64Github Actions
verify-rc-source-js-linux-ubuntu-20.04-amd64Github Actions
verify-rc-source-js-linux-ubuntu-22.04-amd64Github Actions
verify-rc-source-js-macos-amd64Github Actions
verify-rc-source-js-macos-arm64Github Actions
verify-rc-source-python-linux-almalinux-8-amd64Github Actions
verify-rc-source-python-linux-conda-latest-amd64Github Actions
verify-rc-source-python-linux-ubuntu-18.04-amd64Github Actions
verify-rc-source-python-linux-ubuntu-20.04-amd64Github Actions
verify-rc-source-python-linux-ubuntu-22.04-amd64Github Actions
verify-rc-source-python-macos-amd64Github Actions
verify-rc-source-python-macos-arm64Github Actions
verify-rc-source-python-macos-conda-amd64Github Actions
verify-rc-source-ruby-linux-almalinux-8-amd64Github Actions
verify-rc-source-ruby-linux-conda-latest-amd64Github Actions
verify-rc-source-ruby-linux-ubuntu-18.04-amd64Github Actions
verify-rc-source-ruby-linux-ubuntu-20.04-amd64Github Actions
verify-rc-source-ruby-linux-ubuntu-22.04-amd64Github Actions
verify-rc-source-ruby-macos-amd64Github Actions
verify-rc-source-ruby-macos-arm64Github Actions
verify-rc-source-windowsGithub Actions
wheel-macos-big-sur-cp310-arm64Github Actions
wheel-macos-big-sur-cp310-universal2Github Actions
wheel-macos-big-sur-cp38-arm64Github Actions
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp310-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp310-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp310-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp310-amd64Github Actions
wheel-manylinux2014-cp310-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions

@cyb70289

cyb70289 commented May 31, 2022

Copy link
Copy Markdown
Contributor

There are many xsimd related errors, @kou, will you comment perferred fixes? Thanks.

@wesm

wesm commented May 31, 2022

Copy link
Copy Markdown
MemberAuthor

As I recall, the project was pinned to a bundled version that was a few weeks before 8.1 was released. Feel free to take control of this PR from me

@kou

kou commented May 31, 2022

Copy link
Copy Markdown
Member

I'll push fixes later. Please wait for a while.

@kou
kouforce-pushed the ARROW-12626-toolchain-xsimd branch from 4a006ad to 902e81cCompareJune 1, 2022 20:45
@kou

kou commented Jun 1, 2022

Copy link
Copy Markdown
Member

@github-actions crossbow submit conan-* conda-linux-gcc-*

@github-actions

github-actionsBot commented Jun 1, 2022

Copy link
Copy Markdown

Revision: 902e81c2914a362b168bdcc0a0f1d500cbb2bd0e

Submitted crossbow builds: ursacomputing/crossbow @ actions-76b5b8c64b

TaskStatus
conan-maximumGithub Actions
conan-minimumGithub Actions
conda-linux-gcc-py310-arm64Azure
conda-linux-gcc-py310-cpuAzure
conda-linux-gcc-py310-cudaAzure
conda-linux-gcc-py310-ppc64leAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py37-ppc64leAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py38-ppc64leAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-linux-gcc-py39-ppc64leAzure

@kou

kou commented Jun 1, 2022

Copy link
Copy Markdown
Member

@github-actions crossbow submit conda-osx-* conda-win-*

@github-actions

github-actionsBot commented Jun 1, 2022

Copy link
Copy Markdown

Revision: addd3ac86d496ef35888777fed2b34cec0bd4150

Submitted crossbow builds: ursacomputing/crossbow @ actions-603614ae49

TaskStatus
conda-osx-arm64-clang-py310Azure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py310Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2017-py310Azure
conda-win-vs2017-py37-r40Azure
conda-win-vs2017-py37-r41Azure
conda-win-vs2017-py38Azure
conda-win-vs2017-py39Azure

@kou

kou commented Jun 1, 2022

Copy link
Copy Markdown
Member

I've opened a pull request to update xsimd in vcpkg: microsoft/vcpkg#25030

@kou

kou commented Jun 1, 2022

Copy link
Copy Markdown
Member

@github-actions crossbow submit conan-*

@github-actions

github-actionsBot commented Jun 1, 2022

Copy link
Copy Markdown

Revision: 309e42a0d3ac52a58f07471f1327c0c42426d93d

Submitted crossbow builds: ursacomputing/crossbow @ actions-983084a5d3

TaskStatus
conan-maximumGithub Actions
conan-minimumGithub Actions

@kou

kou commented Jun 1, 2022

Copy link
Copy Markdown
Member

conda-win-* failures aren't related to this. They are failed at least since 2022-05-30: https://lists.apache.org/thread/s6ggw17162cgoyk8wm1yg5g5mqv60j5q
@xhochy Could you confirm the conda-win-* failures?

@kou

kou commented Jun 3, 2022

Copy link
Copy Markdown
Member

@github-actions crossbow submit wheel-*

@github-actions

github-actionsBot commented Jun 3, 2022

Copy link
Copy Markdown

Revision: d100b397e9fbc7bf359688146c1e1a74a2980717

Submitted crossbow builds: ursacomputing/crossbow @ actions-9de9fb80c7

TaskStatus
wheel-macos-big-sur-cp310-arm64Github Actions
wheel-macos-big-sur-cp310-universal2Github Actions
wheel-macos-big-sur-cp38-arm64Github Actions
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp310-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp310-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp310-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp310-amd64Github Actions
wheel-manylinux2014-cp310-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions

@kou
kouforce-pushed the ARROW-12626-toolchain-xsimd branch from da2798f to 160ec95CompareJune 3, 2022 02:01
@kou

kou commented Jun 3, 2022

Copy link
Copy Markdown
Member

@github-actions crossbow submit wheel-*

@github-actions

github-actionsBot commented Jun 3, 2022

Copy link
Copy Markdown

Revision: 160ec959af76ed6ef6073ac00fa5fe1e7d49db18

Submitted crossbow builds: ursacomputing/crossbow @ actions-082fa4d9b1

TaskStatus
wheel-macos-big-sur-cp310-arm64Github Actions
wheel-macos-big-sur-cp310-universal2Github Actions
wheel-macos-big-sur-cp38-arm64Github Actions
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp310-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp310-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp310-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp310-amd64Github Actions
wheel-manylinux2014-cp310-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions

@kou

kou commented Jun 3, 2022

Copy link
Copy Markdown
Member

@github-actions crossbow java-jars

@kou

kou commented Jun 12, 2022

Copy link
Copy Markdown
Member

@github-actions crossbow submit wheel-windows-cp38-amd64

@github-actions

github-actionsBot commented Jun 12, 2022

Copy link
Copy Markdown

Revision: 21b3427

Submitted crossbow builds: ursacomputing/crossbow @ actions-729a5ecccd

TaskStatus
wheel-windows-cp38-amd64Github Actions

@kou

kou commented Jun 12, 2022

Copy link
Copy Markdown
Member

@github-actions crossbow submit wheel-windows-cp38-amd64

@github-actions

github-actionsBot commented Jun 12, 2022

Copy link
Copy Markdown

Revision: 083fdba

Submitted crossbow builds: ursacomputing/crossbow @ actions-1fd3333176

TaskStatus
wheel-windows-cp38-amd64Github Actions

@kou

kou commented Jun 12, 2022

Copy link
Copy Markdown
Member

@github-actions crossbow submit wheel-windows-cp38-amd64

@github-actions

github-actionsBot commented Jun 12, 2022

Copy link
Copy Markdown

Revision: 2c2cf17

Submitted crossbow builds: ursacomputing/crossbow @ actions-f57e2a50a4

TaskStatus
wheel-windows-cp38-amd64Github Actions

@kou

kou commented Jun 12, 2022

Copy link
Copy Markdown
Member

@github-actions crossbow submit wheel-windows-cp38-amd64

@github-actions

github-actionsBot commented Jun 12, 2022

Copy link
Copy Markdown

Revision: 080fc8f

Submitted crossbow builds: ursacomputing/crossbow @ actions-c871540910

TaskStatus
wheel-windows-cp38-amd64Github Actions

@kou

kou commented Jun 12, 2022

Copy link
Copy Markdown
Member

@github-actions crossbow submit wheel-windows-cp38-amd64

@github-actions

github-actionsBot commented Jun 12, 2022

Copy link
Copy Markdown

Revision: be9daf2

Submitted crossbow builds: ursacomputing/crossbow @ actions-d8403e63aa

TaskStatus
wheel-windows-cp38-amd64Github Actions

@kou

kou commented Jun 12, 2022

Copy link
Copy Markdown
Member

@github-actions crossbow submit wheel-windows-cp38-amd64

@github-actions

github-actionsBot commented Jun 12, 2022

Copy link
Copy Markdown

Revision: e5bf8ca

Submitted crossbow builds: ursacomputing/crossbow @ actions-c62ae2d49f

TaskStatus
wheel-windows-cp38-amd64Github Actions

@kou

kou commented Jun 13, 2022

Copy link
Copy Markdown
Member

@github-actions crossbow submit wheel-windows-cp38-amd64

@github-actions

github-actionsBot commented Jun 13, 2022

Copy link
Copy Markdown

Revision: 4edd5b0

Submitted crossbow builds: ursacomputing/crossbow @ actions-3056f2001d

TaskStatus
wheel-windows-cp38-amd64Github Actions

@kou

kou commented Jun 13, 2022

Copy link
Copy Markdown
Member

@github-actions crossbow submit wheel-windows-*

@github-actions

github-actionsBot commented Jun 13, 2022

Copy link
Copy Markdown

Revision: 4edd5b0

Submitted crossbow builds: ursacomputing/crossbow @ actions-aa101b6926

TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions

@assignUser

assignUser commented Jun 13, 2022

Copy link
Copy Markdown
Member

2h 51min vs 1h 🥳 🚀

@kou

kou commented Jun 13, 2022

Copy link
Copy Markdown
Member

docker build didn't use pulled images as cache as @kszucs said.
I added the following workflow as a workaround (see the description of this pull request for details, I've updated):

if! docker pull;then
docker build
fi
docker run

This is ready to merge now. I'll merge this tomorrow.

@kou
kou merged commit d63c16b into apache:masterJun 14, 2022
@koukou mentioned this pull request Jun 28, 2022
Closed
This was referenced Jul 26, 2022
Closed
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@koukoukou left review comments
@cyb70289cyb70289cyb70289 approved these changes

Assignees

No one assigned

Labels

Component: C++

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@wesm@cyb70289@kou@kszucs@assignUser