ARROW-15417: [Python][Packaging] Use vcpkg manifest to install wheel dependencies; downgrade AWS SDK by building the bundled version by kszucs · Pull Request #12227 · apache/arrow

kszucs · 2022-01-21T23:13:06Z

No description provided.

kszucs · 2022-01-21T23:13:36Z

@github-actions crossbow submit wheel-windows-cp37-amd64

github-actions · 2022-01-21T23:14:38Z

Revision: f9c227d34ce4890d313f8983e267b9074b10f11c

Submitted crossbow builds: ursacomputing/crossbow @ actions-1424

Task	Status
wheel-windows-cp37-amd64

kszucs · 2022-01-22T09:49:03Z

@github-actions crossbow submit -g wheel

github-actions · 2022-01-22T09:50:18Z

Revision: 48c0c87302d4c78e359824cf9b76cf2a8dad88b2

Submitted crossbow builds: ursacomputing/crossbow @ actions-1427

Task	Status
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp310-universal2
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-big-sur-cp39-universal2
wheel-macos-high-sierra-cp310-amd64
wheel-macos-high-sierra-cp37-amd64
wheel-macos-high-sierra-cp38-amd64
wheel-macos-high-sierra-cp39-amd64
wheel-macos-mavericks-cp310-amd64
wheel-macos-mavericks-cp37-amd64
wheel-macos-mavericks-cp38-amd64
wheel-macos-mavericks-cp39-amd64
wheel-manylinux2010-cp310-amd64
wheel-manylinux2010-cp37-amd64
wheel-manylinux2010-cp38-amd64
wheel-manylinux2010-cp39-amd64
wheel-manylinux2014-cp310-amd64
wheel-manylinux2014-cp310-arm64
wheel-manylinux2014-cp37-amd64
wheel-manylinux2014-cp37-arm64
wheel-manylinux2014-cp38-amd64
wheel-manylinux2014-cp38-arm64
wheel-manylinux2014-cp39-amd64
wheel-manylinux2014-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp37-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

kszucs · 2022-01-22T10:09:59Z

Need to rebuild the windows images.

kszucs · 2022-01-22T19:47:49Z

@github-actions crossbow submit wheel-manylinux* wheel-macos*

github-actions · 2022-01-22T19:48:56Z

Revision: c41df578108c99ad88fbf03782502ec8d60fb572

Submitted crossbow builds: ursacomputing/crossbow @ actions-1428

Task	Status
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp310-universal2
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-big-sur-cp39-universal2
wheel-macos-high-sierra-cp310-amd64
wheel-macos-high-sierra-cp37-amd64
wheel-macos-high-sierra-cp38-amd64
wheel-macos-high-sierra-cp39-amd64
wheel-macos-mavericks-cp310-amd64
wheel-macos-mavericks-cp37-amd64
wheel-macos-mavericks-cp38-amd64
wheel-macos-mavericks-cp39-amd64
wheel-manylinux2010-cp310-amd64
wheel-manylinux2010-cp37-amd64
wheel-manylinux2010-cp38-amd64
wheel-manylinux2010-cp39-amd64
wheel-manylinux2014-cp310-amd64
wheel-manylinux2014-cp310-arm64
wheel-manylinux2014-cp37-amd64
wheel-manylinux2014-cp37-arm64
wheel-manylinux2014-cp38-amd64
wheel-manylinux2014-cp38-arm64
wheel-manylinux2014-cp39-amd64
wheel-manylinux2014-cp39-arm64

kszucs · 2022-01-23T02:59:06Z

@github-actions crossbow submit wheel-manylinux* wheel-macos*

github-actions · 2022-01-23T03:00:24Z

Revision: f1828588a4a9a5fe2649e533e9733edf8de6985c

Submitted crossbow builds: ursacomputing/crossbow @ actions-1429

Task	Status
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp310-universal2
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-big-sur-cp39-universal2
wheel-macos-high-sierra-cp310-amd64
wheel-macos-high-sierra-cp37-amd64
wheel-macos-high-sierra-cp38-amd64
wheel-macos-high-sierra-cp39-amd64
wheel-macos-mavericks-cp310-amd64
wheel-macos-mavericks-cp37-amd64
wheel-macos-mavericks-cp38-amd64
wheel-macos-mavericks-cp39-amd64
wheel-manylinux2010-cp310-amd64
wheel-manylinux2010-cp37-amd64
wheel-manylinux2010-cp38-amd64
wheel-manylinux2010-cp39-amd64
wheel-manylinux2014-cp310-amd64
wheel-manylinux2014-cp310-arm64
wheel-manylinux2014-cp37-amd64
wheel-manylinux2014-cp37-arm64
wheel-manylinux2014-cp38-amd64
wheel-manylinux2014-cp38-arm64
wheel-manylinux2014-cp39-amd64
wheel-manylinux2014-cp39-arm64

kszucs · 2022-01-23T03:11:22Z

@github-actions crossbow submit wheel-macos-big-sur-*

github-actions · 2022-01-23T03:12:34Z

Revision: a7b868de86487b566939694d3021893287d7210a

Submitted crossbow builds: ursacomputing/crossbow @ actions-1430

Task	Status
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp310-universal2
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-big-sur-cp39-universal2

kszucs · 2022-01-23T10:35:24Z

@github-actions crossbow submit wheel-manylinux2014-cp38-amd64

github-actions · 2022-01-23T10:36:22Z

Revision: 11d069496e41bc883c857d535bbc5c3417a54f71

Submitted crossbow builds: ursacomputing/crossbow @ actions-1431

Task	Status
wheel-manylinux2014-cp38-amd64

github-actions · 2022-01-23T13:39:00Z

https://issues.apache.org/jira/browse/ARROW-15417

kszucs · 2022-01-24T00:51:43Z

@github-actions crossbow submit java-jars

github-actions · 2022-01-24T00:52:50Z

Revision: 678c672

Submitted crossbow builds: ursacomputing/crossbow @ actions-1461

Task	Status
java-jars

kszucs · 2022-01-24T01:27:23Z

@github-actions crossbow submit java-jars

github-actions · 2022-01-24T01:28:38Z

Revision: 8d81adc

Submitted crossbow builds: ursacomputing/crossbow @ actions-1462

Task	Status
java-jars

kou

https://github.com/ursacomputing/crossbow/runs/4915602118?check_suite_focus=true#step:6:1075

Link errors are happen. It seems missing symbols are provided by Windows' system DLL. We may need to add more link flags in build_awssdk() in cpp/cmake_modules/ThirdpartyToolchain.cmake.

kszucs · 2022-01-24T02:35:06Z

If you don't mind I'll leave the duplicated boost-crc line there because it would invalidate the docker cache for both the manylinux and windows images taking hours to rebuild :)

kszucs · 2022-01-24T02:35:29Z

@github-actions crossbow submit wheel-windows-*

github-actions · 2022-01-24T02:36:45Z

Revision: a9f4c1d

Submitted crossbow builds: ursacomputing/crossbow @ actions-1463

Task	Status
wheel-windows-cp310-amd64
wheel-windows-cp37-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

kou · 2022-01-24T02:41:12Z

If you don't mind I'll leave the duplicated boost-crc line there because it would invalidate the docker cache for both the manylinux and windows images taking hours to rebuild :)

I see. We can clean up this after we release 7.0.0.

kszucs · 2022-01-24T03:51:28Z

@github-actions crossbow submit java-jars

github-actions · 2022-01-24T03:52:40Z

Revision: f272079

Submitted crossbow builds: ursacomputing/crossbow @ actions-1464

Task	Status
java-jars

kszucs · 2022-01-24T10:06:44Z

@github-actions crossbow submit java-jars

github-actions · 2022-01-24T10:07:49Z

Revision: 92b41d5

Submitted crossbow builds: ursacomputing/crossbow @ actions-1469

Task	Status
java-jars

kszucs

+1

ursabot · 2022-01-24T11:01:33Z

Benchmark runs are scheduled for baseline = 8e34b64 and contender = a8c0b1f. a8c0b1f is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.91% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.17% ⬆️0.09%] ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

### Rationale for this change In #12227 we decided to use a bundled version of the AWS SDK when compiling Python wheels, in order to downgrade the AWS SDK version. Now that we have fixed S3 finalization issues (#36442), it should be ok to rely on the vcpkg-installed version of the AWS SDK again. ### What changes are included in this PR? Remove use of bundled AWS SDK and use S3 vcpkg feature for requirements. ### Are these changes tested? On CI and via crossbow ### Are there any user-facing changes? No * Closes: #36752 Authored-by: Raúl Cumplido <raulcumplido@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

…apache#36925) ### Rationale for this change In apache#12227 we decided to use a bundled version of the AWS SDK when compiling Python wheels, in order to downgrade the AWS SDK version. Now that we have fixed S3 finalization issues (apache#36442), it should be ok to rely on the vcpkg-installed version of the AWS SDK again. ### What changes are included in this PR? Remove use of bundled AWS SDK and use S3 vcpkg feature for requirements. ### Are these changes tested? On CI and via crossbow ### Are there any user-facing changes? No * Closes: apache#36752 Authored-by: Raúl Cumplido <raulcumplido@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

github-actionsBot added the Component: C++ label Jan 22, 2022

kszucs changed the title ~~[Python][Packaging] Fail on any import errors in the windows wheel tests~~[Python][Packaging] Use vcpkg manifest and update vcpkg versionJan 22, 2022

kszucs changed the title ~~[Python][Packaging] Use vcpkg manifest and update vcpkg version~~[Python][Packaging] Use vcpkg manifest to install dependencies; update vcpkg version; use bundled AWS SDKJan 23, 2022

kszucs changed the title ~~[Python][Packaging] Use vcpkg manifest to install dependencies; update vcpkg version; use bundled AWS SDK~~ARROW-15417: [Python][Packaging] Use vcpkg manifest to install dependencies; update vcpkg version; use bundled AWS SDKJan 23, 2022

apache deleted a comment from github-actionsBotJan 23, 2022

kszucs added 11 commits January 23, 2022 14:39

[Python][Packaging] Fail on any import errors in the windows wheel tests
40a78dd

Swtich to manifest file
8834fb5

Update windows image
858dd1c

Use vcpkg manifest and bump vcpkg's version
262db71

Uncomment dockerfile
907136c

Remove 'remove orc' from jni script
5daa05a

Install in vcpkg manifest mode but compile arrow in classic vcpkg mode
9217bae

Ignore whitespace in vcpkg patch
b47538a

Escape path in windows docker img
d3da8b2

Switch to use vcpkg manifest features
e07e07b

Parquet to default features
a7ebec0

kszucs added 3 commits January 24, 2022 01:17

Remove vcpkg.json from dockerignore
f4fba4d

Remove unconditional image push
3b5eb40

Update the JNI dockerfile to use the vcpkg manifest
678c672

Need to list additional features for java-jars
8d81adc

kou reviewed Jan 24, 2022
View reviewed changes

Comment threadci/vcpkg/vcpkg.json Outdated

Missing windows libraries for bundled AWS SDK
a9f4c1d

kszucs changed the title ~~ARROW-15417: [Python][Packaging] Use vcpkg manifest to install dependencies; use bundled AWS SDK~~ARROW-15417: [Python][Packaging] Use vcpkg manifest to install wheel dependencies; use bundled AWS SDKJan 24, 2022

kszucs changed the title ~~ARROW-15417: [Python][Packaging] Use vcpkg manifest to install wheel dependencies; use bundled AWS SDK~~ARROW-15417: [Python][Packaging] Use vcpkg manifest to install wheel dependencies; downgrade AWS SDK by building the bundled versionJan 24, 2022

Missing boost multiprecision for gandiva
f272079

Sort
92b41d5

Use ubuntu-latest in java-jars template
f5d9139

kszucs commented Jan 24, 2022
View reviewed changes

Comment threaddev/tasks/python-wheels/github.linux.amd64.yml Outdated
Comment threaddev/tasks/python-wheels/travis.linux.arm64.yml Outdated

kszucs closed this in a8c0b1fJan 24, 2022

pitrou mentioned this pull request Jul 18, 2023
[Python] Remove AWS SDK bundling when building wheels #36752
Closed

raulcd mentioned this pull request Jul 28, 2023
GH-36752: [Python] Remove AWS SDK bundling when building wheels #36925
Merged

Uh oh!

Conversation

kszucs commented Jan 21, 2022

Uh oh!

kszucs commented Jan 21, 2022

Uh oh!

github-actionsBot commented Jan 21, 2022

Uh oh!

kszucs commented Jan 22, 2022

Uh oh!

github-actionsBot commented Jan 22, 2022

Uh oh!

kszucs commented Jan 22, 2022

Uh oh!

kszucs commented Jan 22, 2022

Uh oh!

github-actionsBot commented Jan 22, 2022

Uh oh!

kszucs commented Jan 23, 2022

Uh oh!

github-actionsBot commented Jan 23, 2022

Uh oh!

kszucs commented Jan 23, 2022

Uh oh!

github-actionsBot commented Jan 23, 2022

Uh oh!

kszucs commented Jan 23, 2022

Uh oh!

github-actionsBot commented Jan 23, 2022

Uh oh!

github-actionsBot commented Jan 23, 2022

Uh oh!

kszucs commented Jan 24, 2022

Uh oh!

github-actionsBot commented Jan 24, 2022

Uh oh!

kszucs commented Jan 24, 2022

Uh oh!

github-actionsBot commented Jan 24, 2022

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kszucs commented Jan 24, 2022

Uh oh!

kszucs commented Jan 24, 2022

Uh oh!

github-actionsBot commented Jan 24, 2022

Uh oh!

kou commented Jan 24, 2022

Uh oh!

kszucs commented Jan 24, 2022

Uh oh!

github-actionsBot commented Jan 24, 2022

Uh oh!

kszucs commented Jan 24, 2022

Uh oh!

github-actionsBot commented Jan 24, 2022

Uh oh!

kszucs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ursabot commented Jan 24, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ursabot commented Jan 24, 2022•
edited
Loading