Skip to content

GH-35193: [Python][Packaging] Enable GCS on Windows wheels - #35255

Merged
assignUser merged 8 commits into
apache:mainfrom
raulcd:GH-35193
May 16, 2023
Merged

GH-35193: [Python][Packaging] Enable GCS on Windows wheels#35255
assignUser merged 8 commits into
apache:mainfrom
raulcd:GH-35193

Conversation

@raulcd

@raulcdraulcd commented Apr 20, 2023•
edited
Loading

Copy link
Copy Markdown
Member

Rationale for this change

GCS could be enabled on Windows wheels.

What changes are included in this PR?

Enabling GCS on Windows wheel

Are these changes tested?

Crossbow jobs for wheels run tests for GCS now.
I have tested locally that I can install the built wheel and I can import GcsFileSystem:

Python 3.9.12 (main, Apr 4 2022, 05:22:27) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Warning:
This Python interpreter is in a conda environment, but the environment has
not been activated. Libraries may fail to load. To activate this environment
please see https://conda.io/activation
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyarrow.fs import GcsFileSystem
>>> fs = GcsFileSystem(access_token='abc',target_service_account='service_account@apache',credential_token_expiration=datetime.now(),default_bucket_location='us-west2',scheme='https', endpoint_override='localhost:8999')
>>> fs.default_bucket_location
'us-west2'
>>> fs.create_dir('hello')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pyarrow\_fs.pyx", line 593, in pyarrow._fs.FileSystem.create_dir
File "pyarrow\error.pxi", line 113, in pyarrow.lib.check_status
PermissionError: [Errno 13] google::cloud::Status(UNAUTHENTICATED: Permanent error GetBucketMetadata: Could not create a OAuth2 access token to authenticate the request. The request was not sent, as such an access token is required to complete the request successfully. Learn more about Google Cloud authentication at https://cloud.google.com/docs/authentication. The underlying error message was: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.). Detail: [errno 13] Permission denied

Are there any user-facing changes?

No but Windows wheels should contain ARROW_GCS

@github-actions

github-actionsBot commented Apr 20, 2023

Copy link
Copy Markdown

@github-actions

github-actionsBot commented Apr 20, 2023

Copy link
Copy Markdown

⚠️ GitHub issue #35193has been automatically assigned in GitHub to PR creator.

@github-actionsgithub-actionsBot added the awaiting committer review Awaiting committer review label Apr 20, 2023
@raulcd

raulcd commented Apr 20, 2023

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-windows-cp311-amd64

@github-actions

github-actionsBot commented Apr 20, 2023

Copy link
Copy Markdown

Revision: 5246fc5be2ed942a5688b29a05700c0673d820d2

Submitted crossbow builds: ursacomputing/crossbow @ actions-43da90de90

TaskStatus
wheel-windows-cp311-amd64Github Actions

@raulcd

raulcd commented Apr 20, 2023

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-windows-cp311-amd64

@github-actions

github-actionsBot commented Apr 20, 2023

Copy link
Copy Markdown 
Command '['git', 'clone', 'https://github.com/ursacomputing/crossbow', '/tmp/tmptzeyqnko/crossbow']' returned non-zero exit status 128.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/4756107770

@raulcd

raulcd commented Apr 20, 2023

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-windows-cp311-amd64

@github-actions

github-actionsBot commented Apr 20, 2023

Copy link
Copy Markdown

Revision: a68e3737990ccd7b7bc3d41236f5156c139c1759

Submitted crossbow builds: ursacomputing/crossbow @ actions-e67a78785a

TaskStatus
wheel-windows-cp311-amd64Github Actions

@raulcd

raulcd commented Apr 27, 2023

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-windows-cp39-amd64

@github-actions

github-actionsBot commented Apr 27, 2023

Copy link
Copy Markdown

Revision: a68e3737990ccd7b7bc3d41236f5156c139c1759

Submitted crossbow builds: ursacomputing/crossbow @ actions-cd5876c7ff

TaskStatus
wheel-windows-cp39-amd64Github Actions

@raulcd

raulcd commented Apr 27, 2023

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-windows-cp39-amd64

@github-actions

github-actionsBot commented Apr 27, 2023

Copy link
Copy Markdown

Revision: cb95927568e66d1be2044ff73e00fbe98ddf0318

Submitted crossbow builds: ursacomputing/crossbow @ actions-bb930e9684

TaskStatus
wheel-windows-cp39-amd64Github Actions

@raulcd

raulcd commented Apr 27, 2023

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-windows-*

@raulcd
raulcd marked this pull request as ready for review April 27, 2023 14:44
@raulcdraulcd changed the title WIP: GH-35193: [Python][Packaging] Enable GCS on Windows wheelsGH-35193: [Python][Packaging] Enable GCS on Windows wheelsApr 27, 2023
@github-actions

github-actionsBot commented Apr 27, 2023

Copy link
Copy Markdown

Revision: 8b0ecc226649fb165e89e39b81a859290f223af7

Submitted crossbow builds: ursacomputing/crossbow @ actions-af464cb5b5

TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions

@raulcd

raulcd commented May 10, 2023

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-windows-*

@github-actions

github-actionsBot commented May 10, 2023

Copy link
Copy Markdown

Revision: b2b891a

Submitted crossbow builds: ursacomputing/crossbow @ actions-5e23d788f1

TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions

@raulcd

raulcd commented May 11, 2023

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-windows-*

@github-actions

github-actionsBot commented May 11, 2023

Copy link
Copy Markdown

Revision: 50efe4e

Submitted crossbow builds: ursacomputing/crossbow @ actions-20194149b9

TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions

@raulcd

raulcd commented May 11, 2023

Copy link
Copy Markdown
MemberAuthor

@westonpace while enabling GCS for the Windows wheels I realised Appveyor wasn't really adding ARROW_GCS=ON and there is a failure on the arrow-gcsfs-test on Appveyor if I enable it, example:

[ FAILED ] GcsIntegrationTest.CreateDirRecursiveFolderOnly (0 ms)
[ RUN ] GcsIntegrationTest.CreateDirRecursiveBucketAndFolder
C:/projects/arrow/cpp/src/arrow/filesystem/gcsfs_test.cc(170): error: Expected equality of these values:
Testbench()->error()
Which is: "Could not start GCS emulator. Used the following list of python interpreter names: 3.10 (exe not found)"
""
[ FAILED ] GcsIntegrationTest.CreateDirRecursiveBucketAndFolder (0 ms)
[ RUN ] GcsIntegrationTest.CreateDirUri

I can move fixing Appveyor to it's own issue/PR as it's slightly different from enabling GCS for the Windows wheels, which as we can see on the wheels jobs is successful. Do you think Appveyor should be fixed here? Any idea what the issue/fix is?

@westonpace

westonpace commented May 11, 2023

Copy link
Copy Markdown
Member

Do you think Appveyor should be fixed here? Any idea what the issue/fix is?

It would be nice to have at least one Windows CI job that uses GCS. I don't know if the others do or not.

Any idea what the issue/fix is?

Testbench is a standalone python project that emulates GCS. This is very similar to how minio emulates S3. The C++ unit tests launch testbench (using boost::process). Since testbench is a python project the C++ tests need to be able to locate and run python. It appears, on Windows, this is not working.

Here is what the code to look for python looks like today:

 std::vector<std::string> names{"python3", "python"};
// If the build script or application developer provides a value in the PYTHON
// environment variable, then just use that.
if (const auto* env = std::getenv("PYTHON")) {
names = {env};
}
auto error = std::string(
"Could not start GCS emulator."
" Used the following list of python interpreter names:");
for (const auto& interpreter : names) {
auto exe_path = bp::search_path(interpreter);
error += " " + interpreter;
if (exe_path.empty()) {
error += " (exe not found)";
continue;
}

So maybe it's as easy as setting the PYTHON environment variable to point to a valid python executable?

@raulcd

raulcd commented May 12, 2023

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit wheel-windows-*

@github-actions

github-actionsBot commented May 12, 2023

Copy link
Copy Markdown

Revision: e1d51a8

Submitted crossbow builds: ursacomputing/crossbow @ actions-20cbf52fd6

TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions

@raulcd

raulcd commented May 12, 2023

Copy link
Copy Markdown
MemberAuthor

Thanks @westonpace, this should be ready for review now

@raulcd
raulcd requested a review from westonpaceMay 12, 2023 11:19
kou
kou reviewed May 12, 2023
View reviewed changes
Comment threadci/scripts/install_gcs_testbench.bat Outdated
@github-actionsgithub-actionsBot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels May 12, 2023
@github-actionsgithub-actionsBot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels May 12, 2023
@kou

kou commented May 12, 2023

Copy link
Copy Markdown
Member

@github-actions crossbow submit wheel-windows-*

@github-actions

github-actionsBot commented May 12, 2023

Copy link
Copy Markdown

Revision: cf41c23

Submitted crossbow builds: ursacomputing/crossbow @ actions-fe9442d7db

TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions

kou
kou approved these changes May 12, 2023
View reviewed changes

@koukou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@github-actionsgithub-actionsBot added awaiting merge Awaiting merge and removed awaiting changes Awaiting changes labels May 12, 2023
@assignUser
assignUser merged commit 8be70c1 into apache:mainMay 16, 2023
@ursabot

ursabot commented May 16, 2023

Copy link
Copy Markdown

Benchmark runs are scheduled for baseline = 2d76d9a and contender = 8be70c1. 8be70c1 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.81% ⬆️0.0%] test-mac-arm
[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.15% ⬆️0.03%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 8be70c13 ec2-t3-xlarge-us-east-2
[Finished] 8be70c13 test-mac-arm
[Finished] 8be70c13 ursa-i9-9960x
[Finished] 8be70c13 ursa-thinkcentre-m75q
[Finished] 2d76d9a5 ec2-t3-xlarge-us-east-2
[Finished] 2d76d9a5 test-mac-arm
[Finished] 2d76d9a5 ursa-i9-9960x
[Finished] 2d76d9a5 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

@westonpace

westonpace commented May 18, 2023

Copy link
Copy Markdown
Member

Thanks @raulcd for enabling this!

@koukou mentioned this pull request Sep 29, 2023
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@westonpacewestonpaceAwaiting requested review from westonpace

Assignees

No one assigned

Labels

awaiting mergeAwaiting merge

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Python] Windows wheel is built without GCS support

5 participants

@raulcd@westonpace@kou@ursabot@assignUser
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
GH-35193: [Python][Packaging] Enable GCS on Windows wheels by raulcd · Pull Request #35255 · apache/arrow · GitHub
Skip to content
GH-35193: [Python][Packaging] Enable GCS on Windows wheels - #35255
Merged
assignUser merged 8 commits into
apache:mainfrom 
raulcd:GH-35193
May 16, 2023
Merged
GH-35193: [Python][Packaging] Enable GCS on Windows wheels#35255
assignUser merged 8 commits into
apache:mainfrom 
raulcd:GH-35193
Conversation
@raulcd
@raulcdraulcd
commented
Apr 20, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Rationale for this change
GCS could be enabled on Windows wheels.
What changes are included in this PR?
Enabling GCS on Windows wheel
Are these changes tested?
Crossbow jobs for wheels run tests for GCS now.

I have tested locally that I can install the built wheel and I can import GcsFileSystem:
Python 3.9.12 (main, Apr 4 2022, 05:22:27) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Warning:
This Python interpreter is in a conda environment, but the environment has
not been activated. Libraries may fail to load. To activate this environment
please see https://conda.io/activation
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyarrow.fs import GcsFileSystem
>>> fs = GcsFileSystem(access_token='abc',target_service_account='service_account@apache',credential_token_expiration=datetime.now(),default_bucket_location='us-west2',scheme='https', endpoint_override='localhost:8999')
>>> fs.default_bucket_location
'us-west2'
>>> fs.create_dir('hello')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pyarrow\_fs.pyx", line 593, in pyarrow._fs.FileSystem.create_dir
File "pyarrow\error.pxi", line 113, in pyarrow.lib.check_status
PermissionError: [Errno 13] google::cloud::Status(UNAUTHENTICATED: Permanent error GetBucketMetadata: Could not create a OAuth2 access token to authenticate the request. The request was not sent, as such an access token is required to complete the request successfully. Learn more about Google Cloud authentication at https://cloud.google.com/docs/authentication. The underlying error message was: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.). Detail: [errno 13] Permission denied

Are there any user-facing changes?
No but Windows wheels should contain ARROW_GCS
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #35193has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting committer review
Awaiting committer review
label
Apr 20, 2023
@raulcd
raulcd
commented
Apr 20, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp311-amd64
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

Revision: 5246fc5be2ed942a5688b29a05700c0673d820d2
Submitted crossbow builds: ursacomputing/crossbow @ actions-43da90de90
TaskStatus
wheel-windows-cp311-amd64Github Actions
@raulcd
raulcd
commented
Apr 20, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp311-amd64
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

Command '['git', 'clone', 'https://github.com/ursacomputing/crossbow', '/tmp/tmptzeyqnko/crossbow']' returned non-zero exit status 128.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/4756107770

@raulcd
raulcd
commented
Apr 20, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp311-amd64
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

Revision: a68e3737990ccd7b7bc3d41236f5156c139c1759
Submitted crossbow builds: ursacomputing/crossbow @ actions-e67a78785a
TaskStatus
wheel-windows-cp311-amd64Github Actions
@raulcd
raulcd
commented
Apr 27, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp39-amd64
@github-actions
github-actionsBot
commented
Apr 27, 2023

Copy link


Copy Markdown

Revision: a68e3737990ccd7b7bc3d41236f5156c139c1759
Submitted crossbow builds: ursacomputing/crossbow @ actions-cd5876c7ff
TaskStatus
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
Apr 27, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp39-amd64
@github-actions
github-actionsBot
commented
Apr 27, 2023

Copy link


Copy Markdown

Revision: cb95927568e66d1be2044ff73e00fbe98ddf0318
Submitted crossbow builds: ursacomputing/crossbow @ actions-bb930e9684
TaskStatus
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
Apr 27, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@raulcd
raulcd
marked this pull request as ready for review
April 27, 2023 14:44
@raulcdraulcd
changed the title
WIP: GH-35193: [Python][Packaging] Enable GCS on Windows wheelsGH-35193: [Python][Packaging] Enable GCS on Windows wheelsApr 27, 2023
@github-actions
github-actionsBot
commented
Apr 27, 2023

Copy link


Copy Markdown

Revision: 8b0ecc226649fb165e89e39b81a859290f223af7
Submitted crossbow builds: ursacomputing/crossbow @ actions-af464cb5b5
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 10, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 10, 2023

Copy link


Copy Markdown

Revision: b2b891a
Submitted crossbow builds: ursacomputing/crossbow @ actions-5e23d788f1
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 11, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 11, 2023

Copy link


Copy Markdown

Revision: 50efe4e
Submitted crossbow builds: ursacomputing/crossbow @ actions-20194149b9
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 11, 2023

Copy link


Copy Markdown

MemberAuthor
@westonpace while enabling GCS for the Windows wheels I realised Appveyor wasn't really adding ARROW_GCS=ON and there is a failure on the arrow-gcsfs-test on Appveyor if I enable it, example:
[ FAILED ] GcsIntegrationTest.CreateDirRecursiveFolderOnly (0 ms)
[ RUN ] GcsIntegrationTest.CreateDirRecursiveBucketAndFolder
C:/projects/arrow/cpp/src/arrow/filesystem/gcsfs_test.cc(170): error: Expected equality of these values:
Testbench()->error()
Which is: "Could not start GCS emulator. Used the following list of python interpreter names: 3.10 (exe not found)"
""
[ FAILED ] GcsIntegrationTest.CreateDirRecursiveBucketAndFolder (0 ms)
[ RUN ] GcsIntegrationTest.CreateDirUri

I can move fixing Appveyor to it's own issue/PR as it's slightly different from enabling GCS for the Windows wheels, which as we can see on the wheels jobs is successful. Do you think Appveyor should be fixed here? Any idea what the issue/fix is?
@westonpace
westonpace
commented
May 11, 2023

Copy link


Copy Markdown

Member
Do you think Appveyor should be fixed here? Any idea what the issue/fix is?
It would be nice to have at least one Windows CI job that uses GCS. I don't know if the others do or not.
Any idea what the issue/fix is?
Testbench is a standalone python project that emulates GCS. This is very similar to how minio emulates S3. The C++ unit tests launch testbench (using boost::process). Since testbench is a python project the C++ tests need to be able to locate and run python. It appears, on Windows, this is not working.
Here is what the code to look for python looks like today:
 std::vector<std::string> names{"python3", "python"};
// If the build script or application developer provides a value in the PYTHON
// environment variable, then just use that.
if (const auto* env = std::getenv("PYTHON")) {
names = {env};
}
auto error = std::string(
"Could not start GCS emulator."
" Used the following list of python interpreter names:");
for (const auto& interpreter : names) {
auto exe_path = bp::search_path(interpreter);
error += " " + interpreter;
if (exe_path.empty()) {
error += " (exe not found)";
continue;
}

So maybe it's as easy as setting the PYTHON environment variable to point to a valid python executable?
@raulcd
raulcd
commented
May 12, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 12, 2023

Copy link


Copy Markdown

Revision: e1d51a8
Submitted crossbow builds: ursacomputing/crossbow @ actions-20cbf52fd6
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 12, 2023

Copy link


Copy Markdown

MemberAuthor
Thanks @westonpace, this should be ready for review now
@raulcd
raulcd
requested a review
from westonpaceMay 12, 2023 11:19
kou
kou
reviewed
May 12, 2023
View reviewed changes
Comment threadci/scripts/install_gcs_testbench.bat
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting committer review
Awaiting committer review
labels
May 12, 2023
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
awaiting changes
Awaiting changes
and removed

awaiting changes
Awaiting changes
awaiting change review
Awaiting change review
labels
May 12, 2023
@kou
kou
commented
May 12, 2023

Copy link


Copy Markdown

Member
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 12, 2023

Copy link


Copy Markdown

Revision: cf41c23
Submitted crossbow builds: ursacomputing/crossbow @ actions-fe9442d7db
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
kou
kou
approved these changes
May 12, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting changes
Awaiting changes
labels
May 12, 2023
@assignUser
assignUser
merged commit 8be70c1
into
apache:mainMay 16, 2023
@ursabot
ursabot
commented
May 16, 2023

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 2d76d9a and contender = 8be70c1. 8be70c1 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Finished ⬇️0.81% ⬆️0.0%] test-mac-arm

[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.15% ⬆️0.03%] ursa-thinkcentre-m75q

Buildkite builds:

[Finished] 8be70c13 ec2-t3-xlarge-us-east-2

[Finished] 8be70c13 test-mac-arm

[Finished] 8be70c13 ursa-i9-9960x

[Finished] 8be70c13 ursa-thinkcentre-m75q

[Finished] 2d76d9a5 ec2-t3-xlarge-us-east-2

[Finished] 2d76d9a5 test-mac-arm

[Finished] 2d76d9a5 ursa-i9-9960x

[Finished] 2d76d9a5 ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True

test-mac-arm: Supported benchmark langs: C++, Python, R

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@westonpace
westonpace
commented
May 18, 2023

Copy link


Copy Markdown

Member
Thanks @raulcd for enabling this!
@koukou
mentioned this pull request
Sep 29, 2023

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@westonpacewestonpaceAwaiting requested review from westonpace

Assignees


No one assigned


Labels

awaiting mergeAwaiting merge

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[Python] Windows wheel is built without GCS support


5 participants

@raulcd@westonpace@kou@ursabot@assignUser

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GH-35193: [Python][Packaging] Enable GCS on Windows wheels by raulcd · Pull Request #35255 · apache/arrow · GitHub
Skip to content
GH-35193: [Python][Packaging] Enable GCS on Windows wheels - #35255
Merged
assignUser merged 8 commits into
apache:mainfrom 
raulcd:GH-35193
May 16, 2023
Merged
GH-35193: [Python][Packaging] Enable GCS on Windows wheels#35255
assignUser merged 8 commits into
apache:mainfrom 
raulcd:GH-35193
Conversation
@raulcd
@raulcdraulcd
commented
Apr 20, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Rationale for this change
GCS could be enabled on Windows wheels.
What changes are included in this PR?
Enabling GCS on Windows wheel
Are these changes tested?
Crossbow jobs for wheels run tests for GCS now.

I have tested locally that I can install the built wheel and I can import GcsFileSystem:
Python 3.9.12 (main, Apr 4 2022, 05:22:27) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Warning:
This Python interpreter is in a conda environment, but the environment has
not been activated. Libraries may fail to load. To activate this environment
please see https://conda.io/activation
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyarrow.fs import GcsFileSystem
>>> fs = GcsFileSystem(access_token='abc',target_service_account='service_account@apache',credential_token_expiration=datetime.now(),default_bucket_location='us-west2',scheme='https', endpoint_override='localhost:8999')
>>> fs.default_bucket_location
'us-west2'
>>> fs.create_dir('hello')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pyarrow\_fs.pyx", line 593, in pyarrow._fs.FileSystem.create_dir
File "pyarrow\error.pxi", line 113, in pyarrow.lib.check_status
PermissionError: [Errno 13] google::cloud::Status(UNAUTHENTICATED: Permanent error GetBucketMetadata: Could not create a OAuth2 access token to authenticate the request. The request was not sent, as such an access token is required to complete the request successfully. Learn more about Google Cloud authentication at https://cloud.google.com/docs/authentication. The underlying error message was: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.). Detail: [errno 13] Permission denied

Are there any user-facing changes?
No but Windows wheels should contain ARROW_GCS
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #35193has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting committer review
Awaiting committer review
label
Apr 20, 2023
@raulcd
raulcd
commented
Apr 20, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp311-amd64
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

Revision: 5246fc5be2ed942a5688b29a05700c0673d820d2
Submitted crossbow builds: ursacomputing/crossbow @ actions-43da90de90
TaskStatus
wheel-windows-cp311-amd64Github Actions
@raulcd
raulcd
commented
Apr 20, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp311-amd64
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

Command '['git', 'clone', 'https://github.com/ursacomputing/crossbow', '/tmp/tmptzeyqnko/crossbow']' returned non-zero exit status 128.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/4756107770

@raulcd
raulcd
commented
Apr 20, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp311-amd64
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

Revision: a68e3737990ccd7b7bc3d41236f5156c139c1759
Submitted crossbow builds: ursacomputing/crossbow @ actions-e67a78785a
TaskStatus
wheel-windows-cp311-amd64Github Actions
@raulcd
raulcd
commented
Apr 27, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp39-amd64
@github-actions
github-actionsBot
commented
Apr 27, 2023

Copy link


Copy Markdown

Revision: a68e3737990ccd7b7bc3d41236f5156c139c1759
Submitted crossbow builds: ursacomputing/crossbow @ actions-cd5876c7ff
TaskStatus
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
Apr 27, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp39-amd64
@github-actions
github-actionsBot
commented
Apr 27, 2023

Copy link


Copy Markdown

Revision: cb95927568e66d1be2044ff73e00fbe98ddf0318
Submitted crossbow builds: ursacomputing/crossbow @ actions-bb930e9684
TaskStatus
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
Apr 27, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@raulcd
raulcd
marked this pull request as ready for review
April 27, 2023 14:44
@raulcdraulcd
changed the title
WIP: GH-35193: [Python][Packaging] Enable GCS on Windows wheelsGH-35193: [Python][Packaging] Enable GCS on Windows wheelsApr 27, 2023
@github-actions
github-actionsBot
commented
Apr 27, 2023

Copy link


Copy Markdown

Revision: 8b0ecc226649fb165e89e39b81a859290f223af7
Submitted crossbow builds: ursacomputing/crossbow @ actions-af464cb5b5
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 10, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 10, 2023

Copy link


Copy Markdown

Revision: b2b891a
Submitted crossbow builds: ursacomputing/crossbow @ actions-5e23d788f1
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 11, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 11, 2023

Copy link


Copy Markdown

Revision: 50efe4e
Submitted crossbow builds: ursacomputing/crossbow @ actions-20194149b9
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 11, 2023

Copy link


Copy Markdown

MemberAuthor
@westonpace while enabling GCS for the Windows wheels I realised Appveyor wasn't really adding ARROW_GCS=ON and there is a failure on the arrow-gcsfs-test on Appveyor if I enable it, example:
[ FAILED ] GcsIntegrationTest.CreateDirRecursiveFolderOnly (0 ms)
[ RUN ] GcsIntegrationTest.CreateDirRecursiveBucketAndFolder
C:/projects/arrow/cpp/src/arrow/filesystem/gcsfs_test.cc(170): error: Expected equality of these values:
Testbench()->error()
Which is: "Could not start GCS emulator. Used the following list of python interpreter names: 3.10 (exe not found)"
""
[ FAILED ] GcsIntegrationTest.CreateDirRecursiveBucketAndFolder (0 ms)
[ RUN ] GcsIntegrationTest.CreateDirUri

I can move fixing Appveyor to it's own issue/PR as it's slightly different from enabling GCS for the Windows wheels, which as we can see on the wheels jobs is successful. Do you think Appveyor should be fixed here? Any idea what the issue/fix is?
@westonpace
westonpace
commented
May 11, 2023

Copy link


Copy Markdown

Member
Do you think Appveyor should be fixed here? Any idea what the issue/fix is?
It would be nice to have at least one Windows CI job that uses GCS. I don't know if the others do or not.
Any idea what the issue/fix is?
Testbench is a standalone python project that emulates GCS. This is very similar to how minio emulates S3. The C++ unit tests launch testbench (using boost::process). Since testbench is a python project the C++ tests need to be able to locate and run python. It appears, on Windows, this is not working.
Here is what the code to look for python looks like today:
 std::vector<std::string> names{"python3", "python"};
// If the build script or application developer provides a value in the PYTHON
// environment variable, then just use that.
if (const auto* env = std::getenv("PYTHON")) {
names = {env};
}
auto error = std::string(
"Could not start GCS emulator."
" Used the following list of python interpreter names:");
for (const auto& interpreter : names) {
auto exe_path = bp::search_path(interpreter);
error += " " + interpreter;
if (exe_path.empty()) {
error += " (exe not found)";
continue;
}

So maybe it's as easy as setting the PYTHON environment variable to point to a valid python executable?
@raulcd
raulcd
commented
May 12, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 12, 2023

Copy link


Copy Markdown

Revision: e1d51a8
Submitted crossbow builds: ursacomputing/crossbow @ actions-20cbf52fd6
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 12, 2023

Copy link


Copy Markdown

MemberAuthor
Thanks @westonpace, this should be ready for review now
@raulcd
raulcd
requested a review
from westonpaceMay 12, 2023 11:19
kou
kou
reviewed
May 12, 2023
View reviewed changes
Comment threadci/scripts/install_gcs_testbench.bat
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting committer review
Awaiting committer review
labels
May 12, 2023
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
awaiting changes
Awaiting changes
and removed

awaiting changes
Awaiting changes
awaiting change review
Awaiting change review
labels
May 12, 2023
@kou
kou
commented
May 12, 2023

Copy link


Copy Markdown

Member
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 12, 2023

Copy link


Copy Markdown

Revision: cf41c23
Submitted crossbow builds: ursacomputing/crossbow @ actions-fe9442d7db
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
kou
kou
approved these changes
May 12, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting changes
Awaiting changes
labels
May 12, 2023
@assignUser
assignUser
merged commit 8be70c1
into
apache:mainMay 16, 2023
@ursabot
ursabot
commented
May 16, 2023

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 2d76d9a and contender = 8be70c1. 8be70c1 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Finished ⬇️0.81% ⬆️0.0%] test-mac-arm

[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.15% ⬆️0.03%] ursa-thinkcentre-m75q

Buildkite builds:

[Finished] 8be70c13 ec2-t3-xlarge-us-east-2

[Finished] 8be70c13 test-mac-arm

[Finished] 8be70c13 ursa-i9-9960x

[Finished] 8be70c13 ursa-thinkcentre-m75q

[Finished] 2d76d9a5 ec2-t3-xlarge-us-east-2

[Finished] 2d76d9a5 test-mac-arm

[Finished] 2d76d9a5 ursa-i9-9960x

[Finished] 2d76d9a5 ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True

test-mac-arm: Supported benchmark langs: C++, Python, R

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@westonpace
westonpace
commented
May 18, 2023

Copy link


Copy Markdown

Member
Thanks @raulcd for enabling this!
@koukou
mentioned this pull request
Sep 29, 2023

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@westonpacewestonpaceAwaiting requested review from westonpace

Assignees


No one assigned


Labels

awaiting mergeAwaiting merge

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[Python] Windows wheel is built without GCS support


5 participants

@raulcd@westonpace@kou@ursabot@assignUser

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GH-35193: [Python][Packaging] Enable GCS on Windows wheels by raulcd · Pull Request #35255 · apache/arrow · GitHub
Skip to content
GH-35193: [Python][Packaging] Enable GCS on Windows wheels - #35255
Merged
assignUser merged 8 commits into
apache:mainfrom 
raulcd:GH-35193
May 16, 2023
Merged
GH-35193: [Python][Packaging] Enable GCS on Windows wheels#35255
assignUser merged 8 commits into
apache:mainfrom 
raulcd:GH-35193
Conversation
@raulcd
@raulcdraulcd
commented
Apr 20, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Rationale for this change
GCS could be enabled on Windows wheels.
What changes are included in this PR?
Enabling GCS on Windows wheel
Are these changes tested?
Crossbow jobs for wheels run tests for GCS now.

I have tested locally that I can install the built wheel and I can import GcsFileSystem:
Python 3.9.12 (main, Apr 4 2022, 05:22:27) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Warning:
This Python interpreter is in a conda environment, but the environment has
not been activated. Libraries may fail to load. To activate this environment
please see https://conda.io/activation
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyarrow.fs import GcsFileSystem
>>> fs = GcsFileSystem(access_token='abc',target_service_account='service_account@apache',credential_token_expiration=datetime.now(),default_bucket_location='us-west2',scheme='https', endpoint_override='localhost:8999')
>>> fs.default_bucket_location
'us-west2'
>>> fs.create_dir('hello')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pyarrow\_fs.pyx", line 593, in pyarrow._fs.FileSystem.create_dir
File "pyarrow\error.pxi", line 113, in pyarrow.lib.check_status
PermissionError: [Errno 13] google::cloud::Status(UNAUTHENTICATED: Permanent error GetBucketMetadata: Could not create a OAuth2 access token to authenticate the request. The request was not sent, as such an access token is required to complete the request successfully. Learn more about Google Cloud authentication at https://cloud.google.com/docs/authentication. The underlying error message was: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.). Detail: [errno 13] Permission denied

Are there any user-facing changes?
No but Windows wheels should contain ARROW_GCS
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #35193has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting committer review
Awaiting committer review
label
Apr 20, 2023
@raulcd
raulcd
commented
Apr 20, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp311-amd64
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

Revision: 5246fc5be2ed942a5688b29a05700c0673d820d2
Submitted crossbow builds: ursacomputing/crossbow @ actions-43da90de90
TaskStatus
wheel-windows-cp311-amd64Github Actions
@raulcd
raulcd
commented
Apr 20, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp311-amd64
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

Command '['git', 'clone', 'https://github.com/ursacomputing/crossbow', '/tmp/tmptzeyqnko/crossbow']' returned non-zero exit status 128.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/4756107770

@raulcd
raulcd
commented
Apr 20, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp311-amd64
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

Revision: a68e3737990ccd7b7bc3d41236f5156c139c1759
Submitted crossbow builds: ursacomputing/crossbow @ actions-e67a78785a
TaskStatus
wheel-windows-cp311-amd64Github Actions
@raulcd
raulcd
commented
Apr 27, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp39-amd64
@github-actions
github-actionsBot
commented
Apr 27, 2023

Copy link


Copy Markdown

Revision: a68e3737990ccd7b7bc3d41236f5156c139c1759
Submitted crossbow builds: ursacomputing/crossbow @ actions-cd5876c7ff
TaskStatus
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
Apr 27, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp39-amd64
@github-actions
github-actionsBot
commented
Apr 27, 2023

Copy link


Copy Markdown

Revision: cb95927568e66d1be2044ff73e00fbe98ddf0318
Submitted crossbow builds: ursacomputing/crossbow @ actions-bb930e9684
TaskStatus
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
Apr 27, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@raulcd
raulcd
marked this pull request as ready for review
April 27, 2023 14:44
@raulcdraulcd
changed the title
WIP: GH-35193: [Python][Packaging] Enable GCS on Windows wheelsGH-35193: [Python][Packaging] Enable GCS on Windows wheelsApr 27, 2023
@github-actions
github-actionsBot
commented
Apr 27, 2023

Copy link


Copy Markdown

Revision: 8b0ecc226649fb165e89e39b81a859290f223af7
Submitted crossbow builds: ursacomputing/crossbow @ actions-af464cb5b5
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 10, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 10, 2023

Copy link


Copy Markdown

Revision: b2b891a
Submitted crossbow builds: ursacomputing/crossbow @ actions-5e23d788f1
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 11, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 11, 2023

Copy link


Copy Markdown

Revision: 50efe4e
Submitted crossbow builds: ursacomputing/crossbow @ actions-20194149b9
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 11, 2023

Copy link


Copy Markdown

MemberAuthor
@westonpace while enabling GCS for the Windows wheels I realised Appveyor wasn't really adding ARROW_GCS=ON and there is a failure on the arrow-gcsfs-test on Appveyor if I enable it, example:
[ FAILED ] GcsIntegrationTest.CreateDirRecursiveFolderOnly (0 ms)
[ RUN ] GcsIntegrationTest.CreateDirRecursiveBucketAndFolder
C:/projects/arrow/cpp/src/arrow/filesystem/gcsfs_test.cc(170): error: Expected equality of these values:
Testbench()->error()
Which is: "Could not start GCS emulator. Used the following list of python interpreter names: 3.10 (exe not found)"
""
[ FAILED ] GcsIntegrationTest.CreateDirRecursiveBucketAndFolder (0 ms)
[ RUN ] GcsIntegrationTest.CreateDirUri

I can move fixing Appveyor to it's own issue/PR as it's slightly different from enabling GCS for the Windows wheels, which as we can see on the wheels jobs is successful. Do you think Appveyor should be fixed here? Any idea what the issue/fix is?
@westonpace
westonpace
commented
May 11, 2023

Copy link


Copy Markdown

Member
Do you think Appveyor should be fixed here? Any idea what the issue/fix is?
It would be nice to have at least one Windows CI job that uses GCS. I don't know if the others do or not.
Any idea what the issue/fix is?
Testbench is a standalone python project that emulates GCS. This is very similar to how minio emulates S3. The C++ unit tests launch testbench (using boost::process). Since testbench is a python project the C++ tests need to be able to locate and run python. It appears, on Windows, this is not working.
Here is what the code to look for python looks like today:
 std::vector<std::string> names{"python3", "python"};
// If the build script or application developer provides a value in the PYTHON
// environment variable, then just use that.
if (const auto* env = std::getenv("PYTHON")) {
names = {env};
}
auto error = std::string(
"Could not start GCS emulator."
" Used the following list of python interpreter names:");
for (const auto& interpreter : names) {
auto exe_path = bp::search_path(interpreter);
error += " " + interpreter;
if (exe_path.empty()) {
error += " (exe not found)";
continue;
}

So maybe it's as easy as setting the PYTHON environment variable to point to a valid python executable?
@raulcd
raulcd
commented
May 12, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 12, 2023

Copy link


Copy Markdown

Revision: e1d51a8
Submitted crossbow builds: ursacomputing/crossbow @ actions-20cbf52fd6
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 12, 2023

Copy link


Copy Markdown

MemberAuthor
Thanks @westonpace, this should be ready for review now
@raulcd
raulcd
requested a review
from westonpaceMay 12, 2023 11:19
kou
kou
reviewed
May 12, 2023
View reviewed changes
Comment threadci/scripts/install_gcs_testbench.bat
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting committer review
Awaiting committer review
labels
May 12, 2023
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
awaiting changes
Awaiting changes
and removed

awaiting changes
Awaiting changes
awaiting change review
Awaiting change review
labels
May 12, 2023
@kou
kou
commented
May 12, 2023

Copy link


Copy Markdown

Member
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 12, 2023

Copy link


Copy Markdown

Revision: cf41c23
Submitted crossbow builds: ursacomputing/crossbow @ actions-fe9442d7db
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
kou
kou
approved these changes
May 12, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting changes
Awaiting changes
labels
May 12, 2023
@assignUser
assignUser
merged commit 8be70c1
into
apache:mainMay 16, 2023
@ursabot
ursabot
commented
May 16, 2023

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 2d76d9a and contender = 8be70c1. 8be70c1 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Finished ⬇️0.81% ⬆️0.0%] test-mac-arm

[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.15% ⬆️0.03%] ursa-thinkcentre-m75q

Buildkite builds:

[Finished] 8be70c13 ec2-t3-xlarge-us-east-2

[Finished] 8be70c13 test-mac-arm

[Finished] 8be70c13 ursa-i9-9960x

[Finished] 8be70c13 ursa-thinkcentre-m75q

[Finished] 2d76d9a5 ec2-t3-xlarge-us-east-2

[Finished] 2d76d9a5 test-mac-arm

[Finished] 2d76d9a5 ursa-i9-9960x

[Finished] 2d76d9a5 ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True

test-mac-arm: Supported benchmark langs: C++, Python, R

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@westonpace
westonpace
commented
May 18, 2023

Copy link


Copy Markdown

Member
Thanks @raulcd for enabling this!
@koukou
mentioned this pull request
Sep 29, 2023

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@westonpacewestonpaceAwaiting requested review from westonpace

Assignees


No one assigned


Labels

awaiting mergeAwaiting merge

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[Python] Windows wheel is built without GCS support


5 participants

@raulcd@westonpace@kou@ursabot@assignUser

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
GH-35193: [Python][Packaging] Enable GCS on Windows wheels by raulcd · Pull Request #35255 · apache/arrow · GitHub
Skip to content
GH-35193: [Python][Packaging] Enable GCS on Windows wheels - #35255
Merged
assignUser merged 8 commits into
apache:mainfrom 
raulcd:GH-35193
May 16, 2023
Merged
GH-35193: [Python][Packaging] Enable GCS on Windows wheels#35255
assignUser merged 8 commits into
apache:mainfrom 
raulcd:GH-35193
Conversation
@raulcd
@raulcdraulcd
commented
Apr 20, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Rationale for this change
GCS could be enabled on Windows wheels.
What changes are included in this PR?
Enabling GCS on Windows wheel
Are these changes tested?
Crossbow jobs for wheels run tests for GCS now.

I have tested locally that I can install the built wheel and I can import GcsFileSystem:
Python 3.9.12 (main, Apr 4 2022, 05:22:27) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Warning:
This Python interpreter is in a conda environment, but the environment has
not been activated. Libraries may fail to load. To activate this environment
please see https://conda.io/activation
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyarrow.fs import GcsFileSystem
>>> fs = GcsFileSystem(access_token='abc',target_service_account='service_account@apache',credential_token_expiration=datetime.now(),default_bucket_location='us-west2',scheme='https', endpoint_override='localhost:8999')
>>> fs.default_bucket_location
'us-west2'
>>> fs.create_dir('hello')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pyarrow\_fs.pyx", line 593, in pyarrow._fs.FileSystem.create_dir
File "pyarrow\error.pxi", line 113, in pyarrow.lib.check_status
PermissionError: [Errno 13] google::cloud::Status(UNAUTHENTICATED: Permanent error GetBucketMetadata: Could not create a OAuth2 access token to authenticate the request. The request was not sent, as such an access token is required to complete the request successfully. Learn more about Google Cloud authentication at https://cloud.google.com/docs/authentication. The underlying error message was: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.). Detail: [errno 13] Permission denied

Are there any user-facing changes?
No but Windows wheels should contain ARROW_GCS
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #35193has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting committer review
Awaiting committer review
label
Apr 20, 2023
@raulcd
raulcd
commented
Apr 20, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp311-amd64
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

Revision: 5246fc5be2ed942a5688b29a05700c0673d820d2
Submitted crossbow builds: ursacomputing/crossbow @ actions-43da90de90
TaskStatus
wheel-windows-cp311-amd64Github Actions
@raulcd
raulcd
commented
Apr 20, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp311-amd64
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

Command '['git', 'clone', 'https://github.com/ursacomputing/crossbow', '/tmp/tmptzeyqnko/crossbow']' returned non-zero exit status 128.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/4756107770

@raulcd
raulcd
commented
Apr 20, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp311-amd64
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

Revision: a68e3737990ccd7b7bc3d41236f5156c139c1759
Submitted crossbow builds: ursacomputing/crossbow @ actions-e67a78785a
TaskStatus
wheel-windows-cp311-amd64Github Actions
@raulcd
raulcd
commented
Apr 27, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp39-amd64
@github-actions
github-actionsBot
commented
Apr 27, 2023

Copy link


Copy Markdown

Revision: a68e3737990ccd7b7bc3d41236f5156c139c1759
Submitted crossbow builds: ursacomputing/crossbow @ actions-cd5876c7ff
TaskStatus
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
Apr 27, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp39-amd64
@github-actions
github-actionsBot
commented
Apr 27, 2023

Copy link


Copy Markdown

Revision: cb95927568e66d1be2044ff73e00fbe98ddf0318
Submitted crossbow builds: ursacomputing/crossbow @ actions-bb930e9684
TaskStatus
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
Apr 27, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@raulcd
raulcd
marked this pull request as ready for review
April 27, 2023 14:44
@raulcdraulcd
changed the title
WIP: GH-35193: [Python][Packaging] Enable GCS on Windows wheelsGH-35193: [Python][Packaging] Enable GCS on Windows wheelsApr 27, 2023
@github-actions
github-actionsBot
commented
Apr 27, 2023

Copy link


Copy Markdown

Revision: 8b0ecc226649fb165e89e39b81a859290f223af7
Submitted crossbow builds: ursacomputing/crossbow @ actions-af464cb5b5
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 10, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 10, 2023

Copy link


Copy Markdown

Revision: b2b891a
Submitted crossbow builds: ursacomputing/crossbow @ actions-5e23d788f1
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 11, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 11, 2023

Copy link


Copy Markdown

Revision: 50efe4e
Submitted crossbow builds: ursacomputing/crossbow @ actions-20194149b9
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 11, 2023

Copy link


Copy Markdown

MemberAuthor
@westonpace while enabling GCS for the Windows wheels I realised Appveyor wasn't really adding ARROW_GCS=ON and there is a failure on the arrow-gcsfs-test on Appveyor if I enable it, example:
[ FAILED ] GcsIntegrationTest.CreateDirRecursiveFolderOnly (0 ms)
[ RUN ] GcsIntegrationTest.CreateDirRecursiveBucketAndFolder
C:/projects/arrow/cpp/src/arrow/filesystem/gcsfs_test.cc(170): error: Expected equality of these values:
Testbench()->error()
Which is: "Could not start GCS emulator. Used the following list of python interpreter names: 3.10 (exe not found)"
""
[ FAILED ] GcsIntegrationTest.CreateDirRecursiveBucketAndFolder (0 ms)
[ RUN ] GcsIntegrationTest.CreateDirUri

I can move fixing Appveyor to it's own issue/PR as it's slightly different from enabling GCS for the Windows wheels, which as we can see on the wheels jobs is successful. Do you think Appveyor should be fixed here? Any idea what the issue/fix is?
@westonpace
westonpace
commented
May 11, 2023

Copy link


Copy Markdown

Member
Do you think Appveyor should be fixed here? Any idea what the issue/fix is?
It would be nice to have at least one Windows CI job that uses GCS. I don't know if the others do or not.
Any idea what the issue/fix is?
Testbench is a standalone python project that emulates GCS. This is very similar to how minio emulates S3. The C++ unit tests launch testbench (using boost::process). Since testbench is a python project the C++ tests need to be able to locate and run python. It appears, on Windows, this is not working.
Here is what the code to look for python looks like today:
 std::vector<std::string> names{"python3", "python"};
// If the build script or application developer provides a value in the PYTHON
// environment variable, then just use that.
if (const auto* env = std::getenv("PYTHON")) {
names = {env};
}
auto error = std::string(
"Could not start GCS emulator."
" Used the following list of python interpreter names:");
for (const auto& interpreter : names) {
auto exe_path = bp::search_path(interpreter);
error += " " + interpreter;
if (exe_path.empty()) {
error += " (exe not found)";
continue;
}

So maybe it's as easy as setting the PYTHON environment variable to point to a valid python executable?
@raulcd
raulcd
commented
May 12, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 12, 2023

Copy link


Copy Markdown

Revision: e1d51a8
Submitted crossbow builds: ursacomputing/crossbow @ actions-20cbf52fd6
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 12, 2023

Copy link


Copy Markdown

MemberAuthor
Thanks @westonpace, this should be ready for review now
@raulcd
raulcd
requested a review
from westonpaceMay 12, 2023 11:19
kou
kou
reviewed
May 12, 2023
View reviewed changes
Comment threadci/scripts/install_gcs_testbench.bat
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting committer review
Awaiting committer review
labels
May 12, 2023
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
awaiting changes
Awaiting changes
and removed

awaiting changes
Awaiting changes
awaiting change review
Awaiting change review
labels
May 12, 2023
@kou
kou
commented
May 12, 2023

Copy link


Copy Markdown

Member
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 12, 2023

Copy link


Copy Markdown

Revision: cf41c23
Submitted crossbow builds: ursacomputing/crossbow @ actions-fe9442d7db
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
kou
kou
approved these changes
May 12, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting changes
Awaiting changes
labels
May 12, 2023
@assignUser
assignUser
merged commit 8be70c1
into
apache:mainMay 16, 2023
@ursabot
ursabot
commented
May 16, 2023

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 2d76d9a and contender = 8be70c1. 8be70c1 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Finished ⬇️0.81% ⬆️0.0%] test-mac-arm

[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.15% ⬆️0.03%] ursa-thinkcentre-m75q

Buildkite builds:

[Finished] 8be70c13 ec2-t3-xlarge-us-east-2

[Finished] 8be70c13 test-mac-arm

[Finished] 8be70c13 ursa-i9-9960x

[Finished] 8be70c13 ursa-thinkcentre-m75q

[Finished] 2d76d9a5 ec2-t3-xlarge-us-east-2

[Finished] 2d76d9a5 test-mac-arm

[Finished] 2d76d9a5 ursa-i9-9960x

[Finished] 2d76d9a5 ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True

test-mac-arm: Supported benchmark langs: C++, Python, R

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@westonpace
westonpace
commented
May 18, 2023

Copy link


Copy Markdown

Member
Thanks @raulcd for enabling this!
@koukou
mentioned this pull request
Sep 29, 2023

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@westonpacewestonpaceAwaiting requested review from westonpace

Assignees


No one assigned


Labels

awaiting mergeAwaiting merge

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[Python] Windows wheel is built without GCS support


5 participants

@raulcd@westonpace@kou@ursabot@assignUser

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GH-35193: [Python][Packaging] Enable GCS on Windows wheels by raulcd · Pull Request #35255 · apache/arrow · GitHub
Skip to content
GH-35193: [Python][Packaging] Enable GCS on Windows wheels - #35255
Merged
assignUser merged 8 commits into
apache:mainfrom 
raulcd:GH-35193
May 16, 2023
Merged
GH-35193: [Python][Packaging] Enable GCS on Windows wheels#35255
assignUser merged 8 commits into
apache:mainfrom 
raulcd:GH-35193
Conversation
@raulcd
@raulcdraulcd
commented
Apr 20, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Rationale for this change
GCS could be enabled on Windows wheels.
What changes are included in this PR?
Enabling GCS on Windows wheel
Are these changes tested?
Crossbow jobs for wheels run tests for GCS now.

I have tested locally that I can install the built wheel and I can import GcsFileSystem:
Python 3.9.12 (main, Apr 4 2022, 05:22:27) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Warning:
This Python interpreter is in a conda environment, but the environment has
not been activated. Libraries may fail to load. To activate this environment
please see https://conda.io/activation
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyarrow.fs import GcsFileSystem
>>> fs = GcsFileSystem(access_token='abc',target_service_account='service_account@apache',credential_token_expiration=datetime.now(),default_bucket_location='us-west2',scheme='https', endpoint_override='localhost:8999')
>>> fs.default_bucket_location
'us-west2'
>>> fs.create_dir('hello')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pyarrow\_fs.pyx", line 593, in pyarrow._fs.FileSystem.create_dir
File "pyarrow\error.pxi", line 113, in pyarrow.lib.check_status
PermissionError: [Errno 13] google::cloud::Status(UNAUTHENTICATED: Permanent error GetBucketMetadata: Could not create a OAuth2 access token to authenticate the request. The request was not sent, as such an access token is required to complete the request successfully. Learn more about Google Cloud authentication at https://cloud.google.com/docs/authentication. The underlying error message was: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.). Detail: [errno 13] Permission denied

Are there any user-facing changes?
No but Windows wheels should contain ARROW_GCS
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #35193has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting committer review
Awaiting committer review
label
Apr 20, 2023
@raulcd
raulcd
commented
Apr 20, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp311-amd64
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

Revision: 5246fc5be2ed942a5688b29a05700c0673d820d2
Submitted crossbow builds: ursacomputing/crossbow @ actions-43da90de90
TaskStatus
wheel-windows-cp311-amd64Github Actions
@raulcd
raulcd
commented
Apr 20, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp311-amd64
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

Command '['git', 'clone', 'https://github.com/ursacomputing/crossbow', '/tmp/tmptzeyqnko/crossbow']' returned non-zero exit status 128.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/4756107770

@raulcd
raulcd
commented
Apr 20, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp311-amd64
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

Revision: a68e3737990ccd7b7bc3d41236f5156c139c1759
Submitted crossbow builds: ursacomputing/crossbow @ actions-e67a78785a
TaskStatus
wheel-windows-cp311-amd64Github Actions
@raulcd
raulcd
commented
Apr 27, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp39-amd64
@github-actions
github-actionsBot
commented
Apr 27, 2023

Copy link


Copy Markdown

Revision: a68e3737990ccd7b7bc3d41236f5156c139c1759
Submitted crossbow builds: ursacomputing/crossbow @ actions-cd5876c7ff
TaskStatus
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
Apr 27, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp39-amd64
@github-actions
github-actionsBot
commented
Apr 27, 2023

Copy link


Copy Markdown

Revision: cb95927568e66d1be2044ff73e00fbe98ddf0318
Submitted crossbow builds: ursacomputing/crossbow @ actions-bb930e9684
TaskStatus
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
Apr 27, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@raulcd
raulcd
marked this pull request as ready for review
April 27, 2023 14:44
@raulcdraulcd
changed the title
WIP: GH-35193: [Python][Packaging] Enable GCS on Windows wheelsGH-35193: [Python][Packaging] Enable GCS on Windows wheelsApr 27, 2023
@github-actions
github-actionsBot
commented
Apr 27, 2023

Copy link


Copy Markdown

Revision: 8b0ecc226649fb165e89e39b81a859290f223af7
Submitted crossbow builds: ursacomputing/crossbow @ actions-af464cb5b5
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 10, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 10, 2023

Copy link


Copy Markdown

Revision: b2b891a
Submitted crossbow builds: ursacomputing/crossbow @ actions-5e23d788f1
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 11, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 11, 2023

Copy link


Copy Markdown

Revision: 50efe4e
Submitted crossbow builds: ursacomputing/crossbow @ actions-20194149b9
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 11, 2023

Copy link


Copy Markdown

MemberAuthor
@westonpace while enabling GCS for the Windows wheels I realised Appveyor wasn't really adding ARROW_GCS=ON and there is a failure on the arrow-gcsfs-test on Appveyor if I enable it, example:
[ FAILED ] GcsIntegrationTest.CreateDirRecursiveFolderOnly (0 ms)
[ RUN ] GcsIntegrationTest.CreateDirRecursiveBucketAndFolder
C:/projects/arrow/cpp/src/arrow/filesystem/gcsfs_test.cc(170): error: Expected equality of these values:
Testbench()->error()
Which is: "Could not start GCS emulator. Used the following list of python interpreter names: 3.10 (exe not found)"
""
[ FAILED ] GcsIntegrationTest.CreateDirRecursiveBucketAndFolder (0 ms)
[ RUN ] GcsIntegrationTest.CreateDirUri

I can move fixing Appveyor to it's own issue/PR as it's slightly different from enabling GCS for the Windows wheels, which as we can see on the wheels jobs is successful. Do you think Appveyor should be fixed here? Any idea what the issue/fix is?
@westonpace
westonpace
commented
May 11, 2023

Copy link


Copy Markdown

Member
Do you think Appveyor should be fixed here? Any idea what the issue/fix is?
It would be nice to have at least one Windows CI job that uses GCS. I don't know if the others do or not.
Any idea what the issue/fix is?
Testbench is a standalone python project that emulates GCS. This is very similar to how minio emulates S3. The C++ unit tests launch testbench (using boost::process). Since testbench is a python project the C++ tests need to be able to locate and run python. It appears, on Windows, this is not working.
Here is what the code to look for python looks like today:
 std::vector<std::string> names{"python3", "python"};
// If the build script or application developer provides a value in the PYTHON
// environment variable, then just use that.
if (const auto* env = std::getenv("PYTHON")) {
names = {env};
}
auto error = std::string(
"Could not start GCS emulator."
" Used the following list of python interpreter names:");
for (const auto& interpreter : names) {
auto exe_path = bp::search_path(interpreter);
error += " " + interpreter;
if (exe_path.empty()) {
error += " (exe not found)";
continue;
}

So maybe it's as easy as setting the PYTHON environment variable to point to a valid python executable?
@raulcd
raulcd
commented
May 12, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 12, 2023

Copy link


Copy Markdown

Revision: e1d51a8
Submitted crossbow builds: ursacomputing/crossbow @ actions-20cbf52fd6
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 12, 2023

Copy link


Copy Markdown

MemberAuthor
Thanks @westonpace, this should be ready for review now
@raulcd
raulcd
requested a review
from westonpaceMay 12, 2023 11:19
kou
kou
reviewed
May 12, 2023
View reviewed changes
Comment threadci/scripts/install_gcs_testbench.bat
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting committer review
Awaiting committer review
labels
May 12, 2023
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
awaiting changes
Awaiting changes
and removed

awaiting changes
Awaiting changes
awaiting change review
Awaiting change review
labels
May 12, 2023
@kou
kou
commented
May 12, 2023

Copy link


Copy Markdown

Member
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 12, 2023

Copy link


Copy Markdown

Revision: cf41c23
Submitted crossbow builds: ursacomputing/crossbow @ actions-fe9442d7db
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
kou
kou
approved these changes
May 12, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting changes
Awaiting changes
labels
May 12, 2023
@assignUser
assignUser
merged commit 8be70c1
into
apache:mainMay 16, 2023
@ursabot
ursabot
commented
May 16, 2023

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 2d76d9a and contender = 8be70c1. 8be70c1 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Finished ⬇️0.81% ⬆️0.0%] test-mac-arm

[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.15% ⬆️0.03%] ursa-thinkcentre-m75q

Buildkite builds:

[Finished] 8be70c13 ec2-t3-xlarge-us-east-2

[Finished] 8be70c13 test-mac-arm

[Finished] 8be70c13 ursa-i9-9960x

[Finished] 8be70c13 ursa-thinkcentre-m75q

[Finished] 2d76d9a5 ec2-t3-xlarge-us-east-2

[Finished] 2d76d9a5 test-mac-arm

[Finished] 2d76d9a5 ursa-i9-9960x

[Finished] 2d76d9a5 ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True

test-mac-arm: Supported benchmark langs: C++, Python, R

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@westonpace
westonpace
commented
May 18, 2023

Copy link


Copy Markdown

Member
Thanks @raulcd for enabling this!
@koukou
mentioned this pull request
Sep 29, 2023

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@westonpacewestonpaceAwaiting requested review from westonpace

Assignees


No one assigned


Labels

awaiting mergeAwaiting merge

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[Python] Windows wheel is built without GCS support


5 participants

@raulcd@westonpace@kou@ursabot@assignUser

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GH-35193: [Python][Packaging] Enable GCS on Windows wheels by raulcd · Pull Request #35255 · apache/arrow · GitHub
Skip to content
GH-35193: [Python][Packaging] Enable GCS on Windows wheels - #35255
Merged
assignUser merged 8 commits into
apache:mainfrom 
raulcd:GH-35193
May 16, 2023
Merged
GH-35193: [Python][Packaging] Enable GCS on Windows wheels#35255
assignUser merged 8 commits into
apache:mainfrom 
raulcd:GH-35193
Conversation
@raulcd
@raulcdraulcd
commented
Apr 20, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Rationale for this change
GCS could be enabled on Windows wheels.
What changes are included in this PR?
Enabling GCS on Windows wheel
Are these changes tested?
Crossbow jobs for wheels run tests for GCS now.

I have tested locally that I can install the built wheel and I can import GcsFileSystem:
Python 3.9.12 (main, Apr 4 2022, 05:22:27) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Warning:
This Python interpreter is in a conda environment, but the environment has
not been activated. Libraries may fail to load. To activate this environment
please see https://conda.io/activation
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyarrow.fs import GcsFileSystem
>>> fs = GcsFileSystem(access_token='abc',target_service_account='service_account@apache',credential_token_expiration=datetime.now(),default_bucket_location='us-west2',scheme='https', endpoint_override='localhost:8999')
>>> fs.default_bucket_location
'us-west2'
>>> fs.create_dir('hello')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pyarrow\_fs.pyx", line 593, in pyarrow._fs.FileSystem.create_dir
File "pyarrow\error.pxi", line 113, in pyarrow.lib.check_status
PermissionError: [Errno 13] google::cloud::Status(UNAUTHENTICATED: Permanent error GetBucketMetadata: Could not create a OAuth2 access token to authenticate the request. The request was not sent, as such an access token is required to complete the request successfully. Learn more about Google Cloud authentication at https://cloud.google.com/docs/authentication. The underlying error message was: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.). Detail: [errno 13] Permission denied

Are there any user-facing changes?
No but Windows wheels should contain ARROW_GCS
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #35193has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting committer review
Awaiting committer review
label
Apr 20, 2023
@raulcd
raulcd
commented
Apr 20, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp311-amd64
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

Revision: 5246fc5be2ed942a5688b29a05700c0673d820d2
Submitted crossbow builds: ursacomputing/crossbow @ actions-43da90de90
TaskStatus
wheel-windows-cp311-amd64Github Actions
@raulcd
raulcd
commented
Apr 20, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp311-amd64
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

Command '['git', 'clone', 'https://github.com/ursacomputing/crossbow', '/tmp/tmptzeyqnko/crossbow']' returned non-zero exit status 128.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/4756107770

@raulcd
raulcd
commented
Apr 20, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp311-amd64
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

Revision: a68e3737990ccd7b7bc3d41236f5156c139c1759
Submitted crossbow builds: ursacomputing/crossbow @ actions-e67a78785a
TaskStatus
wheel-windows-cp311-amd64Github Actions
@raulcd
raulcd
commented
Apr 27, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp39-amd64
@github-actions
github-actionsBot
commented
Apr 27, 2023

Copy link


Copy Markdown

Revision: a68e3737990ccd7b7bc3d41236f5156c139c1759
Submitted crossbow builds: ursacomputing/crossbow @ actions-cd5876c7ff
TaskStatus
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
Apr 27, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp39-amd64
@github-actions
github-actionsBot
commented
Apr 27, 2023

Copy link


Copy Markdown

Revision: cb95927568e66d1be2044ff73e00fbe98ddf0318
Submitted crossbow builds: ursacomputing/crossbow @ actions-bb930e9684
TaskStatus
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
Apr 27, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@raulcd
raulcd
marked this pull request as ready for review
April 27, 2023 14:44
@raulcdraulcd
changed the title
WIP: GH-35193: [Python][Packaging] Enable GCS on Windows wheelsGH-35193: [Python][Packaging] Enable GCS on Windows wheelsApr 27, 2023
@github-actions
github-actionsBot
commented
Apr 27, 2023

Copy link


Copy Markdown

Revision: 8b0ecc226649fb165e89e39b81a859290f223af7
Submitted crossbow builds: ursacomputing/crossbow @ actions-af464cb5b5
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 10, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 10, 2023

Copy link


Copy Markdown

Revision: b2b891a
Submitted crossbow builds: ursacomputing/crossbow @ actions-5e23d788f1
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 11, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 11, 2023

Copy link


Copy Markdown

Revision: 50efe4e
Submitted crossbow builds: ursacomputing/crossbow @ actions-20194149b9
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 11, 2023

Copy link


Copy Markdown

MemberAuthor
@westonpace while enabling GCS for the Windows wheels I realised Appveyor wasn't really adding ARROW_GCS=ON and there is a failure on the arrow-gcsfs-test on Appveyor if I enable it, example:
[ FAILED ] GcsIntegrationTest.CreateDirRecursiveFolderOnly (0 ms)
[ RUN ] GcsIntegrationTest.CreateDirRecursiveBucketAndFolder
C:/projects/arrow/cpp/src/arrow/filesystem/gcsfs_test.cc(170): error: Expected equality of these values:
Testbench()->error()
Which is: "Could not start GCS emulator. Used the following list of python interpreter names: 3.10 (exe not found)"
""
[ FAILED ] GcsIntegrationTest.CreateDirRecursiveBucketAndFolder (0 ms)
[ RUN ] GcsIntegrationTest.CreateDirUri

I can move fixing Appveyor to it's own issue/PR as it's slightly different from enabling GCS for the Windows wheels, which as we can see on the wheels jobs is successful. Do you think Appveyor should be fixed here? Any idea what the issue/fix is?
@westonpace
westonpace
commented
May 11, 2023

Copy link


Copy Markdown

Member
Do you think Appveyor should be fixed here? Any idea what the issue/fix is?
It would be nice to have at least one Windows CI job that uses GCS. I don't know if the others do or not.
Any idea what the issue/fix is?
Testbench is a standalone python project that emulates GCS. This is very similar to how minio emulates S3. The C++ unit tests launch testbench (using boost::process). Since testbench is a python project the C++ tests need to be able to locate and run python. It appears, on Windows, this is not working.
Here is what the code to look for python looks like today:
 std::vector<std::string> names{"python3", "python"};
// If the build script or application developer provides a value in the PYTHON
// environment variable, then just use that.
if (const auto* env = std::getenv("PYTHON")) {
names = {env};
}
auto error = std::string(
"Could not start GCS emulator."
" Used the following list of python interpreter names:");
for (const auto& interpreter : names) {
auto exe_path = bp::search_path(interpreter);
error += " " + interpreter;
if (exe_path.empty()) {
error += " (exe not found)";
continue;
}

So maybe it's as easy as setting the PYTHON environment variable to point to a valid python executable?
@raulcd
raulcd
commented
May 12, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 12, 2023

Copy link


Copy Markdown

Revision: e1d51a8
Submitted crossbow builds: ursacomputing/crossbow @ actions-20cbf52fd6
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 12, 2023

Copy link


Copy Markdown

MemberAuthor
Thanks @westonpace, this should be ready for review now
@raulcd
raulcd
requested a review
from westonpaceMay 12, 2023 11:19
kou
kou
reviewed
May 12, 2023
View reviewed changes
Comment threadci/scripts/install_gcs_testbench.bat
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting committer review
Awaiting committer review
labels
May 12, 2023
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
awaiting changes
Awaiting changes
and removed

awaiting changes
Awaiting changes
awaiting change review
Awaiting change review
labels
May 12, 2023
@kou
kou
commented
May 12, 2023

Copy link


Copy Markdown

Member
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 12, 2023

Copy link


Copy Markdown

Revision: cf41c23
Submitted crossbow builds: ursacomputing/crossbow @ actions-fe9442d7db
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
kou
kou
approved these changes
May 12, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting changes
Awaiting changes
labels
May 12, 2023
@assignUser
assignUser
merged commit 8be70c1
into
apache:mainMay 16, 2023
@ursabot
ursabot
commented
May 16, 2023

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 2d76d9a and contender = 8be70c1. 8be70c1 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Finished ⬇️0.81% ⬆️0.0%] test-mac-arm

[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.15% ⬆️0.03%] ursa-thinkcentre-m75q

Buildkite builds:

[Finished] 8be70c13 ec2-t3-xlarge-us-east-2

[Finished] 8be70c13 test-mac-arm

[Finished] 8be70c13 ursa-i9-9960x

[Finished] 8be70c13 ursa-thinkcentre-m75q

[Finished] 2d76d9a5 ec2-t3-xlarge-us-east-2

[Finished] 2d76d9a5 test-mac-arm

[Finished] 2d76d9a5 ursa-i9-9960x

[Finished] 2d76d9a5 ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True

test-mac-arm: Supported benchmark langs: C++, Python, R

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@westonpace
westonpace
commented
May 18, 2023

Copy link


Copy Markdown

Member
Thanks @raulcd for enabling this!
@koukou
mentioned this pull request
Sep 29, 2023

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@westonpacewestonpaceAwaiting requested review from westonpace

Assignees


No one assigned


Labels

awaiting mergeAwaiting merge

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[Python] Windows wheel is built without GCS support


5 participants

@raulcd@westonpace@kou@ursabot@assignUser

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
GH-35193: [Python][Packaging] Enable GCS on Windows wheels by raulcd · Pull Request #35255 · apache/arrow · GitHub
Skip to content
GH-35193: [Python][Packaging] Enable GCS on Windows wheels - #35255
Merged
assignUser merged 8 commits into
apache:mainfrom 
raulcd:GH-35193
May 16, 2023
Merged
GH-35193: [Python][Packaging] Enable GCS on Windows wheels#35255
assignUser merged 8 commits into
apache:mainfrom 
raulcd:GH-35193
Conversation
@raulcd
@raulcdraulcd
commented
Apr 20, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Rationale for this change
GCS could be enabled on Windows wheels.
What changes are included in this PR?
Enabling GCS on Windows wheel
Are these changes tested?
Crossbow jobs for wheels run tests for GCS now.

I have tested locally that I can install the built wheel and I can import GcsFileSystem:
Python 3.9.12 (main, Apr 4 2022, 05:22:27) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Warning:
This Python interpreter is in a conda environment, but the environment has
not been activated. Libraries may fail to load. To activate this environment
please see https://conda.io/activation
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyarrow.fs import GcsFileSystem
>>> fs = GcsFileSystem(access_token='abc',target_service_account='service_account@apache',credential_token_expiration=datetime.now(),default_bucket_location='us-west2',scheme='https', endpoint_override='localhost:8999')
>>> fs.default_bucket_location
'us-west2'
>>> fs.create_dir('hello')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "pyarrow\_fs.pyx", line 593, in pyarrow._fs.FileSystem.create_dir
File "pyarrow\error.pxi", line 113, in pyarrow.lib.check_status
PermissionError: [Errno 13] google::cloud::Status(UNAUTHENTICATED: Permanent error GetBucketMetadata: Could not create a OAuth2 access token to authenticate the request. The request was not sent, as such an access token is required to complete the request successfully. Learn more about Google Cloud authentication at https://cloud.google.com/docs/authentication. The underlying error message was: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.). Detail: [errno 13] Permission denied

Are there any user-facing changes?
No but Windows wheels should contain ARROW_GCS
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

⚠️ GitHub issue #35193has been automatically assigned in GitHub to PR creator.
@github-actionsgithub-actionsBot
added
the 
awaiting committer review
Awaiting committer review
label
Apr 20, 2023
@raulcd
raulcd
commented
Apr 20, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp311-amd64
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

Revision: 5246fc5be2ed942a5688b29a05700c0673d820d2
Submitted crossbow builds: ursacomputing/crossbow @ actions-43da90de90
TaskStatus
wheel-windows-cp311-amd64Github Actions
@raulcd
raulcd
commented
Apr 20, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp311-amd64
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

Command '['git', 'clone', 'https://github.com/ursacomputing/crossbow', '/tmp/tmptzeyqnko/crossbow']' returned non-zero exit status 128.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/4756107770

@raulcd
raulcd
commented
Apr 20, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp311-amd64
@github-actions
github-actionsBot
commented
Apr 20, 2023

Copy link


Copy Markdown

Revision: a68e3737990ccd7b7bc3d41236f5156c139c1759
Submitted crossbow builds: ursacomputing/crossbow @ actions-e67a78785a
TaskStatus
wheel-windows-cp311-amd64Github Actions
@raulcd
raulcd
commented
Apr 27, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp39-amd64
@github-actions
github-actionsBot
commented
Apr 27, 2023

Copy link


Copy Markdown

Revision: a68e3737990ccd7b7bc3d41236f5156c139c1759
Submitted crossbow builds: ursacomputing/crossbow @ actions-cd5876c7ff
TaskStatus
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
Apr 27, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-cp39-amd64
@github-actions
github-actionsBot
commented
Apr 27, 2023

Copy link


Copy Markdown

Revision: cb95927568e66d1be2044ff73e00fbe98ddf0318
Submitted crossbow builds: ursacomputing/crossbow @ actions-bb930e9684
TaskStatus
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
Apr 27, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@raulcd
raulcd
marked this pull request as ready for review
April 27, 2023 14:44
@raulcdraulcd
changed the title
WIP: GH-35193: [Python][Packaging] Enable GCS on Windows wheelsGH-35193: [Python][Packaging] Enable GCS on Windows wheelsApr 27, 2023
@github-actions
github-actionsBot
commented
Apr 27, 2023

Copy link


Copy Markdown

Revision: 8b0ecc226649fb165e89e39b81a859290f223af7
Submitted crossbow builds: ursacomputing/crossbow @ actions-af464cb5b5
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 10, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 10, 2023

Copy link


Copy Markdown

Revision: b2b891a
Submitted crossbow builds: ursacomputing/crossbow @ actions-5e23d788f1
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 11, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 11, 2023

Copy link


Copy Markdown

Revision: 50efe4e
Submitted crossbow builds: ursacomputing/crossbow @ actions-20194149b9
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 11, 2023

Copy link


Copy Markdown

MemberAuthor
@westonpace while enabling GCS for the Windows wheels I realised Appveyor wasn't really adding ARROW_GCS=ON and there is a failure on the arrow-gcsfs-test on Appveyor if I enable it, example:
[ FAILED ] GcsIntegrationTest.CreateDirRecursiveFolderOnly (0 ms)
[ RUN ] GcsIntegrationTest.CreateDirRecursiveBucketAndFolder
C:/projects/arrow/cpp/src/arrow/filesystem/gcsfs_test.cc(170): error: Expected equality of these values:
Testbench()->error()
Which is: "Could not start GCS emulator. Used the following list of python interpreter names: 3.10 (exe not found)"
""
[ FAILED ] GcsIntegrationTest.CreateDirRecursiveBucketAndFolder (0 ms)
[ RUN ] GcsIntegrationTest.CreateDirUri

I can move fixing Appveyor to it's own issue/PR as it's slightly different from enabling GCS for the Windows wheels, which as we can see on the wheels jobs is successful. Do you think Appveyor should be fixed here? Any idea what the issue/fix is?
@westonpace
westonpace
commented
May 11, 2023

Copy link


Copy Markdown

Member
Do you think Appveyor should be fixed here? Any idea what the issue/fix is?
It would be nice to have at least one Windows CI job that uses GCS. I don't know if the others do or not.
Any idea what the issue/fix is?
Testbench is a standalone python project that emulates GCS. This is very similar to how minio emulates S3. The C++ unit tests launch testbench (using boost::process). Since testbench is a python project the C++ tests need to be able to locate and run python. It appears, on Windows, this is not working.
Here is what the code to look for python looks like today:
 std::vector<std::string> names{"python3", "python"};
// If the build script or application developer provides a value in the PYTHON
// environment variable, then just use that.
if (const auto* env = std::getenv("PYTHON")) {
names = {env};
}
auto error = std::string(
"Could not start GCS emulator."
" Used the following list of python interpreter names:");
for (const auto& interpreter : names) {
auto exe_path = bp::search_path(interpreter);
error += " " + interpreter;
if (exe_path.empty()) {
error += " (exe not found)";
continue;
}

So maybe it's as easy as setting the PYTHON environment variable to point to a valid python executable?
@raulcd
raulcd
commented
May 12, 2023

Copy link


Copy Markdown

MemberAuthor
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 12, 2023

Copy link


Copy Markdown

Revision: e1d51a8
Submitted crossbow builds: ursacomputing/crossbow @ actions-20cbf52fd6
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
@raulcd
raulcd
commented
May 12, 2023

Copy link


Copy Markdown

MemberAuthor
Thanks @westonpace, this should be ready for review now
@raulcd
raulcd
requested a review
from westonpaceMay 12, 2023 11:19
kou
kou
reviewed
May 12, 2023
View reviewed changes
Comment threadci/scripts/install_gcs_testbench.bat
Outdated

@github-actionsgithub-actionsBot
added

awaiting changes
Awaiting changes
and removed

awaiting committer review
Awaiting committer review
labels
May 12, 2023
@github-actionsgithub-actionsBot
added

awaiting change review
Awaiting change review
awaiting changes
Awaiting changes
and removed

awaiting changes
Awaiting changes
awaiting change review
Awaiting change review
labels
May 12, 2023
@kou
kou
commented
May 12, 2023

Copy link


Copy Markdown

Member
@github-actions crossbow submit wheel-windows-*
@github-actions
github-actionsBot
commented
May 12, 2023

Copy link


Copy Markdown

Revision: cf41c23
Submitted crossbow builds: ursacomputing/crossbow @ actions-fe9442d7db
TaskStatus
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp311-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions
kou
kou
approved these changes
May 12, 2023
View reviewed changes
@koukou
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
@github-actionsgithub-actionsBot
added

awaiting merge
Awaiting merge
and removed

awaiting changes
Awaiting changes
labels
May 12, 2023
@assignUser
assignUser
merged commit 8be70c1
into
apache:mainMay 16, 2023
@ursabot
ursabot
commented
May 16, 2023

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 2d76d9a and contender = 8be70c1. 8be70c1 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Finished ⬇️0.81% ⬆️0.0%] test-mac-arm

[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.15% ⬆️0.03%] ursa-thinkcentre-m75q

Buildkite builds:

[Finished] 8be70c13 ec2-t3-xlarge-us-east-2

[Finished] 8be70c13 test-mac-arm

[Finished] 8be70c13 ursa-i9-9960x

[Finished] 8be70c13 ursa-thinkcentre-m75q

[Finished] 2d76d9a5 ec2-t3-xlarge-us-east-2

[Finished] 2d76d9a5 test-mac-arm

[Finished] 2d76d9a5 ursa-i9-9960x

[Finished] 2d76d9a5 ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True

test-mac-arm: Supported benchmark langs: C++, Python, R

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@westonpace
westonpace
commented
May 18, 2023

Copy link


Copy Markdown

Member
Thanks @raulcd for enabling this!
@koukou
mentioned this pull request
Sep 29, 2023

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@koukoukou approved these changes
@assignUserassignUserAwaiting requested review from assignUserassignUser is a code owner
@westonpacewestonpaceAwaiting requested review from westonpace

Assignees


No one assigned


Labels

awaiting mergeAwaiting merge

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

[Python] Windows wheel is built without GCS support


5 participants

@raulcd@westonpace@kou@ursabot@assignUser