Skip to content

ARROW-14892: [Python][C++] GCS Bindings - #12763

Merged
pitrou merged 85 commits into
apache:masterfrom
emkornfield:gcsp
Jun 12, 2022
Merged

ARROW-14892: [Python][C++] GCS Bindings#12763
pitrou merged 85 commits into
apache:masterfrom
emkornfield:gcsp

Conversation

@emkornfield

@emkornfieldemkornfield commented Mar 31, 2022

Copy link
Copy Markdown
Contributor

Incorporate GCS file system into python and other bug fixes.

Bugs/Other changes:

  • Add GCS bindings mostly based on AWS bindings in Python and associated unit tests
  • Tell was incorrect, it double counted when the stream was constructed with an offset.
  • Missed setting the define in config.cmake which means FileSystemFromUri was never tested and didn't compile this is now fixed
  • Refine logic for GetFileInfo with a single path to recognize prefixes followed by a slash as a directory. This allows datasets to work as expected with a toy dataset generated on local-filesystem and copied to the cloud (I believe this is typical of how other systems write to GCS as well.
  • Switch convention for creating directories to always end in "/" and make use of this as another indicator. From testing with a sample iceberg table it appears this is the convention used for hive-partitioning, so I assume this is common practice for other Hive related writers (i.e. what we want to support).
  • Fix bug introduced in a5e45ce which caused failures when a deletion occurred on a bucket (not an object in the bucket).
  • Ensure output streams are closed on destruction (this is consistent with S3)

@emkornfield
emkornfield marked this pull request as draft March 31, 2022 08:26
@emkornfield

Copy link
Copy Markdown
ContributorAuthor

@pitrou@kszucs this still isn't ready for review but I have it compiling locally. I was wondering if there are cross-bow or other actions I should be taking to verify packaging?

@github-actions

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

@emkornfield
emkornfield marked this pull request as ready for review April 2, 2022 06:23
@emkornfield

Copy link
Copy Markdown
ContributorAuthor

CC @coryan for C++ changes.

Comment threadcpp/src/arrow/filesystem/gcsfs.cc Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assume offset into this function is 1000. Without the offset parameter passed to GcsInputStream its Tell() function will return 0 when you are in fact reading byte 1000. That seems like the wrong semantics to me, but maybe it is the expected behavior?

@emkornfieldemkornfieldApr 2, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Empirically the underlying tell returns 1000. I was observing doubling of expected tell value. The python test that found this wrote N bytes then seeked to N/2 and tried reading. The reading called the FS tell which returned N which caused zeo bytes to be read

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked at the GCS code and it does appear to keep the ReadAt offset to return for tell but I might have missed something

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack. SGTM.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@emkornfield I'm curious this wasn't caught by the C++ tests. Is it possible to enhance the generic filesystem tests to cover this?

@emkornfieldemkornfieldApr 19, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, sorry for the tell case we can certainly add a test case. Filed ARROW-16226 to track this.

Comment threadcpp/src/arrow/filesystem/gcsfs.cc Outdated
@emkornfield

emkornfield commented Apr 4, 2022

Copy link
Copy Markdown
ContributorAuthor

when running docker locally the python tests seem to hang forever when attempting to reach GCS test_bench which explains the timeouts, not sure if this could be some sort of config missing for docker? (all tests pass when run without docker).

This could also be a test-bench versioning issue.

It turns out this was testbench not getting installed properly into the conda env.

@emkornfield

Copy link
Copy Markdown
ContributorAuthor

Also CC @jorisvandenbossche if you have time to look

@emkornfield

Copy link
Copy Markdown
ContributorAuthor

@github-actions crossbow submit -g python

@emkornfield

Copy link
Copy Markdown
ContributorAuthor

@github-actions crossbow submit -g wheel

@pitrou
pitrou self-requested a review April 13, 2022 17:00
@emkornfield

Copy link
Copy Markdown
ContributorAuthor

Ping @pitrou to see if you have time to review. Also @rok it seems you have been looking at GCS stuff recently.

@voutilad

Copy link
Copy Markdown

FWIW, @emkornfield I had to revert your last commit (608b6ecc7f1841762462f66d06ce880fd9bb02a2) in order to get tests to work. It seems if the testbench suite doesn't properly initialize, the GcsFileSystem tests hang until the ctest timeout. (This is using archery docker build ubuntu-cpp.)

@coryan

Copy link
Copy Markdown
Contributor

FWIW, @emkornfield I had to revert your last commit (608b6ec) in order to get tests to work. It seems if the testbench suite doesn't properly initialize,

You may be running into postmanlabs/httpbin#673 which we worked around in googleapis/storage-testbench#301 . The latest release should have these fixes.

@pitroupitrou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @emkornfield . You'll find a bunch of comments below.
Also, can you ensure you rebase on the latest git master?

Comment thread.github/workflows/cpp.yml Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@emkornfield Did you mean to add this? ARROW-16102 is fixed already.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, ARROW-16102 was fixed in between this was posted for review and it got reviewed. this will be removed in the rebase.

Comment threadci/scripts/python_test.sh Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this there? Is this PR actually ready? Did you perhaps forget to push some followup changes?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seemed like an ugly hack. I was hoping someone could point me to a better solution. I will try removing after the rebase to see if the tests now pass, as the prior install command (install_gcs_testbench.sh) has been updated slightly at HEAD

Comment threadcpp/src/arrow/filesystem/gcsfs.h Outdated
Comment threadcpp/src/arrow/filesystem/gcsfs.cc Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this constructor is necessary, just let C++ define it implicitly for you?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needed for std::make_shared

Comment threadcpp/src/arrow/filesystem/gcsfs.cc Outdated
Comment threadpython/pyarrow/_gcsfs.pyx Outdated
Comment threadpython/pyarrow/_gcsfs.pyx Outdated
Comment threadpython/pyarrow/_gcsfs.pyx Outdated
Comment threadpython/pyarrow/_gcsfs.pyx Outdated
Comment threadpython/pyarrow/tests/conftest.py Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why this message? Is "gcs test bench" an actual command?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

most copy and past from s3, I rephrased slightly.

@emkornfield

Copy link
Copy Markdown
ContributorAuthor

@pitrou thanks for the thorough review. I think I addressed most comments with the exception of the cleanup of directory logic, which I will try to address in a little bit (I left TODO place markers there). I've also rebased, and changed the destructor on OutputStream to close the file it is isn't closed.

@emkornfield

Copy link
Copy Markdown
ContributorAuthor

@pitrou I addressed the TODO's I left in there for simplification. Please let me know if the code is now easier to read, there was a bunch of superfluous code. I am still not sure why testbench needs to be installed twice.

@pitroupitrou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update @emkornfield !

Comment threadpython/pyarrow/tests/conftest.py Outdated
Comment threadpython/pyarrow/tests/conftest.py Outdated
Comment threadpython/pyarrow/_gcsfs.pyx Outdated
Comment threaddev/archery/archery/cli.py Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems the option is duplicated now? (I'm curious that click doesn't complain about it)

Comment threadcpp/src/arrow/filesystem/gcsfs.h Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you plan on solving this TODO here?

Comment threadcpp/src/arrow/filesystem/gcsfs.h Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note a default-constructed credentials could trivially be anonymous...

Comment threadci/scripts/python_test.sh Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left a comment about this in conftest.py below.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed this to check for the existence of the script and use the existing install comment. I tried multiple approaches with docker but could get the environment variable set.

Comment threadci/scripts/python_build.sh Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like this one is duplicate?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a test for FileSystemFromUri somewhere in this file?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a test to make sure it can instantiate the file system. there are detailed test for the FromUri implementations on GCS already.

Comment thread.github/workflows/cpp.yml Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you revert this change because ARROW_GCS is OFF by default in ci/scripts/cpp_build.sh?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

Comment threadcpp/src/arrow/util/config.h.cmake Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

Comment threaddev/tasks/tasks.yml Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a strong opinion for this but I like adding a new variable for gcs for readability:

{% for macos_version, macos_codename, arrow_s3, arrow_gcs in [("10.9", "mavericks", "OFF", "OFF"),
("10.13", "high-sierra", "ON", "ON")] %}

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

(I had a typo: "I have a strong opinion" -> "I don't have a strong opinion")

Comment threadcpp/src/arrow/filesystem/gcsfs.cc Outdated
@emkornfield

Copy link
Copy Markdown
ContributorAuthor

@github-actions crossbow submit -g wheel

@emkornfield

Copy link
Copy Markdown
ContributorAuthor

@pitrou and @kou I believe I addressed all feedback you provided on the last review (sorry I had to force push my branch). One sticking point seems to be GCS Test bench, CI/Crossbow is running now there might be a few other places it needs to be installed.

@pitrou

Copy link
Copy Markdown
Member

@github-actions crossbow submit wheel-macos-big-sur*

@github-actions

Copy link
Copy Markdown

Revision: 922e4ef

Submitted crossbow builds: ursacomputing/crossbow @ actions-2176483478

TaskStatus
wheel-macos-big-sur-cp310-arm64Github Actions
wheel-macos-big-sur-cp310-universal2Github Actions
wheel-macos-big-sur-cp38-arm64Github Actions
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions

@pitrou

Copy link
Copy Markdown
Member

Ok, the situation on our macOS wheel builds is a bit horrible, but it's pointless to try to improve it here, so I'll revert the last two commits.

@pitrou

Copy link
Copy Markdown
Member

@github-actions crossbow submit wheel-macos-*

@github-actions

Copy link
Copy Markdown

Revision: 0f1f1da

Submitted crossbow builds: ursacomputing/crossbow @ actions-041204a60b

TaskStatus
wheel-macos-big-sur-cp310-arm64Github Actions
wheel-macos-big-sur-cp310-universal2Github Actions
wheel-macos-big-sur-cp38-arm64Github Actions
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp310-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp310-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions

@pitrou

Copy link
Copy Markdown
Member

@github-actions crossbow submit -g nightly

@github-actions

Copy link
Copy Markdown

Revision: 0f1f1da

Submitted crossbow builds: ursacomputing/crossbow @ actions-8c3920b06e

TaskStatus
almalinux-8-amd64Github Actions
almalinux-8-arm64TravisCI
almalinux-9-amd64Github Actions
almalinux-9-arm64TravisCI
amazon-linux-2-amd64Github Actions
centos-7-amd64Github Actions
centos-8-stream-amd64Github Actions
centos-8-stream-arm64TravisCI
conan-maximumGithub Actions
conan-minimumGithub Actions
conda-cleanAzure
conda-linux-gcc-py310-arm64Azure
conda-linux-gcc-py310-cpuAzure
conda-linux-gcc-py310-cudaAzure
conda-linux-gcc-py310-ppc64leAzure
conda-linux-gcc-py37-arm64Azure
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cpu-r41Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py37-ppc64leAzure
conda-linux-gcc-py38-arm64Azure
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py38-ppc64leAzure
conda-linux-gcc-py39-arm64Azure
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-linux-gcc-py39-ppc64leAzure
conda-osx-arm64-clang-py310Azure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py310Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py37-r41Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2017-py310Azure
conda-win-vs2017-py37-r40Azure
conda-win-vs2017-py37-r41Azure
conda-win-vs2017-py38Azure
conda-win-vs2017-py39Azure
debian-bookworm-amd64Github Actions
debian-bookworm-arm64TravisCI
debian-bullseye-amd64Github Actions
debian-bullseye-arm64TravisCI
debian-buster-amd64Github Actions
debian-buster-arm64TravisCI
example-cpp-minimal-build-staticGithub Actions
example-cpp-minimal-build-static-system-dependencyGithub Actions
example-python-minimal-build-fedora-condaGithub Actions
example-python-minimal-build-ubuntu-venvGithub Actions
homebrew-cppGithub Actions
homebrew-r-autobrewGithub Actions
homebrew-r-brewGithub Actions
java-jarsGithub Actions
nugetGithub Actions
python-sdistGithub Actions
test-build-cpp-fuzzGithub Actions
test-build-vcpkg-winGithub Actions
test-conda-cppGithub Actions
test-conda-cpp-valgrindAzure
test-conda-python-3.10Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-kartothek-latestGithub Actions
test-conda-python-3.7-kartothek-masterGithub Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-v3.1.2Github Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-v3.2.0Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-dask-latestGithub Actions
test-conda-python-3.9-dask-masterGithub Actions
test-conda-python-3.9-pandas-masterGithub Actions
test-conda-python-3.9-spark-masterGithub Actions
test-debian-10-cpp-amd64Github Actions
test-debian-10-cpp-i386Github Actions
test-debian-11-cpp-amd64Github Actions
test-debian-11-cpp-i386Github Actions
test-debian-11-go-1.16Azure
test-debian-11-python-3Azure
test-debian-c-glibGithub Actions
test-debian-rubyGithub Actions
test-fedora-35-cppGithub Actions
test-fedora-35-python-3Azure
test-fedora-r-clang-sanitizerAzure
test-r-arrow-backwards-compatibilityGithub Actions
test-r-depsource-bundledAzure
test-r-depsource-systemGithub Actions
test-r-dev-duckdbGithub Actions
test-r-devdocsGithub Actions
test-r-gcc-11Github Actions
test-r-gcc-12Github Actions
test-r-install-localGithub Actions
test-r-linux-as-cranGithub Actions
test-r-linux-rchkGithub Actions
test-r-linux-valgrindAzure
test-r-minimal-buildAzure
test-r-offline-maximalGithub Actions
test-r-offline-minimalAzure
test-r-rhub-debian-gcc-devel-lto-latestAzure
test-r-rhub-debian-gcc-release-custom-ccacheAzure
test-r-rhub-ubuntu-gcc-release-latestAzure
test-r-rocker-r-base-latestAzure
test-r-rstudio-r-base-4.1-opensuse153Azure
test-r-rstudio-r-base-4.2-centos7-devtoolset-8Azure
test-r-rstudio-r-base-4.2-focalAzure
test-r-ubuntu-22.04Github Actions
test-r-versionsGithub Actions
test-skyhook-integrationGithub Actions
test-ubuntu-18.04-cppGithub Actions
test-ubuntu-18.04-cpp-releaseGithub Actions
test-ubuntu-18.04-cpp-staticGithub Actions
test-ubuntu-18.04-r-sanitizerAzure
test-ubuntu-20.04-cppGithub Actions
test-ubuntu-20.04-cpp-14Github Actions
test-ubuntu-20.04-cpp-17Github Actions
test-ubuntu-20.04-cpp-bundledGithub Actions
test-ubuntu-20.04-cpp-thread-sanitizerGithub Actions
test-ubuntu-20.04-python-3Azure
test-ubuntu-22.04-cppGithub Actions
test-ubuntu-c-glibGithub Actions
test-ubuntu-default-docsAzure
test-ubuntu-rubyGithub Actions
ubuntu-bionic-amd64Github Actions
ubuntu-bionic-arm64TravisCI
ubuntu-focal-amd64Github Actions
ubuntu-focal-arm64TravisCI
ubuntu-impish-amd64Github Actions
ubuntu-impish-arm64TravisCI
ubuntu-jammy-amd64Github Actions
ubuntu-jammy-arm64TravisCI
verify-rc-source-cpp-linux-almalinux-8-amd64Github Actions
verify-rc-source-cpp-linux-conda-latest-amd64Github Actions
verify-rc-source-cpp-linux-ubuntu-18.04-amd64Github Actions
verify-rc-source-cpp-linux-ubuntu-20.04-amd64Github Actions
verify-rc-source-cpp-linux-ubuntu-22.04-amd64Github Actions
verify-rc-source-cpp-macos-amd64Github Actions
verify-rc-source-cpp-macos-arm64Github Actions
verify-rc-source-cpp-macos-conda-amd64Github Actions
verify-rc-source-csharp-linux-almalinux-8-amd64Github Actions
verify-rc-source-csharp-linux-conda-latest-amd64Github Actions
verify-rc-source-csharp-linux-ubuntu-18.04-amd64Github Actions
verify-rc-source-csharp-linux-ubuntu-20.04-amd64Github Actions
verify-rc-source-csharp-linux-ubuntu-22.04-amd64Github Actions
verify-rc-source-csharp-macos-amd64Github Actions
verify-rc-source-csharp-macos-arm64Github Actions
verify-rc-source-go-linux-almalinux-8-amd64Github Actions
verify-rc-source-go-linux-conda-latest-amd64Github Actions
verify-rc-source-go-linux-ubuntu-18.04-amd64Github Actions
verify-rc-source-go-linux-ubuntu-20.04-amd64Github Actions
verify-rc-source-go-linux-ubuntu-22.04-amd64Github Actions
verify-rc-source-go-macos-amd64Github Actions
verify-rc-source-go-macos-arm64Github Actions
verify-rc-source-integration-linux-almalinux-8-amd64Github Actions
verify-rc-source-integration-linux-conda-latest-amd64Github Actions
verify-rc-source-integration-linux-ubuntu-18.04-amd64Github Actions
verify-rc-source-integration-linux-ubuntu-20.04-amd64Github Actions
verify-rc-source-integration-linux-ubuntu-22.04-amd64Github Actions
verify-rc-source-integration-macos-amd64Github Actions
verify-rc-source-integration-macos-arm64Github Actions
verify-rc-source-integration-macos-conda-amd64Github Actions
verify-rc-source-java-linux-almalinux-8-amd64Github Actions
verify-rc-source-java-linux-conda-latest-amd64Github Actions
verify-rc-source-java-linux-ubuntu-18.04-amd64Github Actions
verify-rc-source-java-linux-ubuntu-20.04-amd64Github Actions
verify-rc-source-java-linux-ubuntu-22.04-amd64Github Actions
verify-rc-source-java-macos-amd64Github Actions
verify-rc-source-js-linux-almalinux-8-amd64Github Actions
verify-rc-source-js-linux-conda-latest-amd64Github Actions
verify-rc-source-js-linux-ubuntu-18.04-amd64Github Actions
verify-rc-source-js-linux-ubuntu-20.04-amd64Github Actions
verify-rc-source-js-linux-ubuntu-22.04-amd64Github Actions
verify-rc-source-js-macos-amd64Github Actions
verify-rc-source-js-macos-arm64Github Actions
verify-rc-source-python-linux-almalinux-8-amd64Github Actions
verify-rc-source-python-linux-conda-latest-amd64Github Actions
verify-rc-source-python-linux-ubuntu-18.04-amd64Github Actions
verify-rc-source-python-linux-ubuntu-20.04-amd64Github Actions
verify-rc-source-python-linux-ubuntu-22.04-amd64Github Actions
verify-rc-source-python-macos-amd64Github Actions
verify-rc-source-python-macos-arm64Github Actions
verify-rc-source-python-macos-conda-amd64Github Actions
verify-rc-source-ruby-linux-almalinux-8-amd64Github Actions
verify-rc-source-ruby-linux-conda-latest-amd64Github Actions
verify-rc-source-ruby-linux-ubuntu-18.04-amd64Github Actions
verify-rc-source-ruby-linux-ubuntu-20.04-amd64Github Actions
verify-rc-source-ruby-linux-ubuntu-22.04-amd64Github Actions
verify-rc-source-ruby-macos-amd64Github Actions
verify-rc-source-ruby-macos-arm64Github Actions
verify-rc-source-windowsGithub Actions
wheel-macos-big-sur-cp310-arm64Github Actions
wheel-macos-big-sur-cp310-universal2Github Actions
wheel-macos-big-sur-cp38-arm64Github Actions
wheel-macos-big-sur-cp39-arm64Github Actions
wheel-macos-big-sur-cp39-universal2Github Actions
wheel-macos-high-sierra-cp310-amd64Github Actions
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-high-sierra-cp38-amd64Github Actions
wheel-macos-high-sierra-cp39-amd64Github Actions
wheel-macos-mavericks-cp310-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-macos-mavericks-cp38-amd64Github Actions
wheel-macos-mavericks-cp39-amd64Github Actions
wheel-manylinux2010-cp310-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp310-amd64Github Actions
wheel-manylinux2014-cp310-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-windows-cp310-amd64Github Actions
wheel-windows-cp37-amd64Github Actions
wheel-windows-cp38-amd64Github Actions
wheel-windows-cp39-amd64Github Actions

@kou

kou commented Jun 11, 2022

Copy link
Copy Markdown
Member

We need to fix wheel-*-cp37-* failures such as wheel-manylinux2014-cp37-amd64:

https://github.com/ursacomputing/crossbow/runs/6842267198?check_suite_focus=true

=================================== FAILURES ===================================
_______________ TestConvertMetadata.test_rangeindex_doesnt_warn ________________
self = <pyarrow.tests.test_pandas.TestConvertMetadata object at 0x7f9961d90d90>
def test_rangeindex_doesnt_warn(self):
# ARROW-5606: pandas 0.25 deprecated private _start/stop/step
# attributes -> can be removed if support < pd 0.25 is dropped
df = pd.DataFrame(np.random.randn(4, 2), columns=['a', 'b'])
with pytest.warns(None) as record:
_check_pandas_roundtrip(df, preserve_index=True)
> assert len(record) == 0
E assert 4 == 0
E + where 4 = len(WarningsChecker(record=True))
usr/local/lib/python3.7/site-packages/pyarrow/tests/test_pandas.py:229: AssertionError
_______________ TestConvertMetadata.test_multiindex_doesnt_warn ________________
self = <pyarrow.tests.test_pandas.TestConvertMetadata object at 0x7f9961d2a710>
def test_multiindex_doesnt_warn(self):
# ARROW-3953: pandas 0.24 rename of MultiIndex labels to codes
columns = pd.MultiIndex.from_arrays([['one', 'two'], ['X', 'Y']])
df = pd.DataFrame([(1, 'a'), (2, 'b'), (3, 'c')], columns=columns)
with pytest.warns(None) as record:
_check_pandas_roundtrip(df, preserve_index=True)
> assert len(record) == 0
E assert 6 == 0
E + where 6 = len(WarningsChecker(record=True))
usr/local/lib/python3.7/site-packages/pyarrow/tests/test_pandas.py:280: AssertionError

@emkornfield

Copy link
Copy Markdown
ContributorAuthor

These seem unrelated to this PR? If so I can open up JIRA to track

@pitrou

pitrou commented Jun 12, 2022

Copy link
Copy Markdown
Member

@kou Those seem unrelated to this PR. I need to do a last review pass and then this PR can be merged.

@pitrou

Copy link
Copy Markdown
Member

@github-actions crossbow submit -g wheel--cp37-

@github-actions

Copy link
Copy Markdown
Invalid group(s) {'wheel-*-cp37-*'}. Must be one of {'linux-amd64', 'verify-rc', 'conda', 'nightly-packaging', 'packaging', 'vcpkg', 'linux', 'nightly', 'example', 'fuzz', 'r', 'verify-rc-source-macos', 'wheel', 'verify-rc-source', 'verify-rc-jars', 'cpp', 'integration', 'c-glib', 'ruby', 'python', 'test', 'homebrew', 'verify-rc-wheels', 'linux-arm64', 'example-cpp', 'conan', 'example-python', 'verify-rc-source-linux', 'nightly-release', 'nightly-tests', 'verify-rc-binaries'}
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/2482888639

@pitrou

Copy link
Copy Markdown
Member

@github-actions crossbow submit wheel--cp37-

@github-actions

Copy link
Copy Markdown

Revision: b8336ef

Submitted crossbow builds: ursacomputing/crossbow @ actions-d604965942

TaskStatus
wheel-macos-high-sierra-cp37-amd64Github Actions
wheel-macos-mavericks-cp37-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-windows-cp37-amd64Github Actions

@pitrou
pitrou merged commit 7b5912d into apache:masterJun 12, 2022
@pitrou

Copy link
Copy Markdown
Member

Ok, some JIRAs will probably have to be opened for the wheel test failures.

@kou

kou commented Jun 12, 2022

Copy link
Copy Markdown
Member

Oh, sorry.

@emkornfield

Copy link
Copy Markdown
ContributorAuthor

@kou@pitrou thank you very much for all your help on this one.

@grisaitis

Copy link
Copy Markdown

for documentation, should i open a new issue? v excited about this :)

the docs source: https://github.com/apache/arrow/blob/master/docs/source/python/filesystems.rst

@kou

kou commented Jun 20, 2022

Copy link
Copy Markdown
Member

Yes, please.
If you can work on it, please submit a pull request too.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants

@emkornfield@voutilad@coryan@kou@rok@pitrou@kszucs@grisaitis@edponce@jorisvandenbossche