Skip to content

ARROW-1682: [Doc] Expand S3/MinIO fileystem dataset documentation #7517

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
fsaintjacques wants to merge 2 commits into from
Closed

ARROW-1682: [Doc] Expand S3/MinIO fileystem dataset documentation #7517

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0170444
ARROW-1682: [Doc] Expand S3/MinIO fileystem dataset documentation
fsaintjacques Jun 22, 2020
02f3f1f
Address review comments
fsaintjacques Jun 25, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions docs/source/python/dataset.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -325,6 +325,24 @@ The currently available classes are :class:`~pyarrow.fs.S3FileSystem` and
details.


Reading from Minio
------------------

In addition to cloud storage, pyarrow also supports reading from a
`MinIO https://github.com/minio/minio`_ object storage instance emulating S3
APIs. Paired with `toxiproxy https://github.com/shopify/toxiproxy`_, this is
useful for testing or benchmarking.

.. code-block:: python

from pyarrow import fs

# By default, MinIO will listen for unencrypted HTTP traffic.
minio = fs.S3FileSystem(scheme="http", endpoint="localhost:9000")

@pitrou pitrou Jun 25, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a comment that this assumes MinIO is running unencrypted on local port 9000?

dataset = ds.dataset("ursa-labs-taxi-data/", filesystem=minio,
partitioning=["year", "month"])


Manual specification of the Dataset
-----------------------------------

Expand Down