From 0170444df9003957c1f837eb47167734370aae27 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fran=C3=A7ois=20Saint-Jacques?= Date: Mon, 22 Jun 2020 14:15:52 -0400 Subject: [PATCH 1/2] ARROW-1682: [Doc] Expand S3/MinIO fileystem dataset documentation --- docs/source/python/dataset.rst | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/source/python/dataset.rst b/docs/source/python/dataset.rst index ae14e396e16c..15b46698ecf4 100644 --- a/docs/source/python/dataset.rst +++ b/docs/source/python/dataset.rst @@ -325,6 +325,22 @@ The currently available classes are :class:`~pyarrow.fs.S3FileSystem` and details. +Reading from Minio +------------------ + +In addition to cloud storage, pyarrow also supports reading from a MinIO object +storage instance emulating S3 APIs. Paired with toxiproxy, this is useful for +testing or benchmarking. + +.. code-block:: python + + from pyarrow import fs + + minio = fs.S3FileSystem(scheme="http", endpoint="localhost:9000") + dataset = ds.dataset("ursa-labs-taxi-data/", filesystem=minio, + partitioning=["year", "month"]) + + Manual specification of the Dataset ----------------------------------- From 02f3f1ffc438e6f1287ca77ebb8fb441985bcdf9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fran=C3=A7ois=20Saint-Jacques?= Date: Thu, 25 Jun 2020 09:59:51 -0400 Subject: [PATCH 2/2] Address review comments --- docs/source/python/dataset.rst | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/source/python/dataset.rst b/docs/source/python/dataset.rst index 15b46698ecf4..3d9983415cef 100644 --- a/docs/source/python/dataset.rst +++ b/docs/source/python/dataset.rst @@ -328,14 +328,16 @@ details. Reading from Minio ------------------ -In addition to cloud storage, pyarrow also supports reading from a MinIO object -storage instance emulating S3 APIs. Paired with toxiproxy, this is useful for -testing or benchmarking. +In addition to cloud storage, pyarrow also supports reading from a +`MinIO https://github.com/minio/minio`_ object storage instance emulating S3 +APIs. Paired with `toxiproxy https://github.com/shopify/toxiproxy`_, this is +useful for testing or benchmarking. .. code-block:: python from pyarrow import fs + # By default, MinIO will listen for unencrypted HTTP traffic. minio = fs.S3FileSystem(scheme="http", endpoint="localhost:9000") dataset = ds.dataset("ursa-labs-taxi-data/", filesystem=minio, partitioning=["year", "month"])