ARROW-1682: [Doc] Expand S3/MinIO fileystem dataset documentation - #7517
ARROW-1682: [Doc] Expand S3/MinIO fileystem dataset documentation#7517fsaintjacques wants to merge 2 commits into
Conversation
|
Does "for testing and benchmarking" mean that it's not optimal to store arrow/parquet files on Minio for production workloads? |
|
I can't comment on the production quality of MinIO since I've never used it in such scenario. I meant this for reference to other developers who wants to test the S3 bindings without having to use an actual S3 bucket. For example, I have a dataset stored locally. When I want to test the difference between local and cloud storage, I spin a MinIO instance pointing to the same local directory and benchmark accordingly. toxiproxy is used to introduce latency and bandwidth limits mimicking S3. |
|
Thanks, now I understand. So the pairing with toxiproxy is for the testing :)) |
| ------------------ | ||
|
|
||
| In addition to cloud storage, pyarrow also supports reading from a MinIO object | ||
| storage instance emulating S3 APIs. Paired with toxiproxy, this is useful for |
There was a problem hiding this comment.
Can you add hyperlinks to MinIO and toxyproxy?
|
|
||
| from pyarrow import fs | ||
|
|
||
| minio = fs.S3FileSystem(scheme="http", endpoint="localhost:9000") |
There was a problem hiding this comment.
Add a comment that this assumes MinIO is running unencrypted on local port 9000?
No description provided.