Skip to content

ARROW-7761: [C++][Python] Support S3 URIs - #6403

Closed
pitrou wants to merge 3 commits into
apache:masterfrom
pitrou:ARROW-7761-s3-uris
Closed

ARROW-7761: [C++][Python] Support S3 URIs#6403
pitrou wants to merge 3 commits into
apache:masterfrom
pitrou:ARROW-7761-s3-uris

Conversation

@pitrou

Copy link
Copy Markdown
Member

Allow instantiating an S3 filesystem instance from a s3://user:pass@bucket/path?params... URI.

@github-actions

Copy link
Copy Markdown

@fsaintjacques
fsaintjacques self-requested a review February 11, 2020 14:47
@pitrou
pitrouforce-pushed the ARROW-7761-s3-uris branch 3 times, most recently from 2294951 to ddcb28fCompareFebruary 13, 2020 11:30
@fsaintjacques

Copy link
Copy Markdown
Contributor

Works great :D

01:build/ (pr/6403✗) $ time release/dataset-parquet-scan-example file:///home/fsaintjacques/datasets/nyc-tlc/parquet
Table size: 746
real 0m3.248s
user 0m5.832s
sys 0m0.203s
01:build/ (pr/6403✗) $ time release/dataset-parquet-scan-example 's3://123:12345678@nyc-tlc/parquet?scheme=http&endpoint_override=localhost:9000'
Table size: 746
real 0m5.600s
user 0m7.557s
sys 0m0.474s

@fsaintjacques

Copy link
Copy Markdown
Contributor

One thing that I noted in my local integration, I had to explicit the global init of S3. I think it might be worth providing a default or implicitly calling the init step in FileSystemFromUri.

Allow instantiating an S3 filesystem instance from a
`s3://user:pass@bucket/path?params...` URI.
@pitrou

Copy link
Copy Markdown
MemberAuthor

@fsaintjacques Could you try again now?

@fsaintjacques

fsaintjacques commented Feb 17, 2020

Copy link
Copy Markdown
Contributor

I pushed a change to the dataset example, it seems to be lost did you push force before pulling? It was a small change, I can make it again.

@pitrou

Copy link
Copy Markdown
MemberAuthor

Oh, sorry. Yes, I force-pushed after rebasing :-/

@fsaintjacques

Copy link
Copy Markdown
Contributor

It should pass the CI, I'll merge after. The diff of the dataset example shows how well designed the APIs is.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@pitrou@fsaintjacques