[C++] Add filesystem implementation for Amazon S3

[asfimport]

The BSD-licensed C++ code in SFrame (https://github.com/turi-code/SFrame/tree/master/oss_src/fileio) may provide some inspiration.

Reporter: Wes McKinney / @wesm
Assignee: Antoine Pitrou / @pitrou

Related issues:

• [Python/C++] Implement NativeFile interfaces for Amazon S3 (is related to)
• [Python] Run s3fs unit tests in Travis CI (is related to)
• [CI/Python] Have automatized tests for S3 (is related to)
• [Python/C++] Implement NativeFile interfaces for Amazon S3 (supercedes)

PRs and other links:

• GitHub Pull Request #5167

_{Note: This issue was originally created as ARROW-453. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
We might also look at Impala's C++ S3 implementation

[asfimport]

Wes McKinney / @wesm:
See also tensorflow/tensorflow#11089

[asfimport]

Antoine Pitrou / @pitrou:
I'm trying to use MinIO for automated tests. It seems to have a rather delicate protocol implementation...

[asfimport]

Antoine Pitrou / @pitrou:
General question: should the S3 filesystem be anchored on a specific bucket (meaning "path/to/file" is a key inside the bucket) or not (meaning "bucket/path/to/file" gets divided into bucket name and key string)?

Both are possible, and I notice that the Python projects s3fs and fs-s3fs took different decisions here.

@martindurant What's your opinion?

[asfimport]

Martin Durant / @martindurant:
Obviously, I choose to do this one particular way, so you cannot count me as unbiased. To the particular question, I think that a URL like "s3://bucket/path/file" is very convenient at pinpointing a particular object without further qualification.

However, fsspec creates pyarrow-compatible instances for any of the implementations it references (including s3fs and gcsfs), so who do you need to do anything? You are talking about C-land? I'm not sure I am behind repeating all the work and coming up with new file-system standards yet again. Of course, I am a python-first kind of person.

[asfimport]

Antoine Pitrou / @pitrou:
As the title says, this is a C++ issue. We want to be able to read data from remote filesystems directly in C++ without going through Python calls (or even relying on the presence of a Python interpreter), hence why are reimplementing a filesystem layer in C++.

[asfimport]

Wes McKinney / @wesm:
Right, we're implementing things in C++ so that we can use a common implementation in Python, R, Ruby, MATLAB, and any other language that builds bindings to the core Arrow library.

[asfimport]

Martin Durant / @martindurant:
Can you please follow the fsspec model? I think it's a good model and, although young, centralises and codifies a number of FS-related concepts. If you were to rewrite all the implementations there in C++, that would be all to the good, and shouldn't require too much effort on each individual one. 

[asfimport]

Antoine Pitrou / @pitrou:
Which part of the model do you suggest we follow? The filesystem class API? If I look at https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem , there are a lot of methods there and we don't know which ones we'll need.

Or are there any particular gotchas or design points that you want to illuminate us about?

Also you can find our current abstract API here: https://github.com/apache/arrow/blob/master/cpp/src/arrow/filesystem/filesystem.h#L53

[asfimport]

Martin Durant / @martindurant:
Yes, I mean the whole spec, so that you make general-purpose file-system interfaces, rather than just the small part of the API that the internal arrow functions need. That might be part of the arrow mission, to open up data sources local and remote. Otherwise, users would need to use their own language implementations in addition to arrow, when they want to do any management of the remote files (copying, permissions, etc.).

[asfimport]

Wes McKinney / @wesm:
@pitrou one consideration is that data might be spread across multiple buckets with the same credentials, am I understanding that right? As some other data points, TensorFlow's implementation seems to treat the bucket as part of the file path

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/platform/s3/s3_file_system_test.cc

Turi Create (formerly Dato Create) accepts fully qualified URLs including even AWS keys, bucket name, and object name

https://github.com/apple/turicreate/blob/master/src/core/storage/fileio/s3_api.hpp#L177

[asfimport]

Wes McKinney / @wesm:
@martindurant that may be a longer-term aspirational goal but I think we are currently pretty scope-limited to the features we need to be able to read and write datasets in these systems. Maybe later when we have a bigger developer and maintainer community we can fill some of the feature gaps as a one-stop-shop filesystem interface

[asfimport]

Martin Durant / @martindurant:

 Turi Create (formerly Dato Create) accepts fully qualified URLs including even AWS keys, bucket name, and object name

fsspec also allows this, like s3://key:secret@bucket/path/file, which is rare for s3, but more common with ssh or hdfs. Users should probably not be putting their credentials into URLs in the general case.

[asfimport]

Antoine Pitrou / @pitrou:
To be clear, for now I am not putting a URI layer in the filesystem. This can be added separately.

[asfimport]

Antoine Pitrou / @pitrou:
Issue resolved by pull request 5167
#5167