[C++] Allow filesystems to be implemented in separate library (+ move remote filesystems out of libarrow into their own shared libraries)

[jorisvandenbossche]

Currently all our supported file systems in C++ (local, hdfs, s3, google cloud (gs/gcs), and soon azure (abfs)) are incorporated in the core libarrow library. For example, when enabled they are hardcoded in FileSystemFromUri.

There is a desire to be able to separate those filesystems in their own libraries, such that they can be installed separately. The remote filesystems each come with their own (potentially quite large) dependencies, and one typically doesn't need all of them at the same time.
More generally, it might also be nice that filesystems can be implemented externally for filesystems we wouldn't consider including in the main arrow project.

My understanding is that this would require:

• Some mechanism to "register" the filesystem to the core libarrow fs utilities. Especially the parsing from URI (i.e. when the user doesn't pass an already instantiated filesystem object) needs to know for each prefix to dispatch to which filesystem implementation.
• Build some or our own filesystems as separate libraries. I think this would mainly be the cloud filesystems (s3, google cloud, azure), while the local filesystem would always be included in core libarrow (+ some of the composite ones like subtree).

cc @pitrou @zeroshade

[pitrou]

Yes, this sounds reasonable to me.

cc @felipecrv

[zeroshade]

I like this idea a lot and think we should pursue it.

[jorisvandenbossche]

Some mechanism to "register" the filesystem to the core libarrow fs utilities

One aspect of this that is not really clear to me how this would work: when do those filesystems get registered? How does the core libarrow know which separate libraries to load, or what triggers them to be loaded?
For example in Python, you could either do that by letting the user do some manual import (and pyarrow could also abstract that away by already attempting this import under the hood for our own filesystems), or either with entrypoints to discover external ones. But I don't know how that would work in C++?

[pitrou]

In C++, I think it would just be an explicit registration call that the application (or any third-party code) can issue.

[paleolimbot]

Just another +1 on this...I don't know to what extent we will be able to leverage this to simplify our R package build, but I have noted to a few people that most of the time I spend debugging the R package build is because of S3 or GCS. Anything moving filesystems in a direction where they can be plugged in separately would be very welcome (even if the first pass doesn't directly enable runtime plugging from, say, a separate R package).

[bkietz]

The basic sketch of this reorganization is:

• filesystem.h remains part of the public API for libarrow.so, containing
  • class FileInfo, class FileSystem
  • FileSystem FileSystemFromUri(string)
  • void AddFactory(function<FileSystem(string)>)

Some things don't quite fit into this picture, for example I'm not sure how to handle functions like S3FileSystem.region() and resolve_s3_region(). These use the S3 sdk (so we want to build it in the dedicated libarrow_s3fs.so module). However it provides functionality only relevant to S3 since local file systems don't have regions (so it cannot be retrieved from a registry of FromUri functions).

The public API of libarrow.so will need to include some extra interfaces which can be downcast to or lazily initialized in order to access s3's extra functionality. (For completeness: we could instead include a soft code escape hatch in the registry, like a function mapping strings to strings so that s3's could handle "s3: region_for_bucket my-bucket"->"us-east-2")

[jorisvandenbossche]

Some things don't quite fit into this picture, for example I'm not sure how to handle functions like S3FileSystem.region() and resolve_s3_region(). These use the S3 sdk (so we want to build it in the dedicated libarrow_s3fs.so module). However it provides functionality only relevant to S3 since local file systems don't have regions (so it cannot be retrieved from a registry of FromUri functions).

Why is it a problem that the S3 filesystem has some additional APIs? I think this specific example of resolving a region is either used under the hood in FromUri (and then the generic interface doesn't need to know about it), or either used manually in an S3 specific API to resolve the region or to instantiate an S3FileSystem object. But those S3 specific APIs can still exist in the libarrow_s3fs.so library? (and those can be called by the user in code that is specific to S3)

[felipecrv]

Bikeshedding: pass a general FileSystemOptions struct that can contain implementation-specific options when a semantically-overloaded one isn't possible.

- void AddFactory(function<FileSystem(string)>)
+ void AddFactory(function<FileSystem(const FileSystemOptions &)>)

[pitrou]

I think the idea of AddFactory is that it takes a URI.
Though it seems that it should also declare the accepted URI schemes, so that we don't invoke all factories when parsing an URI.

So it would be something like:

// in filesystem.h

using FileSystemFactoryFunc = std::function<
 Result<std::shared_ptr<FileSystem>>(const URI& uri)>;

Status AddFileSystemFactory(std::vector<std::string> uri_schemes, FileSystemFactoryFunc);

I agree that something also needs to be done for ancillary functionality such as resolving a S3 region.

[bkietz]

Why is it a problem that the S3 filesystem has some additional APIs?

This is a less subtle issue in python where it's a more common pattern to provide functions conditionally. In C++ there are a few patterns which could be used, each with their own tradeoffs. I think the most straightforward would be lazily initialized function pointers (this pattern is commonly used to normalize opengl implementations with varying extensions, for example libGLEW):

• s3fs.h is installed alongside filesystem.h
• it contains a minimal S3FileSystem subclass which provides the region accessor
• it also contains a function pointer for resolve_s3_region which is initially null and is assigned an implementation as part of initializing libarrow_s3fs.so

FileSystemOptions

The factory takes string in my example because the main entry point for filesystem creation is FromUri, which takes string (in which all implementation specific options would be encoded).

• It would also be possible to register pairs of functions Filesystem create_fs(Options), Options parse_uri(string) if an explicit options class hierarchy is desired, but I'm not sure that would add value.
• Alternatively we could have a single concrete options class which we parse from a uri string: this would amount to making ParseFileSystemFromUri public, I think. This might be nice for unit testing.