[C++] Add opaque device id identification to InputStream

[asfimport]

For the purpose of collecting input/output statistics, it is important to know to which "device" these stats pertain, so as not to mix e.g. stats for a local NVMe drive, a NFS-attached drive, a S3 filesystem, or a in-memory buffer reader.

I suggest adding to InputStream this API:

/// \brief An opaque unique id for the device underlying this stream.
///
/// Any implementation is free to fill those bytes as it sees fit,
/// but it should be able to uniquely identify each "device"
/// (for example, a specific local drive, or a specific remote network
/// filesystem).
///
/// A suggested format is "<kind>:<bytes>" where "<kind>"
/// is a short string representing the backend kind
/// (for example "local", "s3"...) and "<bytes>" is a
/// backend-dependent string of bytes (for example a
/// `dev_t` for a POSIX local file).
///
/// This is not required to be printable nor human-readable,
/// and may contain NUL characters.
virtual std::string device_id() const = 0;

Reporter: Antoine Pitrou / @pitrou

Related issues:

• [C++] Implement a read range process without caching (is related to)

_{Note: This issue was originally created as ARROW-17917. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
cc @marsupialtail @westonpace @lidavidm for opinions.

[asfimport]

David Li / @lidavidm:
This is in relation to ARROW-17913?

Do we really want statistics for that case, or do we want some more structured, higher level I/O primitives? (e.g. reading a list of ranges, which can be translated to individual reads for nvme/memory-mapping, or coalesced reads on S3)

[asfimport]

Antoine Pitrou / @pitrou:
To the former: yes.

To the latter: I don't know, those sound orthogonal or perhaps complementary.

[asfimport]

Antoine Pitrou / @pitrou:
This might also be useful for OpenTelemetry instrumentation, though I don't know how deep it is advised to go in that matter.

[asfimport]

David Li / @lidavidm:
To me it sounds like having statistics on the input stream won't help you very much, compared to some API that would let readers express their intent in a way that I/O implementations can then service in an optimal way. But it's not clear to me at what level that kind of logic needs to live. (Statistics could let you make a decision about how to read, but then that logic needs to get replicated across a bunch of file formats, and you have to read a bunch of the file before you can make a decision…and what if you're reading 20 files? How do you share the statistics between them, at the level of the file reader?)

So I think I'm missing something about why we want these statistics

[asfimport]

Antoine Pitrou / @pitrou:
I have no precise idea of how statistics would be exposed, but they would probably live at the global level, not at the input stream level. The input stream level would tell you what device it is servicing, and you would look up the collected stats for that device.

[thisisnic]

This issue has been marked as stale because it has had no activity in the past 2 years. Please remove the stale label or comment below, or this issue will be closed in 14 days.