GH-35627: [Format][Integration] Add string-view to arrow format

[bkietz]

String view (and equivalent non-utf8 binary view) is an alternative representation for
variable length strings which offers greater efficiency for several common operations.
This representation is in use by UmbraDB, DuckDB, and Velox. Where those databases use
a raw pointer to out-of-line strings this PR uses a pair of 32 bit integers as a
buffer index and offset, which

• makes explicit the guarantee that lifetime of all character data is equal
  to that of the array which views it, which is critical for confident
  consumption across an interface boundary
• makes the arrays meaningfully serializable and
  venue agnostic; directly usable in shared memory without modification
• allows easy validation

This PR is extracted from #35628 to unblock independent PRs now that the vote has passed, including:

• New types added to Schema.fbs
• Message.fbs amended to support variable buffer counts between string view chunks
• datagen.py extended to produce integration JSON for string view arrays
• Columnar.rst amended with a description of the string view format

• Closes: [Format] Add string view type to the arrow format #35627

[zeroshade]

LGTM

[pitrou]

Please let's please not make a special case for binary-view here. We use signed offsets and lengths everywhere else.

Suggested change

	by a raw pointer. All integers (length, buffer index, and offset) are unsigned
	by a raw pointer. All integers (length, buffer index, and offset) are signed

[bkietz]

We should discuss this on the ML, I'll raise this there

[bkietz]

https://lists.apache.org/thread/x3mh2zh5qlm3fss2716yn0dkortz6z3w

[paleolimbot]

I agree that the standard seems to have adopted signed integers for sizes in every other case. It seems very strange to impose that only for new types?

[bkietz]

It's not a requirement to be imposed on sizes in all cases going forward. It's only relevant to Utf8View because existing implementations of string view already use unsigned integers here, so the concern is whether to keep compatibility with those or maintain the convention of the arrow format. If you'd like to +1 signed integers, please say so on the ML!

[bkietz]

Per the mailing list, I've updated this to signed integers.

[mapleFU]

LGTM, thanks!

[bkietz]

CI failures 1, 2 are unrelated, see #37803

[bkietz]

+1, thanks all!

[alamb]

🎉

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 9d6d501.

There were 4 benchmark results indicating a performance regression:

• Commit Run on ursa-i9-9960x at 2023-09-21 12:33:35Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-11, scale_factor=1
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-10, scale_factor=1
• and 2 more (see the report linked below)

The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.