[fix](arrow-flight) Keep the Doris type of a nested LARGEINT in the Arrow schema

[morningman]

What problem does this PR solve?

Related Jira: DORIS-28389

Problem Summary:

Arrow has no equivalent for LARGEINT, IPV4, IPV6, JSON or VARIANT. Each of them travels over Flight SQL as some other Arrow type and is then indistinguishable from a column that is natively of that type -- a LARGEINT and a STRING both arrive as utf8, an IPV4 and an INT both arrive as int32. The doris_type field metadata is the only thing that tells them apart.

That metadata was attached in get_arrow_schema_from_block and get_arrow_schema_from_expr_ctxs, which only see the top level columns. convert_to_arrow_type recurses into ARRAY / MAP / STRUCT building Arrow types, and Arrow keeps metadata on the Field rather than on the DataType, so every nested element lost it: ListType(item_type) synthesizes a bare "item" field, MapType(key_type, val_type) synthesizes bare "key"/"value" fields, and the STRUCT branch built its fields without going through the metadata helper.

Reading this through a Python ADBC client:

SELECT CAST(495AS LARGEINT) AS scalar_value,
named_struct('count', CAST(495AS LARGEINT)) AS struct_value,
array(CAST(495AS LARGEINT)) AS array_value,
map('k', CAST(495AS LARGEINT)) AS map_value;

Before -- only the top level column is identifiable:

scalar_value: string metadata={'doris_type': 'LARGEINT'}
struct_value: struct<count: string>
count: string metadata={}
array_value: list<item: string>
item: string metadata={}
map_value: map<string, string>
entries: struct<key: string not null, value: string>
key: string metadata={}
value: string metadata={}

After:

scalar_value: string metadata={'doris_type': 'LARGEINT'}
struct_value: struct<count: string>
count: string metadata={'doris_type': 'LARGEINT'}
array_value: list<item: string>
item: string metadata={'doris_type': 'LARGEINT'}
map_value: map<string, string>
entries: struct<key: string not null, value: string>
key: string metadata={}
value: string metadata={'doris_type': 'LARGEINT'}

A nested IPV4 was the worse case of the same defect: it arrives as its 32 bits read as a signed int32 (192.168.1.1 as -1062731519) with nothing left to say it was ever an address.

What is changed?

convert_to_arrow_type now builds the child Fields through create_arrow_field_with_metadata at every level, for ARRAY, MAP and STRUCT. The names and the nullability are exactly the ones Arrow's own constructors produced -- "item" nullable, "key" non-nullable, "value" nullable -- so only the metadata is new, and the record batch builders, which are made from this same schema (FromBlockToRecordBatchConverter reads _schema->field(idx)->type()), are unaffected. DataType::Equals ignores metadata by default, so the batch still type-matches the schema.

The metadata lookup is also completed: JSONB and VARIANT are serialized as utf8 too and were carrying no doris_type at all, not even at the top level. The values (JSON, VARIANT) match what the FE reports for the same column under ARROW:FLIGHT:SQL:TYPE_NAME in FlightSqlSchemaHelper.

No value or Arrow type changes -- this is metadata only.

Release note

Fix Arrow Flight SQL losing the Doris logical type (doris_type field metadata) of LARGEINT / IPV4 / IPV6 nested inside STRUCT, ARRAY and MAP, which made a nested LARGEINT indistinguishable from a STRING for ADBC clients. JSON and VARIANT columns now carry the same metadata as well.

Check List (For Author)

• Test

  • Unit Test

  New be/test/format/arrow/arrow_row_batch_test.cpp, 7 cases:
  nested LARGEINT in STRUCT / ARRAY / MAP value / MAP key; every level of
  array<struct<largeint>> and map<string, array<array<largeint>>>;
  nested IPV4 / IPV6 / JSON / VARIANT; a negative control asserting STRING and
  INT carry no metadata nested or not; an assertion that the nested types still
  compare equal to arrow::list(utf8()) / arrow::map(utf8(), utf8()) with
  check_metadata=false and differ only with check_metadata=true; and one
  that runs convert_to_arrow_batch over the new schema and checks
  ValidateFull(), schema equality including metadata, and the values.

  Verified the assertions are not vacuous: with the arrow_row_batch.cpp change
  reverted, 5 of the 7 fail; the negative control and the batch-building guard
  correctly stay green.

  • Manual test (add detailed scripts or steps below)

  Built BE locally and ran the Jira's repro through
  adbc_driver_flightsql (Python), plus a real table with
  largeint, array<largeint>, struct<count:largeint, name:string>,
  map<string,largeint>, ipv4 and json columns. Nested fields now carry
  doris_type at every depth, a STRING sibling inside the same STRUCT correctly
  carries none, and the values are unchanged (LARGEINT extremes round trip
  losslessly as text).

• Behavior changed:

  • Yes.

  The Arrow schema returned over Flight SQL now carries doris_type field
  metadata on nested fields, and on top level JSON / VARIANT columns. Arrow
  types, field names, nullability and values are unchanged, so a client that
  ignores metadata sees no difference.

• Does this need documentation?

  • No.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FP36MDAsUyKSSDQXQohQsK

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[yiguolei]

wait, I will refactor these types

[morningman]

@yiguolei Sure. Before you start, here is what we need from that refactor, so that our follow-up work can build on it instead of fighting it. Context: we are preparing a proposal that makes MySQL and Arrow Flight SQL equal front ends over one session layer, and its type-mapping part depends on exactly the code this PR touches (convert_to_arrow_type, create_arrow_field_with_metadata and the schema entry points in arrow_row_batch.cpp).

1. One enumerable mapping, table-testable. Whatever shape the refactor takes (per-DataType virtuals, serde methods, a registry), please keep a single place that answers "Doris type -> Arrow storage type + field metadata" for every type, so that a table-driven BE UT can walk all types and the FE can validate its mirror (FlightSqlSchemaHelper.getArrowType, to be extracted) against the same golden file. Today the FE copy already disagrees with the BE in four places (TIMESTAMPTZ zone, TIMEV2, VARBINARY, AGG_STATE); we want that class of drift to be caught by a test, not by a client.
2. Field metadata at every nesting level. Arrow keeps metadata on Field, not on DataType, so ARRAY item / MAP key and value / STRUCT children have to be built through the metadata helper (what this PR fixes). We will add ARROW:extension:name next to doris_type for the types Arrow has no native equivalent for (LARGEINT, IPV4, IPV6, JSON, VARIANT); adding a metadata key should be a one-line change, not an edit in every branch.
3. Storage types stay as they are. The same functions feed the Spark/Flink connector read path (memory_scratch_sink_operator), Python UDF/UDAF/UDTF and the Parquet transformer, so a storage-type change is a wire-format change for connectors. Our plan is to keep the current shapes (LARGEINT -> utf8, IPV4 -> int32, IPV6 -> utf8, DECIMALV2 -> decimal128(27,9), DATETIME -> naive timestamp with the unit chosen by scale, TIMESTAMPTZ with its zone) and only add metadata. If the refactor wants to change any of them, please make it an explicit, separately reviewed change rather than a side effect.
4. Schema authoritative over batches. A Flight stream has one schema, so a builder must never swap a column's array type per batch (utf8 -> large_utf8), see [fix](arrow-flight) Fix utf8/large string schema-array mismatch for oversized columns #65789. If the string/binary builders are touched, keep the split-by-rows approach.
5. AGG_STATE. It currently goes through get_serialized_type() and lands in utf8 carrying non-UTF-8 bytes; if the refactor handles it, binary is the right target.
6. Keep the schema entry points (get_arrow_schema_from_block / get_arrow_schema_from_expr_ctxs, or equivalents): the FE fetches the result schema from the BE (fetchArrowFlightSchema) and forwards it to clients as is.

Could you share the intended scope and rough timing? We would rebase our type-mapping PRs on top of it, and it also settles whether this PR should go in first or be folded into the refactor.