[Docs][Format] Variant extension: align `typed_value` primitive type mappings with the Parquet shredding spec

[sdf-jkl]

Describe the enhancement requested

The Primitive Type Mappings table in the Variant canonical extension is the normative list of Arrow types allowed for typed_value. The only place the table is referenced from is the typed_value storage rule, so its whole role is defining legal typed_value types:

arrow/docs/source/format/CanonicalExtensions.rst

Line 453 in f10c93c

* A field named ``typed_value`` which can be a :ref:`variant_primitive_type_mapping` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``

The table appears to be derived from the Variant encoding types table (https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types), but its role corresponds to the shredding spec's Shredded Value Types table (https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types), which defines what a typed_value column may be. The two tables differ, so the Arrow table currently allows storage with no valid Parquet shredded representation:

• Uint8 -> Int16, Uint16 -> Int32, Uint32 -> Int64: the shredding spec only permits signed integer typed_value columns, so unsigned Arrow storage can never round trip (Arrow Uint -> Parquet INT -> Arrow Int). A Uint8 column also cannot represent the full value domain of its claimed variant type (int16), so it can only act as a partial storage that forces fallback to value. See the discussion in variant: Support Arrow UInt8/UInt16/UInt32 typed values arrow-rs#10417 (issue: Parquet Variant rejects canonical Arrow UInt8/UInt16/UInt32 typed values arrow-rs#10416).
• Null -> Null: the shredding table has no null row; a variant null must be encoded in the value column, so a Null typed_value column has no Parquet representation either.

In practice the unsigned path already produces non-conformant files: Arrow Go writes unsigned typed_value columns to Parquet with INT(8/16/32, false) annotations, which the shredding spec does not allow.

Proposal: re-derive the table from the shredding spec's Shredded Value Types table. Each row maps one variant type to its Parquet shredded type and the Arrow storage type(s) able to represent its full value domain. The table then defines reconstruction (a typed_value column of the listed Arrow type holds values of exactly the listed variant type) and the write-side lowering (the Parquet column is the only valid Parquet representation).

Variant type	Parquet type	Arrow typed_value type
boolean	BOOLEAN	Boolean
int8	INT32, INT(8, true)	Int8
int16	INT32, INT(16, true)	Int16
int32	INT32	Int32
int64	INT64	Int64
float	FLOAT	Float32
double	DOUBLE	Float64
decimal4, 1 <= P <= 9, 0 <= S <= P	INT32, DECIMAL(P, S)	Decimal32(P, S)
decimal8, 10 <= P <= 18, 0 <= S <= P	INT64, DECIMAL(P, S)	Decimal64(P, S)
decimal16, 19 <= P <= 38, 0 <= S <= P	BYTE_ARRAY / FIXED_LEN_BYTE_ARRAY, DECIMAL(P, S)	Decimal128(P, S)
date	INT32, DATE	Date32
time	INT64, TIME(false, MICROS)	Time64(us)
timestamptz(6)	INT64, TIMESTAMP(true, MICROS)	Timestamp(us, UTC)
timestamptz(9)	INT64, TIMESTAMP(true, NANOS)	Timestamp(ns, UTC)
timestampntz(6)	INT64, TIMESTAMP(false, MICROS)	Timestamp(us)
timestampntz(9)	INT64, TIMESTAMP(false, NANOS)	Timestamp(ns)
binary	BYTE_ARRAY	Binary / LargeBinary / BinaryView
string	BYTE_ARRAY, STRING	Utf8 / LargeUtf8 / Utf8View
uuid	FIXED_LEN_BYTE_ARRAY[len=16], UUID	UUID extension type

The decimal precision bands follow the decimal table in VariantEncoding.md: precision alone selects the physical type (the bands are disjoint, so the narrowest type is required), and scale must satisfy 0 <= S <= P. P and S are shared across a row, so the bands constrain both the Parquet DECIMAL annotation and the Arrow type. Arrow otherwise permits negative scale and low-precision wide decimals, and Parquet permits both low-precision wide columns and BYTE_ARRAY decimals above precision 38.

This also resolves three existing ambiguities: the current Time64 row gains its required unit (variant time is micros only), the timestamp rows gain explicit units, and the decimal rows gain their precision bounds.

Related: #50620, apache/parquet-format#591

Component(s)

Format

[wgtmac]

unsigned Arrow storage can never round trip (Arrow Uint -> Parquet INT -> Arrow Int)

I think this is possible. I'm not familiar with arrow-rs. For arrow-cpp, the Parquet writer serializes the Arrow schema into footer when the writer property store_schema is enabled (which is the default value) and then Parquet reader deserializes that schema to restore their original form on read.

Null -> Null: the shredding table has no null row

This is a good argument. Parquet indeed has a UNKNOWN logical type to represent a column with no known type. It would be useful when user-hinted column cannot be shredded. So it seems like a missing piece in the Parquet shredding spec?

cc @zeroshade @alamb

[zeroshade]

For arrow-cpp, the Parquet writer serializes the Arrow schema into footer when the writer property store_schema is enabled (which is the default value) and then Parquet reader deserializes that schema to restore their original form on read.

arrow-go does the same using the same property information and is thus compatible with arrow-cpp (and the impls based on it like pyarrow) writing the arrow schema into the footer.

In practice the unsigned path already produces non-conformant files: Arrow Go writes unsigned typed_value columns to Parquet with INT(8/16/32, false) annotations, which the shredding spec does not allow.

Can you file this as an issue on arrow-go so we can track and fix this? We should definitely make sure that arrow-go is spec conformant

[alamb]

For arrow-cpp, the Parquet writer serializes the Arrow schema into footer when the writer property store_schema is enabled (which is the default value) and then Parquet reader deserializes that schema to restore their original form on read.

arrow-go does the same using the same property information and is thus compatible with arrow-cpp (and the impls based on it like pyarrow) writing the arrow schema into the footer.

Arrow-rs does the same thing

It looks to me like @sdf-jkl observation is that there are certain arrow types that can not be round tripped through a Parquet Variant column without some sort of additional metadata

Specifically something like UInt8 --> Variant must be stored as Variant::Int16 (so the knowledge that the column only has UInt8 values is lost)

As mentioned above, the normal Parquet logical types have the same problem, so the C++ writer adds extra metadata in the file so even though a value might be stored as Parquet type Int 16, Arrow readers know to try and read it as UInt8. This metadata convention seems to be commonly followed.

Maybe we can think about additional metadata writers could add if they want to recreate the original schema.

I am not sure how important this case is, though, because if you had data that was originally highly typed and structured you probably wouldn't be using Variant 🤔

[sdf-jkl]

I am not sure how important this case is, though, because if you had data that was originally highly typed and structured you probably wouldn't be using Variant 🤔

@alamb That's the point I was trying to make. Variant's physical types only include signed integers, same goes for the typed_value type support in the Shredding spec.

I am proposing to drop unsigned integers from the typed_value allowed types for Arrow since they always have to be written as signed.

Null -> Null: the shredding table has no null row

This is a good argument. Parquet indeed has a UNKNOWN logical type to represent a column with no known type. It would be useful when user-hinted column cannot be shredded. So it seems like a missing piece in the Parquet shredding spec?

@wgtmac Same goes for the Null type, I'm proposing to drop it from the Arrow table rather than add a null row to the Parquet spec. A null in typed_value already carries meaning: the row's value couldn't be shredded into the column type (or the field is missing, for object fields). An UNKNOWN column is always null, so it could never mark a row as shredded, and a Variant null would still have to be encoded in value as 00. Worse, for object fields typed_value null + value null means the field is missing, so shredding null into typed_value would make "field present and null" indistinguishable from "field missing". I don't think the null row is a missing piece of the shredding spec, that state is reserved by the null-signaling protocol itself.

Can you file this as an issue on arrow-go so we can track and fix this? We should definitely make sure that arrow-go is spec conformant

@zeroshade before filing I want to pin the correct behavior. I think Arrow should not support unsigned int typed_value at all, which would make the fix different.

[alamb]

I am proposing to drop unsigned integers from the typed_value allowed types for Arrow since they always have to be written as signed.

so is the proposed fix just an additional caveat to the spec that says typed_value has to be an arrow type that has a 1:1 mapping with the parquet types?

[sdf-jkl]

I am proposing to drop unsigned integers from the typed_value allowed types for Arrow since they always have to be written as signed.

so is the proposed fix just an additional caveat to the spec that says typed_value has to be an arrow type that has a 1:1 mapping with the parquet types?

Yes, that would be it

[HuaHuaY]

So, is the final conclusion that we should update the Arrow documentation? How should we go about driving this forward?

[HuaHuaY]

I’ve found another confusing issue. The Arrow specification maps decimal32 to decimal4, decimal64 to decimal8, decimal128 to decimal16. However, the Parquet documentation maps types based on decimal precision at https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types. Given the earlier discussion, I feel we should defer to the Parquet standard here as well?

[sdf-jkl]

@HuaHuaY yes, that makes sense. I updated the proposed table in the issue description, the precision bounds now constrain both the Parquet and the Arrow columns.

There is a related gap upstream: the shredding spec never states the decimal precision rules at all. The bands only appear in the Decimal table in VariantEncoding.md, while the decimal rows in VariantShredding.md are unbounded DECIMAL(P, S). So the Arrow table has to carry the bounds explicitly, and VariantShredding.md could use the same clarification or a cross-reference.

So, is the final conclusion that we should update the Arrow documentation? How should we go about driving this forward?

Update the Arrow doc and make an upstream Parquet PR to add decimal precision clarification/cross-reference to the shredding schema.

[sdf-jkl]

@alamb @zeroshade @wgtmac Please confirm if you think this is the correct path and I'll start working on the Arrow spec PR and submit an issue in arrow-go.