Is your feature request related to a problem or challenge?
datafusion-substrait's producer does not support Arrow Decimal32 or Decimal64.
Any DataFusion plan with a Decimal32/Decimal64 field or literal causes
to_substrait_plan to fail with:
NotImplemented("Unsupported cast type: ")
NotImplemented("Unsupported literal: Decimal32(..)")
Describe the solution you'd like
Substrait has one abstract Decimal { precision, scale } and distinguishes
physical widths via type_variation_reference, same mechanism already used for
Decimal256. So:
- Add
DECIMAL_32_TYPE_VARIATION_REF / DECIMAL_64_TYPE_VARIATION_REF in
variation_const.rs, plus matching entries in Arrow's
format/substrait/extension_types.yaml.
- Add
Decimal32 / Decimal64 arms in to_substrait_type_from_field and
to_substrait_literal, mirroring the existing Decimal128 arms.
- Extend
round_trip_types and round_trip_literals with Decimal32(9, 2)
and Decimal64(18, 2) cases.
- [optionally] Make
consumer/types.rs::from_substrait_type (and the literal consumer)
branch on type_variation_reference to pick the narrow Arrow width, falling
back to precision when the anchor is absent.
Describe alternatives you've considered
None
Additional context
No response
Is your feature request related to a problem or challenge?
datafusion-substrait's producer does not support ArrowDecimal32orDecimal64.Any DataFusion plan with a
Decimal32/Decimal64field or literal causesto_substrait_planto fail with:NotImplemented("Unsupported cast type: ")NotImplemented("Unsupported literal: Decimal32(..)")Describe the solution you'd like
Substrait has one abstract
Decimal { precision, scale }and distinguishesphysical widths via
type_variation_reference, same mechanism already used forDecimal256. So:DECIMAL_32_TYPE_VARIATION_REF/DECIMAL_64_TYPE_VARIATION_REFinvariation_const.rs, plus matching entries in Arrow'sformat/substrait/extension_types.yaml.Decimal32/Decimal64arms into_substrait_type_from_fieldandto_substrait_literal, mirroring the existingDecimal128arms.round_trip_typesandround_trip_literalswithDecimal32(9, 2)and
Decimal64(18, 2)cases.consumer/types.rs::from_substrait_type(and the literal consumer)branch on
type_variation_referenceto pick the narrow Arrow width, fallingback to precision when the anchor is absent.
Describe alternatives you've considered
None
Additional context
No response