Skip to content

Decimal32/Decimal64 support in Substrait Producer #21779

Description

@matal-nvidia

Is your feature request related to a problem or challenge?

datafusion-substrait's producer does not support Arrow Decimal32 or Decimal64.
Any DataFusion plan with a Decimal32/Decimal64 field or literal causes
to_substrait_plan to fail with:

NotImplemented("Unsupported cast type: ")
NotImplemented("Unsupported literal: Decimal32(..)")

Describe the solution you'd like

Substrait has one abstract Decimal { precision, scale } and distinguishes
physical widths via type_variation_reference, same mechanism already used for
Decimal256. So:

  1. Add DECIMAL_32_TYPE_VARIATION_REF / DECIMAL_64_TYPE_VARIATION_REF in
    variation_const.rs, plus matching entries in Arrow's
    format/substrait/extension_types.yaml.
  2. Add Decimal32 / Decimal64 arms in to_substrait_type_from_field and
    to_substrait_literal, mirroring the existing Decimal128 arms.
  3. Extend round_trip_types and round_trip_literals with Decimal32(9, 2)
    and Decimal64(18, 2) cases.
  4. [optionally] Make consumer/types.rs::from_substrait_type (and the literal consumer)
    branch on type_variation_reference to pick the narrow Arrow width, falling
    back to precision when the anchor is absent.

Describe alternatives you've considered

None

Additional context

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions