Skip to content

FFI Scalar function wrapper does not preserve Scalar/Array-ness of a ColumnarValue #18201

Description

@paleolimbot

Is your feature request related to a problem or challenge?

In the FFI_ScalarUDF, I noticed that the Scalar/Array nature of an argument is not preserved. This is not a bug, as scalar values are correctly repeated to the number of rows, but it does make the FFI interface of limited value for what I'd like to do with it (because some geometry predicates are significantly faster if we "prepare" a scalar value once per batch instead of once per row).

Describe the solution you'd like

I think we can use a comparison between the num_rows argument and length of the wrapper array to decide if an argument is a scalar or an array (with a corner case when there's num_rows of 1). This would break the FFI on the "new function producer but old function caller" end although I don't think it would segfault anything if it's of interest.

Describe alternatives you've considered

Our own FFI (or just building everything at once)

Additional context

let args = args
.into_iter()
.map(|arr| {
from_ffi(arr.array, &arr.schema.0)
.map(|v| ColumnarValue::Array(arrow::array::make_array(v)))
})
.collect::<std::result::Result<_, _>>();

let args = args
.into_iter()
.map(|v| v.to_array(number_rows))

PR where I discovered this: apache/sedona-db#228

cc @kylebarron who may run across this issue

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions