Skip to content

perf: Optimize array_positions() for scalar needle - #20770

Merged
alamb merged 7 commits into
apache:mainfrom
neilconway:neilc/optimize-array-positions
Mar 18, 2026
Merged

perf: Optimize array_positions() for scalar needle#20770
alamb merged 7 commits into
apache:mainfrom
neilconway:neilc/optimize-array-positions

Conversation

@neilconway

@neilconwayneilconway commented Mar 7, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

array_positions previously compared the needle against each row's sub-array individually. When the needle is a scalar (the common case), we can do a single bulk arrow_ord::cmp::not_distinct comparison against the entire flat values buffer and then walk the result bitmap, which is significantly faster: the speedup on the array_positions() microbenchmarks ranges from 5x to 40x, depending on the size of the array.

The same pattern has already been applied to array_position (#20532), and previously to other array UDFs.

What changes are included in this PR?

  • Add benchmarks for array_positions.
  • Implement bulk-comparison optimization
  • Refactor array_position's existing fast path slightly for consistency
  • Code cleanup to use "haystack" and "needle" consistently, not vague terms like "list_array" and "element"
  • Add unit tests for array_positions with sliced ListArrays, for peace of mind
  • Add unit tests for sliced lists and sliced lists with nulls for the new array_positions fast path.

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

AI usage

Multiple AI tools were used to iterate on this PR. I have reviewed and understand the resulting code.

@github-actionsgithub-actionsBot added documentation Improvements or additions to documentation functions Changes to functions implementation labels Mar 7, 2026
@neilconway

Copy link
Copy Markdown
ContributorAuthor

Benchmarks:

 ┌────────────────┬────────┬────────┬─────────┐
│ Benchmark │ Before │ After │ Speedup │
├────────────────┼────────┼────────┼─────────┤
│ found_once/10 │ 1.46ms │ 39µs │ 37x │
├────────────────┼────────┼────────┼─────────┤
│ found_many/10 │ 1.59ms │ 67µs │ 24x │
├────────────────┼────────┼────────┼─────────┤
│ not_found/10 │ 1.28ms │ 32µs │ 40x │
├────────────────┼────────┼────────┼─────────┤
│ found_once/100 │ 1.97ms │ 140µs │ 14x │
├────────────────┼────────┼────────┼─────────┤
│ found_many/100 │ 4.06ms │ 501µs │ 8x │
├────────────────┼────────┼────────┼─────────┤
│ not_found/100 │ 1.82ms │ 132µs │ 14x │
├────────────────┼────────┼────────┼─────────┤
│ found_once/500 │ 4.05ms │ 601µs │ 6.7x │
├────────────────┼────────┼────────┼─────────┤
│ found_many/500 │ 12.0ms │ 2.51ms │ 4.8x │
├────────────────┼────────┼────────┼─────────┤
│ not_found/500 │ 3.82ms │ 588µs │ 6.5x │
└────────────────┴────────┴────────┴─────────┘

@github-actionsgithub-actionsBot added the sqllogictest SQL Logic Tests (.slt) label Mar 7, 2026
@neilconway

Copy link
Copy Markdown
ContributorAuthor

FYI @comphead -- this PR implements a similar optimization for array_positions() that we previously shipped for array_position (#20532).

Comment threaddatafusion/functions-nested/src/position.rs
Comment threaddatafusion/functions-nested/src/position.rs Outdated
Comment threaddatafusion/functions-nested/src/position.rs Outdated

@compheadcomphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @neilconway numbers look great, I left some minors

neilconwayand others added 2 commits March 16, 2026 16:23
Co-authored-by: Oleks V <comphead@users.noreply.github.com>
@neilconway

Copy link
Copy Markdown
ContributorAuthor

@comphead Thanks for the review! I addressed your comments.

@alamb
alamb added this pull request to the merge queueMar 18, 2026
@alamb

Copy link
Copy Markdown
Contributor

🚀

Merged via the queue into apache:main with commit b6b542eMar 18, 2026
31 checks passed
@neilconway
neilconway deleted the neilc/optimize-array-positions branch March 18, 2026 13:30
de-bgunter pushed a commit to de-bgunter/datafusion that referenced this pull request Mar 24, 2026
## Which issue does this PR close?
- Closesapache#20769.
## Rationale for this change
`array_positions` previously compared the needle against each row's
sub-array individually. When the needle is a scalar (the common case),
we can do a single bulk `arrow_ord::cmp::not_distinct` comparison
against the entire flat values buffer and then walk the result bitmap,
which is significantly faster: the speedup on the `array_positions()`
microbenchmarks ranges from 5x to 40x, depending on the size of the
array.
The same pattern has already been applied to `array_position` (apache#20532),
and previously to other array UDFs.
## What changes are included in this PR?
- Add benchmarks for `array_positions`.
- Implement bulk-comparison optimization
- Refactor `array_position`'s existing fast path slightly for
consistency
- Code cleanup to use "haystack" and "needle" consistently, not vague
terms like "list_array" and "element"
- Add unit tests for `array_positions` with sliced ListArrays, for peace
of mind
- Add unit tests for sliced lists and sliced lists with nulls for the
new `array_positions` fast path.
## Are these changes tested?
Yes.
## Are there any user-facing changes?
No.
## AI usage
Multiple AI tools were used to iterate on this PR. I have reviewed and
understand the resulting code.
---------
Co-authored-by: Oleks V <comphead@users.noreply.github.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationfunctionsChanges to functions implementationsqllogictestSQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Optimize array_positions() for scalar needle

3 participants

@neilconway@alamb@comphead