Skip to content

[improvement](parquet) Fuse dictionary selection planning - #66508

Closed
Gabriel39 wants to merge 6 commits into
apache:branch-4.1from
Gabriel39:dev/parquet-v2-dictionary-selection-plan-4.1
Closed

[improvement](parquet) Fuse dictionary selection planning#66508
Gabriel39 wants to merge 6 commits into
apache:branch-4.1from
Gabriel39:dev/parquet-v2-dictionary-selection-plan-4.1

Conversation

@Gabriel39

@Gabriel39Gabriel39 commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Predicate-only dictionary filtering currently expands definition levels and row selection into ColumnSelectVector, then scans that row-oriented state again to rebuild physical selection ranges and the selected NULL layout. Fragmented nullable filters spend significant CPU in these two planning passes before dictionary IDs are decoded.

Same-column conjuncts are combined into one dictionary-id bitmap before row decoding. Their incoming row selection therefore remains identity, which previously kept this common shape on the two-pass planner.

What is changed?

  • Build physical dictionary selection ranges, selected NULL layout, and the filtered-row count directly while merging definition-level runs with the input filter.
  • Pass the prepared plan into dictionary-index filtering instead of expanding and rescanning ColumnSelectVector.
  • Plan identity-input dictionary selections directly when definition levels are materially fragmented.
  • Use a bounded run-search fast path for no-NULL clustered filters, then switch to one linear scan after 32 runs.
  • Keep small batches, dense nullable inputs, compact identity inputs, and sparse clustered nullable inputs on the existing planner.
  • Add a self-validating 220-case dictionary-selection benchmark matrix, including 20 identity-input cases.

This PR does not change predicate decomposition, supported data types, decode strategy, or merge-read behavior. It is stacked on #66504.

Performance

Release build, 65,536 rows, CPU-pinned runs with 7 repetitions. The table reports median CPU time; lower is better.

Input selectionNULL layoutNULL rateExisting plannerSelected plannerChange
IdentityAlternating1%11,066 ns8,383 ns-24.24%
IdentityAlternating10%98,943 ns60,185 ns-39.17%
IdentityAlternating50%580,955 ns319,267 ns-45.04%
IdentityAlternating90%87,733 ns56,741 ns-35.33%
IdentityNo NULL0%1,451 ns1,458 ns+0.47%
IdentityClustered1%1,488 ns1,497 ns+0.59%

The last two negative controls are routed to the existing planner; their sub-1% differences are measurement noise from identical code paths.

Validation

  • 7 targeted ASAN Parquet dictionary and nullable-selection tests passed.
  • All 220 dictionary-selection scenarios completed and self-validated identical physical ranges, selected NULL layout, and filtered-row counts.
  • All affected C/C++ files pass clang-format 16.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39

Copy link
Copy Markdown
ContributorAuthor

run buildall

@Gabriel39

Copy link
Copy Markdown
ContributorAuthor

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z.
Workflow run: https://github.com/apache/doris/actions/runs/31020463260

Please trigger /review again after that time.

@Gabriel39

Copy link
Copy Markdown
ContributorAuthor

run buildall

@Gabriel39

Copy link
Copy Markdown
ContributorAuthor

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z.
Workflow run: https://github.com/apache/doris/actions/runs/31065828575

Please trigger /review again after that time.

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 65.23% (362/555) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage58.33% (24737/42411)
Line Coverage42.59% (249504/585801)
Region Coverage38.64% (197096/510130)
Branch Coverage40.03% (90279/225521)

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@Gabriel39@hello-stephen