Skip to content

Prune values across the query executor instead of on the calling thread - #19513

Open
xiangfu0 wants to merge 2 commits into
masterfrom
xiangfu0/data-3221-12-parallel-value-pruning
Open

Prune values across the query executor instead of on the calling thread#19513
xiangfu0 wants to merge 2 commits into
masterfrom
xiangfu0/data-3221-12-parallel-value-pruning

Conversation

@xiangfu0

@xiangfu0 xiangfu0 commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Value-based segment pruning uses the query executor for lists larger than 10 segments, bounded by the query's execution-thread limit. Small lists and calls without an executor retain serial execution.

ValueBasedSegmentPruner owns the shared serial/parallel loops, task-local caches, deadline handling, interruption checks, and per-segment fetch-context acquisition/release. BloomFilterSegmentPruner delegates to that implementation while retaining its prefetch planning and cleanup. Duplicate execution code and stale threshold comments are removed. Parallel results may have a different order.

Validation

  • Built and tested against Apache master, independently of the DATA-3221 heap stack.
  • 46 tests passed across ColumnValueSegmentPrunerTest, BloomFilterSegmentPrunerTest, SegmentPrunerServiceTest, and QueryMultiThreadingUtilsTest.
  • Covers selected-set equivalence, executor use, serial fallback, expired deadlines, worker failures, interruption between segments, and Bloom fetch-context cleanup on success, failure, and cancellation.
  • Spotless, Checkstyle, license formatting, and license checks passed for pinot-core.
  • The additional strict compiler-warning build stopped in unchanged ZstandardDecompressor.java:51 because the Zstandard dependency references an unavailable JetBrains NotNull annotation. Isolated compilation reproduced this on the exact master source; normal compilation succeeds, and supplying the annotation jar also makes strict compilation succeed.

@xiangfu0 xiangfu0 added the performance Related to performance optimization label Sep 9, 2026
@xiangfu0
xiangfu0 force-pushed the xiangfu0/data-3221-12-parallel-value-pruning branch from 8c6e534 to 785926d Compare September 9, 2026 01:56
@xiangfu0
xiangfu0 force-pushed the xiangfu0/data-3221-11-metadata-only-pruning branch from 6f1c691 to 9a69689 Compare September 9, 2026 01:56
Use the query executor for value pruning when the segment count exceeds
the parallel threshold, with task-local caches and bounded concurrency.
Keep the serial path for small inputs or when no executor is supplied.

Stop interrupted workers before visiting another segment so cancellation
can finish while the deadline helper waits for workers to release segments.
Cover executor use, fallback, deadlines, worker failures and interruption.
@xiangfu0
xiangfu0 changed the base branch from xiangfu0/data-3221-11-metadata-only-pruning to master September 10, 2026 23:02
@xiangfu0
xiangfu0 force-pushed the xiangfu0/data-3221-12-parallel-value-pruning branch from 785926d to c375b71 Compare September 10, 2026 23:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance Related to performance optimization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant