feat: optimize parquet reads with page-level filtering

[liangjie3138]

Purpose

Linked issue: close#137

Implement multi-level filtering optimization for Parquet file reading. By leveraging ColumnIndex statistics, the reader can skip non-matching data at the bucket, row group, and page levels, reducing I/O and decoding overhead.

Main Features

1. Page-level data filtering

   • ColumnIndexFilter: Filters data pages based on Parquet ColumnIndex min/max statistics. Supports predicates such as EQUAL, NOT_EQUAL, LESS_THAN, GREATER_THAN, IN, IS_NULL, and compound predicates with AND/OR.
   • PageFilteredRowGroupReader: Reads row groups after page-level filtering:
     • I/O layer: Skips non-matching pages through the data_page_filter callback.
     • Decoding layer: Skips rows through SkipRecords/ReadRecords.

2. Page-level prefetching
   Computes the byte ranges of required pages based on RowRanges and OffsetIndex, and uses ArrowPreBuffer for asynchronous prefetching.

Tests

• bucket_select_converter_test.cpp: Covers various predicate combinations, Timestamp type, and Cartesian product computation.
• column_index_filter_test.cpp: Covers all predicate types (EQUAL, IN, LESS_THAN, GREATER_THAN, IS_NULL, etc.) and AND/OR compound predicates.
• page_filtered_row_group_reader_test.cpp: Verifies filtering correctness, edge cases, and prefetching behavior.

API and Format

No public API changes. No impact on storage format or protocol.

Documentation

Not applicable.

Generative AI Tooling

Claude Code (Opus 4.6)

[CLAassistant]

All committers have signed the CLA.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 2 committers have signed the CLA.

❌ liangjie.liang
❌ liangjie3138

---

liangjie.liang seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[lxy-9602]

Thank you for your contribution! This is a highly complex and important feature, and your work on it is greatly appreciated.

Given the large scope of this PR, would it be possible to split it into smaller, focused changes? For example, separating the bucket predicate logic from the Parquet point lookup improvements could make each part easier to review and move forward incrementally.

Also, could you please fix the CI failures first so we can begin the review process?

We truly recognize the effort behind this change and look forward to helping get it merged smoothly.

[Copilot]

Pull request overview

Implements multi-level Parquet read optimizations (bucket selection + page/row-group filtering) by leveraging Parquet page indexes (ColumnIndex/OffsetIndex) and adding page-level prefetching to reduce I/O and decode work.

Changes:

• Added page-index-based filtering infrastructure (ColumnIndexFilter, RowRanges) and a page-filtered row-group reader with page-range prefetch support.
• Integrated page-level filtering/prefetch into ParquetFileBatchReader/FileReaderWrapper, and enabled writing page indexes via a new writer option.
• Added bucket-id derivation from predicates (BucketSelectConverter) and expanded scan bucket filtering to support multiple buckets.

Reviewed changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
src/paimon/format/parquet/row_ranges.h	Introduces RowRanges abstraction for page/row-range selection.
src/paimon/format/parquet/row_ranges.cpp	Implements range union/intersection/overlap/add logic used by page filtering.
src/paimon/format/parquet/parquet_writer_builder.cpp	Enables Parquet page index writing behind an option.
src/paimon/format/parquet/parquet_format_defs.h	Adds new read/write options for page-index functionality.
src/paimon/format/parquet/parquet_file_batch_reader.h	Adds page-index filtering API and logging member.
src/paimon/format/parquet/parquet_file_batch_reader.cpp	Wires page-level filtering + eager prepare to start prebuffer earlier.
src/paimon/format/parquet/page_filtered_row_group_reader.h	Declares page-filtered row group read + page-range computation.
src/paimon/format/parquet/page_filtered_row_group_reader.cpp	Implements decode skipping + page-range prefetch logic for filtered reads.
src/paimon/format/parquet/page_filtered_row_group_reader_test.cpp	Adds end-to-end tests for page filtering and page-range computation.
src/paimon/format/parquet/file_reader_wrapper.h	Extends wrapper to support page-filtered RG reads and page-range prebuffering.
src/paimon/format/parquet/file_reader_wrapper.cpp	Implements page-filtered RG scheduling + unified PreBufferRanges prefetch.
src/paimon/format/parquet/column_index_filter.h	Adds ColumnIndex-based predicate evaluation for page selection.
src/paimon/format/parquet/column_index_filter.cpp	Implements ColumnIndex-based page matching and RowRanges generation.
src/paimon/format/parquet/column_index_filter_test.cpp	Adds RowRanges unit tests + ColumnIndexFilter integration tests.
src/paimon/format/parquet/CMakeLists.txt	Registers new parquet sources/tests; adds Arrow source include path.
src/paimon/core/operation/key_value_file_store_scan.cpp	Derives bucket filter from predicates when not explicitly set.
src/paimon/core/operation/file_store_scan.h	Changes bucket filter to optional<set<int32_t>>; adds helpers.
src/paimon/core/operation/file_store_scan.cpp	Updates bucket filtering logic to handle multiple buckets.
src/paimon/core/operation/bucket_select_converter.h	Declares predicate→bucket-id derivation helper.
src/paimon/core/operation/bucket_select_converter.cpp	Implements bucket-id derivation compatible with Java hashing.
src/paimon/core/operation/bucket_select_converter_test.cpp	Adds tests for bucket derivation across predicate shapes/types.
src/paimon/core/operation/merge_file_split_read.cpp	Refactors loops to index-based iteration.
src/paimon/core/operation/abstract_split_read.cpp	Refactors loop to index-based iteration.
src/paimon/core/mergetree/compact/sort_merge_reader_with_min_heap.cpp	Refactors loop to index-based iteration.
src/paimon/common/utils/arrow/arrow_input_stream_adapter.h	Tracks outstanding async reads for safe destruction.
src/paimon/common/utils/arrow/arrow_input_stream_adapter.cpp	Waits for pending futures; prunes finished futures.
src/paimon/CMakeLists.txt	Registers new core operation source + test.
cmake_modules/arrow.diff	Patches Arrow Parquet reader to add PreBufferRanges/WhenBufferedRanges and cached page-range reads.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[lucasfang]

+1

[lxy-9602]

+1

[lxy-9602]

+1