[enhancement](scan) Optimize Parquet V2 predicate filtering and fixed-binary decimal decoding by Gabriel39 · Pull Request #66396 · apache/doris

Gabriel39 · 2026-08-03T14:01:48Z

What problem does this PR solve?

Related PR: #66360, #66379

Problem Summary:

This PR brings two complementary Parquet V2 optimizations from branch-4.1 to master:

reduce predicate-filtering overhead and correctly propagate late runtime filters through the scanner and reader layers;
avoid unnecessarily decoding same-scale fixed-binary decimals through Int256 when their physical width permits a narrower native type.

Parquet V2 predicate filtering (#66360)

Keep identity selection-vector state implicit and compact selected rows in bulk.
Retain the selection scratch high-water mark across scanner batches and specialize the first compaction from implicit identity.
Refresh late runtime-filter requests at safe row-group boundaries.
Re-run footer-statistics pruning and reset adaptive predicate state for unopened row groups after a refresh.
Preserve real COUNT(*) carrier values while runtime filters are pending.
Initialize refreshed JNI predicates and attribute refresh work to TableReader, FileReader, and Parquet profiles.
Preserve Hudi and Paimon child-reader predicate state.
Remove query-scoped dictionary-filter cache state.
Share immutable VDirectInPredicate pruning materialization across split-local expression clones.
Add correctness-checked selection-vector and direct-IN lifecycle microbenchmarks.

Fixed-binary decimal decoding (#66379)

Decode same-scale Parquet FIXED_LEN_BYTE_ARRAY decimals with int32_t, int64_t, or Int128, selected from the physical width, instead of always using Int256.
Use unaligned full-width big-endian loads for 4-, 8-, and 16-byte values while preserving sign extension for shorter widths.
Validate target precision before narrowing and preserve both permissive and strict conversion behavior.
Keep rescaling and wider values on the existing Int256 path.
Cover positive and negative precision boundaries, shorter signed inputs, strict rollback, permissive null marking, and the complete int32 source domain.

Performance

The results below come from the original branch-4.1 PR benchmarks in #66360 and #66379. This PR applies the same implementation to master; the performance benchmarks were not re-run as part of this forward-port.

Selection-vector processing

The benchmark validates every surviving original row ID after the timed region. It used the same Clang -O3 -DNDEBUG -mavx2 benchmark source for the branch base, the pre-fix PR, and the final implementation, with one pinned CPU, three warmups, eight adjacent A-B-B-A quartets, and at least 0.3 seconds per invocation. Negative percentages are improvements.

Operation	Final selectivity	Final vs pre-fix PR	Final vs branch base
Identity initialization	100%	-15.23%	-99.12%
Row filter	1%	-24.23%	-23.76%
Row filter	50%	-16.10%	-34.50%
Row filter	90%	-45.91%	-45.95%
Row filter	100%	-31.72%	-17.32%
Successive filters	1%	-33.25%	-35.79%
Successive filters	50%	-29.80%	-35.10%
Successive filters	90%	-25.27%	-25.16%
Successive filters	100%	-24.93%	-23.72%

Compared with the branch base, identity initialization improved by 99.12%, row filtering improved by 17.32%-45.95%, and successive filtering improved by 23.72%-35.79%. All final-vs-base paired-ratio CVs were at most 5.85%.

Direct-IN expression lifecycle

FileScannerExpr/direct_in_clone_prepare_open isolates deep-clone, prepare, and open for an already prepared direct-IN runtime filter. Set construction and the original fragment prepare/open are outside the timed region. The shared and forced-rematerialization implementations ran in the same Release binary on one pinned CPU, with 10 repetitions and at least 0.5 seconds per repetition.

IN values	Rematerialize median	Shared median	Speedup
128	207.470 us	1.634 us	126.9x
1,024	1.674 ms	1.642 us	1,019.5x
8,192	13.514 ms	1.672 us	8,082.2x
65,536	108.337 ms	1.650 us	65,663.1x

The shared path remains approximately constant because split-local clones reuse immutable pruning state instead of rebuilding it for every split.

Reader-level regression guardrail

The Parquet reader benchmark covered nullable INT32 predicate scans with a lazy payload for both PLAIN and dictionary encoding. Across 1%, 10%, 50%, and 90% selectivity, CPU-time changes ranged from -1.34% to +1.41% with mixed signs, so it did not detect a material aggregate reader-level regression.

Fixed-binary decimal decoding

The benchmark decoded 65,536 values per iteration through DataTypeDecimalSerDe::read_column_from_parquet, pinned to one CPU, with three warmups followed by 10 repetitions in A-B-B-A order.

Target / physical width	Before median CPU	After median CPU	Speedup	CPU reduction
Decimal32 / 4 bytes	1,359,514 ns	88,527 ns	15.36x	93.49%
Decimal64 / 8 bytes	1,634,757 ns	90,609 ns	18.04x	94.46%
Decimal128 / 16 bytes	2,206,272 ns	152,823 ns	14.44x	93.07%

The optimized path was 14.44x-18.04x faster in this benchmark, reducing CPU time by 93.07%-94.46%. The benchmark host was heavily loaded and CPU frequency scaling was enabled, so the exact ratios are noisy; however, the before/after median ranges did not overlap in any A-B-B-A stage.

Validation on master

705 filtered ASAN BE unit tests from 53 suites passed, covering Parquet V2, FileScannerV2, TableReader, Hudi/Paimon/JNI readers, SelectionVector, direct-IN pruning, and decimal SerDe.
BE formatting and strict clang-format checks passed.
git diff --check passed.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

…pache#66360) Backport the selected Parquet V2 direct-predicate filtering changes from on this branch. - keep identity selection-vector state implicit and compact selected rows in bulk - retain the selection scratch high-water mark across scanner batches and specialize first compaction from implicit identity - refresh late runtime-filter requests at safe row-group boundaries - re-run footer-statistics pruning and reset adaptive predicate state for unopened row groups after a refresh - preserve real COUNT(*) carrier values while runtime filters are pending - initialize refreshed JNI predicates and attribute refresh work to TableReader/FileReader/Parquet profiles - preserve Hudi/Paimon child-reader predicate state - remove query-scoped dictionary-filter cache state - share immutable `VDirectInPredicate` pruning materialization across split-local expression clones - add correctness-checked selection and direct-IN lifecycle microbenchmarks - `./run-be-ut.sh --run --filter='FileScannerV2Test.*:*Parquet*:*TableReaderTest.*:Hudi*ReaderTest.*:Paimon*ReaderTest.*:SelectionVectorTest.*:DictionaryFilterCostTest.*' -j48` - 639 tests from 47 test suites passed under ASAN - targeted late-RF, COUNT(*), dictionary-snapshot, shared-IN-state, and SelectionVector tests - 19 tests from 5 test suites passed under ASAN - Release benchmark build and smoke run - expected registrations: 228 decoder, 92 kernel, 25 selection, 167 reader, and 8 expression-lifecycle cases - all 25 selection and 8 expression-lifecycle cases executed with zero benchmark errors - `git diff --check` The final benchmark source validates every surviving original row ID after the timed region. Base, pre-fix PR, and final binaries use the same benchmark source and Clang `-O3 -DNDEBUG -mavx2` on the same host. Each comparison uses one pinned CPU, three warmups, eight adjacent A-B-B-A quartets, and at least 0.3 seconds per invocation. The table reports median paired CPU-time ratios; negative values are improvements. | Operation | Final selectivity | Final vs pre-fix PR | Final vs branch base | |---|---:|---:|---:| | Identity initialization | 100% | -15.23% | -99.12% | | Row filter | 1% | -24.23% | -23.76% | | Row filter | 50% | -16.10% | -34.50% | | Row filter | 90% | -45.91% | -45.95% | | Row filter | 100% | -31.72% | -17.32% | | Successive filters | 1% | -33.25% | -35.79% | | Successive filters | 50% | -29.80% | -35.10% | | Successive filters | 90% | -25.27% | -25.16% | | Successive filters | 100% | -24.93% | -23.72% | All final-vs-base paired-ratio CVs are at most 5.85%. The previous 16.43%/59.94% dense row-filter regressions and 27.92%-61.05% successive-filter regressions are no longer present. Retaining `_owned` avoids repeated value initialization; the implicit-identity specialization removes the remaining source/coordinate branches from the first compaction. `FileScannerExpr/direct_in_clone_prepare_open` isolates deep-clone, prepare, and open for an already prepared direct-IN runtime filter. Set construction and the original fragment prepare/open are outside the timed region. Shared and forced-rematerialization implementations run in the same Release binary on one pinned CPU, with 10 repetitions and at least 0.5 seconds per repetition. | IN values | Rematerialize median | Shared median | Speedup | |---:|---:|---:|---:| | 128 | 207.470 us | 1.634 us | 126.9x | | 1,024 | 1.674 ms | 1.642 us | 1,019.5x | | 8,192 | 13.514 ms | 1.672 us | 8,082.2x | | 65,536 | 108.337 ms | 1.650 us | 65,663.1x | The shared path remains approximately constant because split clones reuse the immutable, fragment-originated pruning state; the rematerialization path scales linearly with runtime-filter cardinality. The earlier reader-level comparison used an identical Parquet-only Release benchmark binary on the same host, with one pinned CPU, warm cache, three warmups, A-B-B-A order, 10 repetitions, and at least one second per repetition. It covers nullable INT32 predicate scans with a lazy payload for PLAIN and dictionary encoding. | Encoding | Selectivity | CPU time vs base | Wall time vs base | |---|---:|---:|---:| | Dictionary | 1% | +0.44% | +0.46% | | Dictionary | 10% | +1.13% | +1.22% | | Dictionary | 50% | +1.41% | +1.51% | | Dictionary | 90% | +0.41% | +0.53% | | PLAIN | 1% | -0.65% | -0.66% | | PLAIN | 10% | +0.32% | +0.38% | | PLAIN | 50% | -1.21% | -1.18% | | PLAIN | 90% | -1.34% | -1.29% | The reader-level point estimates span -1.34% to +1.41% CPU time with mixed signs, so this benchmark did not detect a material aggregate regression. It starts at `format::parquet::ParquetReader`; it does not cover scanner scheduling or end-to-end SQL execution.

## Proposed changes - Decode same-scale Parquet `FIXED_LEN_BYTE_ARRAY` decimals with an `int32_t`, `int64_t`, or `Int128` source selected from the physical width instead of always using `Int256`. - Use unaligned full-width big-endian loads for 4-, 8-, and 16-byte values, while preserving sign extension for shorter widths. - Validate the target decimal precision before narrowing, preserve permissive/strict conversion behavior, and keep rescaling and wider values on the existing `Int256` path. - Add regression coverage for positive and negative precision boundaries, shorter signed inputs, strict rollback, permissive null marking, and the complete int32 source domain. ## Validation - ASAN BE unit tests: 8 tests from `DataTypeSerDeParquetTest` passed. - Release microbenchmark: 65,536 values per iteration through `DataTypeDecimalSerDe::read_column_from_parquet`, pinned to one CPU. Each stage used 3 warmups followed by 10 repetitions in ABBA order. | Target / physical width | Before median CPU | After median CPU | Speedup | CPU reduction | | --- | ---: | ---: | ---: | ---: | | Decimal32 / 4 bytes | 1,359,514 ns | 88,527 ns | 15.36x | 93.49% | | Decimal64 / 8 bytes | 1,634,757 ns | 90,609 ns | 18.04x | 94.46% | | Decimal128 / 16 bytes | 2,206,272 ns | 152,823 ns | 14.44x | 93.07% | The benchmark host was heavily loaded and CPU frequency scaling was enabled, so the exact ratios are noisy. However, the before/after median ranges did not overlap in any ABBA stage. A final optimized-build smoke run measured median CPU times of 95,077 ns, 94,635 ns, and 145,417 ns with CPU CVs of 0.41%, 1.76%, and 0.64%, respectively.

hello-stephen · 2026-08-03T14:01:53Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Gabriel39 · 2026-08-03T14:02:00Z

run buildall

Gabriel39 · 2026-08-03T14:02:02Z

/review

hello-stephen · 2026-08-03T17:15:33Z

TPC-H: Total hot run time: 28910 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 474b0c7123934178b22a16278bbf77c120d8be9f, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17615	3961	4111	3961
q2	2038	321	194	194
q3	10319	1467	806	806
q4	4688	477	339	339
q5	7509	832	557	557
q6	183	171	140	140
q7	752	811	595	595
q8	9337	1615	1555	1555
q9	5304	4041	4042	4041
q10	6762	1623	1363	1363
q11	495	382	337	337
q12	723	580	454	454
q13	18151	3364	2713	2713
q14	261	257	240	240
q15	q16	737	723	651	651
q17	950	1019	1005	1005
q18	6837	5614	5614	5614
q19	1345	1330	1039	1039
q20	772	702	608	608
q21	5877	2540	2399	2399
q22	429	355	299	299
Total cold run time: 101084 ms
Total hot run time: 28910 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4370	4200	4180	4180
q2	286	313	204	204
q3	4552	4942	4330	4330
q4	2170	2261	1414	1414
q5	4205	4110	4108	4108
q6	224	172	129	129
q7	1681	1570	1421	1421
q8	2976	2155	2101	2101
q9	7447	7393	7532	7393
q10	4319	4267	3911	3911
q11	561	398	354	354
q12	700	723	519	519
q13	3290	3552	2987	2987
q14	296	298	280	280
q15	q16	682	717	621	621
q17	1285	1268	1266	1266
q18	7900	7104	7176	7104
q19	1146	1133	1118	1118
q20	2204	2188	1892	1892
q21	5224	4585	4382	4382
q22	508	459	411	411
Total cold run time: 56026 ms
Total hot run time: 50125 ms

hello-stephen · 2026-08-03T17:26:19Z

TPC-DS: Total hot run time: 168998 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 474b0c7123934178b22a16278bbf77c120d8be9f, data reload: false
query5	4326	613	461	461
query6	457	219	200	200
query7	4871	543	326	326
query8	329	184	174	174
query9	8748	4046	4048	4046
query10	441	361	285	285
query11	5789	2218	1979	1979
query12	145	97	95	95
query13	1254	582	420	420
query14	6089	4703	4357	4357
query14_1	3805	3791	3804	3791
query15	226	209	177	177
query16	1002	505	476	476
query17	951	723	579	579
query18	2488	480	349	349
query19	214	202	149	149
query20	105	103	101	101
query21	238	155	136	136
query22	13012	13000	12887	12887
query23	17531	16446	15984	15984
query23_1	16141	16205	16044	16044
query24	7587	1702	1256	1256
query24_1	1271	1241	1217	1217
query25	570	458	398	398
query26	1330	375	213	213
query27	2574	590	385	385
query28	4529	2063	2063	2063
query29	1073	621	520	520
query30	341	267	221	221
query31	1114	1071	950	950
query32	117	76	59	59
query33	518	288	239	239
query34	1170	1099	647	647
query35	728	754	639	639
query36	776	795	702	702
query37	143	114	94	94
query38	1832	1642	1584	1584
query39	835	839	790	790
query39_1	785	782	774	774
query40	236	166	144	144
query41	64	63	62	62
query42	92	89	90	89
query43	311	311	276	276
query44	1409	798	777	777
query45	187	173	166	166
query46	1051	1176	695	695
query47	1558	1578	1444	1444
query48	458	414	297	297
query49	574	418	300	300
query50	1068	429	345	345
query51	10399	10422	10555	10422
query52	82	88	74	74
query53	249	267	186	186
query54	272	260	223	223
query55	74	73	67	67
query56	313	317	295	295
query57	1036	1011	909	909
query58	295	259	251	251
query59	1540	1590	1450	1450
query60	303	266	236	236
query61	149	147	148	147
query62	398	325	265	265
query63	231	195	201	195
query64	2820	1019	844	844
query65	3884	3819	3813	3813
query66	1822	466	363	363
query67	28192	28103	28010	28010
query68	3209	1520	1084	1084
query69	406	289	267	267
query70	884	819	761	761
query71	351	320	312	312
query72	3024	2817	2037	2037
query73	825	800	434	434
query74	4622	4471	4323	4323
query75	2426	2332	1983	1983
query76	2381	1127	795	795
query77	335	369	273	273
query78	11165	11154	10598	10598
query79	1403	1152	761	761
query80	663	544	468	468
query81	452	327	283	283
query82	568	150	116	116
query83	403	319	307	307
query84	331	161	132	132
query85	934	618	509	509
query86	322	255	203	203
query87	1785	1786	1687	1687
query88	3744	2844	2812	2812
query89	402	323	286	286
query90	2031	202	199	199
query91	199	189	159	159
query92	64	62	55	55
query93	1644	1608	924	924
query94	538	345	304	304
query95	805	511	558	511
query96	1048	770	358	358
query97	2458	2450	2324	2324
query98	200	200	193	193
query99	715	723	596	596
Total cold run time: 254988 ms
Total hot run time: 168998 ms

hello-stephen · 2026-08-03T17:31:15Z

ClickBench: Total hot run time: 23.86 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 474b0c7123934178b22a16278bbf77c120d8be9f, data reload: false
query1	0.01	0.01	0.00
query2	0.10	0.05	0.05
query3	0.25	0.14	0.13
query4	1.61	0.14	0.13
query5	0.23	0.22	0.21
query6	1.16	0.83	0.80
query7	0.04	0.01	0.01
query8	0.05	0.04	0.04
query9	0.38	0.31	0.33
query10	0.54	0.61	0.54
query11	0.19	0.15	0.13
query12	0.16	0.14	0.14
query13	0.46	0.46	0.47
query14	1.01	0.98	0.98
query15	0.63	0.58	0.60
query16	0.31	0.31	0.32
query17	1.10	1.08	1.07
query18	0.22	0.20	0.20
query19	2.01	1.99	1.95
query20	0.01	0.02	0.01
query21	15.44	0.21	0.15
query22	4.88	0.05	0.05
query23	16.13	0.32	0.13
query24	2.95	0.41	0.31
query25	0.11	0.06	0.04
query26	0.72	0.21	0.15
query27	0.04	0.04	0.04
query28	3.54	0.81	0.34
query29	12.46	4.00	3.20
query30	0.27	0.16	0.17
query31	2.77	0.55	0.31
query32	3.23	0.60	0.49
query33	3.12	3.23	3.20
query34	15.55	3.94	3.27
query35	3.18	3.20	3.24
query36	0.54	0.43	0.43
query37	0.10	0.06	0.07
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.18	0.15	0.14
query41	0.11	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 95.95 s
Total hot run time: 23.86 s

github-actions · 2026-08-03T17:39:55Z

Codex automated review failed and did not complete.

Error: Review step was failure (possibly timeout or cancelled)
Workflow run: https://github.com/apache/doris/actions/runs/30820747155

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

hello-stephen · 2026-08-03T18:32:51Z

BE UT Coverage Report

Increment line coverage 89.37% (353/395) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	59.31% (25740/43401)
Line Coverage	43.40% (258629/595899)
Region Coverage	39.06% (204965/524699)
Branch Coverage	40.41% (93541/231485)

github-actions · 2026-08-04T01:41:27Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-08-04T01:41:30Z

PR approved by anyone and no changes requested.

…-binary decimal decoding (apache#66396) ### What problem does this PR solve? Related PR: apache#66360, apache#66379 Problem Summary: This PR brings two complementary Parquet V2 optimizations from `branch-4.1` to `master`: 1. reduce predicate-filtering overhead and correctly propagate late runtime filters through the scanner and reader layers; 2. avoid unnecessarily decoding same-scale fixed-binary decimals through `Int256` when their physical width permits a narrower native type. #### Parquet V2 predicate filtering (apache#66360) - Keep identity selection-vector state implicit and compact selected rows in bulk. - Retain the selection scratch high-water mark across scanner batches and specialize the first compaction from implicit identity. - Refresh late runtime-filter requests at safe row-group boundaries. - Re-run footer-statistics pruning and reset adaptive predicate state for unopened row groups after a refresh. - Preserve real `COUNT(*)` carrier values while runtime filters are pending. - Initialize refreshed JNI predicates and attribute refresh work to TableReader, FileReader, and Parquet profiles. - Preserve Hudi and Paimon child-reader predicate state. - Remove query-scoped dictionary-filter cache state. - Share immutable `VDirectInPredicate` pruning materialization across split-local expression clones. - Add correctness-checked selection-vector and direct-IN lifecycle microbenchmarks. #### Fixed-binary decimal decoding (apache#66379) - Decode same-scale Parquet `FIXED_LEN_BYTE_ARRAY` decimals with `int32_t`, `int64_t`, or `Int128`, selected from the physical width, instead of always using `Int256`. - Use unaligned full-width big-endian loads for 4-, 8-, and 16-byte values while preserving sign extension for shorter widths. - Validate target precision before narrowing and preserve both permissive and strict conversion behavior. - Keep rescaling and wider values on the existing `Int256` path. - Cover positive and negative precision boundaries, shorter signed inputs, strict rollback, permissive null marking, and the complete int32 source domain. ### Performance The results below come from the original `branch-4.1` PR benchmarks in apache#66360 and apache#66379. This PR applies the same implementation to `master`; the performance benchmarks were not re-run as part of this forward-port. #### Selection-vector processing The benchmark validates every surviving original row ID after the timed region. It used the same Clang `-O3 -DNDEBUG -mavx2` benchmark source for the branch base, the pre-fix PR, and the final implementation, with one pinned CPU, three warmups, eight adjacent A-B-B-A quartets, and at least 0.3 seconds per invocation. Negative percentages are improvements. | Operation | Final selectivity | Final vs pre-fix PR | Final vs branch base | | --- | ---: | ---: | ---: | | Identity initialization | 100% | -15.23% | -99.12% | | Row filter | 1% | -24.23% | -23.76% | | Row filter | 50% | -16.10% | -34.50% | | Row filter | 90% | -45.91% | -45.95% | | Row filter | 100% | -31.72% | -17.32% | | Successive filters | 1% | -33.25% | -35.79% | | Successive filters | 50% | -29.80% | -35.10% | | Successive filters | 90% | -25.27% | -25.16% | | Successive filters | 100% | -24.93% | -23.72% | Compared with the branch base, identity initialization improved by 99.12%, row filtering improved by 17.32%-45.95%, and successive filtering improved by 23.72%-35.79%. All final-vs-base paired-ratio CVs were at most 5.85%. #### Direct-IN expression lifecycle `FileScannerExpr/direct_in_clone_prepare_open` isolates deep-clone, prepare, and open for an already prepared direct-IN runtime filter. Set construction and the original fragment prepare/open are outside the timed region. The shared and forced-rematerialization implementations ran in the same Release binary on one pinned CPU, with 10 repetitions and at least 0.5 seconds per repetition. | IN values | Rematerialize median | Shared median | Speedup | | ---: | ---: | ---: | ---: | | 128 | 207.470 us | 1.634 us | 126.9x | | 1,024 | 1.674 ms | 1.642 us | 1,019.5x | | 8,192 | 13.514 ms | 1.672 us | 8,082.2x | | 65,536 | 108.337 ms | 1.650 us | 65,663.1x | The shared path remains approximately constant because split-local clones reuse immutable pruning state instead of rebuilding it for every split. #### Reader-level regression guardrail The Parquet reader benchmark covered nullable INT32 predicate scans with a lazy payload for both PLAIN and dictionary encoding. Across 1%, 10%, 50%, and 90% selectivity, CPU-time changes ranged from -1.34% to +1.41% with mixed signs, so it did not detect a material aggregate reader-level regression. #### Fixed-binary decimal decoding The benchmark decoded 65,536 values per iteration through `DataTypeDecimalSerDe::read_column_from_parquet`, pinned to one CPU, with three warmups followed by 10 repetitions in A-B-B-A order. | Target / physical width | Before median CPU | After median CPU | Speedup | CPU reduction | | --- | ---: | ---: | ---: | ---: | | Decimal32 / 4 bytes | 1,359,514 ns | 88,527 ns | 15.36x | 93.49% | | Decimal64 / 8 bytes | 1,634,757 ns | 90,609 ns | 18.04x | 94.46% | | Decimal128 / 16 bytes | 2,206,272 ns | 152,823 ns | 14.44x | 93.07% | The optimized path was 14.44x-18.04x faster in this benchmark, reducing CPU time by 93.07%-94.46%. The benchmark host was heavily loaded and CPU frequency scaling was enabled, so the exact ratios are noisy; however, the before/after median ranges did not overlap in any A-B-B-A stage. ### Validation on master - 705 filtered ASAN BE unit tests from 53 suites passed, covering Parquet V2, FileScannerV2, TableReader, Hudi/Paimon/JNI readers, SelectionVector, direct-IN pruning, and decimal SerDe. - BE formatting and strict `clang-format` checks passed. - `git diff --check` passed. ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [x] No. - [ ] Yes. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

Gabriel39 added 2 commits August 3, 2026 21:09

Gabriel39 marked this pull request as ready for review August 3, 2026 14:06

Gabriel39 requested a review from yiguolei as a code owner August 3, 2026 14:06

Gabriel39 added the dev/4.1.4-merged label Aug 3, 2026

Gabriel39 changed the title ~~[enhancement](scan) Forward-port Parquet V2 predicate and decimal optimizations~~[enhancement](scan) Optimize Parquet V2 predicate filtering and fixed-binary decimal decodingAug 3, 2026

yiguolei approved these changes Aug 4, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 4, 2026

github-actionsBot added the reviewed label Aug 4, 2026

Gabriel39 merged commit b03968f into apache:masterAug 4, 2026
33 of 35 checks passed

HappenLee mentioned this pull request Aug 11, 2026
[feature](query) Generate TopN filters from aggregation #66652
Open

Navigation Menu

Uh oh!

[enhancement](scan) Optimize Parquet V2 predicate filtering and fixed-binary decimal decoding - #66396

[enhancement](scan) Optimize Parquet V2 predicate filtering and fixed-binary decimal decoding#66396
Gabriel39 merged 2 commits into
apache:masterfrom
Gabriel39:dev/parquet-v2-optimizations-master

Gabriel39 commented Aug 3, 2026•
edited
Loading

Uh oh!

hello-stephen commented Aug 3, 2026

Uh oh!

Gabriel39 commented Aug 3, 2026

Uh oh!

Gabriel39 commented Aug 3, 2026

Uh oh!

hello-stephen commented Aug 3, 2026

Uh oh!

hello-stephen commented Aug 3, 2026

Uh oh!

hello-stephen commented Aug 3, 2026

Uh oh!

github-actionsBot commented Aug 3, 2026

Uh oh!

hello-stephen commented Aug 3, 2026

Uh oh!

github-actionsBot commented Aug 4, 2026

Uh oh!

github-actionsBot commented Aug 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Gabriel39 commented Aug 3, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Parquet V2 predicate filtering (#66360)

Fixed-binary decimal decoding (#66379)

Performance

Selection-vector processing

Direct-IN expression lifecycle

Reader-level regression guardrail

Fixed-binary decimal decoding

Validation on master

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Aug 3, 2026

Uh oh!

Gabriel39 commented Aug 3, 2026

Uh oh!

Gabriel39 commented Aug 3, 2026

Uh oh!

hello-stephen commented Aug 3, 2026

Uh oh!

hello-stephen commented Aug 3, 2026

Uh oh!

hello-stephen commented Aug 3, 2026

Uh oh!

github-actionsBot commented Aug 3, 2026

Uh oh!

hello-stephen commented Aug 3, 2026

BE UT Coverage Report

Uh oh!

github-actionsBot commented Aug 4, 2026

Uh oh!

github-actionsBot commented Aug 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Gabriel39 commented Aug 3, 2026•
edited
Loading