GH-50216: [C++][Parquet] Add RleBitPackedToBitmapDecoder by AntoinePrv · Pull Request #50217 · apache/arrow

AntoinePrv · 2026-06-18T09:25:44Z

Rationale for this change

Add a RleBitPackedToBitmapDecoder capable of decoding a Parquet mixed Rle / BitPacked byte stream directly into a bitmap. This is an optimization to be used when target and source bit_width = 1 that will improve decoding nullable columns in a follow-up PR.

What changes are included in this PR?

New classes, that reuse previous Runs and parser.
Their API is slighly different from the existing one as it fits a partilcular case.

RleRunToBitmapDecoder
BitPackedRunToBitmapDecoder
RleRunToBitmapDecoder

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

GitHub Issue: [C++][Parquet] Rle-Bitpacked to Bitmap decoder #50216

github-actions · 2026-06-18T09:30:29Z

⚠️ GitHub issue #50216has been automatically assigned in GitHub to PR creator.

Copilot

Pull request overview

Adds a specialized path for decoding Parquet mixed RLE / BitPacked streams with bit_width=1 directly into Arrow bitmaps, aiming to speed up validity/definition-level decoding.

Changes:

Added bitmap-oriented decoders for RleRun and BitPackedRun, plus RleBitPackedToBitmapDecoder for mixed streams.
Refactored RleBitPackedParser to provide ParseWithCallable() and adjusted RleBitPackedDecoder to use it.
Added new bit utilities (Get32Bits, CopyBits) and comprehensive unit tests for the new bitmap decoders.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
cpp/src/arrow/util/rle_encoding_internal.h	Adds `RleRun::value()`, moves callable-based parsing into the parser, and updates `RleBitPackedDecoder` accordingly.
cpp/src/arrow/util/rle_bitmap_internal.h	Introduces bitmap span + run-to-bitmap decoders and the new `RleBitPackedToBitmapDecoder`.
cpp/src/arrow/util/rle_bitmap_test.cc	Adds tests covering alignment, chunking, reset, and truncated input scenarios for bitmap decoding.
cpp/src/arrow/util/CMakeLists.txt	Registers the new test file in the util test target.
cpp/src/arrow/util/bit_util.h	Adds `Get32Bits` and `CopyBits` helpers used by the new bitmap decoder implementation.
cpp/src/arrow/util/bit_util_test.cc	Adds unit tests for `Get32Bits` and `CopyBits`.

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

+ void Reset(const RunType& run) noexcept {
+ values_left_ = run.values_count();
+ if (run.value_little_endian() == 0) {
+ value_pattern_ = uint8_t{0};
+ } else {
+ value_pattern_ = uint8_t{0xFF};
+ }
+ }


AntoinePrv · 2026-06-18T12:26:10Z

@pitrou this is ready

pitrou · 2026-06-18T13:28:09Z

 rle_size_t batch_size) -> rle_size_t {
 using ControlFlow = RleBitPackedParser::ControlFlow;

+ if (ARROW_PREDICT_FALSE(batch_size == 0 || exhausted())) {


Is there a particular reason this became necessary?

Came from the Copilot review, which I found to be relevant. I may not be a possible path in practice in all of Arrow, but better safe than sorry.

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

pitrou

Thanks for the update @AntoinePrv !

pitrou · 2026-06-22T14:32:21Z

+/// Make a vector of `size` pseudo-random bytes, deterministic for a given `seed`.
+std::vector<uint8_t> MakeRandomBytes(size_t size, uint32_t seed = 56) {
+ std::vector<uint8_t> bytes(size);
+ std::minstd_rand gen(seed);


Hmm, the default_random_engine might give better randomness, though not sure it's useful here.

I picked this one because it is small and cheap and I assume sufficient here.

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

+ void Reset(const RunType& run) noexcept {
+ values_left_ = run.values_count();
+ if (run.value_little_endian() == 0) {
+ value_pattern_ = uint8_t{0};
+ } else {
+ value_pattern_ = uint8_t{0xFF};
+ }
+ }


AntoinePrv · 2026-06-23T12:18:19Z

@pitrou this is ready

pitrou · 2026-06-23T13:32:17Z

@AntoinePrv Note there is a compilation error on MSVC.

pitrou · 2026-06-23T13:41:44Z

@github-actions crossbow submit -g cpp

pitrou

+1, but need to fix the MSVC-based CI build.

github-actions · 2026-06-23T13:44:59Z

Revision: 6fc9785

Submitted crossbow builds: ursacomputing/crossbow @ actions-fb93408670

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-debian-13-cpp-amd64
test-debian-13-cpp-i386
test-debian-experimental-cpp-gcc-15
test-fedora-42-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

pitrou · 2026-06-23T14:37:51Z

Other related failures:

this compilation error on Ubuntu 22.04
these Valgrind errors: I think you just have to implement a PrintTo for the new test parameter type(s) such that GTest doesn't read uninitialized memory

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

AntoinePrv · 2026-06-24T07:31:47Z

@github-actions crossbow submit -g cpp

github-actions · 2026-06-24T07:34:41Z

Revision: 441408d

Submitted crossbow builds: ursacomputing/crossbow @ actions-632a7d9657

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-debian-13-cpp-amd64
test-debian-13-cpp-i386
test-debian-experimental-cpp-gcc-15
test-fedora-42-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

AntoinePrv · 2026-06-25T09:50:58Z

@github-actions crossbow submit -g cpp

github-actions · 2026-06-25T09:57:41Z

Revision: e64924b

Submitted crossbow builds: ursacomputing/crossbow @ actions-3b12a273d8

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-debian-13-cpp-amd64
test-debian-13-cpp-i386
test-debian-experimental-cpp-gcc-15
test-fedora-42-cpp
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-bundled
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-bundled-offline
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer

pitrou · 2026-06-25T10:46:02Z

Thanks for fixing CI @AntoinePrv , I'll merge now!

raulcd · 2026-06-25T11:54:53Z

This PR seems to have broken the documentation CI jobs:

+ doxygen
/arrow/cpp/apidoc /
/arrow/cpp/src/arrow/util/bit_util.h:249: error: Unsupported xml/html tag <typename> found (warning treated as error, aborting now)
Error: `docker compose --file=/home/runner/work/arrow/arrow/compose.yaml run --rm conda-python-docs` exited with a non-zero exit code 1, see the process log above.

I'll open an issue, I am trying to understand what is the problem

edit: opened: #50257

conbench-apache-arrow · 2026-06-25T17:10:54Z

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit e74bb67.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them.

…e#50217) ### Rationale for this change Add a `RleBitPackedToBitmapDecoder` capable of decoding a Parquet mixed Rle / BitPacked byte stream directly into a bitmap. This is an optimization to be used when target and source bit_width = 1 that will improve decoding nullable columns in a follow-up PR. ### What changes are included in this PR? New classes, that reuse previous Runs and parser. Their API is slighly different from the existing one as it fits a partilcular case. - `RleRunToBitmapDecoder` - `BitPackedRunToBitmapDecoder` - `RleRunToBitmapDecoder` ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: apache#50216 Authored-by: AntoinePrv <AntoinePrv@users.noreply.github.com> Signed-off-by: Antoine Pitrou <antoine@python.org>

AntoinePrv requested a review from pitrou as a code owner June 18, 2026 09:25

CopilotAI review requested due to automatic review settings June 18, 2026 09:25

Copilotstarted reviewing on behalf of AntoinePrv June 18, 2026 09:28 View session

github-actionsBot added the awaiting review Awaiting review label Jun 18, 2026

CopilotAI reviewed Jun 18, 2026
View reviewed changes

Comment threadcpp/src/arrow/util/rle_encoding_internal.h Outdated
Comment threadcpp/src/arrow/util/rle_bitmap_internal.h
Comment threadcpp/src/arrow/util/bit_util.h Outdated
Comment threadcpp/src/arrow/util/rle_encoding_internal.h

github-actionsBot added awaiting committer review Awaiting committer review Component: C++ and removed awaiting review Awaiting review labels Jun 18, 2026

CopilotAI review requested due to automatic review settings June 18, 2026 12:03

Copilotstarted reviewing on behalf of AntoinePrv June 18, 2026 12:04 View session

CopilotAI reviewed Jun 18, 2026
View reviewed changes

Comment threadcpp/src/arrow/util/rle_bitmap_internal.h Outdated
Comment on lines +169 to +176
void Reset(const RunType& run) noexcept {
values_left_ = run.values_count();
if (run.value_little_endian() == 0) {
value_pattern_ = uint8_t{0};
} else {
value_pattern_ = uint8_t{0xFF};
}
}

pitrou reviewed Jun 18, 2026
View reviewed changes

CopilotAI review requested due to automatic review settings June 22, 2026 09:22

AntoinePrv force-pushed the rle-bitmap branch from 07f2371 to 086af88Compare June 22, 2026 09:22

Copilotstarted reviewing on behalf of AntoinePrv June 22, 2026 09:23 View session

CopilotAI reviewed Jun 22, 2026
View reviewed changes

Comment threadcpp/src/arrow/util/rle_bitmap_internal.h

pitrou reviewed Jun 22, 2026
View reviewed changes

Comment threadcpp/src/arrow/util/bit_util.h Outdated

pitrou reviewed Jun 22, 2026
View reviewed changes

CopilotAI review requested due to automatic review settings June 23, 2026 11:57

Copilotstarted reviewing on behalf of AntoinePrv June 23, 2026 11:58 View session

CopilotAI reviewed Jun 23, 2026
View reviewed changes

Comment threadcpp/src/arrow/util/rle_bitmap_internal.h
Comment on lines +83 to +90
void Reset(const RunType& run) noexcept {
values_left_ = run.values_count();
if (run.value_little_endian() == 0) {
value_pattern_ = uint8_t{0};
} else {
value_pattern_ = uint8_t{0xFF};
}
}

AntoinePrv added 7 commits June 23, 2026 14:17

Add CopyBits
ef5dbbc

Add Get32Bits
109723b

Add RleRun::value
c4ee2f9

Add Rle bitmap
aa97150

Simplify parser decoder maping
00ef6d6

Add BitPackedRunToBitMapDecoder
cf5ac4e

Extra check
c0692c1

Adjust for review
6fc9785

AntoinePrv force-pushed the rle-bitmap branch from c80e478 to 6fc9785Compare June 23, 2026 12:18

pitrou approved these changes Jun 23, 2026
View reviewed changes

Fix uniform int distribution type
441408d

CopilotAI review requested due to automatic review settings June 23, 2026 14:34

Copilotstarted reviewing on behalf of AntoinePrv June 23, 2026 14:35 View session

CopilotAI reviewed Jun 23, 2026
View reviewed changes

Comment threadcpp/src/arrow/util/rle_bitmap_internal.h
Comment threadcpp/src/arrow/util/rle_bitmap_internal.h
Comment threadcpp/src/arrow/util/CMakeLists.txt

Fix uninitialized buffer
4235aa7

Add PrintTo
e64924b

CopilotAI review requested due to automatic review settings June 25, 2026 09:21

Copilotstarted reviewing on behalf of AntoinePrv June 25, 2026 09:23 View session

CopilotAI reviewed Jun 25, 2026
View reviewed changes

Comment threadcpp/src/arrow/util/rle_bitmap_internal.h

pitrou merged commit e74bb67 into apache:mainJun 25, 2026
53 of 55 checks passed

pitrou removed the awaiting committer review Awaiting committer review label Jun 25, 2026

pitrou mentioned this pull request Jun 25, 2026
[C++][Parquet] Rle-Bitpacked to Bitmap decoder #50216
Closed

AntoinePrv deleted the rle-bitmap branch June 25, 2026 10:51

raulcd mentioned this pull request Jun 25, 2026
[CI][Docs] doxygen fails with unsupported xml/html tag <typename> found on /arrow/cpp/src/arrow/util/bit_util.h:249 #50257
Closed

Uh oh!

Conversation

AntoinePrv commented Jun 18, 2026• edited by github-actions BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actionsBot commented Jun 18, 2026

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

AntoinePrv commented Jun 18, 2026

Uh oh!

pitrouJun 18, 2026

Choose a reason for hiding this comment

Uh oh!

AntoinePrvJun 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pitrouJun 22, 2026

Choose a reason for hiding this comment

Uh oh!

AntoinePrvJun 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

AntoinePrv commented Jun 23, 2026

Uh oh!

pitrou commented Jun 23, 2026

Uh oh!

pitrou commented Jun 23, 2026

Uh oh!

pitrou left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jun 23, 2026

Uh oh!

pitrou commented Jun 23, 2026

AntoinePrv commented Jun 18, 2026•
edited by github-actions Bot
Loading

raulcd commented Jun 25, 2026•
edited
Loading