GH-48701: [C++][Parquet] Add ALPpd encoding - #48345

Open

prtkgaur wants to merge 156 commits into

apache:mainfrom

prtkgaur:gh540-alp-pseudoDecimal-encoding

Commits on Sep 9, 2026

Add alp code

sfc-gh-pgaur
and
dhirhank
committed
3b7222b
Integrate ALP with arrow
sfc-gh-pgaur
committed
083834c
Add alp benchmark
sfc-gh-pgaur
committed
2c8ba36
Add datasets for alp benchmarking
sfc-gh-pgaur
committed
086030f
Update cmake file
sfc-gh-pgaur
committed
a52a207
Move hpp files to h
sfc-gh-pgaur
committed
aebb970
Update flow digram and layout digram to use ASCII and not unicode characters
sfc-gh-pgaur
committed
25c97fc
Rename cpp files to cc
sfc-gh-pgaur
committed
46cb129
Update documentation to align with arrow's doxygen style
sfc-gh-pgaur
committed
bba24bd
Adapt methods and variable names to arrow style
sfc-gh-pgaur
committed
7218d1f
Update the tests to adhere to arrow style code
sfc-gh-pgaur
committed
3107ab5
Update callers
sfc-gh-pgaur
committed
d77f23a
Fuse FOR and decode loop
sfc-gh-pgaur
committed
35cc319
Reduce memory allocation in the decompress call
sfc-gh-pgaur
committed
b606185
Attempt at making decoding faster with SIMD
sfc-gh-pgaur
committed
5393d0b
Revert "Attempt at making decoding faster with SIMD"
sfc-gh-pgaur
committed
795e3dd
Move cpp files to cc
sfc-gh-pgaur
committed
8768f7d
Move data file to parquet-testing submodule
sfc-gh-pgaur
committed
d94b800
Update path to the data file
sfc-gh-pgaur
committed
bf6f344
Adapt files names to arrow convention
sfc-gh-pgaur
committed
d99f3b2
File rename
sfc-gh-pgaur
committed
13412f9
Obtain compressed size and number of elements from page header
sfc-gh-pgaur
committed
5bc1499
Fix namespace depth
sfc-gh-pgaur
committed
048b952
Better pack the compression block header
sfc-gh-pgaur
committed
4db50fb
Rename class
sfc-gh-pgaur
committed
329ac14
Rearrage field for vector metadata for better packing
sfc-gh-pgaur
committed
d2de57d
Add spec files
sfc-gh-pgaur
committed
df0d73f
Add more tests
sfc-gh-pgaur
committed
8226378
Move number of elements metadata info to uber header
sfc-gh-pgaur
committed
cb81f5e
Keep version as 1
sfc-gh-pgaur
committed
af5f73e
Tests pass
sfc-gh-pgaur
committed
815f74c
Get rid of numElements and derive it for each vector
sfc-gh-pgaur
committed
ef2935f
Update the md files
sfc-gh-pgaur
committed
ce116d5
Fix ubsan error
sfc-gh-pgaur
committed
4ed790f
Use safe copy
sfc-gh-pgaur
committed
736d235
Add more unit tests
sfc-gh-pgaur
committed
bfcc0d8
Templatize AlpEncodedVectorInfo
sfc-gh-pgaur
committed
2594dee
Fix
sfc-gh-pgaur
committed
93b8c98
[SpecReview] Num values in page is at max int32
sfc-gh-pgaur
committed
9ffd507
[SpecReview] Rename layout to IntegerEncoding
sfc-gh-pgaur
committed
eb1dd91
[SpecReview] Rename enum
sfc-gh-pgaur
committed
f713264
[SpecReview] Make AlpHeader exact 8 bytes
sfc-gh-pgaur
committed
78dbc79
[SpecReview] Move all vector data towards the start
sfc-gh-pgaur
committed
4e10645
[SpecReview][Metadata] Introduce wrapper to read metadata
sfc-gh-pgaur
committed
c9914c1
[SpecReview] Separate out ALP and FOR metadata
sfc-gh-pgaur
committed
9417a32
[SpecReview] Remove unnecessary field
sfc-gh-pgaur
committed
c6d74e9
[FutureFlexibility] Add the FOR specific code behind switch
sfc-gh-pgaur
committed
2961e78
Add unit test
sfc-gh-pgaur
committed
45c7951
ALP: Implement offset-based layout for O(1) random access
sfc-gh-pgaur
committed
6662bfb
ALP: Update documentation for offset-based layout
sfc-gh-pgaur
committed
e9cc9be
ALP: Add CreateSamplingPreset and EncodeWithPreset APIs
sfc-gh-pgaur
committed
44f565e
ALP: Update all comments and docs for offset-based layout
sfc-gh-pgaur
committed
bd0a83a
ALP: Add cross-implementation reference blob generator
sfc-gh-pgaur
committed
87203b3
ALP: Add generate_alp_parquet utility for creating ALP test parquet files
sfc-gh-pgaur
committed
9b05062
ALP: Remove version field from header (8B → 7B)
sfc-gh-pgaur
committed
3a0325d
Update parquet-testing submodule with ALP test data files
sfc-gh-pgaur
committed
dd8b7e7
ALP: Add C++ correctness tests for ALP-encoded parquet files
sfc-gh-pgaur
committed
b945b30
ALP: Add cross-language tests reading Java-generated ALP parquet files
sfc-gh-pgaur
committed
37857af
ALP: Add float32 cross-language tests and generator support
sfc-gh-pgaur
committed
011301c
Address code review comments
sfc-gh-pgaur
committed
7cad264
Address code review comments: endianness assert and decode bounds checks
sfc-gh-pgaur
committed
cdf6b68
Address code review comments: renames, API cleanup, and simplifications
sfc-gh-pgaur
committed
2543c45
Fix build errors from code review changes
sfc-gh-pgaur
committed
569bbc7
Split EstimateCompressedSize loop into batched passes for vectorization
sfc-gh-pgaur
committed
e840738
Address review nits: uint64_t→int64_t, spell out ALP, use uint8_t for byte spans
sfc-gh-pgaur
committed
e052a5f
Move decoder creation outside benchmark loop
sfc-gh-pgaur
committed
fd0fb64
Rename alp_wrapper.h/.cc to alp_codec.h/.cc
sfc-gh-pgaur
committed
af50909
Convert ARROW_CHECK to Status returns on decode/load paths
sfc-gh-pgaur
committed
05d1621
Add precondition docs to Store methods
sfc-gh-pgaur
committed
7326dcb
Use signed integer types consistently per Arrow style guide
sfc-gh-pgaur
committed
75df582
Move out parameters to last position and remove unused enforce_mode
sfc-gh-pgaur
committed
6abcddf
Add ALP encoding to parquet.thrift Encoding enum
sfc-gh-pgaur
committed
3d28124
Remove ALP spec files; spec belongs in parquet-format repo
sfc-gh-pgaur
committed
9395dfe
Rename decomp/comp params to input/output; remove ARROW_CHECK(false) crash paths
sfc-gh-pgaur
committed
f2d2ceb
Copy header fields individually instead of bulk memcpy
sfc-gh-pgaur
committed
2910b7a
Replace StaticVector with std::vector for flexible vector size
sfc-gh-pgaur
committed
cdab797
Make ALP vector_size configurable on the encode path
sfc-gh-pgaur
committed
70ebcbd
Add parquet-level round-trip test for ALP at non-default vector sizes
sfc-gh-pgaur
committed
78964d8
Address reviewer feedback: use int64_t, reorder out-params last, clean up naming
sfc-gh-pgaur
committed
4b86833
Add sign branching to FastRound for correct rounding of negative values
sfc-gh-pgaur
committed
2bcd1dc
Address reviewer feedback: SafeLoad/SafeStore, struct→class conversion
sfc-gh-pgaur
committed
0b3ade8
Convert AlpEncodedVector and AlpEncodedVectorView to private members with getters
sfc-gh-pgaur
committed
0599a78
Remove unused AlpMetadataCache class and redundant GetNumElements()
sfc-gh-pgaur
committed
5f75916
Remove private inheritance from AlpCompression and AlpInlines
sfc-gh-pgaur
committed
78045ac
Replace trailing return type with explicit Result<AlpHeader> in LoadHeader
sfc-gh-pgaur
committed
64118ef
Use SafeLoadAs for all header fields in LoadHeader
sfc-gh-pgaur
committed
7111215
Validate header fields in LoadHeader
sfc-gh-pgaur
committed
8098e05
Change AlpCodec public API from byte size to element count
sfc-gh-pgaur
committed
f0a44ad
Use uint8_t* consistently in AlpCodec API instead of char*
sfc-gh-pgaur
committed
10974af
Replace trailing return types with explicit return types
sfc-gh-pgaur
committed
ab2d8ac
Cite ALP paper §3.1.2 for combination tie-break rule
sfc-gh-pgaur
committed
6d3fec1
Pin AlpMode underlying type to uint8_t for serialization safety
sfc-gh-pgaur
committed
4f328e0
Address reviewer nits: doc tweaks, sizeof(member_), incremental-decode TODO
sfc-gh-pgaur
committed
f11765e
Convert AlpSampler uint64_t to int64_t
sfc-gh-pgaur
committed
43dd67d
Remove GH-48701 reference from incremental-decode TODO
sfc-gh-pgaur
committed
c36054c
Document actual crash/abort behavior in AlpCodec encode-path docs
sfc-gh-pgaur
committed
c728d24
Use bit_util::CeilDiv for vector-count computations in AlpCodec
sfc-gh-pgaur
committed
89ca220
Replace memcmp with IsBitwiseEqual helper in alp_test.cc
sfc-gh-pgaur
committed
d588488
Type-parameterize alp_test integration tests and fix TruncatedData
sfc-gh-pgaur
committed
f881cd1
Type-parameterize AlpEncodingAdHoc tests in encoding_test.cc
sfc-gh-pgaur
committed
2b614b6
Widen RandomData test range to exercise ALP exception fallback path
sfc-gh-pgaur
committed
9284906
Remove uncompiled generate_alp_parquet and generate_reference_blobs
sfc-gh-pgaur
committed
ef72129
Add incremental-encode TODO and snake_case rename in AlpEncoder
sfc-gh-pgaur
committed
7ef8c92
Rename EncodeAlp parameter combinations to preset
sfc-gh-pgaur
committed
6bc9b0f
Use bit_util::IsPowerOf2 and BytesForBits instead of raw bit manipulation
sfc-gh-pgaur
committed
3736ebb
Remove remaining 'non-templated' references in ALP comments
sfc-gh-pgaur
committed
1b196f4
Remove GetBitPackedSize wrapper, inline bit_util::BytesForBits at callers
sfc-gh-pgaur
committed
dfcb54e
Code review changes
sfc-gh-pgaur
committed
f1e62a2
Convert AlpCodec public API from ARROW_CHECK aborts to Status returns
sfc-gh-pgaur
committed
6430bda
Use arrow::csv::TableReader for ALP benchmark dataset loading
sfc-gh-pgaur
committed
a44a24a
Reorder AlpEncoded(For)VectorInfo so kStoredSize uses sizeof(member_)
sfc-gh-pgaur
committed
b94fb47
Make ALP num_exceptions and exception positions unsigned
sfc-gh-pgaur
committed
f83a31e
Enforce the spec's [3, 15] log_vector_size range in ALP
sfc-gh-pgaur
committed
86a67e6
Use array matchers in ALP tests; note PLAIN fallback
sfc-gh-pgaur
committed
e758841
Document the [3, 15] vector_size range in AlpCodec docs
sfc-gh-pgaur
committed
4250fff
Put EncodeAlp output parameters last
sfc-gh-pgaur
committed
37f55ce
Test that ALP rejects malformed page headers
sfc-gh-pgaur
committed
88cb72e
Use std::countr_zero instead of __builtin_ctz in ALP
sfc-gh-pgaur
committed
bd0027a
Apply clang-format 18 to the ALP sources
sfc-gh-pgaur
committed
20bb3b0
Validate ALP vector metadata before it steers the decode
sfc-gh-pgaur
committed
2776b23
Correct inaccurate ALP comments; drop unused UnsafeResize
sfc-gh-pgaur
committed
f13481e
Type-parameterize ALP compression tests; tighten round-trip asserts
sfc-gh-pgaur
committed
24c496e
Drop the ALP benchmark that needs out-of-tree datasets
sfc-gh-pgaur
committed
6c0f63a
Test ALP against the parquet-testing conformance file
sfc-gh-pgaur
committed
6b0d0bb
Put ALP behind an opt-in writer flag
sfc-gh-pgaur
committed
5620a1f
Tighten the ALP tests and document the encoding
sfc-gh-pgaur
committed
722ad9c
Validate the ALP element count and the offset chain
sfc-gh-pgaur
committed
82bc28d
Take the ALP value count from the page's own header
sfc-gh-pgaur
committed
721fbed
Encode an all-null ALP page as a bare header
sfc-gh-pgaur
committed
14fc981
Build the ALP sources once, and under meson too
sfc-gh-pgaur
committed
2327713
Serialize ALP little-endian instead of refusing to build
sfc-gh-pgaur
committed
37bcba5
Move AlpMode out of the public header
sfc-gh-pgaur
committed
b4785f7
Test the ALP element-count and offset-chain validation
sfc-gh-pgaur
committed
db22698
Cover batched ALP decoding, and trim the encoder TODOs
sfc-gh-pgaur
committed
3a8b853
Decode ALP pages one vector at a time
sfc-gh-pgaur
committed
26e5289
Move the ALP end-to-end tests to their own file
sfc-gh-pgaur
committed
7e078b6
Name the ALP headers _internal.h
sfc-gh-pgaur
committed
ef5559a
Fold the ad-hoc ALP vector-size test into the typed suite
sfc-gh-pgaur
committed
df72402
Stop allocating scratch on every ALP vector decode
sfc-gh-pgaur
committed
80c2e08
Drop dead ALP code, link the TODOs, unpause the benchmarks
sfc-gh-pgaur
committed
af50476
Copy the ALP thrift comment from parquet-format verbatim
sfc-gh-pgaur
committed
1356770
Export AlpCodec and concatenate the ALP namespaces
sfc-gh-pgaur
committed
be90517
Drop the AlpSampler members that were compile-time constants
sfc-gh-pgaur
committed
88cec11
Guard the loop pragmas behind the compiler that honours them
sfc-gh-pgaur
committed
b2f53f7
Export the nested VectorReader as well as its enclosing class
sfc-gh-pgaur
committed
187fa13
Regenerate the thrift bindings so they carry the ALP enum value
sfc-gh-pgaur
committed
1eb5593
Reject an ALP page whose values outlive its definition levels
sfc-gh-pgaur
committed
7977b5a
Size the sample buffer without going through a double
sfc-gh-pgaur
committed
2724559
Say in prose what the target type may be
sfc-gh-pgaur
committed
5fc824c
Export the decode member templates from the shared library
sfc-gh-pgaur
committed
05e90cf
Cast the element count the ALP test passes
sfc-gh-pgaur
committed
27cc07d
Export the internal ALP classes the test links against
sfc-gh-pgaur
committed
8d273c6
Widen the shift the ALP test takes a modulus with
sfc-gh-pgaur
committed
daa1f8f
Give the ALP roundtrip helper a name of its own
sfc-gh-pgaur
committed
bb66b0f
Read the stored size through its accessor
sfc-gh-pgaur
committed
58a72d5
Skip the byte copies that have nothing to copy
sfc-gh-pgaur
committed
e123d8b