Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
156 commits
Select commit Hold shift + click to select a range
3b7222b
Add alp code
sfc-gh-pgaur Dec 4, 2025
083834c
Integrate ALP with arrow
sfc-gh-pgaur Dec 4, 2025
2c8ba36
Add alp benchmark
sfc-gh-pgaur Dec 4, 2025
086030f
Add datasets for alp benchmarking
sfc-gh-pgaur Dec 4, 2025
a52a207
Update cmake file
sfc-gh-pgaur Dec 4, 2025
aebb970
Move hpp files to h
sfc-gh-pgaur Dec 6, 2025
25c97fc
Update flow digram and layout digram to use ASCII and not unicode cha…
sfc-gh-pgaur Dec 7, 2025
46cb129
Rename cpp files to cc
sfc-gh-pgaur Dec 7, 2025
bba24bd
Update documentation to align with arrow's doxygen style
sfc-gh-pgaur Dec 7, 2025
7218d1f
Adapt methods and variable names to arrow style
sfc-gh-pgaur Dec 7, 2025
3107ab5
Update the tests to adhere to arrow style code
sfc-gh-pgaur Dec 7, 2025
d77f23a
Update callers
sfc-gh-pgaur Dec 7, 2025
35cc319
Fuse FOR and decode loop
sfc-gh-pgaur Dec 7, 2025
b606185
Reduce memory allocation in the decompress call
sfc-gh-pgaur Dec 7, 2025
5393d0b
Attempt at making decoding faster with SIMD
sfc-gh-pgaur Dec 8, 2025
795e3dd
Revert "Attempt at making decoding faster with SIMD"
sfc-gh-pgaur Dec 8, 2025
8768f7d
Move cpp files to cc
sfc-gh-pgaur Dec 8, 2025
d94b800
Move data file to parquet-testing submodule
sfc-gh-pgaur Dec 8, 2025
bf6f344
Update path to the data file
sfc-gh-pgaur Dec 9, 2025
d99f3b2
Adapt files names to arrow convention
sfc-gh-pgaur Dec 15, 2025
13412f9
File rename
sfc-gh-pgaur Dec 15, 2025
5bc1499
Obtain compressed size and number of elements from page header
sfc-gh-pgaur Dec 15, 2025
048b952
Fix namespace depth
sfc-gh-pgaur Dec 16, 2025
4db50fb
Better pack the compression block header
sfc-gh-pgaur Dec 16, 2025
329ac14
Rename class
sfc-gh-pgaur Dec 16, 2025
d2de57d
Rearrage field for vector metadata for better packing
sfc-gh-pgaur Dec 16, 2025
df0d73f
Add spec files
sfc-gh-pgaur Dec 31, 2025
8226378
Add more tests
sfc-gh-pgaur Jan 2, 2026
cb81f5e
Move number of elements metadata info to uber header
sfc-gh-pgaur Jan 12, 2026
af5f73e
Keep version as 1
sfc-gh-pgaur Jan 12, 2026
815f74c
Tests pass
sfc-gh-pgaur Jan 12, 2026
ef2935f
Get rid of numElements and derive it for each vector
sfc-gh-pgaur Jan 12, 2026
ce116d5
Update the md files
sfc-gh-pgaur Jan 12, 2026
4ed790f
Fix ubsan error
sfc-gh-pgaur Jan 13, 2026
736d235
Use safe copy
sfc-gh-pgaur Jan 13, 2026
bfcc0d8
Add more unit tests
sfc-gh-pgaur Jan 13, 2026
2594dee
Templatize AlpEncodedVectorInfo
sfc-gh-pgaur Jan 14, 2026
93b8c98
Fix
sfc-gh-pgaur Jan 14, 2026
9ffd507
[SpecReview] Num values in page is at max int32
sfc-gh-pgaur Jan 14, 2026
eb1dd91
[SpecReview] Rename layout to IntegerEncoding
sfc-gh-pgaur Jan 14, 2026
f713264
[SpecReview] Rename enum
sfc-gh-pgaur Jan 14, 2026
78dbc79
[SpecReview] Make AlpHeader exact 8 bytes
sfc-gh-pgaur Jan 14, 2026
4e10645
[SpecReview] Move all vector data towards the start
sfc-gh-pgaur Jan 15, 2026
c9914c1
[SpecReview][Metadata] Introduce wrapper to read metadata
sfc-gh-pgaur Jan 15, 2026
9417a32
[SpecReview] Separate out ALP and FOR metadata
sfc-gh-pgaur Jan 15, 2026
c6d74e9
[SpecReview] Remove unnecessary field
sfc-gh-pgaur Jan 16, 2026
2961e78
[FutureFlexibility] Add the FOR specific code behind switch
sfc-gh-pgaur Jan 16, 2026
45c7951
Add unit test
sfc-gh-pgaur Jan 17, 2026
6662bfb
ALP: Implement offset-based layout for O(1) random access
sfc-gh-pgaur Jan 26, 2026
e9cc9be
ALP: Update documentation for offset-based layout
sfc-gh-pgaur Jan 26, 2026
44f565e
ALP: Add CreateSamplingPreset and EncodeWithPreset APIs
sfc-gh-pgaur Jan 27, 2026
bd0a83a
ALP: Update all comments and docs for offset-based layout
sfc-gh-pgaur Jan 27, 2026
87203b3
ALP: Add cross-implementation reference blob generator
sfc-gh-pgaur Mar 7, 2026
9b05062
ALP: Add generate_alp_parquet utility for creating ALP test parquet f…
sfc-gh-pgaur Mar 7, 2026
3a0325d
ALP: Remove version field from header (8B → 7B)
sfc-gh-pgaur Mar 7, 2026
dd8b7e7
Update parquet-testing submodule with ALP test data files
sfc-gh-pgaur Mar 7, 2026
b945b30
ALP: Add C++ correctness tests for ALP-encoded parquet files
sfc-gh-pgaur Mar 7, 2026
37857af
ALP: Add cross-language tests reading Java-generated ALP parquet files
sfc-gh-pgaur Mar 7, 2026
011301c
ALP: Add float32 cross-language tests and generator support
sfc-gh-pgaur Mar 8, 2026
7cad264
Address code review comments
sfc-gh-pgaur Apr 22, 2026
cdf6b68
Address code review comments: endianness assert and decode bounds checks
sfc-gh-pgaur Apr 27, 2026
2543c45
Address code review comments: renames, API cleanup, and simplifications
sfc-gh-pgaur Apr 27, 2026
569bbc7
Fix build errors from code review changes
sfc-gh-pgaur Apr 27, 2026
e840738
Split EstimateCompressedSize loop into batched passes for vectorization
sfc-gh-pgaur Apr 27, 2026
e052a5f
Address review nits: uint64_t→int64_t, spell out ALP, use uint8_t for…
sfc-gh-pgaur May 6, 2026
fd0fb64
Move decoder creation outside benchmark loop
sfc-gh-pgaur May 6, 2026
af50909
Rename alp_wrapper.h/.cc to alp_codec.h/.cc
sfc-gh-pgaur May 6, 2026
05d1621
Convert ARROW_CHECK to Status returns on decode/load paths
sfc-gh-pgaur May 6, 2026
7326dcb
Add precondition docs to Store methods
sfc-gh-pgaur May 6, 2026
75df582
Use signed integer types consistently per Arrow style guide
sfc-gh-pgaur May 6, 2026
6abcddf
Move out parameters to last position and remove unused enforce_mode
sfc-gh-pgaur May 7, 2026
3d28124
Add ALP encoding to parquet.thrift Encoding enum
sfc-gh-pgaur May 7, 2026
9395dfe
Remove ALP spec files; spec belongs in parquet-format repo
sfc-gh-pgaur May 13, 2026
f2d2ceb
Rename decomp/comp params to input/output; remove ARROW_CHECK(false) …
sfc-gh-pgaur May 13, 2026
2910b7a
Copy header fields individually instead of bulk memcpy
sfc-gh-pgaur May 13, 2026
cdab797
Replace StaticVector with std::vector for flexible vector size
sfc-gh-pgaur May 13, 2026
70ebcbd
Make ALP vector_size configurable on the encode path
sfc-gh-pgaur May 13, 2026
78964d8
Add parquet-level round-trip test for ALP at non-default vector sizes
sfc-gh-pgaur May 13, 2026
4b86833
Address reviewer feedback: use int64_t, reorder out-params last, clea…
sfc-gh-pgaur May 14, 2026
2bcd1dc
Add sign branching to FastRound for correct rounding of negative values
sfc-gh-pgaur Jun 3, 2026
0b3ade8
Address reviewer feedback: SafeLoad/SafeStore, struct→class conversion
sfc-gh-pgaur Jun 8, 2026
0599a78
Convert AlpEncodedVector and AlpEncodedVectorView to private members …
sfc-gh-pgaur Jun 8, 2026
5f75916
Remove unused AlpMetadataCache class and redundant GetNumElements()
sfc-gh-pgaur Jun 8, 2026
78045ac
Remove private inheritance from AlpCompression and AlpInlines
sfc-gh-pgaur Jun 8, 2026
64118ef
Replace trailing return type with explicit Result<AlpHeader> in LoadH…
sfc-gh-pgaur Jun 8, 2026
7111215
Use SafeLoadAs for all header fields in LoadHeader
sfc-gh-pgaur Jun 8, 2026
8098e05
Validate header fields in LoadHeader
sfc-gh-pgaur Jun 8, 2026
f0a44ad
Change AlpCodec public API from byte size to element count
sfc-gh-pgaur Jun 8, 2026
10974af
Use uint8_t* consistently in AlpCodec API instead of char*
sfc-gh-pgaur Jun 8, 2026
ab2d8ac
Replace trailing return types with explicit return types
sfc-gh-pgaur Jun 25, 2026
6d3fec1
Cite ALP paper §3.1.2 for combination tie-break rule
sfc-gh-pgaur Jun 25, 2026
4f328e0
Pin AlpMode underlying type to uint8_t for serialization safety
sfc-gh-pgaur Jun 25, 2026
f11765e
Address reviewer nits: doc tweaks, sizeof(member_), incremental-decod…
sfc-gh-pgaur Jun 25, 2026
43dd67d
Convert AlpSampler uint64_t to int64_t
sfc-gh-pgaur Jun 25, 2026
c36054c
Remove GH-48701 reference from incremental-decode TODO
sfc-gh-pgaur Jun 25, 2026
c728d24
Document actual crash/abort behavior in AlpCodec encode-path docs
sfc-gh-pgaur Jun 25, 2026
89ca220
Use bit_util::CeilDiv for vector-count computations in AlpCodec
sfc-gh-pgaur Jun 25, 2026
d588488
Replace memcmp with IsBitwiseEqual helper in alp_test.cc
sfc-gh-pgaur Jun 25, 2026
f881cd1
Type-parameterize alp_test integration tests and fix TruncatedData
sfc-gh-pgaur Jun 26, 2026
2b614b6
Type-parameterize AlpEncodingAdHoc tests in encoding_test.cc
sfc-gh-pgaur Jun 26, 2026
9284906
Widen RandomData test range to exercise ALP exception fallback path
sfc-gh-pgaur Jun 26, 2026
ef72129
Remove uncompiled generate_alp_parquet and generate_reference_blobs
sfc-gh-pgaur Jun 26, 2026
7ef8c92
Add incremental-encode TODO and snake_case rename in AlpEncoder
sfc-gh-pgaur Jun 26, 2026
6bc9b0f
Rename EncodeAlp parameter combinations to preset
sfc-gh-pgaur Jun 26, 2026
3736ebb
Use bit_util::IsPowerOf2 and BytesForBits instead of raw bit manipula…
sfc-gh-pgaur Jun 26, 2026
1b196f4
Remove remaining 'non-templated' references in ALP comments
sfc-gh-pgaur Jun 26, 2026
dfcb54e
Remove GetBitPackedSize wrapper, inline bit_util::BytesForBits at cal…
sfc-gh-pgaur Jun 26, 2026
f1e62a2
Code review changes
sfc-gh-pgaur Jun 28, 2026
6430bda
Convert AlpCodec public API from ARROW_CHECK aborts to Status returns
sfc-gh-pgaur Jun 30, 2026
a44a24a
Use arrow::csv::TableReader for ALP benchmark dataset loading
sfc-gh-pgaur Jun 30, 2026
b94fb47
Reorder AlpEncoded(For)VectorInfo so kStoredSize uses sizeof(member_)
sfc-gh-pgaur Jun 30, 2026
f83a31e
Make ALP num_exceptions and exception positions unsigned
sfc-gh-pgaur Aug 12, 2026
86a67e6
Enforce the spec's [3, 15] log_vector_size range in ALP
sfc-gh-pgaur Aug 12, 2026
e758841
Use array matchers in ALP tests; note PLAIN fallback
sfc-gh-pgaur Aug 12, 2026
4250fff
Document the [3, 15] vector_size range in AlpCodec docs
sfc-gh-pgaur Aug 14, 2026
37f55ce
Put EncodeAlp output parameters last
sfc-gh-pgaur Aug 24, 2026
88cb72e
Test that ALP rejects malformed page headers
sfc-gh-pgaur Aug 24, 2026
bd0027a
Use std::countr_zero instead of __builtin_ctz in ALP
sfc-gh-pgaur Aug 24, 2026
20bb3b0
Apply clang-format 18 to the ALP sources
sfc-gh-pgaur Aug 24, 2026
2776b23
Validate ALP vector metadata before it steers the decode
sfc-gh-pgaur Aug 26, 2026
f13481e
Correct inaccurate ALP comments; drop unused UnsafeResize
sfc-gh-pgaur Aug 26, 2026
24c496e
Type-parameterize ALP compression tests; tighten round-trip asserts
sfc-gh-pgaur Aug 27, 2026
6c0f63a
Drop the ALP benchmark that needs out-of-tree datasets
sfc-gh-pgaur Aug 31, 2026
6b0d0bb
Test ALP against the parquet-testing conformance file
sfc-gh-pgaur Aug 31, 2026
5620a1f
Put ALP behind an opt-in writer flag
sfc-gh-pgaur Aug 31, 2026
722ad9c
Tighten the ALP tests and document the encoding
sfc-gh-pgaur Sep 1, 2026
82bc28d
Validate the ALP element count and the offset chain
sfc-gh-pgaur Sep 1, 2026
721fbed
Take the ALP value count from the page's own header
sfc-gh-pgaur Sep 1, 2026
14fc981
Encode an all-null ALP page as a bare header
sfc-gh-pgaur Sep 1, 2026
2327713
Build the ALP sources once, and under meson too
sfc-gh-pgaur Sep 1, 2026
37bcba5
Serialize ALP little-endian instead of refusing to build
sfc-gh-pgaur Sep 1, 2026
b4785f7
Move AlpMode out of the public header
sfc-gh-pgaur Sep 1, 2026
db22698
Test the ALP element-count and offset-chain validation
sfc-gh-pgaur Sep 1, 2026
3a8b853
Cover batched ALP decoding, and trim the encoder TODOs
sfc-gh-pgaur Sep 1, 2026
26e5289
Decode ALP pages one vector at a time
sfc-gh-pgaur Sep 1, 2026
7e078b6
Move the ALP end-to-end tests to their own file
sfc-gh-pgaur Sep 1, 2026
ef5559a
Name the ALP headers _internal.h
sfc-gh-pgaur Sep 1, 2026
df72402
Fold the ad-hoc ALP vector-size test into the typed suite
sfc-gh-pgaur Sep 3, 2026
80c2e08
Stop allocating scratch on every ALP vector decode
sfc-gh-pgaur Sep 3, 2026
af50476
Drop dead ALP code, link the TODOs, unpause the benchmarks
sfc-gh-pgaur Sep 3, 2026
1356770
Copy the ALP thrift comment from parquet-format verbatim
sfc-gh-pgaur Sep 3, 2026
be90517
Export AlpCodec and concatenate the ALP namespaces
sfc-gh-pgaur Sep 3, 2026
88cec11
Drop the AlpSampler members that were compile-time constants
sfc-gh-pgaur Sep 3, 2026
b2f53f7
Guard the loop pragmas behind the compiler that honours them
sfc-gh-pgaur Sep 7, 2026
187fa13
Export the nested VectorReader as well as its enclosing class
sfc-gh-pgaur Sep 7, 2026
1eb5593
Regenerate the thrift bindings so they carry the ALP enum value
sfc-gh-pgaur Sep 7, 2026
7977b5a
Reject an ALP page whose values outlive its definition levels
sfc-gh-pgaur Sep 7, 2026
2724559
Size the sample buffer without going through a double
sfc-gh-pgaur Sep 8, 2026
5fc824c
Say in prose what the target type may be
sfc-gh-pgaur Sep 8, 2026
05e90cf
Export the decode member templates from the shared library
sfc-gh-pgaur Sep 8, 2026
27cc07d
Cast the element count the ALP test passes
sfc-gh-pgaur Sep 8, 2026
8d273c6
Export the internal ALP classes the test links against
sfc-gh-pgaur Sep 8, 2026
daa1f8f
Widen the shift the ALP test takes a modulus with
sfc-gh-pgaur Sep 8, 2026
bb66b0f
Give the ALP roundtrip helper a name of its own
sfc-gh-pgaur Sep 8, 2026
58a72d5
Read the stored size through its accessor
sfc-gh-pgaur Sep 8, 2026
e123d8b
Skip the byte copies that have nothing to copy
sfc-gh-pgaur Sep 9, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions cpp/src/arrow/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -562,6 +562,9 @@ endforeach()

set(ARROW_UTIL_SRCS
util/align_util.cc
util/alp/alp.cc
util/alp/alp_codec.cc
util/alp/alp_sampler.cc
util/async_util.cc
util/atfork_internal.cc
util/basic_decimal.cc
Expand Down
3 changes: 3 additions & 0 deletions cpp/src/arrow/meson.build
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,9 @@ arrow_components = {

arrow_util_srcs = [
'util/align_util.cc',
'util/alp/alp.cc',
'util/alp/alp_codec.cc',
'util/alp/alp_sampler.cc',
'util/async_util.cc',
'util/atfork_internal.cc',
'util/basic_decimal.cc',
Expand Down
2 changes: 2 additions & 0 deletions cpp/src/arrow/util/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,8 @@ add_arrow_test(bit-utility-test
rle_encoding_test.cc
test_common.cc)

add_arrow_test(alp-test SOURCES alp/alp_test.cc)

add_arrow_test(crc32-test
SOURCES
crc32_test.cc
Expand Down
Loading
Loading