Uh oh!
There was an error while loading. Please reload this page.
feat(aggregate): add Java-compatible field aggregators - #463
feat(aggregate): add Java-compatible field aggregators#463liangjie3138 wants to merge 14 commits into
Conversation
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
| pooled_unique_ptr<Bytes> result = | ||
| Bytes::AllocateBytes(serialized.size() * sizeof(T), GetDefaultPool().get()); | ||
| if (!serialized.empty()) { | ||
| std::memcpy(result->data(), serialized.data(), result->size()); |
There was a problem hiding this comment.
Just to clarify: I noticed many places are using GetDefaultPool(). Will the data volume during merging be relatively large? In paimon-cpp, batches are merged together, and data that has not yet been converted into Arrow arrays may stay held in memory for a while. Is it appropriate for that memory to be allocated from a default pool that is separate from the user-provided memory pool?
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…esult Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…lookups Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…:Equals Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… suite Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ses from Java Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…new aggregators Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
21c7ee0 to
428a47dCompareCo-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lxy-9602
commented
Aug 6, 2026
This has already been merged in apache/paimon-cpp#183, so we can close this PR. |
Purpose
Linked issue: close#457
Port the six practical field aggregators missing on the C++ side —
collect,merge_map,nested_update,rbm64,hll_sketch,theta_sketch.Also adds
GenericArray/GenericMap, the in-memoryInternalArray/InternalMapimplementations these aggregators need to build ARRAY and MAPresults.
Tests
18 new unit test cases in
paimon-core-test, covering each aggregator'saggregate / retract / null / type-validation paths.
Three pin behaviour that is easy to "fix" incorrectly, and each was checked by
reverting the corresponding production change and confirming it fails:
BinaryAggMergeFunctionTest.OwnedAccumulatorSurvivesNullInput— drives a realAggregateMergeFunctionso the use-after-free is exercised through theproduction path.
FieldNestedUpdateAggTest.CountLimitCountsNullElementsOfAccumulator— countlimit is measured against the raw element count, matching Java.
FieldIgnoreRetractAggTest.ReversedAggBypassesWrappedOverride— the wrapperdoes not preserve the wrapped aggregator's reversed-aggregation override, also
matching Java.
No integration tests added.
API and Format
Additive only, no signatures changed:
include/paimon/defs.h: four newOptionsconstants —NESTED_KEY,NESTED_KEY_NULL_STRATEGY,NESTED_SEQUENCE_FIELD,COUNT_LIMIT.CoreOptions: aNestedKeyNullStrategyenum plus accessors.FieldAggregator: new virtuals default to the existingAgg/AggReversed.New dependency: Apache DataSketches 5.2.0 (Apache-2.0), resolved through
resolve_dependency()like the other bundled dependencies.Documentation
None.
Generative AI tooling
Generated-by: Claude Code 2.1.220 (Claude Opus 5)