[SPARK-54785][SQL] Add support for binary sketch aggregations in KLL by cboumalh · Pull Request #53548 · apache/spark

cboumalh · 2025-12-19T20:52:00Z

What changes were proposed in this pull request?

This PR adds SQL aggregate functions with their tests for the KLL merge aggregate functions:

kll_merge_agg_bigint
kll_merge_agg_float
kll_merge_agg_double

These aggregate functions merge multiple binary KLL sketch representations.

Initial PRs:

[SPARK-54199][SQL] Add DataFrame API support for new KLL quantiles sketch functions #52900
[SPARK-53991][SQL] Add SQL support for KLL quantiles functions based on DataSketches #52800

Why are the changes needed?

The existing scalar kll_sketch_merge_* functions can only merge two sketches at a time. In distributed computing scenarios where sketches are pre-computed across multiple partitions, time windows, or datasets, users need to merge many sketches together.

Does this PR introduce any user-facing change?

Yes, this PR adds 3 new aggregate functions.

How was this patch tested?

New SQL tests were added to sql/core/src/test/resources/sql-tests/inputs/kllquantiles.sql:

Positive tests:

Merging bigint/float/double sketches from multiple rows
Merging with custom k parameters (400, 300, 500)
NULL value handling

Negative tests:

Type mismatches (passing non-binary types)
Invalid binary data
k parameter validation (too small, too large, NULL, non-constant)

Was this patch authored or co-authored using generative AI tooling?

claude-4.5-sonnet and manual changes.

cboumalh · 2025-12-19T21:03:12Z

cc @dtenedor if you have time to look at this! I think this would be useful (and someone pointed it out here). Thank you for your time, and please let me know your thoughts. I'll update it with the rest of the code (docs, pyspark, dataframe, etc..) if we're moving forward with this.

cboumalh · 2025-12-30T17:51:24Z

@cloud-fan do we want to get this out in 4.1.1 (KLL was part of 4.1)? I can finish it off since 4.1.1 is already RC. Thanks!

github-actions · 2026-01-08T00:22:23Z

JIRA Issue Information

=== Improvement SPARK-54785 ===
Summary: Add support for binary sketch aggregations in KLL
Assignee: None
Status: Open
Affected: ["4.1.0","4.1.1"]

This comment was automatically generated by GitHub Actions

dtenedor

@cboumalh thanks for working on this! The implementation and testing mostly LGTM, we should probably just deduplicate the implementations of the three functions as much as possible to simplify the code.

dtenedor · 2026-01-08T01:03:29Z

@cloud-fan do we want to get this out in 4.1.1 (KLL was part of 4.1)? I can finish it off since 4.1.1 is already RC. Thanks!

@cboumalh @cloud-fan I'd suggest to backport this to 4.1 if possible, since as a couple folks pointed out, the feature is much more usable if you can accumulate multiple previous sketch buffers using aggregate functions. But up to you guys.

cboumalh · 2026-01-08T22:09:29Z

Hi @dtenedor, I addressed the comments and added the rest of the code for this PR. Please let me know of any comments, and thank you for helping with this! I'll be scanning it for things to improve too.

dtenedor

Generally LGTM, just have a few more testing ideas.

dtenedor

LGTM after implementing remaining code review comments.

dtenedor

LGTM, merging to master and 4.1.

This PR adds SQL aggregate functions with their tests for the KLL merge aggregate functions: - `kll_merge_agg_bigint` - `kll_merge_agg_float` - `kll_merge_agg_double` These aggregate functions merge multiple binary KLL sketch representations. Initial PRs: - #52900 - #52800 The existing scalar `kll_sketch_merge_*` functions can only merge two sketches at a time. In distributed computing scenarios where sketches are pre-computed across multiple partitions, time windows, or datasets, users need to merge many sketches together. Yes, this PR adds 3 new aggregate functions. New SQL tests were added to `sql/core/src/test/resources/sql-tests/inputs/kllquantiles.sql`: **Positive tests:** - Merging bigint/float/double sketches from multiple rows - Merging with custom k parameters (400, 300, 500) - NULL value handling **Negative tests:** - Type mismatches (passing non-binary types) - Invalid binary data - k parameter validation (too small, too large, NULL, non-constant) claude-4.5-sonnet and manual changes. Closes#53548 from cboumalh/cboumalh-kll-enhancement. Lead-authored-by: Chris Boumalhab <cboumalh@amazon.com> Co-authored-by: Chris Boumalhab <84485659+cboumalh@users.noreply.github.com> Signed-off-by: Daniel Tenedorio <daniel.tenedorio@databricks.com> (cherry picked from commit fc15f72) Signed-off-by: Daniel Tenedorio <daniel.tenedorio@databricks.com>

zhengruifeng · 2026-05-27T06:30:16Z

+ sketch (range 8-65535). If k is not specified, the merged sketch adopts the k value
+ from the first input sketch.
+
+ .. versionadded:: 4.1.0


normally we don't cherry-pick new feature to old branches, and after this PR, these new functions actually started since 4.1.2.
It is causing confusion when I audit public APIs by comparing 4.2.0 vs 4.1.0.
will fix in #56135

…_agg_* (4.1.0 -> 4.1.2) ### What changes were proposed in this pull request? Fix the `versionadded` / `since` / `ExpressionDescription.since` annotation on `kll_merge_agg_{bigint,float,double}`, changing `4.1.0` to `4.1.2`. Touches three files: - `python/pyspark/sql/functions/builtin.py` — 3 `.. versionadded::` docstrings - `sql/api/src/main/scala/org/apache/spark/sql/functions.scala` — 15 `since` Scaladocs (5 overloads × 3 types) - `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/kllAggregates.scala` — 3 `ExpressionDescription(since = ...)` annotations ### Why are the changes needed? All three places currently claim "Since 4.1.0", but: | Tag | Date | Contains `kll_merge_agg_*`? | |---|---|---| | v4.1.0 | 2025-12-11 | No | | v4.1.1 | 2026-01-02 | No | | v4.1.2 | 2026-05-16 | **Yes** (via cherry-pick `a39c1b8e5e2`) | | v4.2.0-preview2 | 2026-02-05 | Yes (via master commit `fc15f726eab`) | The introducing commit on master (SPARK-54785 / #53548, `fc15f726eab` on 2026-01-12) landed *after* v4.1.0 was tagged, and the branch-4.1 cherry-pick (`a39c1b8e5e2`) shipped first in v4.1.2. No 4.1.0/4.1.1 release contains these functions. `4.1.2` is the earliest stable release in which they are available, so that is the correct value across all three annotation sites. ### Does this PR introduce _any_ user-facing change? Documentation-only change. The rendered Python API ref, Scaladoc, and SQL function reference currently render "New in version 4.1.0" / "Since: 4.1.0", which is misleading; this PR corrects all of them to `4.1.2`. ### How was this patch tested? Doc/annotation-only change. Manually verified each touched site is one of the 15 `kll_merge_agg_*` overload Scaladocs, the 3 KllMergeAgg expression descriptions, or the 3 Python docstrings — sibling annotations on `kll_sketch_agg_*` (which legitimately shipped in v4.1.0) are left unchanged. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (model: claude-opus-4-7) Closes#56135 from zhengruifeng/spark-doc-fixes-dev1. Authored-by: Ruifeng Zheng <ruifengz@apache.org> Signed-off-by: Ruifeng Zheng <ruifengz@foxmail.com>

…_agg_* (4.1.0 -> 4.1.2) ### What changes were proposed in this pull request? Fix the `versionadded` / `since` / `ExpressionDescription.since` annotation on `kll_merge_agg_{bigint,float,double}`, changing `4.1.0` to `4.1.2`. Touches three files: - `python/pyspark/sql/functions/builtin.py` — 3 `.. versionadded::` docstrings - `sql/api/src/main/scala/org/apache/spark/sql/functions.scala` — 15 `since` Scaladocs (5 overloads × 3 types) - `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/kllAggregates.scala` — 3 `ExpressionDescription(since = ...)` annotations ### Why are the changes needed? All three places currently claim "Since 4.1.0", but: | Tag | Date | Contains `kll_merge_agg_*`? | |---|---|---| | v4.1.0 | 2025-12-11 | No | | v4.1.1 | 2026-01-02 | No | | v4.1.2 | 2026-05-16 | **Yes** (via cherry-pick `a39c1b8e5e2`) | | v4.2.0-preview2 | 2026-02-05 | Yes (via master commit `fc15f726eab`) | The introducing commit on master (SPARK-54785 / #53548, `fc15f726eab` on 2026-01-12) landed *after* v4.1.0 was tagged, and the branch-4.1 cherry-pick (`a39c1b8e5e2`) shipped first in v4.1.2. No 4.1.0/4.1.1 release contains these functions. `4.1.2` is the earliest stable release in which they are available, so that is the correct value across all three annotation sites. ### Does this PR introduce _any_ user-facing change? Documentation-only change. The rendered Python API ref, Scaladoc, and SQL function reference currently render "New in version 4.1.0" / "Since: 4.1.0", which is misleading; this PR corrects all of them to `4.1.2`. ### How was this patch tested? Doc/annotation-only change. Manually verified each touched site is one of the 15 `kll_merge_agg_*` overload Scaladocs, the 3 KllMergeAgg expression descriptions, or the 3 Python docstrings — sibling annotations on `kll_sketch_agg_*` (which legitimately shipped in v4.1.0) are left unchanged. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (model: claude-opus-4-7) Closes#56135 from zhengruifeng/spark-doc-fixes-dev1. Authored-by: Ruifeng Zheng <ruifengz@apache.org> Signed-off-by: Ruifeng Zheng <ruifengz@foxmail.com> (cherry picked from commit 63f9c88) Signed-off-by: Ruifeng Zheng <ruifengz@foxmail.com>

Add support for binary sketch aggregations in KLL
c4d5bfa

cboumalh marked this pull request as draft December 19, 2025 20:52

github-actionsBot added the SQL label Dec 19, 2025

cboumalh changed the title ~~Add support for binary sketch aggregations in KLL~~[WIP][SPARK-54785][SQL] Add support for binary sketch aggregations in KLLDec 19, 2025

cboumalh mentioned this pull request Dec 19, 2025
[SPARK-53991][SQL] Add SQL support for KLL quantiles functions based on DataSketches #52800
Closed

cboumalh marked this pull request as ready for review December 19, 2025 21:04

minor test fix
a2c2ca4

cloud-fan reviewed Dec 22, 2025
View reviewed changes

Comment thread...alyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/kllAggregates.scala Outdated

expression desc fix
1db4e91

dtenedor changed the title ~~[WIP][SPARK-54785][SQL] Add support for binary sketch aggregations in KLL~~[SPARK-54785][SQL] Add support for binary sketch aggregations in KLLJan 8, 2026

dtenedor reviewed Jan 8, 2026
View reviewed changes

cboumalhand others added 4 commits January 8, 2026 13:53

Merge branch 'apache:master' into cboumalh-kll-enhancement
d8b60b7

comments fix
011dfa8

minor
a96404a

fix
42a4c6d

cboumalh requested a review from dtenedor January 8, 2026 22:08

fix
2e4ace2

dtenedor reviewed Jan 8, 2026
View reviewed changes

dtenedor approved these changes Jan 8, 2026
View reviewed changes

Chris Boumalhaband others added 3 commits January 9, 2026 00:49

comment fixes
0d23d04

fix
9b7ce57

Merge branch 'apache:master' into cboumalh-kll-enhancement
f222090

dtenedor approved these changes Jan 12, 2026
View reviewed changes

dtenedor closed this in fc15f72Jan 12, 2026

zhengruifeng reviewed May 27, 2026
View reviewed changes

zhengruifeng mentioned this pull request May 27, 2026
[SPARK-57116][SQL][PYTHON][DOC] Fix versionadded/@since for kll_merge_agg_* (4.1.0 -> 4.1.2) #56135
Closed

art000109 mentioned this pull request Jul 27, 2026
[SPARK-XXXXX][PYTHON][DOCS] Added Returns and See Also sections in pyspark.sql.functions documentation. #57529
Open

Uh oh!

Conversation

cboumalh commented Dec 19, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

cboumalh commented Dec 19, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

cboumalh commented Dec 30, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actionsBot commented Jan 8, 2026

JIRA Issue Information

Uh oh!

dtenedor left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dtenedor commented Jan 8, 2026

Uh oh!

cboumalh commented Jan 8, 2026

Uh oh!

dtenedor left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dtenedor left a comment

Choose a reason for hiding this comment

Uh oh!

dtenedor left a comment

Choose a reason for hiding this comment

Uh oh!

zhengruifengMay 27, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cboumalh commented Dec 19, 2025•
edited
Loading

cboumalh commented Dec 19, 2025•
edited
Loading

cboumalh commented Dec 30, 2025•
edited
Loading