[SPARK-58831][SQL] Add bitmap scalar set operation functions

[jiangxt2]

What changes were proposed in this pull request?

This PR adds four scalar set operations for Spark's flat bitmap representation:

• bitmap_and
• bitmap_or
• bitmap_andnot
• bitmap_xor

Each function operates on two BINARY values from the same row. The implementation:

• accepts inputs up to 4096 bytes;
• treats missing trailing bytes as zero;
• returns a newly allocated, fixed-size 4096-byte bitmap;
• propagates NULL inputs;
• rejects oversized inputs with the structured BITMAP_INPUT_TOO_LARGE error;
• does not mutate either input.

The functions are implemented as Catalyst-native binary expressions with both interpreted and whole-stage codegen paths. They are registered in the SQL function registry and exposed through the Scala API, PySpark classic, and Spark Connect.

This PR also adds SQL documentation, Python documentation, a structured error definition, Connect plan and schema golden files, and tests for the new APIs.

Why are the changes needed?

Spark provides functions for constructing, aggregating, and counting flat bitmaps, but it does not provide scalar set operations for combining two bitmap values from the same row.

Without native scalar functions, users need UDFs or external bitmap processing to compute intersection, union, difference, or symmetric difference between precomputed bitmap columns. Native Catalyst expressions provide consistent SQL semantics, structured validation, and code generation, and allow these operations to compose naturally with the existing bitmap aggregate and count functions.

Does this PR introduce any user-facing change?

Yes. It adds four new SQL, Scala, PySpark classic, and Spark Connect functions:

SELECT bitmap_and(left_bitmap, right_bitmap);
SELECT bitmap_or(left_bitmap, right_bitmap);
SELECT bitmap_andnot(left_bitmap, right_bitmap);
SELECT bitmap_xor(left_bitmap, right_bitmap);

bitmap_andnot(left, right) is directional and returns the bits present in left but not in right.

No existing function behavior is changed.

How was this patch tested?

The following targeted suites and build checks passed:

• build/sbt "catalyst/testOnly org.apache.spark.sql.catalyst.expressions.BitmapExpressionUtilsSuite" (10 tests)
• build/sbt "sql/testOnly org.apache.spark.sql.BitmapExpressionsQuerySuite" (19 tests, including interpreted and codegen execution)
• build/sbt "sql/testOnly org.apache.spark.sql.ExpressionsSchemaSuite"
• build/sbt "connect-client-jvm/testOnly org.apache.spark.sql.PlanGenerationTestSuite -- -z 'function bitmap'" (9 tests)
• build/sbt -Phive package
• python/run-tests --testnames "pyspark.sql.tests.test_functions FunctionsTests.test_bitmap_scalar_functions"
• python/run-tests --testnames "pyspark.sql.tests.connect.test_connect_plan SparkConnectPlanTests.test_bitmap_scalar_functions"
• Targeted PySpark doctests for the four new functions (12 tests)

The tests cover variable-length and empty inputs, the 4096-byte boundary, input immutability, null propagation, invalid types, oversized inputs on either side, interpreted and codegen execution, aggregation composition, grouped queries, PySpark classic, and Spark Connect.

Scalafmt, Ruff, JSON validation, golden-file consistency checks, and git diff --check also passed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex and Claude AI

[jiangxt2]

Additional ecosystem references:

For reference, the same four bitmap set operations are also available in several analytical SQL engines:

• Flink 2.3 exposes BITMAP_AND, BITMAP_OR, BITMAP_ANDNOT, and BITMAP_XOR over its native BITMAP type.
• ClickHouse has provided bitmapAnd, bitmapOr, bitmapAndnot, and bitmapXor since v20.1.
• Apache Doris provides BITMAP_AND, BITMAP_OR, BITMAP_AND_NOT, and BITMAP_XOR, and documents bitmap use cases such as exact distinct counting and audience segmentation.
• StarRocks provides the corresponding bitmap_and, bitmap_or, bitmap_andnot, and bitmap_xor functions and documents bitmap use cases such as exact distinct counting and retention analysis.

These engines use different physical bitmap representations, so this comparison does not imply binary-format compatibility. Spark continues to use its existing 4096-byte BINARY flat bitmap, and both operands must use the same bit-position mapping and, when bucketed, represent the same bucket.

These ecosystem references support the completeness and naming of the proposed API. The four scalar functions allow users to compute row-wise intersection, union, difference, and symmetric difference between precomputed bitmaps without implementing equivalent UDFs.

[jiangxt2]

Hi @uros-b, would you mind taking a look at this PR when you have time? All CI checks are green. Thanks! ^_^

[uros-b]

LGTM, thank you @jiangxt2! Adding @cloud-fan / @MaxGekk for additional review on Catalyst expressions / SQL area, and @HyukjinKwon / @zhengruifeng / @Yicong-Huang for PySpark function bindings and Connect.

[cloud-fan]

can you resolve merge conflicts?

[jiangxt2]

can you resolve merge conflicts?

Sure, I’ll take care of the merge conflicts and push an updated branch shortly.

[jiangxt2]

Hi @cloud-fan I resolved the merge conflicts and pushed the updated branch. The PR is now mergeable again.

[cloud-fan]

0 blocking, 0 non-blocking, 0 nits.
The engine semantics, public bindings, documentation, and tests are internally consistent; I found no issues that need author action.

Verification

I traced each public function name through the Scala/Python wrappers and Connect plans into FunctionRegistry, the Catalyst expressions, and the byte-array helpers. I checked unequal and empty input behavior, fixed-size allocation, null propagation, both oversized-input guards, interpreted/code-generated convergence, API versioning, and the focused test expectations. The selected text, contract, and local-efficiency scanners also completed without findings. I did not run tests locally as part of this review.

[cloud-fan]

Merge Summary:

• merged into master cc75de6
• merged into branch-4.x d20be69

Posted by merge_spark_pr.py