[SPARK-58186][SQL] Add bitmap_contains function

[jiangxt2]

What changes were proposed in this pull request?

This PR adds bitmap_contains(bitmap, bit_position), a scalar predicate for Spark's existing flat BinaryType bitmap function family. It is available through SQL, the Scala DataFrame API, PySpark classic, and PySpark Connect.

The function:

• checks a bucket-local bit position in constant time;
• accepts a binary bitmap and a numeric position that is cast to BIGINT using Spark's existing cast semantics;
• propagates SQL NULL for null bitmap or position inputs;
• returns false for negative positions, positions at or above 32768, and positions beyond the actual bitmap byte length.

For an original value, callers use bitmap_bucket_number(value) to select the bitmap bucket and bitmap_bit_position(value) as the position passed to bitmap_contains.

This is a clean resubmission of #57336 against the current Spark codebase. It retains the function-argument documentation requested during the earlier review and expands the coverage for bucket semantics and bucket-aware joins. It keeps Spark's existing flat bitmap representation and does not introduce another bitmap type, serialization format, or external dependency.

This PR is independent of #58066, which proposes scalar bitmap set operations. The APIs are complementary: if both changes are merged, bitmap_contains can also test a bitmap returned by the scalar set operations.

Why are the changes needed?

Spark can construct flat bitmaps, aggregate them across rows, and count their set bits, but it has no built-in predicate for testing whether a specific position is present.

Without this function, users must convert the bitmap to another collection representation or use a UDF. That loses the consistent SQL, Scala, PySpark classic, and Spark Connect API surface provided by a native Catalyst expression.

bitmap_contains completes the existing bitmap workflow and allows precomputed bitmaps to be used directly in filters, CASE WHEN expressions, and bucket-aware membership joins.

Does this PR introduce any user-facing change?

Yes. Released Spark versions do not provide this function, and this PR does not change the behavior of any existing bitmap function.

It adds the following function in SQL, the Scala DataFrame API, PySpark classic, and PySpark Connect:

bitmap_contains(bitmap BINARY, bit_position BIGINT) ->BOOLEAN

The position is local to one bitmap bucket. A query using original values must match the bucket and pass bitmap_bit_position(value) to bitmap_contains.

For example:

WITH bitmaps AS (
SELECT
bitmap_bucket_number(value) AS bucket,
bitmap_construct_agg(bitmap_bit_position(value)) AS bitmap
FROMVALUES (1L), (32769L) AS source(value)
GROUP BY bitmap_bucket_number(value)
)
SELECTprobe.valueFROM bitmaps
JOINVALUES (1L), (2L), (32769L), (32770L) AS probe(value)
ONbitmaps.bucket= bitmap_bucket_number(probe.value)
AND bitmap_contains(bitmaps.bitmap, bitmap_bit_position(probe.value))
ORDER BYprobe.value;

This returns 1 and 32769.

A null bitmap or position produces SQL NULL. A negative position, a position at or above 32768, or a position beyond the actual bitmap byte length produces false.

How was this patch tested?

The patch adds coverage for:

• helper-level lookup of set and unset bits;
• empty and short bitmaps;
• fixed-range and Long boundaries;
• SQL and Scala DataFrame API behavior;
• null propagation;
• numeric casts in ANSI and non-ANSI modes;
• invalid bitmap and position types;
• filter and CASE WHEN predicates;
• bucket-aware join usage across multiple buckets;
• PySpark classic and Connect parity;
• Scala Connect plan generation.

The SQL expression schema and Connect query-test golden files were regenerated.

Targeted validation completed for the production implementation before the final bucket-aware join test rewrite:

• BitmapExpressionUtilsSuite: 9 tests passed.
• BitmapExpressionsQuerySuite: 19 tests passed.
• PlanGenerationTestSuite: 745 tests passed, with 2 ignored.
• ProtoToParsedPlanTestSuite: 750 tests passed.
• PySpark classic functions: 175 tests passed, with 6 dependency-based skips.
• PySpark Connect parity: 123 tests passed, with 4 expected skips.
• Classic and Connect function doctests passed.

The current head, c52d16aa09ce406f38084d4adf7a54d86ef80e94, includes the final bucket-aware join test. Its GitHub CI rollup passed: 30 checks succeeded, and one Kubernetes integration check was skipped.

Current cumulative diff static validation:

• git diff --check: passed.
• Spark Scalafmt validation: passed.
• Spark Scalastyle for main and test sources: passed.
• Ruff on the changed Python files: passed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex and Claude AI

[jiangxt2]

Additional ecosystem context:

Apache Doris and StarRocks expose bitmap_contains, while ClickHouse exposes bitmapContains. This comparison supports the API naming and the value of a native membership predicate only; it does not imply binary-format or semantic compatibility. Those engines use native bitmap representations, while this PR keeps Spark's existing flat BINARY bitmap representation and bucket-local position mapping.

[uros-b]

Thank you @jiangxt2!

[uros-b]

cc @HyukjinKwon who reviewed #57336
also cc @cloud-fan

[jiangxt2]

Hi @cloud-fan, if you have time, would you mind taking a look at this PR? It adds bitmap_contains to Spark’s bitmap function family. Thanks!