Skip to content

[SPARK-56629][PYTHON][TESTS] Add ASV microbenchmark for SQL_COGROUPED_MAP_PANDAS_UDF - #55551

Closed
Yicong-Huang wants to merge 1 commit into
apache:masterfrom
Yicong-Huang:SPARK-56629/bench/cogrouped-map-pandas
Closed

[SPARK-56629][PYTHON][TESTS] Add ASV microbenchmark for SQL_COGROUPED_MAP_PANDAS_UDF#55551
Yicong-Huang wants to merge 1 commit into
apache:masterfrom
Yicong-Huang:SPARK-56629/bench/cogrouped-map-pandas

Conversation

@Yicong-Huang

@Yicong-HuangYicong-Huang commented Apr 25, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Add ASV microbenchmarks CogroupedMapPandasUDFTimeBench / CogroupedMapPandasUDFPeakmemBench for SQL_COGROUPED_MAP_PANDAS_UDF in python/benchmarks/bench_eval_type.py, mirroring the existing cogrouped-arrow benchmarks. Covers 6 scenarios x 4 UDFs (including the 3-arg (key, left, right) variant).

Why are the changes needed?

Baseline for SPARK-55724 so the upcoming CogroupPandasUDFSerializer -> ArrowStreamSerializer refactor can detect regressions.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

ASV run with -a repeat=5:

$ COLUMNS=120 asv run --bench CogroupedMapPandasUDFTimeBench --python=same -a repeat=5
================ ============== ============ =============== ==================
-- udf
---------------- --------------------------------------------------------------
scenario identity_udf concat_udf left_semi_udf key_identity_udf
================ ============== ============ =============== ==================
few_groups_sm 162+-0.8ms 182+-0.7ms 185+-0.6ms 162+-2ms
few_groups_lg 420+-3ms 540+-10ms 512+-6ms 421+-7ms
many_groups_sm 1.37+-0s 1.49+-0s 1.52+-0.01s 1.38+-0s
many_groups_lg 775+-4ms 918+-7ms 902+-5ms 781+-7ms
wide_values 1.09+-0s 1.28+-0.01s 1.14+-0.01s 1.09+-0.01s
multi_key 434+-6ms 489+-7ms 464+-9ms 431+-2ms
================ ============== ============ =============== ==================
$ COLUMNS=120 asv run --bench CogroupedMapPandasUDFPeakmemBench --python=same
================ ============== ============ =============== ==================
-- udf
---------------- --------------------------------------------------------------
scenario identity_udf concat_udf left_semi_udf key_identity_udf
================ ============== ============ =============== ==================
few_groups_sm 470M 472M 471M 470M
few_groups_lg 508M 521M 509M 508M
many_groups_sm 478M 480M 478M 478M
many_groups_lg 499M 505M 498M 499M
wide_values 501M 506M 501M 501M
multi_key 479M 481M 480M 479M
================ ============== ============ =============== ==================

Was this patch authored or co-authored using generative AI tooling?

No.

@HyukjinKwon

Copy link
Copy Markdown
Member

Merged to master.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@Yicong-Huang@HyukjinKwon