[SPARK-55520][TESTS] Regenerate benchmark results by dongjoon-hyun · Pull Request #54313 · apache/spark

dongjoon-hyun · 2026-02-13T16:44:22Z

What changes were proposed in this pull request?

This PR aims to regenerate benchmark results to check the intermediate status as a part of Apache Spark 4.2.0 preparation.

Please note that V2FunctionBenchmark is excluded because it's broken due to NumericEvalContext.evalMode() error currently. It's good to identify this kind of bug as early as possible via this PR.

SPARK-55519 V2FunctionBenchmark is broken

Why are the changes needed?

Apache Spark 4.2.0 introduced many improvements on top of the key dependency differences from Spark 4.1.0:

We updated the benchmark result 4 months ago. So, it's time to make them up-to-date with our actual code and the current infra.

[SPARK-53893][TESTS] Regenerate benchmark results after upgrading to Scala 2.13.17 #52600

- OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Linux 6.11.0-1018-azure
+ OpenJDK 64-Bit Server VM 17.0.18+8-LTS on Linux 6.14.0-1017-azure

- OpenJDK 64-Bit Server VM 21.0.8+9-LTS on Linux 6.11.0-1018-azure
+ OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.14.0-1017-azure

Does this PR introduce any user-facing change?

No. This is a change on benchmark result files.

How was this patch tested?

Manual review.

Was this patch authored or co-authored using generative AI tooling?

No.

dongjoon-hyun · 2026-02-13T17:22:13Z

+ZooKeeperPersistenceEngine with JavaSerializer 66386863209  0.0 6637519.7 1.0X
+FileSystemPersistenceEngine with JavaSerializer 3161316811 0.0 3160685.62.1X
+FileSystemPersistenceEngine with JavaSerializer (lz4) 87389933 0.0 873026.87.6X
+FileSystemPersistenceEngine with JavaSerializer (lzf) 3286 3302 16  0.0 3286382.2 2.0X


This could be a regression to take a look at. Java 17 and 21 are the same.

dongjoon-hyun · 2026-02-13T17:24:48Z

-Indexed 11 12 0 0.1 10698.4 1.0X
-No Index  8 8 00.1  7430.81.4X
+Indexed 12 12 0 0.1 11413.1 1.0X
+No Index 13 14 1  0.1 12820.20.9X


This could be a regression because the ratio is reverted. However, Java 17 result looks fine.

dongjoon-hyun · 2026-02-13T17:25:29Z

-Indexed 11 12 0 0.1 10694.7 1.0X
-No Index  8 80 0.1  7443.81.4X
+Indexed 12 12 0 0.1 11461.0 1.0X
+No Index 13 14  0 0.1 12881.50.9X


Ditto. The ratio is reverted but Java 17 result looks fine.

dongjoon-hyun · 2026-02-13T17:31:09Z

+Compression 10000 times at level 1 without buffer pool 6486536 0.064756.8 1.0X
+Compression 10000 times at level 2 without buffer pool 6886881 0.0 68788.7 0.9X
+Compression 10000 times at level 3 without buffer pool 80481010  0.0 80354.7 0.8X
+Compression 10000 times at level 1 with buffer pool 5805822 0.0 58024.51.1X


These are expected improvement due to buffer pool.

dongjoon-hyun · 2026-02-13T17:31:24Z

+Decompression 10000 times from level 1 without buffer pool 5955973 0.059453.4 1.0X
+Decompression 10000 times from level 2 without buffer pool 5945951 0.059429.2 1.0X
+Decompression 10000 times from level 3 without buffer pool 5955961 0.059501.2 1.0X
+Decompression 10000 times from level 1 with buffer pool 542543 1 0.0 54194.01.1X


ditto. These are expected improvement due to buffer pool.

dongjoon-hyun · 2026-02-13T17:33:53Z

 ------------------------------------------------------------------------------------------------------------------------
-Use HashSet 1 1 0 1530.20.7 1.0X
-Use EnumSet 2 2 0 461.22.2 0.3X
+Use HashSet 1 1 0  761.61.3 1.0X


HashSet seems to be improved by 2x in both Java 17 and 21.

dongjoon-hyun · 2026-02-13T18:42:46Z

-Murmur3_x86_32 3535360.616.5 1.0X
-xxHash 64-bit 2828 0 76.013.2 1.3X
-HiveHasher 4444 0 48.120.8 0.8X
+Murmur3_x86_32 2626079.412.6 1.0X


The ratio is changed because Murmur3_x86_32 is improved .

dongjoon-hyun · 2026-02-13T18:44:07Z

-Spark 349735014 0.3 3497.4 1.4X
-Spark Binary 263726392 0.4 2637.2 1.8X
+Common Codecs 4987499912 0.2 4986.8 1.0X
+Java 40374040 40.2 4036.71.2X


The ratio is reverted because Java 21 becomes faster. Like Java 17, this should be the fastest one.

dongjoon-hyun · 2026-02-13T18:52:37Z

-LongDelta 6526520102.9 9.7 1.2X
+PassThrough 837837180.212.5 1.0X
+RunLengthEncoding 12311231 0 54.5 18.3 0.7X
+DictionaryEncoding 729730292.1 10.91.1X


The ratio of DictionaryEncoding is changed.

dongjoon-hyun · 2026-02-13T18:54:01Z

 Test read with LongType: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
 ------------------------------------------------------------------------------------------------------------------------
-ConstantColumnVector 183918402222.74.5 1.0X
-OnHeapColumnVector 000 1580533.50.0 7096.3X


Previous value 0 looks wrong to me.

dongjoon-hyun · 2026-02-13T18:57:21Z

+ParquetReader Vectorized: DataPageV1 1671714 94.0 10.6 1.0X
+ParquetReader Vectorized: DataPageV2 193197 2 81.5 12.3 0.9X
+ParquetReader Vectorized -> Row: DataPageV1 171179892.1 10.9 1.0X
+ParquetReader Vectorized -> Row: DataPageV2 203204277.612.90.8X


This becomes slower slightly.

dongjoon-hyun · 2026-02-13T18:58:17Z

+ParquetReader Vectorized: DataPageV1 151153 1 103.9  9.6 1.0X
+ParquetReader Vectorized: DataPageV2 156158 2 100.7 9.9 1.0X
+ParquetReader Vectorized -> Row: DataPageV1 162163 1  97.210.3 0.9X
+ParquetReader Vectorized -> Row: DataPageV2 1621632 97.1 10.30.9X


The ratio is reverted.

dongjoon-hyun · 2026-02-13T19:02:52Z

+SQL Parquet Vectorized: DataPageV2 147614794 7.1140.84.9X
+SQL Parquet MR: DataPageV1 3564358226 2.9339.9 2.0X
+SQL Parquet MR: DataPageV2 3578358592.9341.2 2.0X
+ParquetReader Vectorized: DataPageV1 879886711.983.8 8.3X


Previously, this was the fastest but now becomes slower relatively than SQL ORC Vectorized somehow.

dongjoon-hyun · 2026-02-13T19:07:59Z

 50 ints (non-compact): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
 ------------------------------------------------------------------------------------------------------------------------
-In expression 99 100 1 100.7 9.9 1.0X
-InSet expression 132 136 7 75.9 13.2 0.8X
+In expression 156 158 3 64.0 15.6 1.0X


In becomes slower relatively, but it could be due to different GitHub Action CPUs.

dongjoon-hyun · 2026-02-13T19:13:15Z

-First 10 integers using SELECT and LIMIT 69 74 6 0.0 6947861.5 2.1X
-First 10 integers referencing external table in anchor 137 152 14 0.0 13678740.6 1.1X
+First 10 integers using VALUES 162 180 18 0.0 16176649.5 1.0X
+First 10 integers using SELECT 147 157 9 0.0 14653403.4 1.1X


This becomes slower relatively and consistently.

dongjoon-hyun · 2026-02-13T19:16:16Z

-long/nullable int/string to primitive wholestage off 3637 1 2.8362.6 1.0X
-long/nullable int/string to primitive wholestage on 29322 3.5285.8 1.3X
+long/nullable int/string to primitive wholestage off 3232 1 3.2315.9 1.0X
+long/nullable int/string to primitive wholestage on 32386 3.1317.6 1.0X


The value is too small to say there is a difference.

dongjoon-hyun · 2026-02-13T19:22:52Z

The audit is finished. Could you review this PR, @peter-toth ?

dongjoon-hyun · 2026-02-13T19:39:37Z

Thank you, @peter-toth . SPARK-55519 V2FunctionBenchmark is broken and the further investigation will follow later independently.

Merged to master.

### What changes were proposed in this pull request? This PR aims to regenerate benchmark results to check the intermediate status as a part of Apache Spark 4.2.0 preparation. Please note that `V2FunctionBenchmark` is excluded because it's broken due to `NumericEvalContext.evalMode()` error currently. It's good to identify this kind of bug as early as possible via this PR. - [SPARK-55519 `V2FunctionBenchmark` is broken](https://issues.apache.org/jira/browse/SPARK-55519) ### Why are the changes needed? Apache Spark 4.2.0 introduced many improvements on top of the key dependency differences from Spark 4.1.0: - apache#53396 - apache#53582 - apache#53347 - apache#54233 - apache#54292 We updated the benchmark result 4 months ago. So, it's time to make them up-to-date with our actual code and the current infra. - apache#52600 ``` - OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Linux 6.11.0-1018-azure + OpenJDK 64-Bit Server VM 17.0.18+8-LTS on Linux 6.14.0-1017-azure ``` ``` - OpenJDK 64-Bit Server VM 21.0.8+9-LTS on Linux 6.11.0-1018-azure + OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.14.0-1017-azure ``` ### Does this PR introduce _any_ user-facing change? No. This is a change on benchmark result files. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closesapache#54313 from dongjoon-hyun/SPARK-55520. Lead-authored-by: Dongjoon Hyun <dongjoon@apache.org> Co-authored-by: dongjoon-hyun <dongjoon-hyun@users.noreply.github.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

### What changes were proposed in this pull request? This PR aims to regenerate benchmark results to check the intermediate status as a part of Apache Spark 5.0.0 preparation ### Why are the changes needed? To make the benchmark up-to-date and fill the missing gaps in order to help the comparison with the upcoming Apache Spark 4.3.0. **1. Last Update (2026-02-13)** - #54313 **2. Java Version Changes** ``` - OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.14.0-1017-azure + OpenJDK 64-Bit Server VM 21.0.12+8-LTS on Linux 6.17.0-1020-azure ``` ``` - OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.14.0-1017-azure + OpenJDK 64-Bit Server VM 21.0.12+8-LTS on Linux 6.17.0-1020-azure ``` ``` - OpenJDK 64-Bit Server VM 25.0.2+10-LTS on Linux 6.17.0-1008-azure + OpenJDK 64-Bit Server VM 25.0.4+7-LTS on Linux 6.17.0-1020-azure ``` **3. MISSING BENCHMARK RESULT** - #55278 didn't generate the benchmark result at all. - #56291 didn't generate Java 21 and 25 result. - #56291 didn't generate Java 21 and 25 result. - #56485 didn't generate Java 21 and 25 result. - #57232 didn't generate Java 21 and 25 result. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes#57620 from dongjoon-hyun/SPARK-58415. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

[SPARK-55520][TESTS] Regenerate benchmark results
ec4d3cd

dongjoon-hyun commented Feb 13, 2026
View reviewed changes

peter-toth approved these changes Feb 13, 2026
View reviewed changes

dongjoon-hyun closed this in a252c5bFeb 13, 2026

dongjoon-hyun deleted the SPARK-55520 branch February 13, 2026 19:41

dongjoon-hyun mentioned this pull request Feb 27, 2026
[SPARK-55519][SQL][TESTS] Fix V2FunctionBenchmark #54540
Closed

dongjoon-hyun mentioned this pull request Jul 29, 2026
[SPARK-58415][TESTS] Regenerate benchmark results #57620
Closed

Uh oh!

Conversation

dongjoon-hyun commented Feb 13, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyunFeb 13, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Feb 13, 2026

Uh oh!

dongjoon-hyun commented Feb 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dongjoon-hyun commented Feb 13, 2026•
edited
Loading

dongjoon-hyunFeb 13, 2026•
edited
Loading