Skip to content

[SPARK-54822][BUILD] Bump Parquet 1.17.0 - #53582

Closed
pan3793 wants to merge 2 commits into
apache:masterfrom
pan3793:SPARK-54822
Closed

[SPARK-54822][BUILD] Bump Parquet 1.17.0#53582
pan3793 wants to merge 2 commits into
apache:masterfrom
pan3793:SPARK-54822

Conversation

@pan3793

@pan3793pan3793 commented Dec 23, 2025•
edited
Loading

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Parquet Java 1.17.0 Release Notes

https://github.com/apache/parquet-java/releases/tag/apache-parquet-1.17.0

Note, this is the first version that requires Java 11+ (instead of Java 8)

Why are the changes needed?

Keep Parquet up-to-date.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GHA.

Was this patch authored or co-authored using generative AI tooling?

No.

dongjoon-hyun
dongjoon-hyun reviewed Dec 23, 2025
View reviewed changes

@dongjoon-hyundongjoon-hyun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for testing this, @pan3793 . Could you make the CI happy to help Parquet 1.17.0?

@Fokko

Fokko commented Dec 23, 2025

Copy link
Copy Markdown
Contributor

Thanks for the prompt reply @dongjoon-hyun It was my mistake, retriggering the CI should solve it 👍

Fokko
Fokko approved these changes Jan 13, 2026
View reviewed changes

@FokkoFokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The release is out: https://parquet.apache.org/blog/2026/01/13/1.17.0/

Comment threadpom.xml Outdated
Comment threadproject/SparkBuild.scala Outdated
@pan3793
pan3793 marked this pull request as ready for review January 13, 2026 11:58
@pan3793

pan3793 commented Jan 13, 2026

Copy link
Copy Markdown
MemberAuthor

@Fokko thanks, I have removed the staging repo and rebase to fix the conflict

@github-actions

github-actionsBot commented Jan 13, 2026

Copy link
Copy Markdown

JIRA Issue Information

=== Improvement SPARK-54822 ===
Summary: Upgrade Parquet 1.17.0
Assignee: None
Status: Open
Affected: ["4.2.0"]

This comment was automatically generated by GitHub Actions

dongjoon-hyun
dongjoon-hyun approved these changes Jan 14, 2026
View reviewed changes

@dongjoon-hyundongjoon-hyun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Thank you, @pan3793 , @Fokko , and @LuciferYang .

Merged to master for Apache Sprak 4.2.0.

@dongjoon-hyundongjoon-hyun mentioned this pull request Feb 13, 2026
Closed
dongjoon-hyun added a commit that referenced this pull request Feb 13, 2026
@dongjoon-hyun
[SPARK-55520][TESTS] Regenerate benchmark results
a252c5b
### What changes were proposed in this pull request?
This PR aims to regenerate benchmark results to check the intermediate status as a part of Apache Spark 4.2.0 preparation.
Please note that `V2FunctionBenchmark` is excluded because it's broken due to `NumericEvalContext.evalMode()` error currently. It's good to identify this kind of bug as early as possible via this PR.
- [SPARK-55519 `V2FunctionBenchmark` is broken](https://issues.apache.org/jira/browse/SPARK-55519)
### Why are the changes needed?
Apache Spark 4.2.0 introduced many improvements on top of the key dependency differences from Spark 4.1.0:
- #53396
- #53582
- #53347
- #54233
- #54292
We updated the benchmark result 4 months ago. So, it's time to make them up-to-date with our actual code and the current infra.
- #52600
```
- OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Linux 6.11.0-1018-azure
+ OpenJDK 64-Bit Server VM 17.0.18+8-LTS on Linux 6.14.0-1017-azure
```
```
- OpenJDK 64-Bit Server VM 21.0.8+9-LTS on Linux 6.11.0-1018-azure
+ OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.14.0-1017-azure
```
### Does this PR introduce _any_ user-facing change?
No. This is a change on benchmark result files.
### How was this patch tested?
Manual review.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes#54313 from dongjoon-hyun/SPARK-55520.
Lead-authored-by: Dongjoon Hyun <dongjoon@apache.org>
Co-authored-by: dongjoon-hyun <dongjoon-hyun@users.noreply.github.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
rpnkv pushed a commit to rpnkv/spark that referenced this pull request Feb 18, 2026
@dongjoon-hyun@rpnkv
[SPARK-55520][TESTS] Regenerate benchmark results
877e7b0
### What changes were proposed in this pull request?
This PR aims to regenerate benchmark results to check the intermediate status as a part of Apache Spark 4.2.0 preparation.
Please note that `V2FunctionBenchmark` is excluded because it's broken due to `NumericEvalContext.evalMode()` error currently. It's good to identify this kind of bug as early as possible via this PR.
- [SPARK-55519 `V2FunctionBenchmark` is broken](https://issues.apache.org/jira/browse/SPARK-55519)
### Why are the changes needed?
Apache Spark 4.2.0 introduced many improvements on top of the key dependency differences from Spark 4.1.0:
- apache#53396
- apache#53582
- apache#53347
- apache#54233
- apache#54292
We updated the benchmark result 4 months ago. So, it's time to make them up-to-date with our actual code and the current infra.
- apache#52600
```
- OpenJDK 64-Bit Server VM 17.0.16+8-LTS on Linux 6.11.0-1018-azure
+ OpenJDK 64-Bit Server VM 17.0.18+8-LTS on Linux 6.14.0-1017-azure
```
```
- OpenJDK 64-Bit Server VM 21.0.8+9-LTS on Linux 6.11.0-1018-azure
+ OpenJDK 64-Bit Server VM 21.0.10+7-LTS on Linux 6.14.0-1017-azure
```
### Does this PR introduce _any_ user-facing change?
No. This is a change on benchmark result files.
### How was this patch tested?
Manual review.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closesapache#54313 from dongjoon-hyun/SPARK-55520.
Lead-authored-by: Dongjoon Hyun <dongjoon@apache.org>
Co-authored-by: dongjoon-hyun <dongjoon-hyun@users.noreply.github.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@LuciferYangLuciferYangLuciferYang approved these changes
@dongjoon-hyundongjoon-hyundongjoon-hyun approved these changes
+1 more reviewer
@FokkoFokkoFokko approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

BUILD

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@pan3793@Fokko@LuciferYang@dongjoon-hyun