Skip to content

[SPARK-50425][BUILD] Bump Apache Parquet to 1.15.0 - #48970

Closed
Fokko wants to merge 1 commit into
apache:masterfrom
Fokko:fd-parquet-1-15-0
Closed

[SPARK-50425][BUILD] Bump Apache Parquet to 1.15.0#48970
Fokko wants to merge 1 commit into
apache:masterfrom
Fokko:fd-parquet-1-15-0

Conversation

@Fokko

@FokkoFokko commented Nov 26, 2024

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Bumps to the latest version of Parquet.

For the full list of changes, please check the pre-release:

https://github.com/apache/parquet-java/releases/tag/apache-parquet-1.15.0

Including some interesting patches for Spark, such as apache/parquet-java#3030

Why are the changes needed?

To bring the latest features and bug fixes for Apache Spark 4.0.0.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the CIs.

Was this patch authored or co-authored using generative AI tooling?

No.

@Fokko

Copy link
Copy Markdown
ContributorAuthor

Some issue with the Jackson serialization:

Cause: java.lang.RuntimeException: shaded.parquet.com.fasterxml.jackson.databind.exc.InvalidDefinitionException: No serializer found for class org.apache.parquet.schema.LogicalTypeAnnotation$ListLogicalTypeAnnotation and no properties discovered to create BeanSerializer (to avoid exception, disable SerializationFeature.FAIL_ON_EMPTY_BEANS) (through reference chain: org.apache.parquet.hadoop.metadata.ParquetMetadata["fileMetaData"]->org.apache.parquet.hadoop.metadata.FileMetaData["schema"]->org.apache.parquet.schema.MessageType["fields"]->java.util.ArrayList[0]->org.apache.parquet.schema.GroupType["logicalTypeAnnotation"])

Digging deeper

Comment threadproject/SparkBuild.scala Outdated
Comment threadpom.xml Outdated

@FokkoFokko left a comment

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bumping to RC2

@dongjoon-hyundongjoon-hyun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @Fokko . Is there any updates after RC2?

@dongjoon-hyun
dongjoon-hyun marked this pull request as ready for review December 3, 2024 16:06
jackson-databind/2.18.2//jackson-databind-2.18.2.jar
jackson-dataformat-cbor/2.18.2//jackson-dataformat-cbor-2.18.2.jar
jackson-dataformat-yaml/2.18.2//jackson-dataformat-yaml-2.18.2.jar
jackson-datatype-jdk8/2.17.0//jackson-datatype-jdk8-2.17.0.jar

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A nice clean-up.

@dongjoon-hyundongjoon-hyun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Thank you so much for leading Apache Parquet community and this activity, @Fokko .

@dongjoon-hyun

Copy link
Copy Markdown
Member

Merged to master for Apache Spark 4.0.0 on February 2025.

@Fokko
Fokko deleted the fd-parquet-1-15-0 branch December 3, 2024 19:05
@Fokko

Fokko commented Dec 3, 2024

Copy link
Copy Markdown
ContributorAuthor

@dongjoon-hyun thanks for the prompt follow up 🙌

wangyum pushed a commit to wangyum/spark that referenced this pull request Apr 7, 2025
Bumps to the latest version of Parquet.
For the full list of changes, please check the pre-release:
https://github.com/apache/parquet-java/releases/tag/apache-parquet-1.15.0
Including some interesting patches for Spark, such as apache/parquet-java#3030
To bring the latest features and bug fixes for Apache Spark 4.0.0.
No.
Pass the CIs.
No.
Closesapache#48970 from Fokko/fd-parquet-1-15-0.
Authored-by: Fokko <fokko@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@Fokko@dongjoon-hyun