Skip to content

[SPARK-38262][BUILD] Upgrade Google guava to version 30.0-jre - #35584

Closed
bjornjorgensen wants to merge 1 commit into
apache:masterfrom
bjornjorgensen:Google-guava-version-30.0-jre
Closed

[SPARK-38262][BUILD] Upgrade Google guava to version 30.0-jre#35584
bjornjorgensen wants to merge 1 commit into
apache:masterfrom
bjornjorgensen:Google-guava-version-30.0-jre

Conversation

@bjornjorgensen

@bjornjorgensenbjornjorgensen commented Feb 20, 2022•
edited
Loading

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Bump Google guava from version 14.0.1 to 30.0-jre
Release notes for Google guava 30.0

Why are the changes needed?

Spark is using com.google.guava:guava version 14.0.1 which has two security issues.

CVE-2018-10237

CVE-2020-8908

Does this PR introduce any user-facing change?

NO

How was this patch tested?

All existing tests must pass.

@AmplabJenkins

AmplabJenkins commented Feb 21, 2022

Copy link
Copy Markdown

Can one of the admins verify this patch?

medb
medb reviewed Feb 21, 2022
View reviewed changes
Comment threaddev/deps/spark-deps-hadoop-2-hive-2.3
gmetric4j/1.0.10//gmetric4j-1.0.10.jar
gson/2.2.4//gson-2.2.4.jar
guava/14.0.1//guava-14.0.1.jar
guava/30.0-jre//guava-30.0-jre.jar

@medbmedbFeb 21, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason to not use the latest 31.0.1-jre version?

@bjornjorgensenbjornjorgensenFeb 21, 2022

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this PR is just to get rid of two CVE. I do not want to destroy existing code base. If you are a developer who needs and uses some of what is in newer code, then I recommend that you implement this yourself when you need it.

@LuciferYang

LuciferYang commented Feb 21, 2022

Copy link
Copy Markdown
Contributor

In fact, many people have tried similar jobs, such as SPARK-36676, but due to the cascading dependence of third-party libraries on Guava 14.0.1 (such as hive-2.3.9 and hadoop-2.7.4), in order to ensure the compatibility, it is not possible to upgrade the Guava version directly like this pr.

@bjornjorgensen

bjornjorgensen commented Feb 21, 2022

Copy link
Copy Markdown
ContributorAuthor

@LuciferYang Thank you :) I see that none of the other PRs that can be found have come up with the fact that today there are two CVEs for Google guava. What's a little weird about this is that I chose version 30.0-jre which is the first one without any known security holes. All the tests have passed. The difference is that for example 29326 or 33989 the tests fail.

@LuciferYang

LuciferYang commented Feb 21, 2022

Copy link
Copy Markdown
Contributor

@bjornjorgensen Can you test with hadoop-2 profile manually? GA just test with hadoop-3 profile now.

@LuciferYang

LuciferYang commented Feb 21, 2022•
edited
Loading

Copy link
Copy Markdown
Contributor

also ping @sunchao

@bjornjorgensen

bjornjorgensen commented Feb 21, 2022

Copy link
Copy Markdown
ContributorAuthor

@LuciferYang Oh yes, that might explain why my tests get green. But if we look at the problem differently. We actually use hadoop-common-2.7.4.jar
Now hadoop 2.10.1 This edition has fixed another CSV also CVE-2018- 8009 so the question now is whether the solution to the problem might be to update hadoop to version 2.10.1? And remove the link to google guava in spark?

@sunchao

sunchao commented Feb 21, 2022

Copy link
Copy Markdown
Member

Not only Hadoop, but Hive 2.3.x also has dependency on Guava 14.0.1. For Hadoop we can at least change Guava version accordingly but there's no workaround for Hive.

@bjornjorgensen

bjornjorgensen commented Feb 21, 2022•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

I have updated the JIRA for this issue SPARK-38262

I will close this PR now.

Thanks to @medb@LuciferYang and @sunchao for the help on this one :)

@JoshRosenJoshRosen mentioned this pull request Aug 17, 2022
Closed
@pan3793pan3793 mentioned this pull request Aug 15, 2023
Closed
dongjoon-hyun pushed a commit that referenced this pull request Sep 12, 2024
@pan3793@dongjoon-hyun
[SPARK-44811][BUILD] Upgrade Guava to 33.2.1-jre
1f24b2d
### What changes were proposed in this pull request?
This PR upgrades Spark's built-in Guava from 14 to 33.2.1-jre
Currently, Spark uses Guava 14 because the previous built-in Hive 2.3.9 is incompatible with new Guava versions. HIVE-27560 (apache/hive#4542) makes Hive 2.3.10 compatible with Guava 14+ (thanks to LuciferYang)
### Why are the changes needed?
It's a long-standing issue, see prior discussions at #35584, #36231, and #33989
### Does this PR introduce _any_ user-facing change?
Yes, some user-faced error messages changed.
### How was this patch tested?
GA passed.
Closes#42493 from pan3793/guava.
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
prabhjyotsingh pushed a commit to acceldata-io/spark3 that referenced this pull request Feb 8, 2025
@pan3793@prabhjyotsingh
ODP-3257: [SPARK-44811][BUILD] Upgrade Guava to 33.2.1-jre
e5cc252
This PR upgrades Spark's built-in Guava from 14 to 33.2.1-jre
Currently, Spark uses Guava 14 because the previous built-in Hive 2.3.9 is incompatible with new Guava versions. HIVE-27560 (apache/hive#4542) makes Hive 2.3.10 compatible with Guava 14+ (thanks to LuciferYang)
It's a long-standing issue, see prior discussions at apache#35584, apache#36231, and apache#33989
Yes, some user-faced error messages changed.
GA passed.
Closesapache#42493 from pan3793/guava.
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit 1f24b2d)
prabhjyotsingh pushed a commit to acceldata-io/spark3 that referenced this pull request Feb 8, 2025
@pan3793@prabhjyotsingh
ODP-3257: [SPARK-44811][BUILD] Upgrade Guava to 33.2.1-jre
4ab5d7e
This PR upgrades Spark's built-in Guava from 14 to 33.2.1-jre
Currently, Spark uses Guava 14 because the previous built-in Hive 2.3.9 is incompatible with new Guava versions. HIVE-27560 (apache/hive#4542) makes Hive 2.3.10 compatible with Guava 14+ (thanks to LuciferYang)
It's a long-standing issue, see prior discussions at apache#35584, apache#36231, and apache#33989
Yes, some user-faced error messages changed.
GA passed.
Closesapache#42493 from pan3793/guava.
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit 1f24b2d)
prabhjyotsingh pushed a commit to acceldata-io/spark3 that referenced this pull request Feb 8, 2025
@pan3793@prabhjyotsingh
ODP-3257: [SPARK-44811][BUILD] Upgrade Guava to 33.2.1-jre
e8d55d3
This PR upgrades Spark's built-in Guava from 14 to 33.2.1-jre
Currently, Spark uses Guava 14 because the previous built-in Hive 2.3.9 is incompatible with new Guava versions. HIVE-27560 (apache/hive#4542) makes Hive 2.3.10 compatible with Guava 14+ (thanks to LuciferYang)
It's a long-standing issue, see prior discussions at apache#35584, apache#36231, and apache#33989
Yes, some user-faced error messages changed.
GA passed.
Closesapache#42493 from pan3793/guava.
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit 1f24b2d)
(cherry picked from commit e5cc252)
shubhluck pushed a commit to acceldata-io/spark3 that referenced this pull request May 16, 2025
@pan3793@shubhluck
ODP-3257: [SPARK-44811][BUILD] Upgrade Guava to 33.2.1-jre
c8e0c63
This PR upgrades Spark's built-in Guava from 14 to 33.2.1-jre
Currently, Spark uses Guava 14 because the previous built-in Hive 2.3.9 is incompatible with new Guava versions. HIVE-27560 (apache/hive#4542) makes Hive 2.3.10 compatible with Guava 14+ (thanks to LuciferYang)
It's a long-standing issue, see prior discussions at apache#35584, apache#36231, and apache#33989
Yes, some user-faced error messages changed.
GA passed.
Closesapache#42493 from pan3793/guava.
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit 1f24b2d)
senthh pushed a commit to acceldata-io/spark3 that referenced this pull request May 26, 2025
@pan3793@senthh
ODP-3257: [SPARK-44811][BUILD] Upgrade Guava to 33.2.1-jre
4f83f0b
This PR upgrades Spark's built-in Guava from 14 to 33.2.1-jre
Currently, Spark uses Guava 14 because the previous built-in Hive 2.3.9 is incompatible with new Guava versions. HIVE-27560 (apache/hive#4542) makes Hive 2.3.10 compatible with Guava 14+ (thanks to LuciferYang)
It's a long-standing issue, see prior discussions at apache#35584, apache#36231, and apache#33989
Yes, some user-faced error messages changed.
GA passed.
Closesapache#42493 from pan3793/guava.
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit 1f24b2d)
shubhluck pushed a commit to acceldata-io/spark3 that referenced this pull request Sep 3, 2025
@pan3793@shubhluck
ODP-3257: [SPARK-44811][BUILD] Upgrade Guava to 33.2.1-jre
9cae2fe
This PR upgrades Spark's built-in Guava from 14 to 33.2.1-jre
Currently, Spark uses Guava 14 because the previous built-in Hive 2.3.9 is incompatible with new Guava versions. HIVE-27560 (apache/hive#4542) makes Hive 2.3.10 compatible with Guava 14+ (thanks to LuciferYang)
It's a long-standing issue, see prior discussions at apache#35584, apache#36231, and apache#33989
Yes, some user-faced error messages changed.
GA passed.
Closesapache#42493 from pan3793/guava.
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit 1f24b2d)
senthh pushed a commit to acceldata-io/spark3 that referenced this pull request Mar 5, 2026
@pan3793@senthh
OSV-12358|ODP-3257: [SPARK-44811][BUILD] Upgrade Guava to 33.2.1-jre
5c20ef7
This PR upgrades Spark's built-in Guava from 14 to 33.2.1-jre
Currently, Spark uses Guava 14 because the previous built-in Hive 2.3.9 is incompatible with new Guava versions. HIVE-27560 (apache/hive#4542) makes Hive 2.3.10 compatible with Guava 14+ (thanks to LuciferYang)
It's a long-standing issue, see prior discussions at apache#35584, apache#36231, and apache#33989
Yes, some user-faced error messages changed.
GA passed.
Closesapache#42493 from pan3793/guava.
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit 1f24b2d)
basapuram-kumar pushed a commit to acceldata-io/spark3 that referenced this pull request Mar 5, 2026
@senthh@dilrajsourabh
@Sourabh-Dilraj@pan3793@LuciferYang@prabhjyotsingh
OSV-12358 - CVE - Increase guava version to 2.0.2 (#120)
247cede
* #[Automated] Changed version from '3.2.3.5-2' to '3.2.3.6-2'
* Point repositories and distributionManagement to odp-staging-central/staging-release
* ODP-6148: Replace install with package in dev/make-distribution.sh file (#105)
Co-authored-by: dilrajsourabh <dilrajsourabh@gmail.com>
* OSV-12333 - CVE - Increasing hudi version to fix CVE-2020-36183
* OSV-12360 - CVE - Excluding avro from iceberg which pulles older version of avro to fix CVE-2023-39410 (#110)
* OSV-12291 - CVE - Increasing avro version to 1.11.5 to fix CVE-2024-47561 (#111)
* OSV-12299 - CVE - Increasing netty version to 4.1.130.Final to fix CVE-2025-55163 (#112)
* OSV-12299 - CVE - Increasing protobuf version to 3.25.5 to fix CVE-2024-7254
* OSV-12299 - CVE - Increasing gcs-connector version to hadoop3-2.2.26 to fix CVE-2024-7254
* OSV-12373 - CVE - Explicit mention of jettison to fix CVE-2022-45693 (#115)
* OSV-12373 - CVE - Increasing gcs-connector version to hadoop3-2.2.26 to fix CVE-2024-7254
* OSV-12373 - CVE - Explicit mention of jettison to fix CVE-2022-45693
* OSV-12358|ODP-3257: [SPARK-44811][BUILD] Upgrade Guava to 33.2.1-jre
This PR upgrades Spark's built-in Guava from 14 to 33.2.1-jre
Currently, Spark uses Guava 14 because the previous built-in Hive 2.3.9 is incompatible with new Guava versions. HIVE-27560 (apache/hive#4542) makes Hive 2.3.10 compatible with Guava 14+ (thanks to LuciferYang)
It's a long-standing issue, see prior discussions at apache#35584, apache#36231, and apache#33989
Yes, some user-faced error messages changed.
GA passed.
Closesapache#42493 from pan3793/guava.
Authored-by: Cheng Pan <chengpan@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit 1f24b2d)
* OSV-12358|ODP-3257: [SPARK-50468][BUILD] Upgrade Guava to 33.3.1-jre
This pr aims to upgrade guava from 33.2.1-jre to 33.3.1-jre
The full release notes as follows:
- https://github.com/google/guava/releases/tag/v33.3.0
- https://github.com/google/guava/releases/tag/v33.3.1
No
Pass GitHub Actions
No
Closesapache#48976 from LuciferYang/guava-33.3.1-jre.
Lead-authored-by: yangjie01 <yangjie01@baidu.com>
Co-authored-by: YangJie <yangjie01@baidu.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit 6c84f15)
* OSV-12358|ODP-3257: [SPARK-50972][BUILD] Upgrade Guava to 33.4.0
This pr aims to upgrade guava from 33.3.1-jre to 33.4.0-jire.
The full release notes as follows:
- https://github.com/google/guava/releases/tag/v33.4.0
No
Pass GitHub Actions
No
Closesapache#49639 from LuciferYang/guava-33.4.0-jre.
Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit a8a6078)
* ODP-3257: Fix CVE-2023-2976 com.google.guava:guava in spark (docker-integration-test) - v2
(cherry picked from commit 4221e22)
---------
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Co-authored-by: dilrajsourabh <dilrajsourabh@gmail.com>
Co-authored-by: Sourabh-Dilraj <141140754+Sourabh-Dilraj@users.noreply.github.com>
Co-authored-by: Cheng Pan <chengpan@apache.org>
Co-authored-by: yangjie01 <yangjie01@baidu.com>
Co-authored-by: Prabhjyot Singh <prabhjyot@acceldata.io>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@medbmedbmedb left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

BUILD

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@bjornjorgensen@AmplabJenkins@LuciferYang@sunchao@medb