[SPARK-57771][INFRA] Use Python 3.11 in spark-rm release image

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR uses Python 3.11 instead of 3.10 in dev/create-release/spark-rm/Dockerfile for Apache Spark 4.3.0+.

Why are the changes needed?

Since Python 3.10 reaches end-of-life in October 2026 before Apache Spark 4.3.0 release, Apache Spark 3.4.0 dropped it.

• [SPARK-56889][PYTHON][INFRA] Drop Python 3.10 Support #55914

So, ideally, we need to use Python 3.11+ in the release script. However, recently, on 2026-01-23, we share Dockerfile.base for all maintenance releases and ubuntu:jammy has only Python 3.11.0rc1. Since Python 3.11.0rc1 is better than the EOL Python 3.10, this PR upgrades it only for Spark release.

• [SPARK-55115][INFRA] Use composable Dockerfile for release builds #53905

spark/dev/create-release/spark-rm/Dockerfile.base

Line 30 in e28edaf

FROM ubuntu:jammy-20250819

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Manual review.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 4.8

[dongjoon-hyun]

Hi, @cloud-fan , @HyukjinKwon , @zhengruifeng .

Ideally, we need to upgrade to the latest Ubuntu OS in order to use Python 3.11, but it will affect all maintenance releases because we need to change Dockerfile.base. I'm not sure we can take that risk. Especially, we are currently doing Apache Spark 4.2.0 RCs. So, I proposed this intermediate alternative in this PR. Otherwise, we need to do a partial revert of SPARK-55115 for branch-4.x/master.

WDYT?

[cloud-fan]

I'm +1, but what's the test plan? @huaxingao will cut the next RC with python 3.11?

[dongjoon-hyun]

Thank you, @zhengruifeng and @cloud-fan .

I'm targeting Apache Spark 4.3.0 and later. In other words, Apache Spark 4.3.0-preview1 is the first target to test this release change.

I'm +1, but what's the test plan? @huaxingao will cut the next RC with python 3.11?

Apache Spark 4.2.0 and older will use the unchanged Dockerfile.base and their branches' Dockerfiles.

[dongjoon-hyun]

Merged to master/4.x for Apache Spark 4.3.0-preview1.