[SPARK-58439][BUILD][INFRA] Add dedicated validation script for distributions

[nchammas]

What changes were proposed in this pull request?

Add a new Python script, dev/validate-distribution, that runs at the end of dev/make-distribution.sh and checks several basic things about the distribution.

The checks that run depend on the flags that are passed to make-distribution.sh. If a Python package is built, Python-specific tests are run; if an R package is built, R-specific tests are run; etc. I used the same command line flags in the validation script as in the build script.

Why are the changes needed?

This extends the work done in #56453 and checks additional, basic things that would likely fail a release if done incorrectly somehow. Since the script is in Python, it's also a bit easier to maintain.

This will also make it easier to add more checks in the future, should they become relevant. I would add smoke tests for the new CLI, for example, if #57313 is accepted.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Here's what the output looks like when the distribution is fine:

$ ./dev/validate-distribution dist/
Testing distribution at: .../spark/dist
Checking distribution structure... OK
Checking bin/spark-submit --version... OK
Checking bin/spark-shell --version... OK
Checking bin/pyspark --version... OK
Checking bin/spark-sql --version... OK
Checking sbin/start-thriftserver.sh --help... OK
Checking sbin/start-connect-server.sh --help... OK
All distribution validation checks passed.

And here's what it looks like when the distribution has some problems:

$ ./dev/validate-distribution dist/ --pip /tmp/fake-pip
Testing distribution at: .../spark/dist
Checking distribution structure... FAILED
MISSING: LICENSE
MISSING: RELEASE
Checking bin/spark-submit --version... OK
Checking bin/spark-shell --version... OK
Checking bin/pyspark --version... FAILED
Checking bin/spark-sql --version... OK
Checking sbin/start-thriftserver.sh --help... OK
Checking sbin/start-connect-server.sh --help... OK
Checking pyspark-fake.tar.gz for LICENSE and NOTICE... FAILED
pyspark-fake.tar.gz: missing LICENSE at the package root
pyspark-fake.tar.gz: missing NOTICE at the package root
5 check(s) failed:
MISSING: LICENSE
MISSING: RELEASE
bin/pyspark --version: [Errno 13] Permission denied: '.../spark/dist/bin/pyspark'
pyspark-fake.tar.gz: missing LICENSE at the package root
pyspark-fake.tar.gz: missing NOTICE at the package root

Was this patch authored or co-authored using generative AI tooling?

I wrote this with assistance from GitHub Copilot.

[nchammas]

@huaxingao - Helpful or overkill?

[HyukjinKwon]

0 blocking, 0 non-blocking, 0 nits.
Useful, well-scoped build-validation tooling; robust and correctly wired.

Verification

Confirmed make-distribution.sh assembles the command as a bash array conditional on MAKE_PIP/MAKE_R/MAKE_TGZ/MAKE_SPARK_CONNECT (validates only what was built; array form avoids quoting bugs). The script accumulates errors and sys.exit(1)s so a validation failure fails the build, and the smoke-test subprocess calls use a timeout with TimeoutExpired handling (the one reviewer concern, already fixed).

[pan3793]

I personally feel this is overkill ... leave it to others to make a decision.

not a big deal, but this seems to add a hard dependency on Python to the make-distribution.sh script even without --pip.

[nchammas]

@pan3793 - I added a more complete smoke test that catches the kind of problem documented in #58428. ;)