[SPARK-57130][BUILD] make-distribution.sh copies only git-tracked files for python

[pan3793]

What changes were proposed in this pull request?

make-distribution.sh copies only git-tracked files for python folder, when git and cpio commands are available and under a git repo, instead of raw cp.

Why are the changes needed?

I find that sometimes make-distribution.sh produces an unreasonably large tarball because it copies the entire python folder to the dist directory, which may contain generated files, e.g., compiled PySpark docs.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Run dev/make-distribution.sh manually.

Also tested the performance of the new command, on macOS, cpio is slightly slower than raw cp, but good enough.

$ time git ls-files -z "$PWD/python" | cpio -0pdm "target"
42452 blocks
git ls-files -z "$PWD/python" 0.01s user 0.01s system 76% cpu 0.027 total
cpio -0pdm "target" 0.05s user 1.10s system 77% cpu 1.480 total
$ rm -rf target/python
$ time cp -r "$PWD/python" "target" cp -r "$PWD/python" "target" 0.02s user 0.56s system 78% cpu 0.731 total

on Linux, cpio is faster

$ time git ls-files -z "$PWD/python" | cpio -0pdm "target"
46385 blocks
git ls-files -z "$PWD/python" 0.01s user 0.01s system 81% cpu 0.022 total
cpio -0pdm "target" 0.05s user 1.02s system 84% cpu 1.260 total
$ rm -rf target/python
$ time cp -r "$PWD/python" "target"
cp -r "$PWD/python" "target" 0.02s user 0.57s system 73% cpu 0.807 total

Was this patch authored or co-authored using generative AI tooling?

Generated-by: DeepSeek V4 Pro.

[sarutak]

To keep the console clean, how about redirecting the stdout/stderr of command -v to /dev/null?

[pan3793]

it follows the existing command -v pattern, e.g., in line 128 if [ $(command -v git) ]; then

[sarutak]

Got it. Let's fix all the occurrences of such pattern in the separate PR if we should do it.

[sarutak]

I'm concerned about whether cpio behaves the same way in GNU and BSD. Did you confirm this script works on macOS and Linux?

[pan3793]

my testing (also confirmed by asking LLM) shows the BSD cpio has different behavior when involving symlinks, but I think it does not affect the make-distribution.sh case

[sarutak]

I tried your change and I noticed:

• On Linux environment, cpio only copies .coveragerc to $DISTDIR/python. -0 option seems required, which should work with cpio on macOS.
• If we run make-distribution.sh on a directory other than $SPARK_HOME, files under python are't copied to $DISTDIR.

Suggested change

	git ls-files -z "$SPARK_HOME/python"| cpio -pdm"$DISTDIR"
	(cd "$SPARK_HOME"&&git ls-files -z python | cpio -0pdm"$DISTDIR")

[pan3793]

@sarutak thanks for checking, yes, -0 should be used

 Operation modifiers valid in copy-out and copy-pass modes:
-0, --null Filenames in the list are delimited by null

but I think cd is not required as it runs cd "$SPARK_HOME" in lien 169, I tested it by running the make-distribution.sh under $SPARK_HOME/common, it works fine, am I missing something?

[sarutak]

Sorry. It was a misunderstanding on my part about cd $SPARK_HOME. Adding -0 option to cpio is enough.

[dongjoon-hyun]

+1, LGTM. Thank you, @pan3793 and @sarutak .

[pan3793]

thanks for review, merged to master/4.x