[SPARK-46745][INFRA] Purge pip cache in dockerfile

[zhengruifeng]

What changes were proposed in this pull request?

Purge pip cache in dockerfile

Why are the changes needed?

to save 4~5G disk space:

before

https://github.com/zhengruifeng/spark/actions/runs/7541725028/job/20530432798

#45 [39/39] RUN df -h
#45 0.090 Filesystem Size Used Avail Use% Mounted on
#45 0.090 overlay 84G 70G 15G 83% /
#45 0.090 tmpfs 64M 0 64M 0% /dev
#45 0.090 shm 64M 0 64M 0% /dev/shm
#45 0.090 /dev/root 84G 70G 15G 83% /etc/resolv.conf
#45 0.090 tmpfs 7.9G 0 7.9G 0% /proc/acpi
#45 0.090 tmpfs 7.9G 0 7.9G 0% /sys/firmware
#45 0.090 tmpfs 7.9G 0 7.9G 0% /proc/scsi
#45 DONE 2.0s

after

https://github.com/zhengruifeng/spark/actions/runs/7549204209/job/20552796796

#48 [42/43] RUN python3.12 -m pip cache purge
#48 0.670 Files removed: 392
#48 DONE 0.7s
#49 [43/43] RUN df -h
#49 0.075 Filesystem Size Used Avail Use% Mounted on
#49 0.075 overlay 84G 65G 19G 79% /
#49 0.075 tmpfs 64M 0 64M 0% /dev
#49 0.075 shm 64M 0 64M 0% /dev/shm
#49 0.075 /dev/root 84G 65G 19G 79% /etc/resolv.conf
#49 0.075 tmpfs 7.9G 0 7.9G 0% /proc/acpi
#49 0.075 tmpfs 7.9G 0 7.9G 0% /sys/firmware
#49 0.075 tmpfs 7.9G 0 7.9G 0% /proc/scsi

Does this PR introduce any user-facing change?

no, infra-only

How was this patch tested?

ci

Was this patch authored or co-authored using generative AI tooling?

no

[zhengruifeng]

here doesn't make much sense any more, since we install those libraries for each python version

[dongjoon-hyun]

Yes, this will reduce the final status of filesystems.

At the same time, this increases the number of layers, doesn't it?

I'm curious about the download size, @zhengruifeng . Could you check the result of

$ docker images

[zhengruifeng]

Yes, this will reduce the final status of filesystems.

At the same time, this increases the number of layers, doesn't it?

I'm curious about the download size, @zhengruifeng . Could you check the result of

$ docker images

The size of current PR is 11.1GB

ruifeng.zheng@xxx:~/spark$ docker images | grep cleanup
test_cleanup 0.1 97b0f1ca0bb6 43 seconds ago 11.1GB
ruifeng.zheng@xxx:~/spark$

probably we can combine those pip commands to control the number of layers

[dongjoon-hyun]

+1, LGTM. Yes, we can do that later separately.

[dongjoon-hyun]

Merged to master. Thank you, @zhengruifeng and @HyukjinKwon .

[zhengruifeng]

thanks @dongjoon-hyun and @HyukjinKwon for reviews!