[SPARK-45651][Build] Log memory usage of publish snapshot workflow

[EnricoMi]

What changes were proposed in this pull request?

This logs memory consumption while publishing snapshots. This is to investigate whether the suspected high memory usage is the root cause of publish_snapshots failures for master.

Merging this after #43512 allows to run this manually.

Why are the changes needed?

The working assumption is that high memory usage is the root cause. This logging should provide proof or disproof for this assumption. This can be reverted once more is known or SPARK-45651 is fixed.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Locally

Was this patch authored or co-authored using generative AI tooling?

No

[EnricoMi]

CC @LuciferYang@HyukjinKwon

[HyukjinKwon]

If this is just a debug purpose, I would recommend doing this within your fork first ...

[EnricoMi]

I ran this logging in my fork, but cannot run the publish build there, of course: https://github.com/EnricoMi/spark/actions/runs/6638012356/job/18033491439#step:6:434

[HyukjinKwon]

Okey, let's get this in and see

[HyukjinKwon]

Merged to master.

[EnricoMi]

Please trigger the publish_snapshot workflow for the master branch manually.

[LuciferYang]

Please trigger the publish_snapshot workflow for the master branch manually.

https://github.com/apache/spark/actions/runs/6651054633/job/18072315690

[EnricoMi]

The last log entry is

2023-10-26T08:23:55.5712846Z mem: Thu Oct 26 08:23:07 UTC 2023
2023-10-26T08:23:55.5713956Z mem: top - 08:23:35 up 34 min, 0 users, load average: 10.56, 7.82, 6.44
2023-10-26T08:23:55.5714956Z mem: Tasks: 129 total, 4 running, 125 sleeping, 0 stopped, 0 zombie
2023-10-26T08:23:55.5715903Z mem: %Cpu(s): 0.5 us, 23.3 sy, 0.0 ni, 9.7 id, 65.6 wa, 0.0 hi, 1.0 si, 0.0 st
2023-10-26T08:23:55.5716952Z mem: MiB Mem : 6922.0 total, 103.6 free, 6692.7 used, 125.7 buff/cache
2023-10-26T08:23:55.5718023Z mem: MiB Swap: 4096.0 total, 0.0 free, 4096.0 used. 20.2 avail Mem 2023-10-26T08:23:55.5718710Z mem: 2023-10-26T08:23:55.5719344Z mem: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2023-10-26T08:23:55.5720469Z mem: 49 root 20 0 0 0 0 R 14.3 0.0 1:30.13 kswapd0
2023-10-26T08:23:55.5721397Z mem: 1 root 20 0 167724 4352 3072 R 2.0 0.1 0:05.46 systemd
2023-10-26T08:23:55.5722332Z mem: 572 root 20 0 261.6g 37392 8960 S 1.3 0.5 0:10.11 provisi+
2023-10-26T08:23:55.5723285Z mem: 585 root 20 0 1421916 13860 1408 S 1.2 0.2 0:05.43 contain+
2023-10-26T08:23:55.5724228Z mem: 863 root 20 0 365712 13580 3200 D 1.2 0.2 0:08.26 python3
2023-10-26T08:23:55.5725115Z mem: 1643 runner 20 0 3839788 39500 3036 S 0.8 0.6 0:24.63 Runner.+
2023-10-26T08:23:55.5726094Z mem: 821 root 20 0 1454428 20208 3712 S 0.7 0.3 0:03.47 dockerd
2023-10-26T08:23:55.5737404Z mem: 2462 runner 20 0 21.0g 6.0g 3132 S 0.7 89.3 33:02.14 java
2023-10-26T08:23:55.5738711Z mem: 1620 root 20 0 1297076 16768 1664 S 0.6 0.2 1:12.63 provjobd

The java build process consumes 89% of the available memory (6g of 7g). Even 4g of swap is being used by that point. Reducing the memory footprint of the build process should be the right strategy. Additionally, the available swap space could also be increased.

[LuciferYang]

spark/.github/workflows/build_and_test.yml

Line 825 in 7d7afb0

export MAVEN_OPTS="-Xss64m -Xmx2g -XX:ReservedCodeCacheSize=1g -Dorg.slf4j.simpleLogger.defaultLogLevel=WARN"

java-other-versions task use -Xss64m -Xmx2g -XX:ReservedCodeCacheSize=1g.

Maybe we could try adding an env to configure Java options, suppose it's called GA_DEPLOY_MAVEN_OPTS.

When publish_snapshot.yml sets this environment variable, run export MAVEN_OPTS = "$GA_DEPLOY_MAVEN_OPTS" before mvn deploy in the script.

[EnricoMi]

The release-build.sh script defines

export MAVEN_OPTS="-Xss128m -Xmx12g -XX:ReservedCodeCacheSize=1g"

So it could simply allow to override the -Xmx value as

export MAVEN_OPTS="-Xss128m -Xmx${MAVEN_XMX_OPT:-12g} -XX:ReservedCodeCacheSize=1g"

then, publish_snapshot.yml can define

 - name: Publish snapshotenv:
...MAVEN_XMX_OPT: "2g"run: | ./dev/create-release/release-build.sh publish-snapshot

The release-build.sh script should not care about the context it is running in (e.g. Github actions), so it shouldn't define env variables that are context-specific (e.g. GA_DEPLOY_MAVEN_OPTS), but purpose specific (e.g. MAVEN_XMX_OPT).

[EnricoMi]

See #43538.

[EnricoMi]

java-other-versions task use ... -Xmx2g ...

This explains why -Xmx3g in #43364 did not suffice.

[LuciferYang]

java-other-versions task use ... -Xmx2g ...

This explains why -Xmx3g in #43364 did not suffice.

I still underestimated the memory usage.

[EnricoMi]

And, since release-build.sh sets MAVEN_OPTS, that setting of 3g in build/mvn was ignored / overridden in publish_snapshot.yml anyway.

[HyukjinKwon]

i reverted this for now.