HADOOP-19976 Support Apache Repeatable Builds - #8735
Draft
steveloughran wants to merge 21 commits into
Draft
steveloughran wants to merge 21 commits into
steveloughran wants to merge 21 commits into
Conversation
As tested against x86 machine
copy hadoop-yarn/bin/FederationStateStore/MySQL/*.sql into the right place for mvn test builds to *always* include the sql files; otherwise they only end in the jar on the second run (i.e. a dirty build)
- from test jars - from source distros
New dev-support/bin/hadoop-maketar writes a gzipped tar with normalised entry metadata when project.build.outputTimestamp is set, and exactly as before when it is not: --sort=name --mtime=<ts> --owner=0 --group=0 --numeric-owner --format=gnu --mode=go-w, piped to gzip -n --format=gnu avoids the pax atime/ctime headers, and --mode drops the group and other write bits so the archive does not depend on the umask it was built under. Needs GNU tar; bsdtar warns and falls back. Called from the five places that rolled a tarball by hand, which also unifies the three copies of dist-maketar.sh: dist-tar-stitching, the antrun blocks in hadoop-project-dist, hadoop-yarn-project and hadoop-mapreduce-project, and both create-release tar calls. Those antrun blocks are inherited, so they locate the helper through maven.multiModuleProjectDirectory rather than basedir. create-release gains BUILD_TIMESTAMP, left unset here; the command line option that sets it follows. Also fixes what this exposed. hadoop-project-dist has no dist layout directory, so its tar had nothing to archive, but "tar cf - X | gzip > X.tar.gz" returns gzip's exit status, so the error was discarded and every -Dtar build wrote a 45 byte empty archive and reported success. The generated script now skips a module with no layout directory, and pipefail makes real failures fail. BUILDING.txt: jar content does not vary with the architecture, so that precondition covers the binary distribution rather than the jars. AI-Authored-by: claude.ai
More on what is needed for reproduction
claude added it as a param, but also set up repeatable build s.t. it will extract from pom using sed if set to "-", while documenting that "-" meant it was a no-op. this commit uses xmlstarlet, but I'm about to change so that - changes it to be driven by the release profile, rather than parse it into the bash script.
- adding a parent pom broke the sed to pick up hadoop version; moved to xmlstarlet. - checks for read failure and fail meaningfully - use the same tool to replace later grep/sed calls - document the create-release script for others. - asfrelease profile sets --repeatable option
- avoids needing to install maven in docker setup - guarantees identical maven version everywhere
...i.e. don't overwrite it
This reverts commit fdce12d.
+ building.txt discusses mvncache more
it may contain all the atr stuff, but it breaks javadoc
Fails fast if missing, *does not downgrade*. The other gpg checks just turn off signing, not this. Questionable whether the old policy is right (I disagree with it)
steveloughran
marked this pull request as draft
September 9, 2026 22:14
|
(!) A patch to the testing environment has been detected. |
Contributor
Author
|
This is still a WiP; some remaining steps of repeatability to be done... |
Site builds become repeatable
|
(!) A patch to the testing environment has been detected. |
- move to modern site.xml schema - fix to https urls - reproducibility: resource filtering uses utf-8 so works on all locales - maven build warning eliminated
|
(!) A patch to the testing environment has been detected. |
Yetus adds timestamps to things. Address through faketime.
|
(!) A patch to the testing environment has been detected. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
How was this patch tested?
Enhance the build/release process to work under the apache trusted release mechanism.
Goal: add ability to create 100% repeatable builds with binaries identical when built on different systems
docker wiring up with --repeatable and --build-timestamp options
For code changes:
declared according to the connector-specific documentation? Note: Automated CI
testing doesn't cover all cases so manual testing with cloud storage is still
required.
LICENSE,LICENSE-binary,NOTICE-binaryfiles?AI Tooling
If an AI tool was used:
where is the name of the AI tool used.
https://www.apache.org/legal/generative-tooling.html