GH-39214: [Java] Support reproducible build by jbonofre · Pull Request #39215 · apache/arrow

jbonofre · 2023-12-13T14:23:11Z

Rationale for this change

This PR add reproducible builds support.

Are these changes tested?

I tested the builds are reproducible on arrow-vector, arrow-memory, ... using mvn clean verify artifact:compare.

Are there any user-facing changes?

No

Closes: [Java] Provide reproducible builds arrow-java#146
GitHub Issue: ARROW-293: [C++] Implement Arrow IO interfaces for operating system files #146

github-actions · 2023-12-13T14:23:39Z

⚠️ GitHub issue apache/arrow-java#146has been automatically assigned in GitHub to PR creator.

lidavidm

Thanks!

jbonofre · 2023-12-13T20:09:07Z

@lidavidm let me check why GH Action is not happy 😃

lidavidm · 2023-12-13T20:12:49Z

@github-actions crossbow submit java

github-actions · 2023-12-13T20:15:19Z

Revision: 502eb177c8265ff30879bd41db09380d6d52abe7

Submitted crossbow builds: ursacomputing/crossbow @ actions-b2f5994f24

Task	Status
java-jars
verify-rc-source-java-linux-almalinux-8-amd64
verify-rc-source-java-linux-conda-latest-amd64
verify-rc-source-java-linux-ubuntu-20.04-amd64
verify-rc-source-java-linux-ubuntu-22.04-amd64
verify-rc-source-java-macos-amd64

davisusanibar · 2023-12-13T21:54:44Z

Can you provide some commands or steps for validating a reproducible build, such as in the Arrow Format module?

When I tried these commands, it finished ok with or without any changes:
$ mvn clean install $ mvn clean verify artifact:compare ... [INFO] Saved info on build to /Users/dsusanibar/fork/arrow/java/format/target/arrow-format-15.0.0-SNAPSHOT.buildinfo [INFO] Checking against reference build from central... [INFO] Reference buildinfo file not found: it will be generated from downloaded reference artifacts [INFO] Reference build java.version: 11 (from MANIFEST.MF Build-Jdk-Spec) [INFO] Reference build os.name: Unix (from pom.properties newline) [INFO] Minimal buildinfo generated from downloaded artifacts: /Users/dsusanibar/fork/arrow/java/format/target/reference/arrow-format-15.0.0-SNAPSHOT.buildinfo [INFO] Reproducible Build output summary: 5 files ok [INFO] Reproducible Build output comparison saved to /Users/dsusanibar/fork/arrow/java/format/target/arrow-format-15.0.0-SNAPSHOT.buildcompare ...

You are right for the check: mvn clean install and then mvn clean verify artifact:compare do a check.
This PR ensures we have reproducible build, not meaning that some module are not ok already (AFAIR, flight and algorithm were not reproducible).

You can also check with artifact:check-buildplan.

What we might want is to add this to CI somewhere (or possibly the release verification process) so we can make sure it stays working.

@lidavidm good idea. Let me check to add a workflow in GH actions.

jbonofre · 2023-12-18T09:36:30Z

FYI, I'm resuming work on this PR:

checking the build failure on CI
checking to add a GH Action workflow to verify

lidavidm · 2023-12-18T13:07:22Z

checking to add a GH Action workflow to verify

It might be best to add it to an existing workflow, or again, adding it to the release verification script even (what do you think @assignUser?)

jbonofre · 2023-12-18T13:21:21Z

checking to add a GH Action workflow to verify
It might be best to add it to an existing workflow, or again, adding it to the release verification script even (what do you think @assignUser?)

@lidavidm it makes sense, I do like this. Thanks !

assignUser · 2023-12-19T04:47:23Z

How exactly does this work, it compares the current build against a reference build? How does that work for dev versions?

We do have binary verification scripts so it would make sense to add it there, that would automatically also run it. dev/release/verify-release-candidate.sh with `TEST_JARS=1' is run by the jar binary verification script (this downloads previously build nightly jars iirc) cc @raulcd who is a bit more familiar with the release scripts

lidavidm · 2023-12-19T14:11:10Z

https://maven.apache.org/plugins/maven-artifact-plugin/compare-mojo.html

Compare current build output (from package) against reference either previously install-ed or downloaded from a remote repository: comparison results go to .buildcompare file.

So maybe we finally have a useful thing to do in the binary JAR verification step.

jbonofre · 2023-12-19T14:15:20Z

@lidavidm yup, it's what I plan to do (and I used while working on this PR 😄 ).

First I will fix the failing checks (AMD64 Conda Integration Test, etc).

jbonofre · 2023-12-19T14:16:22Z

Error: Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:3.4.1:enforce (enforce-maven-version) on project arrow-bom: Error: Rule 0: org.apache.maven.enforcer.rules.version.RequireMavenVersion failed with message:
Error: Detected Maven Version: 3.5.0 is not in the allowed range [3.6.3,).
Error: -> [Help 1]

That's due to the Apache POM update. I will propose fix as part of this PR.

lidavidm · 2023-12-19T14:20:55Z

See #39279

lidavidm · 2024-01-19T14:04:36Z

@github-actions crossbow submit java

github-actions · 2024-01-19T14:05:12Z

Only contributors can submit requests to this bot. Please ask someone from the community for help with getting the first commit in.
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/7584811545

lidavidm · 2024-01-19T15:05:44Z

Hmm well Archery doesn't like me but it seems CI still has a few issues even after a re-run.

jbonofre · 2024-01-19T15:27:42Z

@lidavidm yup, to be honest, I don't know why it's failing 😄 I will probably try on main branch to compare.
I will do a new pass again, maybe I missed something 😄

kou · 2024-01-19T22:11:08Z

@assignUser Could you take a look at #39215 (comment) and #39215 (comment) ?

#39610 is related.

assignUser · 2024-01-20T03:52:54Z

@kou hm weird asf membership is public for David so the bot should pick it up... if this keeps happening we may need to add a non gh api based method to identify committers (e.g. committer list, IIRC we have a yaml one around somehwere right?)

lidavidm · 2024-01-22T14:46:46Z

I fixed my membership after that. You may want to update the bot's error message to tell people what to do, I only knew because I happened to see someone else ask the same question in a private channel.

jbonofre · 2024-01-25T13:48:42Z

I'm rebasing and doing new tests on this one.

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

…aven-assemly-plugin

lidavidm · 2024-03-31T15:32:01Z

@github-actions crossbow submit java

github-actions · 2024-03-31T15:34:24Z

Revision: e6e5d16

Submitted crossbow builds: ursacomputing/crossbow @ actions-02510129a8

Task	Status
java-jars
verify-rc-source-java-linux-almalinux-8-amd64
verify-rc-source-java-linux-conda-latest-amd64
verify-rc-source-java-linux-ubuntu-20.04-amd64
verify-rc-source-java-linux-ubuntu-22.04-amd64
verify-rc-source-java-macos-amd64

jbonofre · 2024-03-31T16:53:48Z

@lidavidm as I'm back on Arrow, I'm resuming several PRs/works I have in progress 😄 Thanks !

vibhatha · 2024-05-31T23:57:33Z

@jbonofre need any help here?

jbonofre · 2024-06-01T09:16:58Z

I can resume on this one yeah :)

jbonofre · 2024-09-03T07:41:43Z

I'm back on Arrow. I will rebase and retest this PR.

github-actions · 2024-11-26T19:16:06Z

⚠️ GitHub issue #39214 has no components, please add labels for components.

kou · 2024-11-27T00:55:07Z

We're moving java/ to https://github.com/apache/arrow-java .
Could you open a new PR in apache/arrow-java?

jbonofre requested a review from lidavidm as a code owner December 13, 2023 14:23

github-actionsBot added Component: Java awaiting review Awaiting review labels Dec 13, 2023

lidavidm approved these changes Dec 13, 2023
View reviewed changes

github-actionsBot added awaiting merge Awaiting merge and removed awaiting review Awaiting review labels Dec 13, 2023

davisusanibar reviewed Dec 13, 2023
View reviewed changes

Comment threadjava/pom.xml Outdated

davisusanibar reviewed Dec 13, 2023
View reviewed changes

github-actionsBot added awaiting changes Awaiting changes and removed awaiting merge Awaiting merge labels Dec 14, 2023

jbonofre force-pushed the GH-39214 branch from 502eb17 to 7dfca83Compare December 15, 2023 09:34

github-actionsBot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Dec 15, 2023

jbonofre force-pushed the GH-39214 branch from 7dfca83 to ca6c208Compare December 19, 2023 14:22

jbonofre mentioned this pull request Dec 19, 2023
MINOR: [Java] Bump org.apache:apache from 18 to 31 in /java #39279
Closed

jbonofre force-pushed the GH-39214 branch 3 times, most recently from a65906a to fe22e18Compare December 19, 2023 14:57

jbonofre force-pushed the GH-39214 branch from 8414e04 to a812077Compare January 17, 2024 19:25

jbonofre force-pushed the GH-39214 branch 3 times, most recently from 35de715 to 4f4343fCompare January 30, 2024 16:27

jbonofreand others added 6 commits March 30, 2024 06:51

GH-39214: [Java] Support reproducible build
c8b30cb

Update dev/release/test-helper.rb
f0feef8
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

Update dev/release/test-helper.rb
51ce4f7
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

Use 1970-01-01 as project.build.outputTimestamp
fe05eb1

Use 1980-01-01 as project.build.outputTimestamp to avoid issue with m…
2f36124
…aven-assemly-plugin

Update outputTimestamp in test-helper
e6e5d16

jbonofre force-pushed the GH-39214 branch from 4f4343f to e6e5d16Compare March 30, 2024 05:53

assignUser mentioned this pull request Mar 31, 2024
[CI] Update crossbow message about private org visibility #40916
Closed

jbonofre mentioned this pull request May 2, 2024
[Java] java-jars fails with "We have duplicated artifacts attached" apache/arrow-java#92
Closed

kou closed this Nov 27, 2024

Uh oh!

Conversation

jbonofre commented Dec 13, 2023• edited by github-actions BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actionsBot commented Dec 13, 2023

Uh oh!

lidavidm left a comment

Choose a reason for hiding this comment

Uh oh!

jbonofre commented Dec 13, 2023

Uh oh!

lidavidm commented Dec 13, 2023

Uh oh!

github-actionsBot commented Dec 13, 2023

Uh oh!

Uh oh!

davisusanibarDec 13, 2023

Choose a reason for hiding this comment

Uh oh!

davisusanibarDec 13, 2023

Choose a reason for hiding this comment

Uh oh!

jbonofreDec 14, 2023

Choose a reason for hiding this comment

Uh oh!

jbonofreDec 14, 2023

Choose a reason for hiding this comment

Uh oh!

lidavidmDec 14, 2023

Choose a reason for hiding this comment

Uh oh!

jbonofreDec 15, 2023

Choose a reason for hiding this comment

Uh oh!

jbonofre commented Dec 18, 2023

Uh oh!

lidavidm commented Dec 18, 2023

Uh oh!

jbonofre commented Dec 18, 2023

Uh oh!

assignUser commented Dec 19, 2023

Uh oh!

lidavidm commented Dec 19, 2023

Uh oh!

jbonofre commented Dec 19, 2023

Uh oh!

jbonofre commented Dec 19, 2023

Uh oh!

lidavidm commented Dec 19, 2023

Uh oh!

lidavidm commented Jan 19, 2024

Uh oh!

github-actionsBot commented Jan 19, 2024

Uh oh!

lidavidm commented Jan 19, 2024

Uh oh!

jbonofre commented Jan 19, 2024

Uh oh!

kou commented Jan 19, 2024

Uh oh!

assignUser commented Jan 20, 2024

Uh oh!

lidavidm commented Jan 22, 2024

Uh oh!

jbonofre commented Jan 25, 2024

Uh oh!

lidavidm commented Mar 31, 2024

Uh oh!

github-actionsBot commented Mar 31, 2024

Uh oh!

jbonofre commented Mar 31, 2024

Uh oh!

vibhatha commented May 31, 2024

Uh oh!

jbonofre commented Jun 1, 2024

Uh oh!

jbonofre commented Dec 13, 2023•
edited by github-actions Bot
Loading