chore: Generate between split repos and main by lqiu96 · Pull Request #8442 · googleapis/google-cloud-java

lqiu96 · 2022-09-23T03:11:11Z

Following Tomo's script advice:

#!/bin/bash
set -e
modules=$(mvn help:evaluate -Dexpression=project.modules | grep '<.*>.*</.*>' | grep 'java' | sed -e 's/<.*>\(.*\)<\/.*>/\1/g')
for module in $modules; do
echo "Running for ${module}"
rm -rf "${module}"
git clone "https://github.com/googleapis/${module}"
rm -rf "${module}/.git"
echo "Done running for ${module}"
done
./generation/delete_non_generated_samples.sh
echo "Done running script"

For now:

Ignore pom.xml (assuming they are renovate PRs) -- Can create PR to check only pom.xmls
Ignore CHANGELOG.md (from release-please) -- Might need to copy over from split-repos
Ignore README.md -- Can copy over from split repos
Ignore split repos renovate.json configs
Ignore samples/*
git diff --name-only origin/main | grep -v "pom.xml" | grep -v "CHANGELOG.md" | grep -v "README.md" | grep -v "renovate.json" | grep -v "versions.txt" | grep -v "samples/*" > diff.txt

Probably should ignore owlbot files and .github/*

Shows only the name of the files that have changed: diff.txt

Github has issue displaying the diffs. Might need to only add the files listed in the diff.txt

suztomo · 2022-09-23T14:45:04Z

How about limiting first 1000 diff files in the pull request? (We can't see thousands of files anyway)

Once the script is ready and merged to main, we want a GitHub Actions job that updates a branch periodically. We monitor a pull request from the branch to observe the changes.

suztomo · 2022-09-23T15:25:45Z

This PR reveals Java code diff (maybe manually-maintained tests?). Great.

lqiu96 · 2022-09-23T16:18:56Z

How about limiting first 1000 diff files in the pull request? (We can't see thousands of files anyway)

I didn't want to manually limit/exclude certain files. I've taken a look at a few of the diffs locally and I think quite a few of them can be ignored.

Ignoring these files should help us limit the number of diffs we need to check:

CHANGELOG.md (from release-please) -- Might need to copy over from split-repos
README.md -- Can copy over from split repos
renovate.json configs in each split repos
Ignore samples/* -- As we are no longer maintaining them and can look into them later
Ignore owlbot.yamls

Need to check:

The .java files (either manually or from owlbot)

The pom.xml files shouldn't have custom pom.xml, but I can do a separate and specific diff (either in a new PR or a seperate branch) to check each pom.xml. What do you think about the proposed ignored files above? @suztomo

Once the script is ready and merged to main, we want a GitHub Actions job that updates a branch periodically. We monitor a pull request from the branch to observe the changes.

I can look into a GH action once I flush out the script with the diffs that I want to show.

lqiu96 · 2022-09-23T16:34:32Z

Plan forward is two branches

Java only code (from owlbot changes and samples)
Non-Java code

Script will create two branches and create PRs for each branch that show the diffs

This reverts commit 8a616dc.

suztomo · 2022-09-23T18:09:13Z

We need 1 script generation/generate_diff.sh. The two jobs that run upon updating main branch:

1st GitHub Actions job

steps:
- run: |
generation/generate_diff.sh
- run: |
# Commit only Java code.
git commit ./**/*.java -m 'chore: Java code diff from split repos'
- run: |
git push -f --set-upstream origin split-repo-diff-non-java

2nd GitHub Actions job

steps:
- run: |
generation/generate_diff.sh
- run: |
# Commit non-Java code. How??
git commit ???????? -m 'chore: non-java diff from split repos'
- run: |
git push -f --set-upstream origin split-repo-diff-non-java

suztomo · 2022-09-23T20:06:14Z

+git stash
+git checkout "${current_branch}"
+git stash pop
+
+git checkout -b "${diff_non_java_branch}"
+git add .


I like the use of git stash to separete Java change and non-Java changes.

Let me know if you have a better option. I couldn't think besides this that wouldn't require either two jobs or running the script twice.

suztomo · 2022-09-23T20:18:01Z

+ - name: Run generate-diff script
+ run: ./generation/generate_diff.sh
+ env:
+ USERNAME: ${{ github.actor }}


Did it work? If not YOSHI_CODE_BOT_TOKEN is available.
https://github.com/googleapis/google-cloud-java/blob/main/.github/workflows/googleapis_hermetic_sync.yaml#L41

Yep, #8448 and #8449 both have github-actions force-pushed...

suztomo · 2022-09-23T20:22:36Z

+ push:
+ branches:
+ - main
+ pull_request:


I assume you're testing. We don't need this job at pull request checks.

Yep! Just added in fa84ab5 so the CI only runs on non-PR push events (should be merge only)

It appears in the checks. Can you fix this?

Hmm, I'm not sure if I can prevent it from showing up on the checks. It seems to show up on a push event.
I can skip it from running all together.

Would you explain why you need this line 19 " pull_request:" ?

I can skip it from running all together.
Yes, please do so. Remove line 19.

Ah good catch! I've removed the check.

the job is appearing pull request checks

…ain-diff

lqiu96 · 2022-09-26T15:30:11Z

I'll merge this now to have it regenerate the diffs on merge. I'll create a separate PR to split the ITs and other Java code.

* chore: Generate diff from current split repos * chore: Call delete samples script afterwards * chore: Display diff on GH * chore: Push diffs to seperate branches * Revert "chore: Display diff on GH" This reverts commit 8a616dc. * chore: Create seperate branches * chore: Add push to multiple branches * chore: Stash the non staged changes * chore: Delete non-generated samples * chore: Add retry for git clone * chore: Remove untracked filed * chore: Run from main-diff branch * chore: Fix Typo * chore: cleanup branches * chore: Test if it can ci works * chore: Delete branch only if doesn't exist * chore: Set git username/email * chore: Use fetch-depth 0 to allow unshallow update * chore: Only run on merge * chore: Run action only on merge * chore: Remove run on pull_request

lqiu96 added 3 commits September 23, 2022 02:37

chore: Generate diff from current split repos
6bbe456

chore: Call delete samples script afterwards
2dc9964

chore: Display diff on GH
8a616dc

lqiu96 added 3 commits September 23, 2022 17:56

chore: Push diffs to seperate branches
8d74950

Revert "chore: Display diff on GH"
d2b37ac
This reverts commit 8a616dc.

chore: Create seperate branches
2977762

lqiu96 added 11 commits September 23, 2022 18:16

chore: Add push to multiple branches
1e47be8

chore: Stash the non staged changes
eb082d2

chore: Delete non-generated samples
aea3208

chore: Add retry for git clone
7758d31

chore: Remove untracked filed
3e559b5

chore: Run from main-diff branch
f2947db

chore: Fix Typo
76e00b8

chore: cleanup branches
0264eba

chore: Test if it can ci works
5657d28

chore: Delete branch only if doesn't exist
08ddf73

chore: Set git username/email
abb4b92

suztomo previously approved these changes Sep 23, 2022
View reviewed changes

suztomo reviewed Sep 23, 2022
View reviewed changes

chore: Use fetch-depth 0 to allow unshallow update
e2b6761

suztomo reviewed Sep 23, 2022
View reviewed changes

chore: Only run on merge
fa84ab5

suztomo reviewed Sep 23, 2022
View reviewed changes

lqiu96 marked this pull request as ready for review September 23, 2022 20:33

lqiu96 added the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 23, 2022

gcf-owl-botBot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 23, 2022

lqiu96 added 3 commits September 23, 2022 21:00

chore: Run action only on merge
7fa55ff

Merge branch 'main' of github.com:googleapis/google-cloud-java into m…
d2ee138
…ain-diff

chore: Remove run on pull_request
a14bffc

suztomo approved these changes Sep 23, 2022
View reviewed changes

lqiu96 merged commit d01ae48 into mainSep 26, 2022

lqiu96 deleted the main-diff branch September 26, 2022 15:40

Uh oh!

Conversation

lqiu96 commented Sep 23, 2022

Uh oh!

suztomo commented Sep 23, 2022

Uh oh!

suztomo commented Sep 23, 2022

Uh oh!

lqiu96 commented Sep 23, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lqiu96 commented Sep 23, 2022

Uh oh!

suztomo commented Sep 23, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1st GitHub Actions job

2nd GitHub Actions job

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lqiu96Sep 23, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lqiu96Sep 23, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lqiu96 commented Sep 26, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lqiu96 commented Sep 23, 2022•
edited
Loading

suztomo commented Sep 23, 2022•
edited
Loading

lqiu96Sep 23, 2022•
edited
Loading

lqiu96Sep 23, 2022•
edited
Loading