Skip to content

fix(ci): cut the release branch from current main, not the stale run SHA - #61

Merged
sbs44 merged 2 commits into
mainfrom
fix/release-branch-stale-workflow-race
Sep 3, 2026
Merged

fix(ci): cut the release branch from current main, not the stale run SHA#61
sbs44 merged 2 commits into
mainfrom
fix/release-branch-stale-workflow-race

Conversation

@claude

@claudeclaudeBot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Fixes the nightly failure on vscode 1.136.0.

Refs #59

This was not a build failure

Run 33795347082 built both targets greenBuild BradfordCode (macOS arm64) and Build REH server (linux x64) both succeeded. Patches, npm ci and the compile are all fine on 1.136.0; #60 already landed those fixes. The only failing job was Publish release + update versions pin:

! [remote rejected] release/1.136.05923 -> release/1.136.05923 (refusing to allow a
GitHub App to create or update workflow `.github/workflows/claude-build-fix.yml`
without `workflows` permission)

Root cause: the release branch is cut from a stale SHA

The release commit touches only upstream/stable.json and versions/stable/darwin-arm64/latest.json (1 file changed in the log) — it never touches claude-build-fix.yml. The error names a file the push did not modify because GitHub compares the pushed branch's workflow files against the default branch, not the diff of the commits being pushed.

The timeline, from the run's own job timings:

TimeEvent
19:15:23Run starts; actions/checkout pins GITHUB_SHA = 42e1e60 (main's tip)
19:16:45Dependabot #58 merges to main → bf86348, changing onlyclaude-build-fix.yml and claude-code-review.yml
19:15–19:31Builds run (~15 min)
19:31:41Release job cuts release/1.136.05923 from the now-stale 42e1e60 and pushes → rejected

bf86348's parent is exactly 42e1e60, confirming main moved 82 seconds into the run. The branch's workflow files were a version behind main, which reads to GitHub as "updating a workflow".

GITHUB_TOKEN is a GitHub App token with no workflows permission, and there is no workflows: key for permissions: — it cannot be granted in the workflow. So the fix has to remove the drift, not acquire the permission.

This is a live race, not a one-off: the build window is ~15 minutes and dependabot bumps action pins inside .github/workflows/ on a schedule.

The fix

Cut the release branch from current origin/main instead of the run's checked-out SHA, re-applying the two generated release files on top. Side benefits: the auto-release PR merges cleanly, and gh release create --target stops tagging a stale tree.

⚠️ The one-line-that-matters is NOT in this PR — it needs a human

The workflow hunk could not be pushed. The fix loop's own claude[bot] token has the same missing workflows permission as GITHUB_TOKEN:

! [remote rejected] fix/release-branch-stale-workflow-race (refusing to allow a GitHub App
to create or update workflow `.github/workflows/cron-build-and-release.yml`
without `workflows` permission)

The contents API is gated identically (403 Resource not accessible by integration), so there is no transport available to this token. Routing the logic into a shell script would not help — the workflow must still be edited to call it.

So this PR contains the regression test and the docs; a human must apply the workflow hunk below. Please apply it to this branch (it is verified — see below):

git apply this patch to .github/workflows/cron-build-and-release.yml
@@ -812,7 +812,33 @@ jobs:
if git ls-remote --exit-code --heads origin "$branch" >/dev/null 2>&1; then
git push origin --delete "$branch"
fi
+ # Cut the branch from CURRENT origin/main, not this run's checked-out+ # SHA. The build takes ~15 min and main moves under us — dependabot's+ # action-pin bumps land in .github/workflows/. A branch cut from the+ # stale SHA then carries workflow files that differ from the default+ # branch's, and GITHUB_TOKEN is a GitHub App token with no `workflows`+ # permission (there is no such key for `permissions:`, so it cannot be+ # granted), which GitHub refuses at push time:+ # ! [remote rejected] release/X -> release/X (refusing to allow a+ # GitHub App to create or update workflow+ # `.github/workflows/claude-build-fix.yml` without `workflows`+ # permission)+ # GitHub compares the branch's workflow files against the default+ # branch, NOT the diff of the new commit — so a commit touching only+ # the two files below is still rejected. Took run 33795347082 down+ # after both builds had already passed. Regression test:+ # ./dev/test-release-branch-race.sh+ staged=$(mktemp -d)+ cp upstream/stable.json "$staged/stable.json"+ cp versions/stable/darwin-arm64/latest.json "$staged/latest.json"+ git fetch --no-tags origin main+ # Discards the working-tree edits made by the two steps above (saved+ # in $staged); leaves untracked assets/ alone.+ git reset --hard FETCH_HEAD
git checkout -B "$branch"
+ cp "$staged/stable.json" upstream/stable.json+ mkdir -p versions/stable/darwin-arm64+ cp "$staged/latest.json" versions/stable/darwin-arm64/latest.json
git add upstream/stable.json versions/stable/darwin-arm64/latest.json
git commit -m "chore(release): ${RELEASE_VERSION} (vscode ${MS_TAG})"
git push -u origin "$branch"

Worth considering separately: granting the release job a token with workflows permission would also unblock the fix loop the next time this class of bug appears.

Verification

dev/test-release-branch-race.sh (new) extracts and runs the real "Create release branch and push" step out of the workflow against a fake origin whose main advances mid-build, so it cannot drift from the step it guards. It asserts the exact condition GitHub rejects on.

Against the pre-fix step (what is on main today) — reproduces the failure:

FAIL: branch still drifts workflow files -> push would be rejected: .github/workflows/claude-build-fix.yml
FAIL: expected exactly the two release files, got [.github/workflows/claude-build-fix.yml upstream/stable.json versions/...]
FAIL: release branch is not a descendant of current main
exit=1

Against the fixed step:

PASS: workflow files identical to main -> push allowed
PASS: release commit changes exactly the two release files
PASS: regenerated release-file contents survived the branch cut
PASS: release branch descends from current origin/main
PASS: untracked assets/ survived the branch cut
PASS: step emitted branch=release/1.136.05923 to $GITHUB_OUTPUT
exit=0

Pin consistency: upstream/stable.json already pins 1.136.0 / 520fb30b2d3d324b4cb2342f6e88e2cd93751de1, which matches git ls-remote refs/tags/1.136.0. This PR changes no build inputs, so no bump is needed.

CI_BUILD=yes ./dev/ci-verify.sh --commit 520fb30b2d3d324b4cb2342f6e88e2cd93751de1passed (overlay + patches + npm ci + TypeScript compile):

[compile-client] Finished compile-src /vscode/src/tsconfig.json with 0 errors.
ci-verify: PASS (compile, tag 1.136.0)

Tier reached: compile passed. Not run: --full (vscode-min-prepack minify + ASCII hygiene) and packaging/signing.

Note that ci-verify is not the load-bearing check for this change: it exercises patches and the compile, and this PR touches no build inputs. The build was already proven green at this tree by the failing run itself, whose only delta to main is workflow files.

dev/test-release-branch-race.sh extracts and runs the real 'Create release
branch and push' step from cron-build-and-release.yml against a fake origin
whose main advances mid-build, so it cannot drift from the step it guards.
Asserts the condition GitHub actually rejects on: whether the pushed
branch's workflow files differ from the default branch's. Fails against the
current (pre-fix) step; passes once the branch is cut from origin/main.
Also documents the release-job failure class in docs/build-fix.md.
Applies the workflow hunk from the PR body that claude[bot] could not
push (its App token lacks the workflows permission). Verified locally:
./dev/test-release-branch-race.sh fails 3/6 assertions against the
pre-fix step and passes 6/6 with this change; zizmor findings are
unchanged from main.
@claude

claudeBot commented Sep 3, 2026

Copy link
Copy Markdown
ContributorAuthor

Claude finished @sbs44's task in 10m 17s —— View job


Code review

No blocking issues found. Checked for bugs, git-semantics correctness, historical context, prior PR feedback, and code-comment accuracy. (This repo has no CLAUDE.md, so that pass fell back to the conventions established by sibling files.)

I independently exercised the new regression test rather than trusting the PR body:

  • ./dev/test-release-branch-race.sh against this branch: 6/6 PASS, exit 0.
  • Same test with origin/main's version of the workflow step swapped in: FAILs on workflow-file drift, extra file in the release commit, and non-ancestry — so it genuinely reproduces run 33795347082 and is not tautological.
  • Also re-ran under init.defaultBranch=master to check the fake-origin bootstrap isn't environment-dependent: still passes.

git reset --hard / checkout -B leave untracked files alone, so the "leaves untracked assets/ alone" comment is accurate and the later gh release create asset uploads are safe. The release job's checkout uses fetch-depth: 0, so the added git fetch + reset is not fighting a shallow clone.

Non-blocking notes (scored below the reporting bar, recorded for the record)

  1. Release-tag provenance now differs from the built tree. The branch is current main + 2 files rather than the SHA that was compiled, signed and notarized, so gh release create --target "$BRANCH" can tag a tree whose source doesn't match the DMG/ZIP SHA-256 values in the release notes. In practice the mid-build drift is normally just dependabot workflow-pin bumps, and any source commit that lands mid-build already went through its own CI and is already on main — so this doesn't put unbuilt code onto main, it only loosens tag↔binary correspondence. A narrower variant (cut from the run SHA, then git checkout FETCH_HEAD -- .github/workflows/) would preserve parity, but only if GitHub's rule really is "compare branch state to default branch" as the PR asserts — that's the PR's model, not something I could confirm, so it's not a recommendation.

    cp versions/stable/darwin-arm64/latest.json "$staged/latest.json"
    git fetch --no-tags origin main
    # Discards the working-tree edits made by the two steps above (saved
    # in $staged); leaves untracked assets/ alone.
    git reset --hard FETCH_HEAD
    git checkout -B "$branch"
    cp "$staged/stable.json" upstream/stable.json
    mkdir -p versions/stable/darwin-arm64

    --title "BradfordCode ${RELEASE_VERSION}" \
    --notes-file "$notes_file" \
    --target "$BRANCH" \
    "assets/BradfordCode.arm64.${RELEASE_VERSION}.dmg" \

  2. upstream/stable.json is restored unconditionally. If main's pin advanced past this run's MS_TAG between checkout and the fetch (overlapping run, or a hand-merged pin bump), the copy-back silently downgrades it where the old code would have surfaced a merge conflict on the PR. Rare given the daily cron cadence.

  3. The race is narrowed, not closed — a workflow-touching commit landing between the fetch (L834) and the push (L844) still hits the same rejection, with no retry. Seconds-wide vs. the previous ~15 minutes, so this is fine as-is; worth knowing it can still recur.

  4. docs/build-fix.md says the release job "runs ~15 minutes after checkout". The release job's own checkout ran 37s before the failing step in 33795347082; the precise statement is that actions/checkout resolves GITHUB_SHA, pinned at run start — fresh in wall-clock, stale in content. Minor wording, and the rest of the section is accurate.

  5. dev/test-release-branch-race.sh diverges from the four sibling dev/test-*.sh scripts (set -euo vs set -eo, unbraced $var, [ ] vs [[ ]], bespoke pass/bad instead of the shared assert_eq/FAILURES harness), and extracts the step with a hand-rolled Node line scanner rather than the PyYAML parse dev/test-release-timestamp.sh uses for the same job. The extractor throws on "step not found"/"run block not found" and runs bash -n, so it mostly fails loudly; the one silent path is a reformat that changes the run: | body indentation, which would yield an empty script that bash -n accepts.

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.


| branch fix/release-branch-stale-workflow-race

@github-actions

Copy link
Copy Markdown
Contributor

Build green at 8e7367d (run). Ready for review — assigned to @sbs44.

@sbs44
sbs44 merged commit 7ec7c12 into mainSep 3, 2026
8 checks passed
@sbs44
sbs44 deleted the fix/release-branch-stale-workflow-race branch September 3, 2026 20:51
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@sbs44