Skip to content

ci: two-person promotion of a verified candidate to version + latest - #262

Merged
LeeroyHannigan merged 2 commits into
mainfrom
ci/promote-image
Aug 13, 2026
Merged

ci: two-person promotion of a verified candidate to version + latest#262
LeeroyHannigan merged 2 commits into
mainfrom
ci/promote-image

Conversation

@LeeroyHannigan

Copy link
Copy Markdown
Collaborator

Replaces the runbook's manual 'docker buildx imagetools create' promotion with a dispatchable workflow behind the dockerhub protected environment, so promotion gains the same two-person property as the candidate push: the operator who verified the candidate dispatches with the digest they verified, and a different maintainer approves the gate. The registry credential never leaves the environment, removing the dependency on any individual holding the Docker Hub token locally.

Gates, in order, before any credential exists in the job: strict input shapes (semver, sha-<40 hex> candidate tag, sha256 digest); the candidate commit must be contained in origin/main; the version promoted to must equal Cargo.toml's version at that commit; the candidate tag must still resolve to the verified digest (refuses if the tag moved); an existing version tag with a different digest is refused, making released versions immutable through this path. Promotion copies by digest with skopeo --preserve-digests (no rebuild, byte-identical), tags version and latest, and verifies both resolve to the verified digest afterwards.

Gate logic simulated locally with adversarial inputs (injection shapes, malformed tags and digests all rejected); happy path validated against the live sha-c34aac1 candidate (ancestry and Cargo version 0.1.4 agree).

What

Why

Closes #

Testing done

Checklist

  • I have read CONTRIBUTING.md
  • All tests pass (cargo test --workspace)
  • Code is formatted (cargo fmt --check)
  • Clippy is clean (cargo clippy -- -W clippy::pedantic)
  • I have added or updated tests for new functionality
  • I have updated documentation if behavior changed
  • Breaking changes are noted below (if any)
  • If this changes the wire protocol, Storage trait, auth model, on-disk
    format, or public CLI surface, an RFC has been accepted or is linked
    below. Otherwise, an ADR captures the decision (link below).

ADR / RFC:

Breaking changes


By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

Replaces the runbook's manual 'docker buildx imagetools create' promotion
with a dispatchable workflow behind the dockerhub protected environment,
so promotion gains the same two-person property as the candidate push:
the operator who verified the candidate dispatches with the digest they
verified, and a different maintainer approves the gate. The registry
credential never leaves the environment, removing the dependency on any
individual holding the Docker Hub token locally.
Gates, in order, before any credential exists in the job: strict input
shapes (semver, sha-<40 hex> candidate tag, sha256 digest); the candidate
commit must be contained in origin/main; the version promoted to must
equal Cargo.toml's version at that commit; the candidate tag must still
resolve to the verified digest (refuses if the tag moved); an existing
version tag with a different digest is refused, making released versions
immutable through this path. Promotion copies by digest with skopeo
--preserve-digests (no rebuild, byte-identical), tags version and latest,
and verifies both resolve to the verified digest afterwards.
Gate logic simulated locally with adversarial inputs (injection shapes,
malformed tags and digests all rejected); happy path validated against
the live sha-c34aac1 candidate (ancestry and Cargo version 0.1.4 agree).
@robinnsc

robinnsc commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Nice, one small change that could be good would be to put a rollback guard on latest. The last copy step moves latest to the promoted digest unconditionally, so if we ever promote an older line after a newer one has shipped (Ex: say a 0.1.x hotfix once 0.2.0 is out, latest will move backwards). Possible fix options:

  1. Before touching latest, list the repo's existing X.Y.Z tags (skopeo list-tags + a semver sort) and skip or fail the latest copy when the input version isn't the highest
  2. Add a move_latest boolean input defaulting to true, with the same check backing it, would also gives us an explicit escape hatch for hotfix promotions that intentionally shouldn't move latest.

Review finding from robinnsc: the final copy moved `latest` to the
promoted digest unconditionally, so promoting an older line after a newer
one had shipped (a 0.1.x hotfix once 0.2.0 is out) would walk `latest`
backwards and silently downgrade every user who does not pin.
Implements both suggested options, because they are complementary: the
highest-version check is the substance, and the boolean is the escape
hatch that check needs to stay usable.
- New `move_latest` boolean input, default true, validated in the input
gate like every other input rather than trusted as a boolean.
- New gate, placed with the other gates so it still runs before the
registry credential exists in the job: list the repository's released
X.Y.Z tags and refuse to move `latest` unless the promoted version is
the highest.
Semantics chosen so nothing happens silently. When the version is behind
and move_latest is true, the run FAILS and names the higher version plus
the exact remedy, rather than quietly skipping the tag the operator asked
to move. When move_latest is false, the version tag is promoted and
`latest` is left alone, which is the legitimate back-line hotfix path.
The version tag is published either way, so the guard never blocks
shipping a hotfix, only mislabelling it.
Details worth noting for review:
- The promoted version is excluded from the comparison set, so the
ALREADY_PROMOTED re-run (version tag exists at the verified digest,
ensuring only `latest`) is not blocked by its own tag.
- Comparison is `sort -V`, so 0.1.10 correctly beats 0.1.9 where a
lexical compare would not.
- Non-version tags are filtered out: candidate `sha-*` tags, `latest`,
and malformed or prerelease names cannot hold `latest` back.
- The tag-list pipeline carries `|| true` because grep exits 1 on no
match, which under `set -o pipefail` would otherwise kill the step on
a repository with no released versions yet, which is exactly today's
state.
Verification: the gate logic was simulated against nine cases, including
no released versions (today's live state, confirmed: only the three
sha-c34aac1 candidate tags exist, no version tags and no `latest`), a
behind-version promotion with the flag both ways, self-exclusion on
re-run, the 0.1.9 vs 0.1.10 ordering trap, and a tag list seeded with
`v0.1.5`, `0.1`, `1.2.3.4`, `0.2.0-rc1` and `sha-*` junk. All nine
behaved as intended. The verify step and run summary now follow the same
condition, so neither claims `latest` was moved when it was not.
@LeeroyHannigan

Copy link
Copy Markdown
CollaboratorAuthor

Good catch, that's a real hole. Fixed in 860ec7a.

I took both of your options rather than picking one, because they're complementary: the highest-version check is the substance, and the boolean is the escape hatch that check needs in order to stay usable. So there's now a move_latest boolean input (default true, validated in the input gate rather than trusted), and a new gate that lists the repository's released X.Y.Z tags and refuses to move latest unless the promoted version is the highest. It sits with the other gates, so it still runs before the registry credential exists in the job.

On skip-vs-fail I went with fail, deliberately. If the version is behind and move_latest is true, the run stops and names the higher version plus the exact remedy, rather than quietly not doing the thing the operator asked for. With move_latest unchecked, the version tag is promoted and latest is left alone, which is the legitimate hotfix path. The version tag publishes either way, so the guard never blocks shipping a hotfix, only mislabelling it as current.

Three details that took the most care:

  • The promoted version is excluded from the comparison set, otherwise the ALREADY_PROMOTED re-run (version tag already at the verified digest, ensuring only latest) would be blocked by its own tag.
  • Comparison is sort -V, so 0.1.10 beats 0.1.9; a lexical compare gets that backwards.
  • The tag-list pipeline carries || true, because grep exits 1 on no match and under set -o pipefail that would kill the step on a repo with no released versions, which is exactly today's state.

Simulated the gate against nine cases: no released versions, a behind-version promotion with the flag both ways, self-exclusion on re-run, the 0.1.9 vs 0.1.10 ordering trap, and a tag list seeded with v0.1.5, 0.1, 1.2.3.4, 0.2.0-rc1 and sha-* junk to confirm non-version tags can't hold latest back. All nine behaved as intended. I also confirmed the live repo state matches the no-released-versions path: only the three sha-c34aac1 candidate tags exist, no version tags and no latest.

The verify step and the run summary now follow the same condition, so neither claims latest moved when it didn't.

@robinnsc
robinnsc self-requested a review August 13, 2026 20:46
@LeeroyHannigan
LeeroyHannigan added this pull request to the merge queueAug 13, 2026
Merged via the queue into main with commit 4487fd8Aug 13, 2026
17 checks passed
@robinnscrobinnsc mentioned this pull request Aug 13, 2026
7 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@LeeroyHannigan@robinnsc@jcshepherd