fix(security): confirm a Rekor 409 by verifying it, sign before the rollout, publish release artifacts only after the attestation gate

[Cre-eD]

Four related supply-chain fixes. The first two break deploys today; the third
is why nobody notices when they do; the fourth is that "nothing is published"
was not true for images.

A review pass over the first three reproduced two of its findings
empirically. Everything it raised is addressed here; the mechanism in
section 1 changed substantially as a result. See Review round 2 at the end
for the finding-by-finding list.

1. A Rekor 409 on an already-attested image now succeeds

Redeploying an unchanged image digest regenerates a byte-identical SBOM or
provenance predicate. The transparency log already holds that exact entry, so
it answers 409 and the deploy fails on an artifact that is already in the state
the deploy wanted:

cosign attest failed: signing bundle: [POST /api/v1/log/entries][409]
createLogEntryConflict {"code":409,"message":"an equivalent entry already
exists in the transparency log with UUID ..."}
Error: failed to attach SBOM: cosign attest failed: exit status 1

Retrying cannot clear it. Under key-based signing the body is deterministic, so
every attempt reproduces the same conflict and the loop only delays the failure.

RunCosignWithRetry already refused to read a 409 as success, and its reasoning
was right: cosign uploads to Rekor before it pushes to the registry, so a tlog
entry is not proof the artifact was attached. Rather than drop that guarantee,
this closes the gap. On a conflict the loop asks the registry directly, with
cosign verify --key/--certificate-identity-regexp for a signature and
cosign verify-attestation --type <type> ... for an attestation. Artifact
present and verifying means the run is already done, so it returns success
immediately without burning the rest of the retry budget. Otherwise the
previous retry-then-report behaviour is untouched.

Verify, not download. The first draft of this probe used cosign download signature / cosign download attestation. Two problems, both raised in review
and one reproduced against a local registry with cosign v3.0.2:

• cosign v3 defaults to the new bundle format and writes nothing to the legacy
  sha256-<digest>.sig tag. On a signed image, cosign download signature
  returns no signatures associated while cosign verify --key succeeds. The
  download probe is therefore a permanent false negative on current cosign, so
  every conflict would run to exhaustion and fail. Combined with fix 2 below,
  which makes signing a hard predecessor of the Deployment, the exact case this
  PR exists to fix would have degraded from "rolls out, run is red" to "does not
  roll out at all".
• download answers presence, not identity. Any entry under the legacy tag
  counted, so a signature made under a rotated key, or an attestation of the
  same predicate family written by an unrelated workflow, read as proof.
  security.Signing.Verify is opt-in, so nothing downstream re-checked.

verify reads both formats and checks the identity, so a confirmation now means
the artifact is present and ours. The probe's exit code is the whole signal:
cosign prints its verification banner to stderr and nothing to stdout, so the
earlier "clean exit and non-empty stdout" rule was itself a way never to
confirm.

The confirmation stays fail-closed. A probe that errors, times out, or cannot be
built at all leaves the caller on retry-then-report, so it can never convert a
genuine signing failure into a green deploy. It is only built when the config
can name an identity to verify against; without one the caller keeps the plain
retry path rather than accept an artifact it cannot attribute. The predicate
type carries through from the attest invocation, otherwise an existing SBOM
attestation would stand in for a missing provenance one.

The probe is bounded at 30s rather than inheriting the sign/attest budget.
Reusing it made the worst case per artifact 5 x (timeout + timeout) plus
backoff, roughly 20 minutes for the SBOM attacher against 6 minutes before
confirmation existed, and both attachers run per image. It also no longer
inherits SIGSTORE_ID_TOKEN: it mints no certificate, so its environment is
filtered to registry-relevant variables.

Proving the probe actually fires. A fake cosign cannot certify this: the
fake decides what the probe means, which is exactly why the v3 breakage was
invisible to a green suite. So the suite now has an integration test that starts
a real registry:2, pushes an image, signs it for real, and drives the
confirmation path with the genuine cosign binary, plus the negative half over an
unsigned image. The synthetic part is only the 409 itself, because reaching a
real one would mean uploading to a public transparency log. On the run used to
develop this, that test logged:

cosign v3.0.2
legacy `cosign download signature` on a signed image:
  err=exit status 1 stderr="... no signatures associated"

while the verify probe confirmed. The fake was hardened to match: it validates
the probe argv the way real cosign does (identity flag required, --type
required for verify-attestation, image reference must be the one under test),
answers download the way cosign v3 does, and models probe exit status and
stdout separately so exit-0-empty, auth failure and false-negative are all
expressible. Its probe branch also moved below the delay scripting, so probes
cost simulated time like every other invocation.

Sibling symptom. Concurrent deploys of one digest also hit

signing bundle: Post "https://rekor.sigstore.dev/api/v1/log/entries": ...
giving up after 2 attempt(s)

That is cosign exhausting its own HTTP retries against the log, not a conflict.
It used to fall straight through to the fail-fast branch and break the deploy on
the first hiccup. It is now a second retryable class, matched only on a
log-entries POST so an unrelated give-up from a registry or from Fulcio still
fails fast.

The first draft matched the two halves with (?s) and a 400-character gap,
which only required them to appear near each other in the same stream. Review
confirmed all three of these matched: a log-entries POST returning 201 Created
followed by a registry unauthorized, giving up after 3 attempt(s), the same
followed by a Fulcio signingCert give-up, and the same followed by an OIDC
token give-up. None of them falsely succeeded, but a permanent registry-auth
failure was retried five times and the operator was pointed at the
transparency-log runbook. The gap now excludes quotes and newlines, so the match
stays inside one message; every competing give-up names its own quoted URL
first. All three are regression tests.

Retry budget. With two retryable classes the bound goes from 3 attempts to
5. The jittered full-backoff schedule is unchanged (1s base doubling to an 8s
cap), so the worst case grows from about 3s to about 15s of waiting. That only
applies to the genuinely-contended path now: a conflict on an artifact that is
already attached short-circuits on the first attempt and never sleeps at all.

2. Signing no longer runs after the workload has rolled out

BuildAndPushImage models sign, verify, SBOM attestation and provenance
attestation as resources hanging off the image push and returns their fan-in on
ImageOut.AddOpts. The documented contract is that the workload update depends
on them. The ECS path implements it (aws/ecs_fargate.go folds img.AddOpts
into its options). The Kubernetes path did not: BuildAndPushImages copied
AddOpts onto ContainerImage and nothing read it again, so
DeploySimpleContainer created the Deployment with no dependency edge to any
security operation.

Pulumi was therefore free to update the workload first. Observed ordering in a
failing run: rollout completed at 06:42:04, signing failed at 06:42:26, then

Image Signing         SKIPPED
Provenance Generation SKIPPED
Report Uploads        SKIPPED
error: update failed

Worst of both outcomes: an unsigned, unattested image with no SBOM is already
serving traffic, and CI is red so the next deploy looks like the problem rather
than the symptom.

This is the full fix, not a guard. Folding the per-image options into the
options DeploySimpleContainer builds makes signing a precondition of the
rollout, the same way the ECS path already worked. Images SC did not build, and
stacks with security disabled, contribute no options and deploy exactly as
before. Nothing about the security resources themselves changed.

The regression test drives DeploySimpleContainer under Pulumi mocks with a
stand-in gate resource on AddOpts and asserts the registered Deployment lists
it as a dependency. Reverting the one-line wiring fails it with the gate URN
absent from the dependency list.

3. The attestation chain now blocks a release

In push.yaml every step of the chain is continue-on-error: true and the
collector only emits ::warning title=Attestation incomplete::.... A release
can therefore publish a tarball with no SBOM, no signature and no provenance on
a fully green workflow. docs/SECURITY.md tells consumers the opposite: "Every
release produces signed, attested artifacts". Nothing downstream closes the gap
either, since verify-attestations runs after publish and its own header says
it surfaces the regression "without blocking the release that already shipped".

Which path I took, and why. The soft-fail is scoped in the comments as a
14-day bake-in tracked in a HARDENING.md Phase 2 plan. I looked for evidence
that this is still a live decision and found the opposite:

• HARDENING.md does not exist in the repository and never has
  (git log --diff-filter=A -- '**HARDENING.md' is empty)
• no open issue tracks flipping it
• the soft-fail comments landed on 2026-05-21 at the latest, so a 14-day window
  closed roughly three months ago

So this enforces rather than hiding behind a flag defaulting to today's
behaviour. The three collectors in push.yaml now exit 1: the per-platform
tarball chain, the per-image chain, and the sidecar count assertion guarding the
bundle actually uploaded to the CDN (it runs before publish, so a gap there
blocks the upload).

What the chain does is unchanged. Every sub-step deliberately stays soft, so one
failure does not mask the others and the gate reports the whole picture at once
instead of aborting at the first missing piece.

Break glass, with an expiry. ATTESTATION_SOFT_FAIL degrades all three back
to warnings for the duration of a Sigstore or Rekor outage. It is unset by
default, so the enforcing path is what runs.

Review pointed out that the first draft was an unbounded, unattributed and
invisible bypass: vars.* also resolves ORGANIZATION variables, so one set on
the org silently disabled the gate in every repository that reads it, nothing
printed which scope supplied it, there was no expiry or recorded reason, and a
break-glass release was byte-indistinguishable from a clean one at the consumer
end. So the value is now an ISO-8601 UTC expiry date (YYYY-MM-DD) and:

• it stops applying on that date whether or not anyone remembers to remove it
• anything that is not a date, including the historical true, is rejected with
  an ::error and does not bypass
• every gate echoes the value on every run, pass or fail, so a stale value shows
  up in green runs
• a bypassed run writes attestation-breakglass=<date> to the job summary
• an expired value logs a Break-glass expired warning before blocking

docs/SECURITY.md documents the format, the organization-scope caveat and who
may set it.

Sidecar pairing. The bundle assertion compared totals per sidecar kind
against the tarball count, so it passed a bundle in which one tarball carried
two of a kind and another none, which is exactly the shape a half-failed matrix
leg produces. It now pairs each tarball with its own four sidecars, and an empty
sidecar counts as missing.

branch-preview.yaml and build-staging.yml stay warn-only and gain a comment
saying why: neither is a release, no published contract promises their artifacts
carry a complete chain, and blocking them would stall iteration on an upstream
hiccup.

Matrix values in the rewritten gate scripts are read through env rather than
interpolated into the shell.

4. Release image tags are published only after the gate

Fix 3 made the gate block, but for images it blocked too late. The build step
ran docker/build-push-action with push: true and tags: containing both
<img>:latest and :<version>, so the release tags were live before
Sign + attest and before the gate. A failing gate only failed the job and
skipped docker-finalize, which left a mutable :latest pointing at an
unsigned, un-SBOM'd image while dist.simple-container.com still served the
previous version. The tarball path already gates before upload, so this was an
oversight rather than a design choice, and docs/SECURITY.md was left asserting
in writing something that was false for images.

The image now builds to a throwaway :ci-<run>-<attempt> tag, because cosign
signs by digest and needs the manifest in the registry before it can sign
anything. The release tags are applied by a promote step after the gate, and the
throwaway tag is deleted afterwards, including when the gate blocked, since that
is precisely when an unsigned image would otherwise stay reachable under a
predictable name.

Promotion has to preserve the digest, and the obvious tool does not. The
natural shape is docker buildx imagetools create -t <tag> <repo>@<digest>.
Measured against a local registry, given a single platform-specific manifest it
wraps it in a new index: source sha256:dd54c6... came back as
sha256:45b9e3.... The tag would then resolve to a digest that was never
signed, and cosign verify <repo>:latest would fail for every consumer
following the documented verification path. Promotion instead re-PUTs the signed
manifest byte for byte under each release tag, which is what crane tag does
and what the registry v2 API is for, needing nothing on the runner beyond curl
and jq. The step then compares the registry's Docker-Content-Digest against
the signed digest and fails the release if they ever diverge, so a promotion
that stops preserving the digest cannot ship silently.

With this, "the release build fails and nothing is published" is literally true
on both paths, and docs/SECURITY.md says how.

Validation

• go build ./... clean, go vet ./... clean
• go test ./...: 48 packages, all pass, 0 failures. The pkg/api/git
  environment failure noted in the first round no longer reproduces (this run
  used a worktree outside /tmp).
• the integration test in pkg/security/signing was run for real against a
  local registry:2 with cosign v3.0.2: both halves pass, and it is the source
  of the download-versus-verify evidence quoted in section 1. Build-tagged
  integration and skipped under -short, so it does not run in normal CI.
• every workflow touched YAML-parses; actionlint reports the same finding set
  before and after (6 runner-label for the self-hosted runners, 1 pre-existing
  stack-name expression warning)
• the registry-API promotion and the break-glass date logic were both executed
  against a local registry / shell before being written into the workflow: the
  promotion preserves the digest across two tags, and true, a malformed value,
  an expired date, today's date and a future date all behave as documented
• gofmt -l clean on every file in the diff
• no deploys run, nothing pushed to a registry other than a throwaway local one,
  nothing written to a transparency log

Review round 2

Every finding from that review, and what happened to it.

Finding	Status
P0 image tags public before the gate can block them	fixed, section 4
P0 idempotency fix does not fire on cosign v3, and the rollout gate turns that into a blocked deploy	fixed, section 1 (verify probe)
P1 presence vs identity in the probe	fixed by the same change; the probe now checks identity
P1 rekorGiveUpRe over-matches	fixed; three confirmed cases added as want: false
P1 probe reuses the full signing timeout	fixed; capped at 30s, with a test
P1 ATTESTATION_SOFT_FAIL unbounded and invisible	fixed; ISO-8601 expiry, echoed every run, written to the job summary, org-scope documented
P1 the cosign fake cannot catch a broken probe	fixed; argv validation, ProbeExit/ProbeStdout, probe below the delay, plus the real-registry integration test
P2 confirmed conflict returns an empty RekorEntry while reporting success	fixed; SignResult.Confirmed plus an explicit log line
P2 probe inherits SIGSTORE_ID_TOKEN	fixed; environment filtered to an allowlist
P2 sidecar assertion compares totals, not pairs	fixed; per-tarball iteration, empty counts as missing
P2 containsURN accepts a suffix	fixed; equality
P2 ecs_fargate.go missing the nil guard imageSecurityOpts has	fixed
docs drift: SECURITY.md "nothing is published", preview/staging "mirrors push.yaml", preview same-guarantees contradiction, verify-attestations header	all four fixed

Two things worth flagging to a human:

• Found by the new integration test: GenerateKeyPair never honoured its
  outputDir. Without --output-key-prefix, cosign writes the pair into the
  process working directory, so every caller passing a directory failed on the
  chmod that follows, after dropping an encrypted private key into the source
  tree. Fixed, and both filenames are now git-ignored as a second line of
  defence. pkg/security/signing/e2e_test.go has been calling it that way.
• Out of scope, reported not fixed: the review found pre-existing
  private-organisation references on main of this public repository,
  including a real cloud account id in a test fixture. They are untouched here
  and need their own change plus an owner decision.

Note on #383

#383 is red only because its base predates the go directive move to 1.26.6. A
rebase clears it. Not touched here.

[github-actions]

Semgrep Scan Results

Repository: api | Commit: bbae0ad

Check	Status	Details
⚠️ Semgrep	Warning	2 warning(s), 6 total

Scanned at 2026-08-23 11:25 UTC

[github-actions]

Security Scan Results

Repository: api | Commit: bbae0ad

Check	Status	Details
✅ Secret Scan	Pass	No secrets detected
✅ Dependencies (Trivy)	Pass	1 total (no critical/high)
✅ Dependencies (Grype)	Pass	1 total (no critical/high)
📦 SBOM	Generated	523 components (CycloneDX)

Scanned at 2026-08-23 11:26 UTC

[github-actions]

📊 Statement coverage

Measured on the documented included set (see docs/TESTING.md → Coverage scope). Observe-only — no regression gate is enforced yet.

Scope	This PR	main baseline	Δ
Included set (Gold-tier denominator)	90.5%	90.5%	+0.0 pp
Full set (whole repo, transparency)	29.6%	29.1%	+0.5 pp

Baseline: main @ fccdc83