ci: make every release lane fail closed and observable

.agents/skills/release/SKILL.md

@@ -10,46 +10,41 @@ mechanics and failure modes.
 
 ## Release model
 
-- **Changesets, not tags.** Contributors land PRs with `.changeset/*.md` entries (authored via the
- tracked `gen-changesets` skill, `.agents/skills/gen-changesets/SKILL.md`). Versions and CHANGELOGs
- are machine-generated from those entries. Nobody edits a version number by hand.
-- **Two-phase CI flow** on every push to `main` (`.github/workflows/release.yml`):
- 1. *Pending changesets exist* → changesets action runs `pnpm run version:release`
- (= `changeset version`) and opens/updates the **`ci: release packages`** PR.
- 2. *That PR merges* → next run finds no pending changesets but bumped versions → publishes via
- `node scripts/release/changeset-publish-idempotent.mjs`, creates the GitHub Release at tag
- `@pythoughts/pythinker-code@<version>`, and fans out to downstream jobs.
-- **Publishing is CI-only** via npm Trusted Publishing (OIDC, `id-token: write`). The workflow
- deliberately sets **no `NPM_TOKEN`** — changesets prefers a token over OIDC when one is set, so
- adding it would silently downgrade publishing to a long-lived secret. Never "fix" a publish
- failure by adding NPM_TOKEN, and never run `changeset publish` locally.
-- The root `publish` script in `package.json` chains the full local gate
- (`typecheck → lint → sherif → test → build → lint:pkg → changeset publish`) — it exists for gate
- parity, not for actually publishing from a laptop.
+- **Changesets, not manual version edits.** Contributor PRs add `.changeset/*.md`. The changesets
+ action creates or updates the `ci: release packages` PR. Merging that PR publishes the public npm
+ package and creates `@pymodel/pythinker-code@<version>`.
+- **npm publishing is CI-only.** Trusted Publishing uses OIDC. Do not add `NPM_TOKEN`; a token takes
+ precedence over OIDC. Do not run `changeset publish` locally.
+- **Private lanes use the push boundary.** `publishedPackages` only lists packages published to npm.
+ Desktop and VS Code are private workspaces, so `detect-lane-bumps.mjs` compares their versions at
+ `github.event.before` and `github.sha`.
+- **Desktop is tag-driven.** The required `cut-desktop-tag` job creates `desktop-v<version>`, which
+ starts `desktop-release.yml`.
+- **VS Code is isolated.** `vscode-release.yml` supports `workflow_call` and version-checked manual
+ dispatch. Existing registry versions are skipped by the publisher scripts, so recovery is safe.
 
 ## What publishes
 
-`.changeset/config.json` `ignore` list excludes almost every internal package
-(`agent-core`, `pyaos`, `kosong`, `server`, dashboards, web, …). Effective publishable set =
-non-private, non-ignored workspace packages — in practice **`@pythoughts/pythinker-code`** and the
-SDK-adjacent packages not on the ignore list. When adding a workspace package, decide its ignore/
-publish status explicitly, and remember `flake.nix` workspace lists must be updated by hand
-(root `AGENTS.md`).
+`@pymodel/pythinker-code` is the public npm package. Desktop and VS Code package files are private;
+their versions are release signals but changesets does not publish them to npm. When adding a
+workspace, set its `private` and changesets policy explicitly and update `flake.nix`.
 
 ## release.yml job map
 
 | Job | Trigger | Notes |
 |---|---|---|
-| `Release` | every main push | install → build catalog → `pnpm build` → changesets action |
-| `Redeploy code.pythinker.com` | `packages_published == 'true'` | runs `scripts/release/verify-release-consistency.mjs`, then POSTs `DOKPLOY_CDN_DEPLOY_WEBHOOK` (skips with a warning if the secret is unset) |
-| `Update Homebrew tap` | published | `scripts/release/update-brew-formula.mjs` with `TAP_GITHUB_TOKEN` (skips if unset) |
-| `Deploy docs` | published | reusable `docs-deploy.yml` |
-| `Native release artifact` | `pythinker_native_release == 'true'` | reusable `_native-build.yml`, macOS signing/notarization secrets |
-| `Publish native release assets` | native release | `produce-manifest.mjs` then `gh release upload <tag> … --clobber` |
+| `Release` | every main push after CI + Nix | Detect lane versions, build, run changesets |
+| `Cut desktop release tag` | desktop version changed | Required and idempotent; App token makes the tag trigger the desktop workflow |
+| `Publish VS Code extension` | extension version changed | Reusable workflow; six VSIX targets, both registries, provenance |
+| `Native release artifact` | CLI was published | Six signed/tested zips, checksums, provenance |
+| `Publish native release assets` | native builds passed | All-or-nothing immutable upload with `manifest.json` |
+| `Redeploy CDN` + verify | native assets published | Webhook may retry; verification is the hard gate |
+| `Update Homebrew tap` | CLI was published | App token scoped to `homebrew-tap` contents |
+| `Release lane summary` | always | One table with provenance state; fails when an expected enabled lane failed or skipped |
 
-`pythinker_native_release` and the release tag come from
-`apps/pythinker-code/scripts/native/resolve-release.mjs`, driven by the changesets action's
-`publishedPackages` output; the tag format is `@pythoughts/pythinker-code@<version>`.
+Set `RELEASE_LANE_DESKTOP`, `RELEASE_LANE_VSCODE`, `RELEASE_LANE_CDN`, or
+`RELEASE_LANE_BREW` to exactly `disabled` for a conscious temporary opt-out. Missing credentials are
+otherwise errors.
 
 ## Failure modes and known lessons
 
@@ -61,28 +56,39 @@ publish status explicitly, and remember `flake.nix` workspace lists must be upda
  genuinely half-published release — read the log; do not blind-rerun.
 - **Version PR looks wrong.** Never patch the `changeset-release/main` branch by hand. Fix or add
  changesets on `main`; the next workflow run regenerates the PR.
-- **Native builder fails after npm publish succeeded.** npm state is final; native jobs are
-re-runnable against the same workflow run (`gh run rerun <id> --failed`). `--clobber` on asset
-upload makes re-runs safe.
+- **Native builder fails after npm publish succeeded.** npm state is final. Re-run failed jobs from
+ the same run before any assets upload. A complete asset set is an idempotent no-op. A partial set
+must not be filled from a rebuild; keep it or publish a new patch version.
 - **CDN not updated after publish.** `verify-release-consistency.mjs` gates the webhook: local
  `apps/pythinker-code/package.json` version must equal the npm `latest` dist-tag (plus sane
  `beta`/`dev` tags). A mismatch means the checkout in the job predates the release commit or npm
- propagation lag — check `npm view @pythoughts/pythinker-code dist-tags` before touching anything.
+ propagation lag — check `npm view @pymodel/pythinker-code dist-tags` before touching anything.
  Dokploy deploy specifics: see memory `cdn-dokploy-deploy-pipeline`.
 - **`pnpm install` fails in CI or locally.** `engine-strict=true` + Node `>=24.15.0` — check
  `.nvmrc` before debugging anything else.
 - **Pre-push hook** (`scripts/pre-push.sh` via simple-git-hooks) gates local pushes; a hook failure
  is a real gate failure — fix the cause, never `--no-verify`.
 
+## Recovery
+
+| Symptom | Command | Safety |
+|---|---|---|
+| Desktop tag job failed | `git tag desktop-v<VERSION> <RELEASE_SHA> && git push origin desktop-v<VERSION>` | Confirm the tag does not exist first; pushing it starts a public release workflow |
+| VS Code lane partially failed | `gh workflow run vscode-release.yml --ref <RELEASE_SHA> -f expected-version=<VERSION>` | Version is checked; both publishers skip versions already present |
+| Native matrix failed before upload | `gh run rerun <RUN_ID> --failed` | Reuses the same run and commit; do not mix a rebuilt partial asset set |
+| CDN is stale | Re-run the failed `Redeploy CDN` or verification job | Do not republish npm; nightly reconciliation remains red until aligned |
+| Unknown lane drift | `pnpm release:status` | Read-only; queries npm, GitHub Releases, CDN, Marketplace, and Open VSX |
+
 ## Verification commands
 
 ```bash
 gh run list --workflow=release.yml --branch=main -L 3 # workflow health
 gh pr list --search 'ci: release packages in:title' --state open
-npm view @pythoughts/pythinker-code version # published version
-npm view @pythoughts/pythinker-code dist-tags --json
-node scripts/release/verify-release-consistency.mjs # local == npm latest
-gh release view "@pythoughts/pythinker-code@<version>" # assets + manifest.json
+pnpm release:status # all live lanes
+npm view @pymodel/pythinker-code dist-tags --json
+node scripts/release/verify-release-consistency.mjs
+gh release view "@pymodel/pythinker-code@<version>"
+gh attestation verify <artifact> -R PyModel/pythinker-code
 ```
 
 ## Hard rules (mirror tracked contracts)

.github/workflows/_native-build.yml

@@ -32,6 +32,9 @@ on:
 
 permissions:
  contents: read
+ id-token: write
+ attestations: write
+ artifact-metadata: write
 
 jobs:
  native-bundle:
@@ -57,30 +60,32 @@ jobs:
  env:
  # secrets.* isn't allowed in `if:` conditions, so resolve the effective
  # signing gate here (job-level env does allow secrets) and branch on the
- # env var in steps below. Absent/empty secrets -> build unsigned + warn,
- # not a hard failure; a malformed present secret still fails loudly
- # inside macos-keychain-setup (bad base64 / no identity found).
- APPLE_SIGNING_AVAILABLE: ${{ inputs.sign-macos && secrets.APPLE_CERTIFICATE_P12 != '' && secrets.APPLE_CERTIFICATE_PASSWORD != '' }}
+ # env var in steps below. A requested release signature is fail-closed.
+ APPLE_SIGNING_AVAILABLE: ${{ inputs.sign-macos && secrets.APPLE_CERTIFICATE_P12 != '' && secrets.APPLE_CERTIFICATE_PASSWORD != '' && secrets.APPLE_NOTARIZATION_KEY_P8 != '' && secrets.APPLE_NOTARIZATION_KEY_ID != '' && secrets.APPLE_NOTARIZATION_ISSUER_ID != '' }}
 
  steps:
  - name: Checkout
- uses: actions/checkout@v6
+ uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # pinned from v6.0.2
+ with:
+ persist-credentials: false
 
  - name: Setup pnpm
- uses: pnpm/action-setup@v6
+ uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # pinned from v6
 
  - name: Setup Node.js
- uses: actions/setup-node@v7
+ uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # pinned from v7.0.0
  with:
  node-version-file: .nvmrc
  cache: 'pnpm'
 
  - name: Install dependencies
  run: pnpm install --frozen-lockfile
 
- - name: Warn if macOS signing requested but secrets are absent
+ - name: Require macOS signing secrets
  if: runner.os == 'macOS' && inputs.sign-macos && env.APPLE_SIGNING_AVAILABLE != 'true'
- run: echo "::warning::APPLE_CERTIFICATE_P12/APPLE_CERTIFICATE_PASSWORD not configured — building an unsigned, unnotarized darwin bundle."
+ run: |
+ echo "::error::A signed release requires every APPLE_CERTIFICATE_* and APPLE_NOTARIZATION_* secret."
+ exit 1
 
  - name: Setup macOS keychain (release only)
  if: runner.os == 'macOS' && env.APPLE_SIGNING_AVAILABLE == 'true'
@@ -131,12 +136,18 @@ jobs:
  - name: Package native artifact
  run: pnpm --filter @pymodel/pythinker-code run package:native
 
+ - name: Attest native zip provenance
+ if: inputs.sign-macos
+ uses: actions/attest@1e69f48acb82d1966a394da916b4c1698aa569d6 # pinned from v4.2.2
+ with:
+ subject-path: apps/pythinker-code/dist-native/artifacts/pythinker-code-${{ matrix.target }}.zip
+
  - name: Upload native artifact
- uses: actions/upload-artifact@v7
+ uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # pinned from v7
  with:
  name: ${{ inputs.upload-artifact-prefix }}-${{ matrix.target }}
  retention-days: ${{ inputs.retention-days }}
  path: |
  apps/pythinker-code/dist-native/artifacts/pythinker-code-${{ matrix.target }}.zip
  apps/pythinker-code/dist-native/artifacts/pythinker-code-${{ matrix.target }}.zip.sha256
- if-no-files-found: ignore
+ if-no-files-found: error

.github/workflows/ci.yml

@@ -114,6 +114,7 @@ jobs:
  - run: pnpm install --frozen-lockfile
  - run: pnpm run lint
  - run: pnpm run sherif
+ - run: pnpm run test:release
 
  typecheck:
  runs-on: ubuntu-latest

.github/workflows/manual-native-bundle.yml

@@ -5,6 +5,9 @@ on:
 
 permissions:
  contents: read
+ id-token: write
+ attestations: write
+ artifact-metadata: write
 
 jobs:
  build:
@@ -13,9 +16,14 @@ jobs:
  upload-artifact-prefix: pythinker-code-native
  retention-days: 3
  sign-macos: true
+ # One certificate, one notary key, one set of secret names. These are the
+ # names desktop-release.yml already ships signed and notarized from; the
+ # APPLE_CERTIFICATE_*/APPLE_NOTARIZATION_* names this workflow used to read
+ # were never set on this repository, which is why every darwin CLI bundle
+ # so far shipped ad-hoc signed behind a green job.
  secrets:
- APPLE_CERTIFICATE_P12: ${{ secrets.APPLE_CERTIFICATE_P12 }}
- APPLE_CERTIFICATE_PASSWORD: ${{ secrets.APPLE_CERTIFICATE_PASSWORD }}
- APPLE_NOTARIZATION_KEY_P8: ${{ secrets.APPLE_NOTARIZATION_KEY_P8 }}
- APPLE_NOTARIZATION_KEY_ID: ${{ secrets.APPLE_NOTARIZATION_KEY_ID }}
- APPLE_NOTARIZATION_ISSUER_ID: ${{ secrets.APPLE_NOTARIZATION_ISSUER_ID }}
+ APPLE_CERTIFICATE_P12: ${{ secrets.MAC_CSC_LINK }}
+ APPLE_CERTIFICATE_PASSWORD: ${{ secrets.MAC_CSC_KEY_PASSWORD }}
+ APPLE_NOTARIZATION_KEY_P8: ${{ secrets.APPLE_API_KEY_P8 }}
+ APPLE_NOTARIZATION_KEY_ID: ${{ secrets.APPLE_API_KEY_ID }}
+ APPLE_NOTARIZATION_ISSUER_ID: ${{ secrets.APPLE_API_ISSUER }}

.github/workflows/nightly.yml

@@ -20,15 +20,16 @@ jobs:
  runs-on: ubuntu-latest
  steps:
  - name: Checkout
- uses: actions/checkout@v6
+ uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # pinned from v6.0.2
  with:
  fetch-depth: 0
+ persist-credentials: false
 
  - name: Setup pnpm
- uses: pnpm/action-setup@v6
+ uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # pinned from v6
 
  - name: Setup Node.js
- uses: actions/setup-node@v7
+ uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # pinned from v7.0.0
  with:
  node-version-file: .nvmrc
  cache: "pnpm"
@@ -59,3 +60,74 @@ jobs:
  exit 0
  fi
  pnpm changeset publish --tag dev --no-git-tag
+
+ release-reconciliation:
+ name: Reconcile release lanes
+ if: github.repository_owner == 'PyModel'
+ runs-on: ubuntu-latest
+ permissions:
+ contents: read
+ issues: write
+ steps:
+ - name: Checkout
+ uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # pinned from v6.0.2
+ with:
+ persist-credentials: false
+
+ - name: Setup Node.js
+ uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # pinned from v7.0.0
+ with:
+ node-version-file: .nvmrc
+
+ - name: Check release lane status
+ id: release-status
+ env:
+ GITHUB_TOKEN: ${{ github.token }}
+ shell: bash
+ run: |
+ set +e
+ node scripts/release/release-status.mjs > "$RUNNER_TEMP/release-status.md" 2>&1
+ status=$?
+ set -e
+ cat "$RUNNER_TEMP/release-status.md" >> "$GITHUB_STEP_SUMMARY"
+ echo "exit_code=$status" >> "$GITHUB_OUTPUT"
+
+ - name: Sync release drift issue
+ env:
+ GH_TOKEN: ${{ github.token }}
+ RELEASE_STATUS: ${{ steps.release-status.outputs.exit_code }}
+ shell: bash
+ run: |
+ set -euo pipefail
+ title='Release lane drift detected'
+ mapfile -t issues < <(gh issue list \
+ --state open \
+ --limit 100 \
+ --json number,title \
+ --jq '.[] | select(.title == "Release lane drift detected") | .number')
+
+ if [ "$RELEASE_STATUS" = '0' ]; then
+ for issue in "${issues[@]}"; do
+ gh issue close "$issue" --comment 'Automated release reconciliation is clean.'
+ done
+ exit 0
+ fi
+
+ body="$RUNNER_TEMP/release-drift-issue.md"
+ {
+ echo 'The nightly release reconciliation found drift or could not query a release surface.'
+ echo
+ echo "Run: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
+ echo
+ cat "$RUNNER_TEMP/release-status.md"
+ } > "$body"
+
+ if [ "${#issues[@]}" -eq 0 ]; then
+ gh issue create --title "$title" --body-file "$body"
+ else
+ gh issue edit "${issues[0]}" --title "$title" --body-file "$body"
+ for issue in "${issues[@]:1}"; do
+ gh issue close "$issue" --comment "Duplicate of #${issues[0]}."
+ done
+ fi
+ exit "$RELEASE_STATUS"