ci: publish to public PyPI on release + TestPyPI dry-run workflow

[uipreliga]

What

Adds public PyPI publishing alongside the existing private Azure Artifacts feed, using OIDC Trusted Publishing (no long-lived token stored).

release.yml (edited)

• Exposes the release job's version as a job output.
• Uploads the built wheel + sdist as a release-dist artifact.
• New environment-gated publish-pypi job: needs: release, only runs when a version was cut, on GitHub-hosted ubuntu-latest, with id-token: write scoped to just this job. Downloads the same artifact and publishes to pypi.org.
• The Azure Artifacts upload and GHCR image steps are unchanged.

publish-testpypi.yml (new)

• Manually-dispatched dry run of the publish path against test.pypi.org.
• Stamps a throwaway PEP 440 .dev<run#> version so repeated runs never collide; does not bump the version, tag, or touch main / the Azure feed.

Security

• OIDC Trusted Publishing — no PyPI token/secret stored in the repo.
• Publisher is tightly scoped: repo UiPath/coder_eval + workflow filename + environment (pypi / testpypi).
• id-token: write is isolated to the dedicated publish jobs (least privilege), not the broad release job.
• All third-party actions pinned to full commit SHAs; Dependabot's github-actions ecosystem keeps them current.

Required setup before this can publish

• pypi.org pending publisher → workflow release.yml, environment pypi ✅ (configured)
• GitHub environment pypi ✅ (created)
• test.pypi.org pending publisher → workflow publish-testpypi.yml, environment testpypi — needed only if using the TestPyPI dry run.

Testing

End-to-end OIDC can only be exercised once merged to main:

1. TestPyPI dry-run (safe, repeatable) — after setting up the test.pypi.org publisher, dispatch Publish to TestPyPI (dry run).
2. Real release — Actions → Release → patch/minor/major.

🤖 Generated with Claude Code

[github-actions]

Claude encountered an error after 0s —— View job

---

I'll analyze this and get back to you.

[github-actions]

Claude encountered an error after 39s —— View job

---

Code Review in Progress

Todo List:

• Read .github/code_review.md for review standards
• Read CLAUDE.md for project conventions
• Get full diff from origin/main
• Review .github/workflows/publish-testpypi.yml (new file)
• Review .github/workflows/release.yml modifications
• Check cross-file consistency
• Perform "what's missing" analysis
• Post comprehensive review

---

[github-actions]

Claude finished @uipreliga's task in 1m 29s —— View job

---

PR Review: PyPI Publishing CI/CD

I'm reviewing this PR that adds public PyPI publishing to the CI/CD pipeline.

Review Checklist:

• Read .github/code_review.md for review standards
• Read CLAUDE.md for project conventions
• Review git diff for all changes
• Read full context of changed files
• Review release.yml modifications
• Review new publish-testpypi.yml workflow
• Check security considerations (OIDC setup, permissions)
• Verify cross-file consistency
• Analyze what might be missing
• Provide comprehensive feedback

[uipreliga]

Review: coder_eval — pr:6

Scope: pr:6 · branch ci/pypi-publishing · 0b19b26 · 2026-07-09T09:45Z · workflow variant

Change class: complex — adds public-PyPI publishing via OIDC Trusted Publishing (new environment-gated publish-pypi job) plus a TestPyPI dry-run workflow, and moves the release job from the self-hosted uipath-ubuntu-latest pool to GitHub-hosted ubuntu-latest; job dependency/gating (needs + if), OIDC scoping, artifact hand-off, and dev-version stamping require correctness reasoning

coder_eval is in excellent health (9.8/10) with flawless type safety, security, architecture, and harness quality; every open finding is confined to the new CI/release workflows, where the only real risk is that a partially-failed public-PyPI publish is unrecoverable for that version, so the bottom line is: ship after making the PyPI publish idempotent and tightening a few release-doc/reproducibility nits.

Summary

Axis	Score	🔴	🟠	🟡	🔵	Top Issue
1. Code Quality & Style	9.7 / 10	0	0	0	3	Guard if: steps.release.outputs.version != '' copy-pasted across 9 steps in the release job
2. Type Safety	10 / 10	0	0	0	0	—
3. Test Health	9.9 / 10	0	0	0	1	setup-uv is pinned to a SHA in publish-testpypi.yml but floats as @v4 in release.yml (version divergence)
4. Security	10 / 10	0	0	0	0	—
5. Architecture & Design	10 / 10	0	0	0	0	—
6. Error Handling & Resilience	9.4 / 10	0	0	1	1	Real PyPI publish lacks skip-existing, making a partially-failed upload unrecoverable for that version
7. API Surface & Maintainability	9.4 / 10	0	0	1	1	Release workflow header comment omits the new public-PyPI publish target (operator-facing doc is stale)
8. Evaluation Harness Quality	10 / 10	0	0	0	0	—

Overall Score: 9.8 / 10 · Weakest Axis: Error Handling & Resilience at 9.4 / 10
Totals: 🔴 0 · 🟠 0 · 🟡 2 · 🔵 6 across 8 axes.

Blockers

None.

Non-blocking, but please consider before merge

1. [Axis 6] Real PyPI publish lacks skip-existing, making a partially-failed upload unrecoverable for that version (.github/workflows/release.yml:217) — The new publish-pypi job's Publish to PyPI step (lines 217-222) ends with packages-dir: dist/ and NO skip-existing, whereas the sibling TestPyPI publish (publish-testpypi.yml) deliberately sets skip-existing: true (# A re-run at the same dev version is a no-op rather than a hard fail.). Each workflow_dispatch of release.yml cuts a NEW version (semantic-release forces a bump), so the ONLY way to recover a failed PyPI publish for version X is GitHub's 'Re-run failed jobs' on publish-pypi. gh-action-pypi-publish uploads the wheel and sdist as separate files; if the first attempt uploads one file then dies (network blip mid-batch), the re-run hits PyPI's 'File already exists' and hard-fails permanently — you cannot delete files from PyPI nor re-dispatch the same version, so that version is left half-published and unrecoverable. Add skip-existing: true to the Publish to PyPI step to match TestPyPI and make job re-runs idempotent.
2. [Axis 7] Release workflow header comment omits the new public-PyPI publish target (operator-facing doc is stale) (.github/workflows/release.yml:4) — The header comment still enumerates only two publish destinations — line 3-4: # ...build, and publish to the / # UiPath \coder_eval` Azure Artifacts feed (wheel) and GHCR (agent image).— but this PR adds a whole newpublish-pypijob (lines 193-222) that pushes the same wheel+sdist to PUBLIC PyPI (an irreversible, publicly-visible action). An operator reading the header to understand a release's blast radius would not know it now publishes to public PyPI. Update the header (line 4) to add public PyPI, e.g.# UiPath `coder_eval` Azure Artifacts feed (wheel), public PyPI (OIDC Trusted Publishing), and GHCR (agent image).`

Nits

1. [Axis 1] Guard if: steps.release.outputs.version != '' copy-pasted across 9 steps in the release job (.github/workflows/release.yml:109) — The exact string if: steps.release.outputs.version != '' is repeated on 9 separate steps (lines 109, 122, 126, 133, 141, 158, 163, 167, 175). This is a DRY/readability smell: a reader must re-verify each step carries the guard, and dropping it from one of the docker steps would silently push an image on a no-version run. The PR itself already demonstrates the cleaner pattern one level down -- publish-pypi is a separate needs: release job gated ONCE with if: needs.release.outputs.version != '' (line ~213). Consider likewise splitting the wheel-publish + versioned-image build/push steps into a second needs: release job gated once, eliminating the 9 per-step guards. Same-file inconsistency: PyPI publish is a gated job, but the near-identical 'publish an artifact only on a real release' logic for the wheel/Docker image stays inline with repeated guards.
2. [Axis 1] Brittle dual-templated heredoc: shell ${DEV_VERSION} interpolated into an unquoted Python <<PY heredoc (.github/workflows/publish-testpypi.yml:76) — Line 67 opens the heredoc as python3 - <<PY (unquoted), so the shell expands ${DEV_VERSION} into the Python source before Python parses it. Line 76 then mixes both templating layers in one expression: p.write_text(re.sub(pat, f'{key} = "${DEV_VERSION}"', p.read_text(), count=1)) -- ${DEV_VERSION} is shell-substituted while {key} is a Python f-string field. It works only because DEV_VERSION is a controlled <current>.dev<run#> string; any shell-special or quote char injected via the version would corrupt the generated Python. Cleaner and robust: use a quoted heredoc <<'PY' and read the value inside Python via os.environ['DEV_VERSION'] (the step already sets RUN_NUMBER via env at line 61, so this is the established pattern) rather than string-splicing it into source.
3. [Axis 1] Near-duplicate build+publish step sequence across release.yml and publish-testpypi.yml must be kept in sync by hand (.github/workflows/publish-testpypi.yml:48) — The 'Checkout / Set up Python 3.13 / Install uv / uv build / pypa/gh-action-pypi-publish' sequence in publish-testpypi.yml (lines ~48-91) is a near-copy of the corresponding steps in release.yml, including the shared pypa/gh-action-pypi-publish SHA pin ed0c53931b1dc9bd32cbe73a98c7f6766f8a527e # v1.13.0 that now lives in two files and must be bumped in lockstep. GitHub Actions has limited step reuse, so this is a minor note rather than a defect, but a shared composite action (or at minimum a comment cross-referencing the two pin sites) would reduce the risk of the TestPyPI dry-run drifting from the real publish path it is meant to validate.
4. [Axis 3] setup-uv is pinned to a SHA in publish-testpypi.yml but floats as @v4 in release.yml (version divergence) (.github/workflows/publish-testpypi.yml:55) — The dry-run pins uses: astral-sh/setup-uv@38f3f104447c67c051c4a08e39b64a148898af3a # v4.2.0 (publish-testpypi.yml:55) while the real release floats uses: astral-sh/setup-uv@v4 (release.yml:82). Because uv build produces the artifact in both paths, a future @v4 uv release that regresses wheel/sdist metadata would break the real release while the pinned dry-run keeps passing -- the build tool is not held identical across the two paths, weakening the dry-run's value as a build smoke test. Pin both to the same SHA (or float both) so the dry-run reproduces the real build environment.
5. [Axis 6] New public-PyPI publish extends a non-atomic release: main is tagged/pushed (irreversible) before the wheel reaches PyPI (.github/workflows/release.yml:198) — The new publish-pypi job (lines 198-222) has needs: release, so it runs only AFTER the release job has already executed the irreversible git push origin main "v${{ steps.release.outputs.version }}" (line 123, pushed with the app token that bypasses branch protection) and after the Azure feed + GHCR image publishes. A failure in the new publish-pypi job therefore leaves main permanently tagged at v with the package on the Azure feed and GHCR but absent from public PyPI — a partial release with no automatic compensation/rollback. The push-before-publish ordering itself pre-exists on origin/main (not in this diff), but this PR adds public PyPI — the most user-visible artifact — as the LAST, separate-job step past that irreversible push, widening the dangling-release window. Total-failure of publish-pypi is recoverable by re-running the failed job (the release-dist artifact is retained), so severity is Low; document the manual recovery (re-run publish-pypi) or, longer term, move the git push of the tag to AFTER a successful publish of all artifacts (or use a staging tag promoted on success).
6. [Axis 7] Real-PyPI Trusted-Publisher one-time setup is undocumented in release.yml (asymmetric with publish-testpypi.yml) (.github/workflows/release.yml:220) — The publish-pypi step comment (lines 220-221) only asserts # Trusted Publisher is configured on pypi.org for this repo + / # workflow (release.yml) + environment (pypi); no password needed. — it gives no steps to (re)create that publisher. publish-testpypi.yml documents the full one-time pending-publisher config in its header (lines 12-19: Project name / Owner / Repository / Workflow name / Environment). Mirror that block in release.yml so the higher-stakes real-PyPI publisher can be reconstructed if the repo/env is ever recreated; right now the setup for the TEST instance is documented but the setup for the PRODUCTION instance is not.

What's Missing

Parallel paths:

• 🔵 The TestPyPI dry-run stamps the version with a hand-rolled Python regex over pyproject.toml + init.py (publish-testpypi.yml:67-77), whereas the real release writes the version via semantic-release (release.yml:89-106). So the dry-run validates the build + OIDC publish path but NOT the version-writing mechanism it exists to de-risk — a semantic-release version-write regression would pass the dry-run and only surface at real release time. (trigger: .github/workflows/publish-testpypi.yml)(restates: Axis 3: setup-uv pinned to SHA in publish-testpypi.yml but floats @v4 in release.yml)
• 🔵 The new dry-run job omits the SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS env that release.yml (53-58) and docker-publish.yml (50) both set — release.yml's own comment says it is 'Kept (matching pr-checks.yml) so parity is preserved if the job ever moves back to the self-hosted pool.' The new parallel publish path silently breaks that stated parity convention. (trigger: .github/workflows/publish-testpypi.yml)

Tests:

• 🔵 Nothing asserts the version-stamp substitution actually matched: re.sub(..., count=1) at publish-testpypi.yml:76 returns the text UNCHANGED (no error) if the regex misses, so a future pyproject.toml / init.py format drift would build an un-stamped wheel with no failure signal. Add a post-stamp check that the emitted version equals DEV_VERSION. (trigger: .github/workflows/publish-testpypi.yml)(restates: Axis 1: Brittle dual-templated heredoc splicing ${DEV_VERSION} into Python source)
• 🔵 No mechanical guard ties the OIDC Trusted-Publisher config to reality: the publisher is registered against a specific workflow filename + environment name (release.yml:204-206 -> env 'pypi'; publish-testpypi.yml:39-41 -> env 'testpypi'), but nothing (actionlint / a CEnnn-style check) fails if the workflow file is renamed or the environment key drifts — the publish would silently 403 only at real release time. Genuinely hard to fully statically verify (the publisher lives on pypi.org), so at minimum note this coupling. (trigger: .github/workflows/release.yml)

Daily/nightly:

• 🟡 The PR adds public-PyPI publishing (an irreversible, publicly-visible artifact + a claimed public 'coder-eval' name) but states no cross-repo blast radius: whether the downstream coder-eval-uipath / eval-runner consumers should now install from public PyPI vs. the private Azure Artifacts feed the release job still populates, and what the nightly/pipeline install source becomes. The unstated blast radius on the cross-repo install contract is itself the gap. (trigger: .github/workflows/release.yml)

Harness & Lint Improvements

Static checks (lint / type):

• [ce-lint] Add a CE025 workflow-pinning rule. This requires a small framework extension: the ce-lint runner currently only rglob's *.py and ast.parses them, so teach check_paths to also load .github/workflows/*.yml via pyyaml and dispatch a YAML-node rule pass. CE025 then enforces two invariants over every uses: step: (a) every third-party action (anything not actions/* first-party, at reviewer discretion) must pin a full 40-char commit SHA — rejecting astral-sh/setup-uv@v4; and (b) any action referenced in more than one place must resolve to a single SHA repo-wide. This runs in make lint/CI and fails before human review. Prevents: Finding 4 (setup-uv pinned to a SHA in publish-testpypi.yml:55 but floating @v4 in release.yml:82 — the @v4 float fails invariant (a)) and Finding 3 (the duplicated pypa/gh-action-pypi-publish SHA pin living in two files that must be bumped in lockstep — a future drift fails invariant (b)).
• [ce-lint] In the same CE025/CE026 workflow YAML pass, add a rule requiring every step whose uses: is pypa/gh-action-pypi-publish to explicitly set a with.skip-existing key (any value). Forcing the key to be present makes the author consciously choose idempotent-vs-strict upload behavior for each publish step rather than silently inheriting the non-idempotent default. Prevents: Finding 5 (release.yml:217-222 Publish to PyPI omits skip-existing, leaving a partially-failed upload of version X permanently unrecoverable, while the sibling TestPyPI step deliberately sets skip-existing: true). The rule surfaces the omission at lint time.

Harness improvements (not statically reachable):

• Add actionlint (which bundles shellcheck for run: blocks) to make lint / CI so the GitHub Actions workflows and their embedded shell/heredoc scripts are linted alongside the Python tree. Why not static: The repo's four static gates (ce-lint AST over *.py, ruff, pyright, bandit) are Python-only and never parse .github/workflows/*.yml or the bash/Python heredocs embedded in run: steps, so none can see a shell variable being spliced into script source. actionlint+shellcheck is a shell-aware analyzer the repo does not currently run. Prevents: Finding 2 (publish-testpypi.yml:67-76 opens an unquoted python3 - <<PY heredoc and interpolates the shell ${DEV_VERSION} directly into the generated Python f'{key} = "${DEV_VERSION}"' — shellcheck-class heredoc/quoting analysis flags this dual-templating splice).
• Consider adding a dedicated GitHub Actions auditor (e.g. zizmor) to CI for broader supply-chain hardening beyond the two CE025 invariants — its unpinned-uses check independently corroborates the SHA-pinning rule and it audits workflow-permission / injection surfaces the custom CE rule does not cover. Why not static: No workflow-aware static tool exists in the repo's gates today; this is a new CI dependency rather than a tweak to an existing check, and it targets YAML the Python-only gates cannot read. Listed as optional to avoid redundancy with CE025's pinning invariant. Prevents: Reinforces Finding 4 (unpinned astral-sh/setup-uv@v4) and provides defense-in-depth for future workflow-security regressions the point fixes do not address.

Top 5 Priority Actions

1. Add skip-existing: true to the Publish to PyPI step (release.yml:217) so a mid-batch upload failure is recoverable via job re-run instead of leaving that semantic-release-bumped version half-published and permanently stuck — highest-impact resilience fix.
2. Make the release atomic-ish by moving the irreversible git push origin main/tag (release.yml:123) to after all artifact publishes succeed, or at minimum document that recovery from a failed publish-pypi job (release.yml:198) is a re-run of that job, closing the dangling-release window this PR widens by adding public PyPI as the last step.
3. Pin astral-sh/setup-uv to the same SHA in both paths — the dry-run pins @…af3a # v4.2.0 (publish-testpypi.yml:55) while the real release floats @v4 (release.yml:82) — so the TestPyPI dry-run actually reproduces the real build environment it exists to validate.
4. Update the release.yml header comment (line 4) to add the new public-PyPI (OIDC Trusted Publishing) target so operators can gauge a release's true blast radius.
5. Mirror publish-testpypi.yml's one-time Trusted-Publisher setup block into release.yml (near line 220) and de-duplicate the copy-pasted per-step guards / shared pypa-publish SHA pin so the higher-stakes production publisher is documented and the two workflows can't silently drift.

---

Stats: 0 🔴 · 0 🟠 · 2 🟡 · 6 🔵 across 8 axes reviewed.

[uipreliga]

Review: coder_eval — pr:6

Scope: pr:6 · branch ci/pypi-publishing · 0b19b26 · 2026-07-09T09:44Z · workflow variant

Change class: complex — introduces a new public-PyPI OIDC Trusted-Publishing path (new gated publish-pypi job + TestPyPI dry-run workflow with a regex version-stamping script) and moves the release job off the self-hosted runner; correctness/security requires reasoning about permissions, OIDC scoping, and supply-chain pinning

coder_eval's application code is exemplary (perfect Type Safety and Architecture, 9.8/10 overall) and every confirmed finding is confined to the newly-added release/PyPI CI workflows — a widened supply-chain blast radius plus a handful of publish-path fidelity gaps — so the bottom line is a healthy codebase whose only real risks live in the tag-pinned, non-atomic release pipeline that now reaches public PyPI.

Summary

Axis	Score	🔴	🟠	🟡	🔵	Top Issue
1. Code Quality & Style	9.9 / 10	0	0	0	1	SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS is dead/no-op config after the move to ubuntu-latest
2. Type Safety	10 / 10	0	0	0	0	—
3. Test Health	9.9 / 10	0	0	0	1	TestPyPI dry-run diverges from the real publish path (e.g. setup-uv pin), so a green dry-run does not faithfully rehearse / de-risk the real PyPI publish
4. Security	9.4 / 10	0	0	1	1	Public-PyPI wheel is built in the release job through a tag-pinned (mutable) setup-uv and docker actions — widened supply-chain blast radius
5. Architecture & Design	10 / 10	0	0	0	0	—
6. Error Handling & Resilience	9.4 / 10	0	0	1	1	TestPyPI version-stamp uses re.sub (not re.subn+count check), so a version-line format drift silently no-ops under set -euo pipefail (publish-testpypi.yml:76; set -euo is line 63, not 62)
7. API Surface & Maintainability	9.9 / 10	0	0	0	1	release.yml top-of-file header is stale after adding the public-PyPI publish path (still lists only Azure feed + GHCR)
8. Evaluation Harness Quality	9.9 / 10	0	0	0	1	Public-PyPI publish is coupled into the internal multi-registry release job (needs: release), making the release non-atomic and entangled with Azure/GHCR steps

Overall Score: 9.8 / 10 · Weakest Axis: Security at 9.4 / 10
Totals: 🔴 0 · 🟠 0 · 🟡 2 · 🔵 6 across 8 axes.

Blockers

None.

Non-blocking, but please consider before merge

1. [Axis 4] Public-PyPI wheel is built in the release job through a tag-pinned (mutable) setup-uv and docker actions — widened supply-chain blast radius (.github/workflows/release.yml:82) — The PR correctly SHA-pins every NEW action, but the wheel/sdist that is now published to PUBLIC PyPI is built by uv build inside the release job, which installs uv via an unpinned mutable tag: line 82 uses: astral-sh/setup-uv@v4 (contrast publish-testpypi.yml:55 which pins the SAME action to astral-sh/setup-uv@38f3f104447c67c051c4a08e39b64a148898af3a). The docker steps are likewise tag-pinned: line 164 uses: docker/setup-buildx-action@v3, line 168 uses: docker/login-action@v3, line 176 uses: docker/build-push-action@v6. Before this PR a compromise of setup-uv@v4 (repointed tag / upstream account takeover) could only poison the private Azure feed + GHCR; this PR uploads that same dist/ as release-dist (line 134) and publishes it to public PyPI (publish-pypi job), so a tampered build now reaches every pip install coder-eval user — and the same job holds RELEASE_APP_PRIVATE_KEY (line 66, bypasses main branch protection) and the Azure PAT. Pin all four to full 40-char commit SHAs (matching the file's own convention # vX.Y.Z comment) to close the widened blast radius. CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:C/C:N/I:H/A:N
2. [Axis 6] TestPyPI version-stamp uses re.sub (not re.subn+count check), so a version-line format drift silently no-ops under set -euo pipefail (publish-testpypi.yml:76; set -euo is line 63, not 62) (.github/workflows/publish-testpypi.yml:76) — Line 76 stamps the dev version with p.write_text(re.sub(pat, f'{key} = "${DEV_VERSION}"', p.read_text(), count=1)). re.sub returns the input unchanged (NO exception) when pat does not match, so set -euo pipefail at line 62 cannot catch a missed stamp. I verified empirically: on the current double-quoted line version = "0.8.2" the substitution count is 1 (works today), but on any format drift (e.g. single quotes version = '0.8.2', or a move to a dynamic/attr version) the count is 0 and the text is returned untouched with rc=0. A silent miss means the wheel/sdist carries the BASE version instead of <base>.dev<run#>, which then collides with an already-uploaded file on TestPyPI — a collision masked by skip-existing: true (line 92) — so the whole point of the dry run (validating the dev-version + Trusted-Publishing path) passes green while having uploaded nothing / the wrong version. Fix: use re.subn(...) and fail loud when the count is 0 for either file, e.g. new, n = re.subn(pat, ...); assert n == 1, f'version pattern did not match {path}'; p.write_text(new) so a format drift aborts the step instead of publishing a stale/colliding artifact.

Nits

1. [Axis 1] SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS is dead/no-op config after the move to ubuntu-latest (.github/workflows/release.yml:58) — Line 58 SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: "openai-codex-cli-bin,openai-codex" is, per its own comment (lines 54-57: ... on GitHub-hosted runners this is a no-op. Kept (matching pr-checks.yml) so parity is preserved if the job ever moves back to the self-hosted pool.), a no-op now that runs-on: ubuntu-latest. It is retained purely as a speculative hedge against a hypothetical future move back to uipath-ubuntu-latest (YAGNI). It is dead config on the job today; consider removing it (git history + pr-checks.yml preserve the pattern) or leaving only if the parity hedge is genuinely load-bearing. Informational nit — the comment already prevents it from misleading.
2. [Axis 3] TestPyPI dry-run diverges from the real publish path (e.g. setup-uv pin), so a green dry-run does not faithfully rehearse / de-risk the real PyPI publish (.github/workflows/publish-testpypi.yml:92) — publish-testpypi.yml is the pre-release test harness for release.yml's publish-pypi job, but three divergences mean a green dry-run does not exercise the exact real path: (1) skip-existing: true (publish-testpypi.yml:92, # A re-run at the same dev version is a no-op rather than a hard fail.) is ABSENT from the real publish-pypi step (release.yml:222 has only packages-dir: dist/); if the inline regex stamp ever silently no-ops (leaving version = "0.8.2") and that base version is already on TestPyPI, the upload is silently skipped and the job still reports green — a false pass that proves no publish occurred, whereas the real path would hard-fail on the duplicate. (2) setup-uv is SHA-pinned to astral-sh/setup-uv@38f3f104447c67c051c4a08e39b64a148898af3a # v4.2.0 (publish-testpypi.yml:55) while the release build job floats astral-sh/setup-uv@v4 (release.yml:82), so the dry-run builds with a different, pinned-older uv than the real build. (3) The dry-run builds and publishes in a single job (uv build at :81 → publish at :84), while the real path builds in release, hands off via actions/upload-artifact (release.yml:134) and publishes from a separate publish-pypi job via actions/download-artifact (release.yml:212) — so the artifact upload/download handoff the real publish depends on is never exercised by the dry-run. Severity is Low because the OIDC-critical publish step is identical (same pypa/gh-action-pypi-publish@ed0c... # v1.13.0 SHA and trusted-publishing shape at publish-testpypi.yml:84 and release.yml:218) and the failure mode is visible and recoverable (tag/Azure feed/docker all succeed first; only the public-PyPI job fails). To close the gap: drop or justify skip-existing, add a post-publish assertion that a file was uploaded, align the setup-uv ref with the release job, and/or split the dry-run into a build job + download-artifact publish job mirroring release.yml.
3. [Axis 4] TestPyPI version-stamp uses an unquoted heredoc with a shell-expanded value inlined into the Python source (.github/workflows/publish-testpypi.yml:67) — Line 67 opens the heredoc with an UNQUOTED delimiter python3 - <<PY, so the shell expands ${DEV_VERSION} into the python source text before python runs (line 76: p.write_text(re.sub(pat, f'{key} = "${DEV_VERSION}"', p.read_text(), count=1))). DEV_VERSION="${CURRENT_VERSION}.dev${RUN_NUMBER}" (line 66) is built from github.run_number (integer, passed safely via the RUN_NUMBER env var — good) and CURRENT_VERSION read from pyproject.toml. run_number is not attacker-controlled, so this is not exploitable in the normal flow, but the pattern is fragile: a " or newline in the pyproject version string would break out of the f-string and inject arbitrary python. Harden by quoting the delimiter (<<'PY') and reading the value inside python via os.environ["DEV_VERSION"] instead of shell-expanding it into the source, so no repo-content value is ever spliced into executable text. CVSS:3.1/AV:N/AC:H/PR:H/UI:N/S:U/C:N/I:L/A:N
4. [Axis 6] Real publish-pypi job omits skip-existing (present on the TestPyPI sibling), making a partial-upload release version unrecoverable via re-run (.github/workflows/release.yml:217) — The Publish to PyPI step (lines 217-222) passes only packages-dir: dist/ with no skip-existing, whereas the parallel dry-run job sets skip-existing: true (publish-testpypi.yml line 92). gh-action-pypi-publish uploads the wheel and sdist as separate files; if one uploads and the other fails transiently (network/PyPI blip), a GitHub 're-run failed jobs' re-executes this step and twine hard-fails with 400 (file already exists) on the already-uploaded file — so that release version can never be completed on public PyPI without cutting a brand-new version bump. The normal (nothing-uploaded) failure re-runs fine, so this only bites on a partial upload, but the inconsistency with the TestPyPI job is real. Note skip-existing: true does NOT overwrite an existing file (it skips it), so it preserves the 'never silently overwrite a real release' intent while making partial-upload retries recoverable; adding it here is the safe, consistent choice. If the hard-fail-on-any-existing behavior is intentional, add a comment stating why the two publish jobs diverge.
5. [Axis 7] release.yml top-of-file header is stale after adding the public-PyPI publish path (still lists only Azure feed + GHCR) (.github/workflows/release.yml:4) — This PR adds a new publish-pypi job (lines 193-222) that publishes the wheel+sdist to public PyPI, but the workflow's own summary header at lines 3-4 still reads # Manually-triggered release: bump the version, tag, build, and publish to the / # UiPath \coder_eval` Azure Artifacts feed (wheel) and GHCR (agent image).— it never mentions public PyPI, so a maintainer reading the file header gets an incomplete picture of where a release now goes. Separately, the new job documents its trusted-publisher requirement only as a 2-line inline comment (lines 220-221:# Trusted Publisher is configured on pypi.org for this repo +/# workflow (release.yml) + environment (pypi); no password needed.), whereas the sibling publish-testpypi.yml carries a full one-time-setup block enumerating Project name / Owner / Repository / Workflow name / Environment. The higher-stakes production path is thus less documented than the dry run. Fix: add public PyPI (pypi.org, OIDC Trusted Publishing)to the line-3/4 header summary, and add an operator setup block for the real publisher (Project name: coder-eval, Owner: UiPath, Repository: coder_eval, Workflow name: release.yml, Environment: pypi) mirroring publish-testpypi.yml. Note: the setup values that ARE stated (workflow release.yml, environment pypi) correctly match the workflow'senvironment.name: pypi` and filename, so this is a completeness gap, not a mismatch.
6. [Axis 8] Public-PyPI publish is coupled into the internal multi-registry release job (needs: release), making the release non-atomic and entangled with Azure/GHCR steps (.github/workflows/release.yml:198) — publish-pypi (line 198) runs as a separate needs: release job that fires only AFTER the release job has already pushed the tag, uploaded the wheel to the Azure coder_eval feed (line 140 Publish to coder_eval feed), and pushed the versioned GHCR image (line 174 Build and push versioned agent image). If publish-pypi's Publish to PyPI step (line 217) then fails (OIDC/Trusted-Publisher misconfig, PyPI outage), the release is partial: v<version> exists on the Azure feed and GHCR but not on public PyPI, with no rollback (Azure/GHCR are already published and PyPI artifacts cannot be recalled). This is inherent to adding a second public registry and is recoverable by re-running only the failed publish-pypi job (the built dist is retained as the release-dist artifact, so the same wheel republishes without a rebuild), which keeps severity Low. To reduce the window, consider gating the Azure feed publish behind the same environment/OIDC readiness as PyPI, or documenting the 're-run publish-pypi job' recovery step in the workflow header. Note also that unlike TestPyPI (skip-existing: true), the real PyPI step has no skip-existing, so a re-run after a partial upload that actually reached PyPI will surface a 400 'file already exists' — acceptable/intended for the real index but worth being aware of during recovery.

What's Missing

Parallel paths:

• 🟡 publish-testpypi.yml SHA-pins astral-sh/setup-uv (@38f3f10... v4.2.0) but the release.yml build step that produces the wheel actually shipped to public PyPI floats astral-sh/setup-uv@v4 (line 82) — the two supposedly-parallel install paths diverge, and the mutable one is on the production side. (trigger: .github/workflows/release.yml)(restates: Axis 4: tag-pinned setup-uv/docker widens supply-chain blast radius)
• 🔵 The dry-run builds and publishes in one job, while the real path splits build (release job, upload-artifact release-dist at line 134) from publish (publish-pypi job, download-artifact at line 212) — the artifact upload/download handoff the real publish depends on is a parallel path the TestPyPI rehearsal never exercises. (trigger: .github/workflows/publish-testpypi.yml)(restates: Axis 3: TestPyPI dry-run diverges from the real publish path)

Tests:

• 🔵 Both new/changed production workflows ship with no actionlint (or any YAML/shell) CI gate — the heredoc python, the regex version-stamp, and the OIDC job wiring are un-linted, so malformed workflow syntax or shell issues surface only at dispatch time against a real index; an actionlint job in make verify/CI would push this class of defect left. (trigger: .github/workflows/publish-testpypi.yml)
• 🟡 No guard asserts the dev-version stamp regex actually matches the current pyproject.toml / init.py line format; a quote/format drift makes re.sub a silent no-op (rc=0) under set -euo pipefail, so a green dry-run can prove no correct publish occurred — a re.subn+count check is the missing mechanical test. (trigger: .github/workflows/publish-testpypi.yml)(restates: Axis 6: re.sub version-stamp silently no-ops on format drift)

Display & mapping dicts:

• 🔵 release.yml's top-of-file summary header (lines 3-4) still enumerates only the Azure feed + GHCR as publish targets and was not extended when the new public-PyPI target was added — the 'what this workflow publishes' map falls out of date. (trigger: .github/workflows/release.yml)(restates: Axis 7: release.yml header stale after adding public-PyPI path)
• 🔵 The publish-pypi job carries only a 2-line inline trusted-publisher comment, while the sibling publish-testpypi.yml carries a full one-time pending-publisher setup block (Project/Owner/Repository/Workflow/Environment); the operator setup 'map' was not extended for the higher-stakes production publisher. (trigger: .github/workflows/release.yml)(restates: Axis 7: release.yml header stale after adding public-PyPI path)

Daily/nightly:

• 🔵 The PR does not state the cross-repo/distribution blast radius: coder-eval now resolves from public PyPI for any consumer installing off the default index (e.g. docs/tutorials/02-ci-pipeline.md 'uv pip install --system coder-eval', plus external eval-runner / coder-eval-uipath pipelines). Publishing the same version to BOTH the private Azure feed and public PyPI makes index precedence/coexistence load-bearing and unstated. (trigger: .github/workflows/release.yml)
• 🔵 The release path is now non-atomic across three registries (Azure feed + GHCR + public PyPI): a publish-pypi failure leaves v live on Azure/GHCR but absent from PyPI with no rollback — a production-release blast radius introduced by adding the second public registry, recoverable only by re-running the failed job. (trigger: .github/workflows/release.yml)(restates: Axis 8: PyPI publish coupled into the internal multi-registry release job)

Harness & Lint Improvements

Static checks (lint / type):

• [bandit-codeql] Wire a GitHub Actions security linter into CI/make lint — CodeQL's Actions query pack (actions/security-and-quality, which ships an unpinned-tag query) or zizmor (its unpinned-uses audit) — to fail the build on any uses: that references a mutable tag (@vN / branch) instead of a full 40-char commit SHA. This is the load-bearing gate: it turns 'SHA-pin every action' from a reviewer's memory into a mechanical check that fires before merge. Prevents: MEDIUM finding 4 (release.yml:82 setup-uv@v4, :164/:168/:176 docker actions on the now-public-PyPI build path) and the setup-uv-pin sub-point of finding 2 (publish-testpypi.yml:55 pins the identical action, release.yml:82 floats it).
• [bandit-codeql] Enable the template/expression-injection audit in the same Actions linter (zizmor template-injection, or CodeQL's Actions injection queries) to flag run: scripts that splice untrusted or repo-content values into an interpreter — including an unquoted heredoc delimiter (<<PY) whose body inlines a shell ${VAR} into inline Python. The safe form is a quoted delimiter (<<'PY') + reading via os.environ. Prevents: LOW security finding 5 (publish-testpypi.yml:67 unquoted python3 - <<PY inlining ${DEV_VERSION} into the Python source at :76).
• [ce-lint] New CE025 'workflow actions must be SHA-pinned' as an in-repo alternative/complement to the CodeQL route: extend tests/lint/runner.py (currently Python-AST-only — check_paths rglobs *.py, check_file runs ast.parse) to also feed .github/workflows/*.yml to a new text/regex rule family, then have CE025 flag any uses: …@<ref> where <ref> is not 40 hex chars. Slots into tests/lint/rules/ce025_workflow_actions_sha_pinned.py + ALL_RULES in tests/lint/runner.py (next free CE number after CE024). Note the framework extension is real work — the runner has no YAML/text-rule path today. Prevents: Same class as the two bandit-codeql items — finding 4 (release.yml:82/164/168/176) and finding 2's unpinned-setup-uv divergence — kept entirely inside make verify if a third-party actions linter isn't desired.

Harness improvements (not statically reachable):

• Collapse the TestPyPI dry-run and the real publish onto ONE shared build+publish seam so drift is impossible by construction: extract the uv build step and the pypa/gh-action-pypi-publish step into a reusable workflow (.github/workflows/_build-and-publish.yml) or composite action that both release.yml (publish-pypi) and publish-testpypi.yml call with only the index URL / skip-existing differing. This makes the dry-run rehearse the exact real path — same setup-uv ref, same build→upload-artifact→download-artifact→publish handoff — instead of a single-job shortcut. Why not static: A static check can grep for an unpinned tag but cannot verify that two separate workflows exercise the same runtime path (artifact handoff, job graph, publish shape) — that's execution-semantics parity, not a textual pattern. Prevents: Finding 2 (dry-run diverges from real publish: skip-existing, setup-uv pin, single-job vs build→artifact→publish) and finding 8 (publish-pypi coupled into the internal release job) — the reusable seam is the single source of truth for the OIDC publish.
• Add a post-stamp verification step to publish-testpypi.yml (and any real path that stamps a version): after rewriting pyproject.toml/init.py, assert the substitution actually occurred — e.g. re.subn(pat, …, count=1) returning n == 1 per file, or a grep-back that the file now contains <base>.dev<run#> — and fail the job loudly if not, instead of re.sub silently returning the text unchanged under set -euo pipefail. Why not static: Whether the version-line regex matches depends on the runtime content of pyproject.toml (format drift like single-quotes or a dynamic version), so the miss is only observable at execution time; a lint rule broadly banning re.sub in stamping scripts would be too false-positive-prone to be the gate. Prevents: MEDIUM finding 6 (re.sub no-ops on version-line format drift, masked green by skip-existing) and, indirectly, the skip-existing masking called out in findings 2 and 7.
• Reconcile the two publish steps' skip-existing behavior deliberately rather than by accident: either add skip-existing: true to release.yml:217 (so a partial wheel/sdist upload is re-runnable) or add an inline comment on both steps stating why the real index intentionally hard-fails on duplicates while the dry run skips — and, if kept divergent, add a one-line release-recovery note ('re-run only the publish-pypi job; the built dist is retained as the release-dist artifact') to the workflow header. Why not static: This is a semantic parity + operability decision between two jobs and a documentation choice; there is no textual pattern that distinguishes an intentional divergence from a forgotten one. Prevents: LOW finding 7 (real publish-pypi omits skip-existing → partial-upload release unrecoverable via re-run) and the operability half of finding 9 (non-atomic release recovery).

Top 5 Priority Actions

1. Pin the four mutable tag-referenced actions in release.yml to full 40-char commit SHAs — astral-sh/setup-uv@v4 (release.yml:82), docker/setup-buildx-action@v3 (:164), docker/login-action@v3 (:168), docker/build-push-action@v6 (:176) — since this job now builds the wheel published to public PyPI while holding RELEASE_APP_PRIVATE_KEY and the Azure PAT, giving a repointed upstream tag a path to every pip install coder-eval user.
2. Replace re.sub(..., count=1) with re.subn(...) plus an assert n == 1 per file at publish-testpypi.yml:76 so a version-line format drift aborts loudly instead of silently shipping a stale/colliding base-version artifact that skip-existing then masks as a green dry-run.
3. Quote the heredoc delimiter (<<'PY') at publish-testpypi.yml:67 and read the value via os.environ["DEV_VERSION"] instead of shell-splicing ${DEV_VERSION} into the Python source, removing the injection-prone pattern where a stray quote/newline in the version string could execute arbitrary code.
4. Add skip-existing: true (or a comment justifying its absence) to the real Publish to PyPI step at release.yml:217 to match the TestPyPI sibling, so a partial wheel/sdist upload can be recovered by a job re-run rather than forcing a brand-new version bump.
5. Close the remaining release-fidelity/documentation gaps — align the dry-run's setup-uv pin and build-then-publish artifact handoff with the real path (publish-testpypi.yml:55/81-84), and refresh the stale release.yml header (lines 3-4) plus add a full trusted-publisher setup block for the public-PyPI job (release.yml:193-222).

---

Stats: 0 🔴 · 0 🟠 · 2 🟡 · 6 🔵 across 8 axes reviewed.