test: SwapSeam helper + advisory gremlins mutation workflow

[LukasWodka]

Summary

testutil.SwapSeam (generic seam swapper). New internal/testutil package (no shared test-support package existed) with one helper:

funcSwapSeam[Tany](t testing.TB, ptr*T, stubT)

Swaps the value behind a package-level seam variable and LIFO-restores it via t.Cleanup. Generic, so non-function seams (timeouts) work too. Comes with its own tests (100% coverage, including nested-swap restore ordering).

Converted blocks — honest count vs the ticket. The ticket estimated "7+ blocks repo-wide". Reality on today's develop: ~34 hand-rolled save/restore sites repo-wide, of which ~26 live in internal/cli (out of scope here — the WS-B decomposition PRs own that tree) and 8 elsewhere. Of the packages this PR is scoped to:

• internal/submit — 3 blocks, all converted (watch_test.go x2 finalJobStatusTimeout, run_watch_test.go x1 watchJobFn)
• internal/push — 0 blocks found (its tests don't use seam-var stubbing)

The remaining 5 out-of-scope non-cli blocks (internal/api x2, internal/helm x1, internal/nodeboot x1, internal/ui x1) are trivial follow-up candidates once this merges.

mutation.yml (advisory gremlins workflow). Formalizes the manual gremlins ritual that already produced #262/#263/#264:

• workflow_dispatchonly — never on PRs/pushes, never a merge gate
• one package per run via a validated package input (routed through env vars, not raw interpolation)
• advisory: no --threshold-* flags, so lived mutants never fail the job; the run summary lists LIVED/TIMED OUT survivors to triage and the JSON report is uploaded as an artifact
• timeout_coefficient input (default 3) because gremlins' default timeout misclassifies kills as TIMED OUT (reproduced locally)

CONTRIBUTING.md (existed already — extended, not created): a "Test seams" subsection documenting SwapSeam, and a "Mutation testing" section documenting the ritual — one package per run, how to read survivors, which ones are noise, and that real gaps become issues titled test(<pkg>): pin <behavior> (mutation survivor).

Test plan

Ran locally:

• go build ./..., go vet, gofmt -s — clean
• go test -race -cover ./internal/testutil/ ./internal/submit/ ./internal/push/ — green; submit/push coverage unchanged (80.4% / 89.0%), testutil 100%
• errcheck / ineffassign / misspell (CI's pinned versions) on the touched files — clean
• gremlins v0.5.0 exercised locally against internal/slug: default coefficient mistimes-out all mutants; --timeout-coefficient raised → 11 KILLED / 3 LIVED / 2 TIMED OUT, which is what informed the workflow's default input
• YAML parse check on mutation.yml

NOT run locally: the workflow itself on Actions (dispatch-only; can be triggered once merged to develop).

Fixes#295

🤖 Generated with Claude Code

---

Note

Low Risk
Test-only helper and documentation plus a non-gating CI workflow; no production code paths change.

Overview
Adds internal/testutil.SwapSeam, a generic helper that stubs package-level test seams and restores them via t.Cleanup (including LIFO nested swaps). internal/submit watch tests are updated to use it instead of hand-rolled save/stub/restore for watchJobFn and finalJobStatusTimeout.

Introduces a manual-only GitHub Actions workflow Mutation (gremlins) (workflow_dispatch, one package per run, validated inputs, pinned gremlins v0.5.0). Lived mutants do not fail the job; the step summary lists survivors and uploads JSON/log artifacts.

CONTRIBUTING.md gains Test seams (SwapSeam usage) and Mutation testing (ritual, triage, issue titling).

^{Reviewed by Cursor Bugbot for commit 23b3853. Bugbot is set up for automated code reviews on this repo. Configure here.}

[LukasWodka]

👋 Heads-up — Code review queue is at 44 / 30

Above the WIP limit. The team convention is to review existing PRs before opening new work.

Open PRs currently in Code review (oldest first):

• averaging-service#181 — feat(weights): normalize-on-read — mixed cycles average instead of rejecting (SEC-03 §8 step 2) · author: @shujaatTracebloc · no reviewer assigned
• averaging-service#182 — feat(weights): averaging writes SafeTensors (TF + PyTorch) — SEC-03 §8 step 4 · author: @shujaatTracebloc · no reviewer assigned
• backend#1079 — feat(global_meta): edge dataset_meta exposure + attributes contract at ingest (#924 G4a) · author: @divyasinghds · no reviewer assigned
• backend#1086 — docs(rfc): SafeTensors weight-format migration — SEC-03 Phase 1 (RFC 0004) · author: @shujaatTracebloc · no reviewer assigned
• backend#1093 — chore(deps): bump django from 5.2.14 to 5.2.15 · author: @dependabot · no reviewer assigned
• backend#1095 — feat(experiment): configurable preprocessing knobs incl. tabular scaler — RFC 0003 L1 + L1b (#1094) · author: @LukasWodka · no reviewer assigned
• backend#1100 — fix(boot): pin SDK install to tracebloc==0.11.2, drop 404 dev line (#1098) · author: @LukasWodka · reviewer: @saqlainsyed007
• backend#1105 — perf(api): query micro-fixes — notifications N+1, cached data_scientist, composite index, sampling (#975) · author: @aptracebloc · no reviewer assigned
• cli#266 — main - > enhance CLI features and tests · author: @saadqbal · no reviewer assigned
• cli#278 — fix(deps): toolchain go1.26.5 + x/net v0.57.0 — clear 6 reachable vulns; govulncheck CI gate · author: @LukasWodka · reviewer: @saadqbal

Pull from review before opening new work. (This is a nudge from the kanban WIP check, not a block.)

[LukasWodka]

@BugBot run

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 852eb8c. Configure here.}

[saadqbal]

👍 Clean — SwapSeam is a tidy generic over the hand-rolled save/stub/restore, with the right caveats documented (LIFO cleanup, not t.Parallel-safe, test-only). The mutation workflow is workflow_dispatch-only so it can't gate anything, and it input-validates the package/coefficient. Nice.

[saadqbal]

Status: approved, but 2 open Cursor Bugbot findings block merge (develop requires conversation resolution): Pipeline hides gremlins failures (mutation.yml:84) and Zero timeout coefficient accepted (mutation.yml:76). Address/resolve those and it's ready.

[saadqbal]

Re-approving — Bugbot fixes verified: set -o pipefail unmasks a gremlins/test failure through the | tee, and the coefficient validation now rejects zero/all-zeros ((*[1-9]*) gate). Both findings addressed. 👍