feat(parity): CPU-vs-GPU parity harness with arena-stress schedules (ADR 091, T1.2)

[dndungu]

Implements plan T1.2 (zerfoo docs/plan-gpu-training-hardening.md) -- harness #2 of ADR 091: the same op set run CPU-f32 vs GPU-f32, forward AND backward, under interleaved arena-stress schedules. Builds on the gradcheck OpInfo registry (#129), poison mode (#130), the oracle harness (#131), and the SaveForBackward contract + arena Pin/Unpin (#132). Refs #128.

Schedule semantics

phase F: opA.Forward, opB.Forward, ..., opZ.Forward
[reset point]
phase B: opZ.Backward, ..., opB.Backward, opA.Backward

• no-reset -- pure kernel parity, no allocator interference.
• reset-between-fwd-bwd -- the Wolf per-sample-ResetPool hazard: the candidate's arena is Reset() between phases. The runner acts as the graph executor: it wires a pinning Saver into every SaverAware node (Builder semantics) and releases pins after each node's Backward (releaseSaved semantics). Ops honoring the save-for-backward contract or recomputing from live inputs stay correct; raw struct-field caches of arena-backed intermediates read poisoned memory and are flagged as NaN diffs attributed to op + schedule. This is the schedule shape that exposed the zerfoo#842 LayerNorm/QK-norm cached-intermediate corruption class.

To make the registry ops contract-clean, the gradcheck op wrappers now implement graph.SaverAware: opNode saves its cached forward output, layerNormNode saves xhat+inv (no-op for gradcheck/oracle, which never wire a Saver).

Tolerance policy

f32-appropriate defaults: forward atol 1e-6 / rtol 1e-5, gradients rtol 1e-4. MatMul / Softmax / LayerNorm / reductions / Hadamard loosened to grad rtol 1e-3: CPU loops and cuBLAS/parallel kernels legitimately reduce in different orders at f32 -- expected reduction-order divergence is why per-op tolerances exist. Overridable per op (Op.Tol, toleranceOverrides).

CI vs GB10 split

CI (no GPU):

• StressEngine: a CPU engine whose every result is relocated into a host-backed cuda.ArenaPool (the poison-test / WolfHazard pattern), giving CPU tensors GPU lifetime semantics. CPU-vs-CPU+host-arena runs under both schedules with poison on -- the lifetime gate runs on every CI push (TestRun_HostArenaStress_RegistryGreen, with pin-accounting assertions).
• Runner determinism + diff plumbing proven at exact-zero diff (TestRun_CPUvsCPU_ExactParity).
• Both red proofs (below).

GB10 (Spark pod, scripts/parity/):

• TestParity_GPUvsCPU_ArenaStressSchedules_GPU (gated on cuda.Available(), skips cleanly in CI): full registry, both schedules, deliberately small 64 MiB arena via the new compute.SetArenaBytesForTesting hook (byte-granular, bypasses the 1 GB env-var minimum; preferred over env mutation), poison-on-reset enabled. Writes one oracle-style JSON report per schedule.
• TestParity_GPURedProof_GPU: the fixture pair re-proven against the real CUDA arena.
• scripts/parity/{run.sh,parity-pod.yaml,README.md} mirror scripts/oracle + the issue-118 Go-GPU-test pod conventions (golang:1.26-bookworm, CUDA hostPath mounts, base64-delivered script, exit-code-encoded verdict, SKIP = hard failure). Not run in this PR -- DGX execution is the follow-up acceptance step (S1.2.1 / S2.3.1 territory).

Red-proof story (CI-asserted)

• FixtureCachedIntermediateRaw: caches an engine-computed forward intermediate in a struct field WITHOUT SaveForBackward -- the pre-fix LayerNorm shape. Under reset-between-fwd-bwd + poison the harness flags it: gradient diff max_abs = +Inf (NaN sentinel), forward still green (snapshotted pre-reset), attributed to the op and schedule. Asserted red. Under no-reset it passes -- proving single-op-style runs cannot see this bug class.
• FixtureCachedIntermediateContract: identical op honoring the contract. Asserted green under both schedules.
• TestRun_RedProof_RealOpWithoutContractFlagged: a real registry op (Exp) with the contract stripped is flagged the same way -- sensitivity is not a fixture artifact.

Report format

JSON per (candidate, schedule), consistent with the oracle harness: per-op max_abs/max_rel for forward and every input/parameter gradient, passed/failed/errored totals, per-op parity PASS/FAIL log lines for pod-log scraping. oracle.DiffStats gained NaN/Inf-safe JSON marshaling (the poison verdict is +Inf, which strict JSON cannot carry as a number).

DGX run procedure

See scripts/parity/README.md: render parity-pod.yaml with the base64 of run.sh + a RUNID, submit via the Spark HTTP API, poll for Succeeded, collect /home/ndungu/parity/$RUNID/parity-gpu-<schedule>.json, delete the pod.

Gates

• go build ./... clean
• CI vet set clean; gofmt clean on all touched files
• go test ./... -race -count=1 fully green (GPU-gated tests skip cleanly without CUDA)