feat(oracle): PyTorch-as-oracle per-op parity harness -- exchange format + NGC runner (ADR 091, T1.3)

[dndungu]

Implements T1.3 of the GPU training-stack hardening plan (zerfoo docs/plan-gpu-training-hardening.md, decision rationale zerfoo ADR 091): the PyTorch-as-oracle per-op parity harness. The same op, same inputs, through ztensor and through torch (in nvcr.io/nvidia/pytorch:26.02-py3 on the DGX GB10), diffing forward AND backward within per-op tolerances. Catches numerics-convention divergence (fast-math, reduction ordering, eps placement) that ztensor's CPU and GPU engines could share and gradcheck cannot see. Test infrastructure only — the production stack stays pure Go.

Refs #128. Builds on the merged gradcheck registry (#129) and ADR 006.

Bundle format (format_version 1)

One directory per op case (authoritative spec: testing/oracle/bundle.go):

bundles/
generation.json # written/skipped summary
<Op>/
manifest.json # op, torch_expr, dtype, seed, tolerances, tensor refs
input_0.bin ... # op inputs
param_<name>.bin # trainable parameter values (LayerNorm gamma/beta)
upstream.bin # upstream gradient dL/dy fed to Backward
forward.bin # ztensor forward output
grad_input_0.bin ... # ztensor input gradients
grad_param_<name>.bin # ztensor parameter gradients

Tensor files are raw little-endian IEEE-754, row-major, no header; the manifest records shape and dtype (float32/float64, numpy names). Pass criterion per element, identical in testing/oracle/diff.go and scripts/oracle/run_oracle.py: |ztensor − torch| ≤ atol + rtol·|torch|; any NaN fails. First cut records the f32 CPU engine; the GPU-engine variant regenerates bundles through the same format on the DGX.

To run the registry at f32 without duplicating op definitions, the gradcheck op wrappers are now generic over tensor.Float, and gradcheck.NewRegistryNode[T] is the single source of truth for constructor arguments (slopes, axes, reshape targets, layernorm width/eps). Registry() routes through it; no behavior change at float64. A CI test keeps the mapping table and the registry in lockstep.

Op → torch mapping (25 mapped, 1 skipped)

ztensor op	torch expression
Add / Sub / Mul / Div	x0 + x1 / x0 - x1 / x0 * x1 / x0 / x1
Pow	torch.pow(x0, x1)
Tanh / Sigmoid / ReLU	torch.tanh(x0) / torch.sigmoid(x0) / torch.relu(x0)
LeakyReLU	torch.nn.functional.leaky_relu(x0, 0.1)
Exp / Log / Sqrt / Rsqrt / Sin / Cos	torch.exp(x0) etc.
AddScalar / MulScalar	x0 + 0.7 / x0 * -1.3
MatMul	x0 @ x1
Transpose	x0.transpose(0, 1)
Reshape	x0.reshape(3, 2)
Softmax	torch.softmax(x0, 1)
ReduceSum / ReduceMean	x0.sum(dim=1, keepdim=True) / x0.mean(dim=1, keepdim=True)
ReduceMax	x0.amax(dim=1, keepdim=True) (no-ties sampling is load-bearing)
LayerNorm	torch.nn.functional.layer_norm(x0, (4,), weight=gamma.reshape(4), bias=beta.reshape(4), eps=1e-05)

SKIPPED:HadamardTransform — torch has no built-in normalized Walsh-Hadamard transform; hand-building the H matrix in the runner would test our own reimplementation rather than torch.

Red proof (CI-side, no torch dependency)

testing/oracle/redproof_test.go encodes the ztensor#125 fast-math tanh class: a Tanh bundle whose recorded "ztensor output" comes from an unsaturated tanh approximation (grows past |x| > 9 instead of clamping to ±1, like the raw cubic-GELU tanh-arg overflow). The Go reference checker — the same diff logic as the Python runner, with math.Tanh standing in for torch.tanh — must flag it red in both forward and backward (TestRedProofFastMathTanhFails), and the saturating twin must pass green (TestGreenProofTrueTanhPasses). NaN policy pinned by TestDiffNaNFails.

Verified locally (beyond CI)

End-to-end against real torch 2.11 on CPU: oracle-gen → run_oracle.py gives 25/25 PASS (max fwd diff 4.1e-6 on Exp, everything else ≤2.4e-7); corrupting one forward element makes the runner report FAIL and exit 1. Round-trip, endianness pin (f32 1.0 = 00 00 80 3F), determinism (two generations byte-identical), and full-suite go test -race ./... green.

DGX run procedure (the lead runs this serially — NOT run in this PR)

SPARK=http://192.168.86.250:8080
RUNID=$(git rev-parse --short=8 HEAD)
go run github.com/zerfoo/ztensor/testing/oracle/cmd/oracle-gen -out /tmp/oracle/$RUNID/bundles
rsync -av /tmp/oracle/$RUNID/bundles scripts/oracle/run_oracle.py ndungu@192.168.86.250:/home/ndungu/oracle/$RUNID/
curl -s $SPARK/api/v1/resources # GB10 jobs must serialize
sed "s/RUNID/$RUNID/g" scripts/oracle/oracle-pod.yaml | curl -sf -X POST $SPARK/api/v1/pods -H 'Content-Type: application/yaml' --data-binary @-
curl -s $SPARK/api/v1/pods/ztensor-oracle-$RUNID/logs # poll until Succeeded/Failed
rsync -av ndungu@192.168.86.250:/home/ndungu/oracle/$RUNID/report.json /tmp/oracle/$RUNID/
curl -s -X DELETE $SPARK/api/v1/pods/ztensor-oracle-$RUNID

Full procedure + GPU-engine next cut: scripts/oracle/README.md. This harness is the gate for the kernel-numerics work (plan T3.1–T3.4).