feat(arena): ZTENSOR_ARENA_POISON poison-on-reset debug mode (ADR 006, T1.4)

[dndungu]

Implements ADR 006 decision 4 / zerfoo docs/plan-gpu-training-hardening.md T1.4. References #128 (leave open until the two GPU-deferred tests run green on the GB10).

What

ZTENSOR_ARENA_POISON=1 (read once at init; off by default, zero per-Alloc cost when unset) fills every arena region that becomes reusable with a NaN sentinel before it can be handed out again. A node that cached a forward intermediate in a struct field and reads it after reclamation — the zerfoo#842 LayerNorm variance / zerfoo#845 gradient buffer / Wolf QK-norm bug class — now explodes deterministically at the corruption site instead of surfacing as a delayed, non-deterministic training NaN.

Where the fills hook in

• internal/cuda/arena.goReset() — poisons the [resetFloor, offset) span under the arena lock before rewinding (covers GPUEngine.ResetPool / StepScope.Close; buffers below the MarkStepBoundary floor are never poisoned).
• internal/cuda/arena.goFreeArena() — poisons a freed block before it enters the free-list, so a stale read explodes even before reuse.
• Free-list reuse in Alloc() needs no third fill: reused blocks (and split remainders) already hold poison from the paths above; a comment documents this, keeping Alloc free of poison branches.
• MarkStepBoundary itself reclaims nothing (it only records the floor); the reclamation it shapes happens at the next Reset, which is hooked.

Fill mechanism

• Primary (on-device): the elementwise fill kernel, registered by internal/gpuapi/cuda_arena_poison.go at NewCUDAArenaPool via cuda.SetArenaPoisonFill (internal/cuda cannot import the kernels package — it would cycle). The NaN payload survives bit-exact: Float32frombits → floatBits is a pure bits round-trip and the kernel stores, never computes on, the value. Launched on the legacy default stream, which orders before subsequent work on the engine's blocking streams.
• Fallback (host-staged): when no kernel fill is registered, a 4 MiB host staging buffer is cudaMemcpy'd to the device in chunks — synchronous and slow (~one H2D copy per 4 MiB reclaimed per Reset), acceptable for a debug mode and documented in docs/design.md.

Pattern:0x7FF80000 repeated (dtype-agnostic bytes 00 00 F8 7F). Deliberate deviation from the plan's example 0x7FC00000: the canonical f32 qNaN repeated decodes to a large finite f64, while 0x7FF80000 (high half of the canonical f64 qNaN) is a quiet NaN for both f32 and 8-byte-aligned f64 reads; i32 reads see the sentinel 2146959360.

Capture interaction (ADR 004/005)

poisonRegion checks cuda.CaptureActive() and skips the fill with a logged warning while a CUDA graph capture is active — issuing fill kernels or synchronous copies mid-capture would be recorded into the graph or hang the GB10 driver. Regions recycled mid-capture are not poisoned (tested).

CI-tested vs GPU-deferred

CI-tested (CPU-only, host-backed arenas + mockable fill func, all green):

• TestArenaPoison_CachedBufferAfterReset — the mandated demo/regression test: a fake node caches a tensor backed by arena memory, the pool is reset, and the cached read is asserted NaN (bit-exact 0x7FF80000) under poison semantics and clean (1.5) without.
• TestArenaPoisonWord_Pattern — sentinel decodes to NaN as f32 and as repeated f64; byte fill reproduces the word at every phase.
• TestArenaPoison_FreeArenaPoisonsBlock — poison at free time, still present when the free-list hands the block back out.
• TestArenaPoison_ResetRespectsResetFloor — persistent buffers below the floor untouched.
• TestArenaPoison_SkippedDuringCapture — zero fills + exactly one warning during capture.
• TestArenaPoison_ZeroWorkWhenDisabled — no fill attempted on any reuse path when off.
• TestSetArenaPoisonFill_NilRestoresDefault — registration contract.

GPU-deferred (skip in CI via the existing cuda.Available() gating; to run on the GB10 via a Spark pod):

• internal/cuda/arena_poison_gpu_test.goTestArenaPoison_GPU_DefaultHostStagedFill — default host-staged fill against a real device arena (alloc → write → Reset → D2H readback asserts NaN).
• internal/gpuapi/cuda_arena_poison_gpu_test.goTestArenaPoisonKernelFill_GPU — kernel fill writes the exact bit pattern on-device.

Not yet verified on real hardware: this PR has not been run on the DGX (per task constraints); the two GPU tests above are the follow-up artifact for that run.

Docs

docs/design.md: new "Arena poison-on-reset debug mode" section plus a consolidated GPUEngine env-toggle list (ZERFOO_ARENA_SIZE_GB, ZERFOO_OVERFLOW_POOL_RETAIN_GB, ZERFOO_ENABLE_MANAGED_MEM, ZERFOO_DEBUG_GPU/ZERFOO_ARENA_PROFILE, ZTENSOR_ARENA_POISON). The engine also logs a warning at arena construction when the mode is active. ADR 006 (on docs/save-for-backward-adr) already names this mode as decision 4.

Gates

go build ./..., CI vet command, gofmt on all touched files, go test ./... -count=1 — all green locally (darwin/arm64, CUDA unavailable, GPU tests skip).