Uh oh!
There was an error while loading. Please reload this page.
fix: arena frees are reset-epoch guarded -- stale GC-finalizer frees dropped (Bug 11 residual) - #138
Merged
Merged
Conversation
…e a no-op Bug 11 residual (Wolf GB10 batch-3/4 corruption). An arena allocation's lifetime cannot extend across Reset: the rewind reclaims it wholesale and the bump allocator re-issues its bytes to new allocations. But GPUStorage frees are GC-finalizer-driven and can fire arbitrarily late -- the Wolf training loop's first big GC lands around batch 3-4 and releases thousands of dead pre-Reset storages. Each such FreeArena poison-filled (under ZTENSOR_ARENA_POISON=1, verify8: forward softmax scores all-NaN) or free-listed (without poison: double-issued block, silent aliasing -> gradient NaN in the AdamW guard, verify5/6/7) memory owned by LIVE current-epoch tensors. Contract: ArenaPool.Reset advances an epoch; pool-backed GPUStorage captures the epoch at allocation (gpuapi.EpochMemPool) and frees through FreeAtEpoch, which drops arena-range frees whose epoch has passed. The check, poison fill, and free-list insert run in one critical section with Reset's epoch increment, so no interleaving can slip a stale block in. Views propagate the allocation epoch; TrySet resize re-captures it and keeps allocSize in sync. Regression tests: stale free is dropped (no free-list double-issue, no poison of live data), same-epoch free still reuses, storage frees carry the allocation-time epoch through refcounted views.
Uh oh!
There was an error while loading. Please reload this page.
dndungu added a commit
that referenced
this pull request
Jun 12, 2026
…dst-form accumulation policy design.md documents the contracts shipped for the GPU training hardening plan in general terms: - Host-Access Synchronization: host reads/writes of device memory are stream-ordered via per-device registered sync hooks (tensor.RegisterHostAccessSync); the GPU engine registers a capture-guarded stream.Synchronize() (#137). - Arena Reset: pinning (ADR 006 reset floor / MarkStepBoundary) and reset epochs -- FreeAtEpoch drops cross-epoch frees, making GC-finalizer-driven frees safe across Reset (#138). - Engine dst-form accumulation policy: results write into dst's existing storage, never re-homed to the arena. - Graph features: Engine() accessor and the save-for-backward lifetime contract (ADR 006). devlog: end-to-end GB10 validation entry for #137/#138 (two clean runs, zero NaN, accuracy within 0.05pp of the CPU baseline).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
After the host-access stream-ordering fix (#137), Wolf CrossAsset GB10 f32
still corrupted: verify7 hit
adamw: NaN detected in gradient of layer0.ffnB1_biasesat batch 4 (was batch 3 pre-fix); verify8 underZTENSOR_ARENA_POISON=1localized it to the forward pass — softmaxattention scores all-NaN mid-batch-4, i.e. a forward tensor read NaN-filled
(reclaimed) arena memory.
Root cause
An arena allocation's lifetime cannot extend across
Reset: the rewindreclaims it wholesale and the bump allocator re-issues its bytes. But
GPUStoragefrees are GC-finalizer-driven and fire arbitrarily late.The Wolf loop resets per sample; its first big Go GC lands deterministically
around batch 3–4 (heap growth is deterministic — which is why every prior
fix shifted the failing batch by one) and releases thousands of dead
pre-Reset storages. Each stale
FreeArena:tensor (verify8's all-NaN softmax scores), and
a second live tensor — silent aliasing that surfaces as the deterministic
gradient NaN in the AdamW guard (verify5/6/7).
Fix (contract level)
ArenaPool.Resetadvances an epoch. Pool-backedGPUStoragecapturesthe epoch at allocation time (
gpuapi.EpochMemPool) and releases throughFreeAtEpoch, which drops arena-range frees whose epoch has passed — theReset already reclaimed them. The epoch check, poison fill, and free-list
insert run in one critical section with Reset's epoch increment, so a
concurrent Reset cannot interleave. Views propagate the allocation epoch;
TrySetresize re-captures it and keepsallocSizein sync (a stale,larger
allocSizewould free-list bytes of a neighboring live allocation).Fallback/async-overflow pointers are not Reset-reclaimed and free
unconditionally, as before.
Tests
internal/cuda/arena_epoch_test.go(host-backed arena, no GPU): stalecross-epoch free is dropped — no free-list double-issue, no poison fill over
live data; same-epoch free still enters the free-list and is reused.
tensor/gpu_storage_epoch_test.go: storage frees carry the allocation-timeepoch, including through refcounted views.
go build, CI-equivalentgo vet,go test -race ./...green locally.Bug 11 lineage: zerfoo#850, ztensor#134, ztensor#137.