Skip to content

fix(distill): train from in-memory labels; never re-read the just-written file - #25

Merged
ronaldtse merged 2 commits into
mainfrom
fix/seq-labels-tear
Aug 22, 2026
Merged

fix(distill): train from in-memory labels; never re-read the just-written file#25
ronaldtse merged 2 commits into
mainfrom
fix/seq-labels-tear

Conversation

@ronaldtse

Copy link
Copy Markdown
Contributor

Summary

  • Arabic tiny labeled all 11,790 srcs, then the post-labeling re-read of teacher_labels.jsonl inside the same container served a stale volume replica (2 visible pairs of 11,790 written; the persisted file is healthy at 27,324 valid lines across runs)
  • the spec lacked labels_complete so no guard fired: the run trained on 2 pairs and saved garbage weights
  • fix: keep the freshly generated labels in memory and train from them directly; the file is read only on resume runs; the torn-view guard (<50% pairs) is now unconditional; final volume commit after the label loop (the 3200-modulo could skip the tail)

Test plan

  • CI green
  • ara-diac-tiny relaunch logs 'trainable label pairs: ~11790' and trains

Ronald Tse added 2 commits August 22, 2026 13:05
…tten file
Arabic tiny labeled all 11,790 srcs, then the post-label re-read of
teacher_labels.jsonl served a stale volume replica (2 visible pairs)
and the run trained on 2 pairs, saving garbage. The just-generated
labels are now kept in memory and used directly; the file is only read
on resume runs. The torn-view guard is unconditional, and the label
loop gets a final volume commit (the modulo could skip the tail).
The follow-on hole: when the Step-1 resume-read declares everything
done, Step 2 re-reads the file — that read can itself serve a stale
replica (6 valid pairs of 27,324 persisted), and raising there makes
the watchdog relaunch into the identical state forever. On a torn
read the run now regenerates all labels in-container and trains from
the in-memory rows. Labeling extracted into a label_all() closure
shared by the fresh/resume/regenerate paths.
@ronaldtse
ronaldtse merged commit e82f6d5 into mainAug 22, 2026
10 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@ronaldtse