Uh oh!
There was an error while loading. Please reload this page.
fix(distill): retry label-file reads; ara-diac-tiny frozen snapshot - #29
Merged
Conversation
added 8 commits
August 23, 2026 08:11
…data, metrics sources
…pshot Every relaunch's resume-read of teacher_labels.jsonl tore to 3 lines (27,324 persisted), forcing a full relabel each ~40-min eviction cycle — the run could never reach training. Reads now retry 3x keeping the best parse, and ara-diac-tiny points labels_file at a frozen labels_snapshot.jsonl uploaded from a verified-healthy local copy.
The volume snapshot tore on read exactly like the original file (3 visible srcs of 27,324, all three retries), and the retry loop's early-break fired on the first non-empty parse. The verified-healthy 27,324-line snapshot now ships in the repo (6MB) and is read from the image path — image files cannot tear. labels_file accepts absolute in-image paths; the retry loop is best-of-3 without early break.
Probes settled it: the 69MB UTF-8 jsonl reads as 3 parseable srcs both through the image COPY layer and modal volume put (double-encoded bytes, 5.6M broken lines in the accumulated container-written file too). Pure-ASCII transports are immune: the snapshot now ships gz+base64 (~20MB) and is decoded to /tmp at run start; label appends switch to ensure_ascii=True.
str.splitlines() fragments on U+0085/U+2028-class codepoints inside Arabic label text, shredding healthy files into mid-JSON fragments — the original '3 valid srcs' mystery. All label-file reads now split on plain \n (verified locally: 11,793 srcs, 0 bad after the fix).
… GPU until labels are usable The Step-2 label filter hard-rejected anything over 384 bytes — with Arabic's 1450-byte windows that discarded essentially every label (8 of 11,793 survived). This, not volume replication, was the original '2-6 valid pairs' disease. Cap is now 2x spec max_len. Regeneration also ran after the teacher had been moved to CPU — the confirmation step now precedes the free.
…ba commits Both refs 404 (raw.githubusercontent) — the rababa commits are not on any remote yet. Re-add once rababa is pushed; they cannot ride a branch whose CI fetches them.
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan