Uh oh!
There was an error while loading. Please reload this page.
Arabic + Hebrew SOTA stack, SadeedDiac-25 benchmark win, papers - #55
Merged
Conversation
ModernCharTransformer: RoPE + SDPA Flash + mHC residual + AttnRes + RMSNorm + SwiGLU. 113M params at 12L/768d. Selected via cfg.model.arch=modern. Multi-task seg head optional.
Optimizer: MuonAdamWHybrid (Muon Newton-Schulz for 2D weights, AdamW for 1D) + qk_clip_ weight-rescaling callback with anneal. Prevents attention-logit explosion during from-scratch pretrain.
Decoding: trie-constrained beam decoder with per-word exact search. scripts/build_lexicon.py builds the word-to-haraqat-sequences JSON.
Data: combined corpus (GPLv2 Tashkeela-full + Sadeed HF + QCRI EMNLP 2025) built by fetch_data on first call. Graceful fallback when HF_TOKEN unset.
Idempotency: training/resume.py auto-detects latest epoch checkpoint. train_all.py skips done stages via _status.json on checkpoints volume. scripts/status.py queries Modal volumes.
Configs: rababa_arabic_pro{,_pretrain}.yaml → arch=modern, max_len=512, optimizer=muon, with_seg_head=true, root=/datasets/arabic-combined.
References: arXiv:2606.19348 (DS V4), 2607.24653 (Kimi K3), 2512.24880 (mHC), 2507.20534 (MuonClip).…wnload Modal secret 'huggingface' was registered with HF_TOKEN. fetch_data now reads it via env var to authenticate the Sadeed_Tashkeela download.
Pretrain failed because MuonAdamWHybrid is not a torch.optim.Optimizer. WarmupCosine is now a duck-typed scheduler; GradScaler is skipped for Muon (bf16 autocast is enough). Add run_sota_pipeline + sota_pipeline entrypoint: fetch -> pretrain -> train -> export ONNX/TFLite entirely on Modal via .remote() chaining. Idempotent stage skips + volume status. Survives --detach disconnect. Fix Tashkeela-full layout discovery (tashkeela_full_train/ subdirs).
Usage: python scripts/status.py [--watch|--json] [--task rababa_arabic_pro]. Reports Modal app state, stage status JSON, pipeline log, per-epoch checkpoints, and exported artifacts.
…len 512
ModernMultiHeadCharTransformer mirrors ModernCharTransformer's encoder
(RoPE + SDPA + mHC + AttnRes + RMSNorm + SwiGLU) but with a ModuleList
of per-category linear heads for niqqud/dagesh/sin.
Encoder weights are key-compatible with ModernCharTransformer — a single
pretrain checkpoint can fine-tune into either Arabic single-head or
Hebrew multi-head.
rababa_hebrew{,_pretrain}.yaml now use:
arch=modern_multi_head, max_len=512, optimizer=muon (MuonAdamWHybrid)
Smoke-tested: forward + backward + Muon step OK, 2.4M params at 384d/6L.…e won't trip on stale format
- 12L/768d/113M was too big for single-A100 pretrain on 1.7M lines (0 checkpoints in 35min before death). 6L/512d/40M keeps the modern stack but is tractable. - sota_pipeline --force now only wipes arabic-combined for Arabic tasks. Hebrew force-wipe was destroying Arabic corpus when both pipelines ran in parallel.
…on't interfere Both Arabic and Hebrew pipelines write to the same /checkpoints/_status.json. Without task-keying, Hebrew's 'pretrain done' marker caused Arabic's pretrain to skip in non-force mode. Keys are now 'rababa_arabic_pro:pretrain' etc.
ByT5/seq2seq training paths alongside the char-level encoder: Muon optimizer variants (AdaMuon, NorMuon, HTMuon, Spectral Cap), ResFormer, MoE, ELECTRA pretraining, curriculum sampler, EMA, SAM, multi-seed and distillation harnesses, plus eval scripts for DictaBERT, Nakdimon and dNIKUD baselines. Arabic 0.99% DER, Hebrew 17.46% DER (beam 4).
RESULTS.md as ground truth for Arabic (0.99% DER) and Hebrew (17.46% DER, DictaBERT 35.63% on same test), three LaTeX papers (arabic, hebrew, umbrella), and the TODO.publish checklist.
ByT5-base run-002 (full 1.42M-line corpus, 2 epochs) to beat Claude's 1.39 DER on SadeedDiac-25. Checkpoints commit to the Modal volume at every save so preemption no longer discards hours of training; EVAL_DONE marker makes relaunch-after-completion a no-op.
Rejection-sampling fine-tuning on ByT5 r2 (TODO.research/12): sample K=4 per prompt, keep letter-exact-DER winners over greedy, SFT on winners. Selection on the frozen private dev split (1,372 lines, sha256-pinned, byte-identical to r2's held-out val); SadeedDiac-25 measured once at the end. Self-fires when r2's EVAL_DONE marker appears; per-iter volume commits + markers make preemptions resume cleanly.
Steers RAFT iterations: quantifies how much residual DER sits in the word-final iʿrāb zone and which haraqat get confused, from the eval CSV.
Misraj's public corpus leaks the SadeedDiac-25 benchmark (122 exact paragraphs + ~1k near-dup lines found via stride-1 60-char shingles). r3 continues r2 on the decontaminated copy (1M) + MSA replay (150k) to close the classical-Arabic domain gap behind r2's residual errors.
Preemptions every ~2h kept killing the ~4h iter-1 sampling before the iteration marker existed, restarting from zero every time. Winners now persist to the volume every 25 batches (with commit) and resume from the saved prompt index.
r3 lands 2.8429/1.7589 (best non-frontier on SadeedDiac-25). Found eval truncation: 57/1,200 preds cut at 1024B — windowed eval gives the apples-to-apples number. RAFT now targets r3 (run-002) with mid- sampling preemption resume.
Diacritized output is 1.4-1.6x input bytes; max_new_tokens=WINDOW truncated windows mid-word (345-letter input, 200-letter pred). Now WINDOW*2, plus SequenceMatcher haraqat projection onto input letters: 759/1200 word-count mismatches -> 0, zero evaluator skips.
Nested imports made json function-local; sampling state crashed on first access. Module-level import only.
Zero-skip windowed protocol: 2.8126/1.6877 DER, all 1,200 scored. Papers updated with the contamination finding (122 verbatim + ~1k near-dup benchmark paragraphs in Misraj's public corpus) and the survivorship-bias protocol lesson (1.82 was skip-artifact).
Temperature 0, thinking disabled (reasoning mode burns minutes per long paragraph; plain completion matches the published LLM protocol). Checkpointed per-row; reports raw + projected zero-skip protocols.
Clean reproduction (temp 0, plain completion, 1200/1200 responses, 5 long-paragraph retries). The 2026 frontier sits at 2.51 DER, not the published Claude-3.7 1.39; our 580M r3 trails it by ~0.3 DER and splits metrics on the zero-skip protocol.
Distillation rejected on principle: student ceiling = teacher errors, one systematic error poisons the chain, and gold-filtering makes the teacher redundant. Instead: join line-split book text into ~1400-byte paragraph units so iʿrab gets inter-sentence context, eval at the same window with zero-skip projection.
Third crash in the 8000-9000+ region, 350MB short on the A100. Halving per-device activations keeps the resume deterministic past the long-unit region without changing data order or step count.
…M-5.2 Windowed zero-skip at 1400B context: our 580M ByT5 now beats the GLM-5.2 reproduction (2.6911/1.7179) on both Total and Morph DER. Paragraph-unit training closed the context gap; GTPO-GRPO chained on r5 best by the supervisor.
Arabic headline now 2.6775/1.5965 (r5 paragraph-context, windowed zero-skip at 1400B) — beats the verified GLM-5.2 reproduction on both metrics; r5 promoted to current-best teacher.
The GTPO entropy term kept a float32 softmax copy in the autograd graph for 32 teacher-forced 1400B sequences. Weights are constants (GTPO redistribution), so detach them; PROMPTS_PER_STEP 4->2 with GRAD_ACCUM 4->8 keeps the effective batch at 16 prompts/step.
byt5-base teacher-forced activations for 16 seqs x (1400 in + 2800 out) byte-tokens exceed A100 even after halving the micro-batch; the OOM sat in the T5 gated-FFN forward. Checkpointing drops activation memory ~10x for ~30% step time.
At temp 1.0 over ~1400 letters every sample has some letter mismatch; binary mismatch scored ALL samples der=1.0 (mean_r=-1.0 across every group), collapsing advantages to zero — no learning signal. Letters are now SequenceMatcher-aligned and haraqat errors count on gold-vocalized positions, giving a graded reward.
Checkpointing made steps ~3.8 min; a 3h window never reached step 50, so SAVE_EVERY=200 could never fire before the 11h timeout — restart-from-scratch forever. 400 steps still spans the Persian curve shape (best at 200); saves every 50 survive preemption.
The 5.5GB alloc was T5's materialized decoder self-attention (16 x 12 x 2800^2 fp32) on a full-length batch; transformers 4.46 T5 materializes scores without SDPA. Halve micro-batch to 1 prompt (GRAD_ACCUM 16 keeps effective batch), request sdpa with fallback.
Three preemptions in 50 min at 16 min/step meant the step-50 save was unreachable — every restart re-paid model load + dev eval and lost all progress. Now: A100-80GB (less preempted, checkpointing dropped), 700B units (quadratic attention win, pool covers most corpus lines), GROUP 6, 150 steps, save 25, dev 200 lines.
best/ was only saved on dev improvement; the flat GTPO curve meant the final benchmark loaded a nonexistent dir, crashed pre-EVAL_DONE, and left the supervisor relaunching forever.
Entropy-weighted credit does not move Arabic either; dev curve exactly flat, benchmark = r5 within 700B-window protocol noise. Conclusion: knowledge-limited, not policy-limited. Data-side levers (decontam, domain, paragraph context) remain the only proven movers.
Audit: every table cross-checked against RESULTS.md ground truth (r5 = 2.6775/1.5965 canonical; GLM-5.2 zero-skip row added; RAFT rows added; abstract/contributions/discussion/limitations updated; stale Thai + GRPO/RAFT manifest rows fixed). Blog post covers all five languages with the honest frontier-verification framing. Distill prompt now leads with r5 as canonical Arabic teacher.
…, init r5, 24h timeout)
…s43 recipe); decontam + dedupe prep
…still gate now <=2.6 PER
ronaldtseforce-pushed
the
sota-sprint-arabic
branch
from
August 20, 2026 06:16
c619ec4 to
bef1538CompareUh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan