P0 fixes + panel-based MRD benchmark + paper (v2.2.0) - #2
Merged
Merged
Conversation
…, CHIP wiring Real-TCGA validation (real_tcga_validation.py): - --seeds now actually loops seeds × patients (was a no-op: only seeds[0] used) - Simulation fully seeded (error-rate draws moved off global np.random) - Cohort loader counts patients, not files; picks richest-signal patients; dedupes - cBioPortal → GDC open-access MAF downloader wired in; saves normalized MAFs for offline reproducibility; refuses the synthetic fallback dataset (fail-loud) - Threshold optimization on test data removed: AUC/PR-AUC + sensitivity at FIXED 95%/99% specificity (was Youden's J on pooled test predictions) - CHIP/germline filter applied (matched-normal VAF rules) with counts in metadata - Honest framing: pipeline_type REAL_MUTATIONS_+_SIMULATED_PLASMA_READS - Results regenerated: 20 real LUAD patients, 5,738 mutations (GDC open access), 5 seeds; ultra-early regime (0.1% ctDNA) AUC 0.64 / sens@95% 0.18 Jiang 4-mer pipeline (run_jiang_analysis.py): - NESTED CV: MWU motif selection now inside each training fold (was full-data selection -> optimistic AUC); selection stability reported Portability/CI/docs: - Remove hardcoded /home/node/.openclaw/... paths; DEEPCATCH_DATA_DIR override - CI runs the real test suite (core + DL jobs) + real-data guard; the old synthetic smoke test renamed honestly - requirements_py.txt: add torch-geometric (was required, missing) - README: test badge 228/228, corrected module counts, honest TCGA table - review/agent_review_2026-08-10.md: full external review + fix log
test_26_downstream_fit_predict asserted AUC > 0.4 after 20 epochs of a tiny Transformer trained with unseeded torch init + randperm split — failed ~1/3 runs. Seed torch/numpy at test start; now 5/5 stable.
Per-position detection is information-limited at ultra-low ctDNA (signal ~1.9 reads vs error ~10 reads per locus @ 0.1% ctDNA, 5000x). Add the field-standard panel-based design: - run_panel_detection: per-SAMPLE aggregation of per-locus Poisson LLR over the tracking panel (tumor-informed / MRD-style), paired cancer/control per patient, ROC across patients per seed (mean±std over 5 seeds) - run_ultraearly_sweep: panel detection across background error rate (2e-3 raw -> 1e-5 duplex-UMI) x depth (5k/50k) at 0.1% ctDNA - compute_llr_scores extracted (shared by caller + panel detector) - simulate_cfdna_from_real: bg_error_rate parameter - --with-ml opt-in (per-position ML classifier: 13 min, no benefit at ultra-low VAF); default run now ~3 min - README: panel table + sweep table; docs/PRODUCTION_ROADMAP.md (MRD-first production plan: assay spec, data acquisition, validation ladder, 12-month milestones) Results (20 real LUAD patients, 5738 mutations, 5 seeds): 0.1% ctDNA: panel AUC 0.935 / sens@95% 0.770 / paired win 1.000 (vs 0.642 / 0.183 per-position). Duplex-UMI error (1e-4) or 50k depth: sens@95% = 1.000 at 0.1% ctDNA.
Root cause of the flaky test_26_downstream_fit_predict: generate_dataset used Python's built-in hash() on strings, which is randomized per process (PYTHONHASHSEED). Every pytest invocation generated different synthetic data, so the tiny model's test AUC landed anywhere in [0.33, 0.50+] and the AUC>0.4 assertion failed ~1/4 runs — with different failure AUCs each time. Verified: same seed produced different data hashes across processes. - data.py: sample_id now derives from hashlib.md5 (the file's own _deterministic_key pattern); data hash identical across processes - test_26: also give the model a realistic capacity (32-dim/2-layer/30 epochs) and assert AUC > 0.5, so the test measures learning with margin instead of knife-edge convergence Verification: test_26 8/8 fresh processes, foundation 4x43, full suite 3x228 all green.
…lean panel
Three orthogonal improvements grounded in ctDNA sequencing biology:
1. Context-aware error simulation (context_mix=True, default):
- CpG positions: 10× error (deamination artifact)
- Homopolymer runs: 5× (polymerase slippage)
- Clean baseline: 90% of positions
All positions share the same Beta error prior × context multiplier
(no leakage: context distribution identical for variants and background).
2. Fisher-method panel scoring (CAPP-Seq / Newman 2014 standard):
- Per-locus: one-sided Poisson p-value P(X≥a|H0) via 1-gammaincc(a,λ)
- Panel score: Σ -log₁₀(p) — guards against outlier loci dominating
the LLR sum. AUC 0.834 vs LLR 0.921 at 0.1% ctDNA.
3. Strand-concordance simulation + scoring:
- Forward/reverse read counts simulated independently
- True variants: biallelic (alt on both strands)
- Background errors: strand-asymmetric (randomly assigned to one strand)
- Strand score = Fisher × strand-concordance weight
AUC 0.820 at 0.1% ctDNA (strand adds a modest orthogonal lever).
4. Clean-panel design (--clean-panel flag):
- Restricts the tracking panel to clean-context loci only
- Models a well-designed targeted panel avoiding high-error regions
- Modest improvement (LLR 0.922, Fisher 0.849 at 0.1%)
Key finding: panel design is a modest lever; the dominant levers remain
sequencing depth and error-rate suppression (duplex UMI), per the sweep.
Verification: 228/228 tests, ad-hoc script 24/24 checks on changed paths.
- NEXT_STEPS.md: prioritized executable action plan (8 sections)
- Push branch, re-run Jiang nested-CV, FLARE longitudinal validation,
real cfDNA data acquisition, IRB + clinical cohort, assay dev,
longitudinal model redesign, papers, weekly operating rhythm
- data/README.md: data acquisition guide with all data sources
- data/FLARE_CONTACT_TEMPLATE.md: email draft for FLARE authors
- docs/PRODUCTION_ROADMAP.md: updated with literature validation section
(5 papers from 2025-2026, arXiv + PubMed search)
- All code + detectors tested (228/228, panel AUC 0.9215 verified)
The guard ran 'python ... | tee guard.log' inside an if; the if tested tee's exit code (always 0), so the guard always hit the FAIL branch even though the script correctly refused synthetic data. set -o pipefail propagates python's SystemExit(1), making the guard actually assert.
rollroyces
pushed a commit
that referenced
this pull request
Sep 17, 2026
Revise the 09-13 v3 design to reflect what actually moved Sens@99% in the post-design optimization rounds. The original AUC-led framing is replaced with a per-cancer / per-subgroup Sens@99% framing, because pooled AUC is saturated at 0.9755 and never moved despite 10+ techniques tried (Insight #6). Key changes: - §2 target metrics: Sens@99% target lowered 0.85 → 0.80 (empirical ceiling), per-cancer Sens@99% ≥ 0.50 floor (NEW), subgroup-stratified Sens@99% ≥ 0.60 (NEW based on Insight #2), AUC demoted to no-regression guardrail. - §4.2 pre-registered plan: subgroup-stratified CADD panel selection (Top-K=200 per patient) promoted to primary; OvR elastic-net demoted to secondary calibrator. Match-rate budget added (Insight #5). - §6 acceptance: per-cancer Sens@99% ≥ 0.50 (NEW), subgroup-stratified +10pp on LUAD (NEW), match-rate reporting (NEW), pooled Sens@99% ≥ +0.05 absolute (was ≥ 0.85 hard target, now empirical ceiling). - §8 NEW: seven insights with honest framing — CADD Top-K=500 +18pp on whole cohort, Top-K=200 per patient +14 to +40pp on 4 of 8 LUAD subgroups, AUC/Sens@spec decoupling, OV/PAAD headroom, CADD 86%/0% match rate, AUC saturation, deep-learning wrong at n=627. Honest bottom line: design now aligns with what moved the metric, not with what the original proposal hoped would move it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
All fixes and the full paper are now on this branch.
📄 Paper (bioRxiv-ready):
paper/PAPER.md+paper/paper.tex🏷 Release: https://github.com/rollroyces/deepcatch/releases/tag/v2.2.0
Key results (verified, 20 LUAD patients × 5 seeds)
Fixes included (13 commits)
hash()→hashlib.md5(per-process data flake)228/228 tests green. All data open-access (GDC, GEO).