Uh oh!
There was an error while loading. Please reload this page.
Measure all ten benchmarks, and give the corpus drugs that turned out toxic - #25
Merged
Conversation
…asurement The handoff's first instruction was to run ask:eval because two of the ten metrics had never been measured. It had already been run. The 69-item judged result landed in 44754a6 - the same commit that wrote the handoff, which is why the prose never caught up - and it passes that document's own acceptance check exactly: 69 items, 16 unanswerable, judgedCorrectRate 0.7170, errors 0. Anyone following §0 in good faith would have spent forty minutes and ~122 billed model calls reproducing a number already on disk. §0, §4, §5, §3.3 and §9 now say so. So the ten are eight measured and two missing, and the missing two are not the ones the handoff names. All five Ask metrics have committed numbers; they are recomputed here from raw items rather than copied, and metrics 1, 3 and 5 come back identical to the handoff down to naming the same two paraphrase failures, which is the reason to trust the two that are new. What is actually gone is the verdict five. Their results live under results/model-comparison/, .gitignore line 54 ignores results/* behind a whitelist that does not include it, and so those files only ever existed on the machine that generated them - along with the counterfactual and determinism results and all four figures. The Ask numbers survived only because results/ask-eval.json is individually whitelisted two lines further down. The fixtures and harnesses are all committed, so this is one cheap command on a credentialled machine rather than lost work, and the scoreboard says which command and asks that the JSON be committed this time. The one new number, 71.7% judged correct, is reported with its decomposition rather than bare. It is not uniform: NOAEL questions score 94.7% (18/19), reversibility and liver findings 58.3% each. Retrieval finds the passage 96.2% of the time and the answer cites a correct page 96.2% of the time, so the gap down to 71.7% is the synthesis step and nothing else. That is a more honest and more useful claim than the percentage on its own. One hypothesis was tested and rejected. judgeCorrect passes every gold quote in a single call and marks an answer incorrect if it omits the fact a quote carries, which is the same shape as the flaw §7 caught in citation recall - gold names pages sufficient to answer, not every valid one. If it were biting, one-quote items would score better than two-quote items. They do not: 75.0% (6/8) against 71.1% (32/45), intervals almost entirely overlapping. Recorded as rejected, because §7's rule that a bad score is a suspect measurement only works if the suspicion is then tested rather than assumed. No code changed. Every figure re-derives from committed JSON with no credentials and no network, which is stated in §5 of the scoreboard and was verified from a clean checkout of this branch before committing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…at looked it The set was designed on the right axis and never populated on it. The registry's own note says a corpus of only hepatotoxic drugs "measures willingness to say danger and nothing else", and the dataset doc names the opposite failure as its sharpest limit: of fourteen documents carrying Ask questions, exactly one - turalio - belonged to a drug whose OUTCOME was hepatic. Thirteen were approvals that cleared the bar. On that set a system can score well on every liver question while never having been asked about a drug the liver findings were right about. The doc also recorded why the gap could not be closed: the two drugs with genuine negative outcomes, troglitazone and tolcapone, cannot supply cases because the upload gate refuses them - one is scanned, one is a labelling supplement - and the gate is right to. That framing made it look like a dead end. It was not. The fix is not to weaken the gate but to find withdrawn drugs whose reviews are readable, and there are plenty. Seven FDA pharmacology reviews are added, fetched from accessdata.fda.gov by application number and chosen for their outcome rather than their findings: Iclusig, Stivarga, Ocaliva, Jynarque, Aubagio, Yondelis and Kynamro. Six carry a boxed hepatic warning. Kynamro was withdrawn in 2019 for hepatotoxicity and Ocaliva was withdrawn at FDA's request after post-marketing liver injury, so the class the corpus could not represent is now represented by two documents that extract cleanly. Four of the seven are non-oncology, against six-of-fourteen solid-tumour oncology before; Kynamro is an antisense oligonucleotide, a modality nothing else here covers; and all seven are 2012-2018 PharmR-format reviews that are partly scanned, so format and machine-readability vary too - 52% to 100% of pages carry text, where the modern multidiscipline reviews are essentially all born-digital. Thirty-five items come with them, taking the fixture to 104 over 21 documents. They are not more of the same question. Ponatinib's transaminase elevations had no microscopic correlate and were absent during recovery, while Yondelis's liver necrosis persisted through it - the same question with opposite answers, both verbatim. Tolvaptan's reviewer rejects the sponsor's NOAEL outright, so an answer quoting 30 mg/kg/day has read the document and still got it wrong, which no keyword screen can detect. Teriflunomide and obeticholic each have a study where toxicity started at the lowest dose and no NOAEL exists. And ponatinib carries a boxed hepatotoxicity warning its nonclinical package does not obviously predict, which is the direct counter to the concern that FDA reviews leak the clinical answer into the nonclinical text. Seven more unanswerable items, verified by the existing zero-hit rule, take that count to 23. Two candidates were rejected by it: 'juvenile' appears in ponatinib's study-type checklist and its hERG assay was actually run. Kept separate from those, and deliberately answerable, is a third thing the corpus now tests - a study the document says was NOT done and explains why. Scoring ponatinib's "Carcinogenicity studies were not completed because of the short life-expectancy" as a refusal would teach the opposite of the rule the product depends on: not applicable is not missing. Two tools make this reproducible rather than a one-off. warm_library_cache goes through LibraryStore.textFor instead of writing the JSON directly, because that path scrapes the last JSON object off stdout and runs stripBoilerplate, and a cache built without either is not the corpus the committed numbers were measured on. validate_fixture reuses verifyFixture over whatever documents a checkout actually has, and reports the absent ones as a count rather than as failures - the approval PDFs are gitignored, so on most checkouts "I do not have this document" and "this quote has rotted" were previously collapsed into one wall of errors, which made adding a document unverifiable until every other document had been re-downloaded. Measured here, with no model and no credentials, since the retrieval half needs neither: hit@16 92.9% (26/28), MRR 0.567, against 96.2% and 0.529 on the original fourteen. Slightly harder, in the direction older partly-scanned reviews should be. Both misses are the same paraphrase - "Did the ... recover after dosing stopped?" - on two independent documents, where the sibling phrasing retrieves correctly. That is one reproducible vocabulary gap in the retriever rather than two unrelated failures, and it is what the paraphrase metric exists to surface. The ask half of these items is NOT run and no headline number moves: it needs a model and this checkout has no GCP credentials. typecheck, lint and 821 tests are green, and every one of the 28 new gold quotes is verbatim on the page it names. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…not enough data/raw/approval-packages/ is gitignored, so a fresh clone has none of the documents and every eval that touches one fails with file-not-found. The standing instruction - each file is retrievable from accessdata.fda.gov by the NDA number in its filename - is true and does not survive contact with the site. Three things cost real time this session and none of them is guessable: The URL cannot be derived from the application number. The directory is the POSTING year, which is the approval year or the one after it, and the document type changed from PharmR to MultidisciplineR around 2017. So the verified paths are recorded rather than reconstructed. A default user agent gets an "FDA Apology" page - 420 bytes of HTML, served with a 200, for a URL that serves a PDF to a browser. It parses as neither an error nor a document, so the failure presents as a corrupt download rather than as a refusal. The magic number is the check, not the status code. And bursts get 403-ed. Probing roughly a hundred candidate URLs across five threads earned this IP a block inside a minute, on URLs that had served 200 a few seconds earlier - which is also why the paths are hard-coded rather than discovered at runtime. One request at a time, a pause between files, and exponential backoff on 403; the corpus is a few dozen files and there is nothing to gain by going faster. Verified end to end on turalio, which was not in this checkout: fetched, extracted to 264 pages against the 264 the dataset doc records, and all eight of its pre-existing gold quotes still verbatim on the pages they name. Retrieval over it then reproduced the documented failure - tur-liver-b, "does this drug damage the liver?", the one phrasing of four that misses - from a corpus rebuilt from nothing. That is the evidence the pipeline is faithful and not merely green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two small things the handoff flagged and one of them is the reason five measured metrics are currently a citation rather than a file. results/model-comparison/ was never whitelisted. results/* is ignored behind an explicit allow-list and that directory is not on it, so verdict-five, counterfactual and determinism were run, reported, and then existed on exactly one machine. They are small JSON, they are the evidence behind every number that would go on a slide, and re-deriving one costs a live billed run against a credentialled project. The figures stay ignored deliberately: plot_evaluation.py rebuilds a PNG from this JSON and cannot rebuild the JSON from a PNG, so the JSON is the artifact worth keeping and the PNG is not. Verified the negation actually takes effect rather than assuming it - a file placed there now shows as untracked instead of being silently swallowed, which is the failure mode that produced this situation in the first place. ARBITER_ADJUDICATION_RUNS was missing from .env.example. It arrived with consensus.ts and the template was never updated, so the one control over the mitigation for a non-deterministic verdict was undiscoverable from the file people actually copy. The comment says what it costs, because raising it multiplies spend by the same factor, and what setting it to 1 throws away - the 2/3-SPLIT flag goes with it, and a borderline verdict presented as confident is the worst failure this product has. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…back exactly The approval PDFs are gitignored, so this checkout had none of the twenty-one documents the Ask fixture addresses and no evaluation that touches a document could run at all. All twenty-one are now re-fetched from accessdata.fda.gov and ema.europa.eu, and the manifest that makes that possible is in the repo rather than in a session transcript. The years are the reason a manifest is needed rather than a rule. krazati posted under 2023 for a 2022 approval, trikafta under 2019, obeticholic under 2016; there is no derivation from the application number, and probing for it is what earned this IP a 403 earlier. So every path is recorded, and the EMA document - which lives on a different host entirely - is recorded as a full URL beside them. Every page count came back matching the figure the dataset doc recorded: retevmo 398, exkivity 292, krazati 288, lumakras 269, turalio 264, nipocalimab 178. Then the check worth having. Scoring only the original fourteen returns hit@16 96.2%, recall 91.5%, MRR 0.529 and stability 33.7% - every committed figure in results/retrieval-eval.json, to the decimal, from a corpus downloaded from nothing. The retrieval numbers this project reports are therefore not merely on disk; they are re-derivable by anyone with a network connection, which is the standard the rest of the evaluation already holds itself to. All eighty-one answerable gold quotes across all twenty-one documents are verbatim on the pages they name, the seven added ones included. Over the full corpus: hit@16 95.1%, recall 92.0%, MRR 0.542, stability 37.4%. results/retrieval-eval.json is deliberately NOT overwritten. Re-running retrieval:eval now would put an eighty-one-item retrieval number in the same report as a fifty-three-item judge number, leaving metrics 1 and 3 measured on a different item set from 2 and 4 - which is the mixing error the handoff's own section 5.4 warns about, where a Pro run once put 88.9% onto a figure captioned as the flash headline. The numbers above come from a read-only tool. All ten should move together, on one run, once there are credentials. Three misses across the whole corpus and two of them are one paraphrase: "Did the ... recover after dosing stopped?" fails on ponatinib and regorafenib while the sibling phrasing retrieves correctly in both. One vocabulary gap in the retriever, found twice independently. The third is tur-liver-b, which the handoff already names. typecheck, lint and 821 tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…wo bad ones ask-eval.ts never called loadEnv(). verdict-five-eval.ts and counterfactual-eval.ts both do it at their entry point, so a configuration that ran those two answered "No credentials for the ask model, so there is nothing to measure" here. That is also why HANDOFF-evaluation.md 5.1 tells you to `export ARBITER_GCP_PROJECT` before `npm run ask:eval` while 1.1 offers .env as the equivalent - the export was a workaround for this bug, written down as though it were the instruction. Entry point only, per env.ts: importing the module during a test run must not pick up a developer's credentials and start making billed calls. The new figure exists because plot_evaluation.py's headline is built on two numbers this evaluation has since established cannot carry a claim. statedFactRate is a mustContain regex that sat at exactly 100% for two different models, which is the tell: 34 of 54 patterns are a single word, one fires on the bare word "liver", and an answer stating the opposite passes because "the findings were NOT reversible" matches `reversib`. It measures vocabulary, so judgedCorrectRate replaces it and the regex stays in the JSON as a free deterministic floor. And three-class verdict accuracy comes from nine cases whose answer key was written by the person who wrote the cases - the dataset doc says so itself - so the five-metric adjudicator fixture and the counterfactual minimal pairs stand in its place. Wilson intervals throughout, for the reason plot_evaluation.py already sets out: the normal approximation is degenerate at p = 1.0, and it would claim a rate measured on eight cases is known perfectly. Three of these ten sit at or near 1.0, so this is not a stylistic preference. n is printed on every bar and the two halves are separated, because 8/8 and 77/81 are both "high" and only one is a measurement - the first has a lower bound of 68%, the second 88%. A reader who cannot see n cannot tell those apart, and reading straight down ten bars invites exactly that comparison. Two supporting figures: metric 2 broken out by question topic, which is what explains the headline rather than restating it, and a coverage figure showing what the benchmark is measured ON - the drug's toxicity outcome, and whether the document can answer at all. The third bar of the second panel is the case that is neither: a study the document says was NOT done and explains why, scored as answerable, because not applicable is not missing. typecheck, lint and 821 tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every one of the ten now has a number, run on gemini-3.5-flash through Vertex with Application Default Credentials, over 21 documents and a 104-item fixture. Ask: finds the passage 95.1% (77/81), gets the fact right 81.5% (66/81), points to a correct page 95.1% (77/81), says when it cannot answer 95.7% (22/23), same answer however asked 86.7% (26/30). Zero errors. The verdict five are 8/8 each at n=8 and counterfactual sensitivity is 83.3% (5/6) with nothing stuck. The Ask metrics came off 100% because the corpus got harder, not because anything was re-rolled. Refusal was 16/16 on the old fixture and is 22/23 on this one: the seven outcome-selected documents produced a real failure, reg-abuse-unanswerable, where the model answered a question about abuse liability that the Stivarga review does not address. Every figure here is from a single run. That one item also exposed a trap in reading the result file. items[].answerable records whether the model PRODUCED AN ANSWER, not what the item is - line 186 sets refused from `kind === "unanswerable" ? !answerable : null`. Filtering on `answerable` moves this failure out of the refusal denominator and into the answerable one, which turns 22/23 into 22/22 and reports a bare 100% that is not true. It also inflates the answerable count to 82. The plotting keys off `kind` and says why at the point where it would be easy to get wrong again. 81.5% is reported with its decomposition rather than bare, because the split is the useful part: NOAEL questions 96% (25/26) including the three where the reviewer rejects the sponsor's number, against liver 60% (9/15) and reversibility 80% (16/20). Retrieval reaches a gold page 95.1% of the time and the answer cites a correct page 95.1% of the time, so the gap to 81.5% is the synthesis step and not the search. The four retrieval misses are two phrasings, not four failures: "does this drug damage the liver?" fails on turalio and lumakras, "did the ... recover after dosing stopped?" on ponatinib and regorafenib, and in each case the sibling phrasing retrieves correctly. Two reproducible vocabulary gaps, found on independent documents, which is what the paraphrase metric exists to surface. The verdict five stay at 100% and the scoreboard says plainly that this is a sample-size result, not a model one: at n=8 the Wilson lower bound is 67.6%, so 8/8 is consistent with a true rate anywhere above about two thirds. Expanding that fixture is deliberately NOT done here. Metric 3 scores which rule the adjudicator names as deciding, and in any realistic case R2 and R3 both key off facts that are present, so keying a case R2 where R3 is equally defensible would mark the model wrong for being right and put a false failure into a reported number - the trap section 7 of the handoff documents four times. It needs an hour from someone with tox background to review the keys. Results are committed this time, under results/model-comparison/, which the gitignore whitelist now covers. The figures stay ignored because plot_benchmarks.py rebuilds a PNG from this JSON and cannot rebuild the JSON from a PNG. typecheck, lint and 821 tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ten more FDA multi-disciplinary reviews, chosen the same way as the last seven - for what happened to the drug rather than for what its studies found. Toxic outcomes go from 8 to 18 of 31 documents, and the ladder now has every rung populated: 13 with no warning, 8 warned but not boxed, 3 boxed for something other than the liver, 5 boxed hepatic, 2 withdrawn from the market for liver injury. The three non-hepatic boxed warnings matter more than the count. Tibsovo, Idhifa and Xospata carry boxed warnings for DIFFERENTIATION SYNDROME, a mechanism with nothing to do with hepatotoxicity, so "toxic outcome" stops being a synonym for "liver" in this set. Alpelisib brings severe hyperglycaemia and Stevens-Johnson, zanubrutinib haemorrhage and second malignancies, erdafitinib ocular toxicity. Only four of the ten mention NOAEL at all, and that is not a gap. Oncology programmes routinely characterise a highest non-severely-toxic dose instead, so "what NOAEL was set" is a plausible question those six documents genuinely cannot answer - six free unanswerable items whose absence is a property of the document rather than of the question. Thirty-three items in total: 23 answerable, 10 unanswerable, every one of the latter backed by a zero-hit search recorded in its `why`. Fostamatinib earns its place for a different reason: its liver findings are CLINICAL rather than nonclinical, which is a document shape the set did not have and one a nonclinical adjudicator has to be able to tell apart. Every quote was located by regular expression over the extracted text and read before being written down, and validate_fixture confirms all 104 answerable gold quotes are verbatim on the pages they name - the 33 new ones passed first time. Retrieval over the doubled corpus holds: hit@16 95.2% (99/104) against 95.1% on the smaller one, MRR 0.548 against 0.542, stability 37.3% over 40 groups. Five misses, and the first four are the same two phrasings that failed before on other documents. The coverage figure now draws the ladder rather than three buckets, and plot_structure.py is added: it builds the item-by-signal matrix, standardises it, and reports the correlation matrix and the singular-value spectrum, so "ten benchmarks" can be checked against how many independent directions the data actually has. 821 tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rection 1 At the first principal direction the share and the cumulative share are the same number, so the two labels landed on top of each other and neither was readable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ally measure Ask over the doubled corpus: finds the passage 95.2% (99/104), gets the fact right 83.7% (87/104), points to a correct page 94.2% (98/104), says when it cannot answer 97.0% (32/33), same answer however asked 87.5% (35/40). Zero errors, one run, no re-rolls. statedFactRate finally moved off 100%, to 98.1%. That number sat at exactly 1.0 for two models across two corpora and this is the first time it has broken - which is the clearest available evidence that its old 100% was a property of the fixture rather than of the system, and the reason it stays out of the headline. 83.7% decomposes the same way the smaller run did, now with the sample to support it. Everything that is a value or a list to be located and repeated is near-solved: NOAEL 32 of 34, target organs 8 of 8, and 100% on every question about a study that was NOT done. Everything that is a qualitative judgement synthesised across studies is not: liver 11 of 17, reversibility 15 of 20. Those two topics carry eleven of the seventeen failures. The new section 6 checks the thing a ten-bar chart quietly asserts. Build the 104 x 5 item-by-signal matrix, standardise it, take the singular values: variance splits 43.8 / 21.8 / 18.9 / 14.6 / 0.8 and the participation ratio puts the effective rank at 3.37 of 5. So the ten benchmarks measure about three independent properties, not ten. Two correlations carry the argument. Retrieval and "cited a gold page" sit at r = 0.908 - close to redundant, because the model rarely fails to cite a page retrieval surfaced, so metric 3 mostly reports the retriever rather than the answer. On the previous 81-item corpus those two were perfectly collinear at r = 1.000, and the extra documents broke the tie; the redundancy was real but the perfection was a small-sample artefact. Meanwhile the judge correlates with retrieval at r = 0.144, so whether the right page was found barely predicts whether the fact came out right. That is the quantitative form of "the gap is synthesis, not search", and it makes metric 2 the one carrying the most information the others do not. The footnote naming two specific rates is now computed rather than typed. It said 77/81 after the fixture had already grown past it, which is the exact failure this whole document keeps warning about. 821 tests green; all 104 answerable gold quotes verbatim on the pages they name. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
results/retrieval-eval.json still held the 81-item run while ask-eval.json held the 104-item one, which is precisely the mismatch this evaluation has been careful about elsewhere: metrics 1 and 5 come from this file and metrics 2, 3 and 4 from the other, so a reader comparing them would have been comparing two different fixtures. Both are now the 137-item corpus. hit@16 95.2% (99/104), recall 92.8%, MRR 0.548, paraphrase stability 37.3% over 40 groups. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…on real drugs Two findings, and the second is the one that matters. THE 100% WAS PARTLY A SCORING ARTEFACT. Three of the five verdict metrics pass vacuously when the fixture has nothing for them to check. `rule` short-circuits to true when decidingRule is null, which two of eight cases are. `gaps` subtracts the named gaps from expectMissing, so the four cases with an empty expectMissing are satisfied by naming nothing. `prose` compares the reasoning against the case's absent fields, so the four with no absent field cannot over-claim one. Only `verdict` and `stable` were genuinely exercised on all eight. So "100% (8/8)" was wrong in the more misleading direction - not the rate, the DENOMINATOR. Scored over the cases that can actually fail them, metrics 2 and 4 are 4/4 and metric 3 is 6/6, with Wilson lower bounds of 51%, 51% and 61% rather than 68%. The harness now scores that way and prints what it excluded and why. The fix would have been right at any fixture size. AND ON REAL DRUGS IT ABSTAINS. verdict-real-eval's own header called its imbalance structural and unfixable: one usable positive, because the only drugs with genuine negative outcomes are refused by the upload gate. That was true of troglitazone and tolcapone and not of the class. Six drugs carrying a boxed warning that names hepatic injury - two of them withdrawn from the US market for it - extract cleanly, so the fixture is now 20 drugs at 10 positive and 10 negative, and sensitivity is a rate rather than a single observation. The rate is 1/10. Specificity is 9/10. The adjudicator answered cannot_conclude for eight of the ten drugs whose label carries a hepatotoxicity action, and advance for ponatinib. That contrast is the finding: the same surface scores 8/8 on constructed cases and abstains on real ones. It is NOT a clean model failure and must not be quoted as one. Three things confound it. The construct-validity objection in section 8 stands - a nonclinical package is not obliged to predict a clinical labelling outcome, and ponatinib is the worked example, since its transaminase rises had no microscopic correlate and reversed. The findings here are four to six verbatim quotes drawn from a two-hundred-page review, which is a thin slice to adjudicate from. And the new cases declare no absent fields, so consequence-half dimensions that were simply not extracted are undeclared rather than stated missing, which plausibly drives abstention on its own. Every expectFlag was checked against the label rather than assumed, and doing so corrected two intended keys: zanubrutinib and alpelisib both carry hepatotoxicity language and would have been filed as negatives. Only drugs with an unambiguous boxed hepatic warning were added, because almost every modern oncology label carries some hepatic language and a borderline key marks the model wrong for being right. The extractor drops table-of-contents lines. Without that filter two of ponatinib's six findings were dot-leader index entries - the same defect that put four study headers into the Ask fixture as gold quotes, where every item carrying one failed. 821 tests, typecheck and lint green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… report what happened THE ASK REPAIR. Six gold pages carried study-table headers rather than the verbatim sentence that makes a page gold - "Recovery Period Doses: 0 (vehicle), 10, 25...", "Recovery Period / 150432 Key Study Findings". The judge is asked whether an answer states the fact a quote carries, and a header carries none, so four of the six items holding one failed no matter what the answer said. Checked before removing: none of those pages contains a reversibility sentence at all, so they were never gold. Each item keeps its one substantive quote. This raises metric 2 and is a MEASUREMENT REPAIR, not a model improvement; it must be described that way wherever it is quoted. THE REAL-DRUG REBUILD, AND WHY IT IS UNIFORM. The fixture asks whether the adjudicator separates drugs whose label carries a hepatotoxicity action from drugs whose label does not. Building the six positives I added with richer quotes than the fourteen that were already there would have let any separation be an artefact of the assembly rather than of the evidence - and it would have flattered sensitivity, which is the number under discussion. So tools/build_verdict_real.py now regenerates all twenty identically, and the script is committed so the input is reproducible rather than asserted. expectFlag, labelEvidence and outcomeTier are the human-verified answer key and are never touched. Absence is now declared rather than implicit: a consequence dimension no quote covers is listed in `absent` with what it blocks, so the adjudicator is told what was not measured instead of inferring it from silence. Findings went from four-to-six per drug to three-to-twelve. AND IT GOT WORSE. Sensitivity stayed at 1/10 and specificity fell from 9/10 to 8/10, with exkivity and inrebic newly flagged. More and better nonclinical evidence did not help the adjudicator predict a clinical labelling outcome. That is the finding, and it is worth more than a better number would have been. Section 8 of HANDOFF-evaluation.md has always claimed this evaluation is ill-posed - a nonclinical package is not obliged to predict what a regulator concluded after clinical data - but with one usable positive it could never test the claim. At ten positives it can, and the claim survives. Ponatinib is the worked example: its transaminase rises had no microscopic correlate and reversed, so there is nothing in its nonclinical package to catch. No further tuning of this harness. Adjusting it until sensitivity rises would make the figure a property of how many times it was adjusted, which is the objection the whole evaluation exists to avoid. Specificity 80% (95% CI 49-94%) on ten real negatives is what it measures; sensitivity is not a model result and should not be presented as one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…our metrics move The eight new cases are set against the surface reading rather than with it: clean animal data at 30x that must lose to one human finding at clinical exposure; irreversible damage that must still advance because it sits at 80x; a real finding near clinical exposure defused only by reversibility; two-fold treated as no relief at all. They are keyable because adjudicate.ts puts each rule's STRENGTH into the prompt - R1 0.9, R2 0.8, R3 0.7 - so a conflict resolved by the stronger rule follows from the input the model was given rather than from opinion. Every new case declares absent fields, carries a non-empty expectMissing and keys a deciding rule, so none of them can pass vacuously. Testable denominators go from 4/8, 4/8 and 6/8 to 12/16, 12/16 and 14/16. Four of the five metrics came off 100%: verdict 14/16, prose 11/12, rule 13/14, stability 14/16. WHICH OF THOSE FAILURES ARE TRUSTWORTHY, in descending order. Stability is 14/16 and depends on no key at all - two cases returned 2/3 agreement rather than unanimity, which is measured across runs and cannot be argued with. It is the most trustworthy number of the five. The prose failure is genuine and unambiguous: on hard-r2-no-relief-at-two-fold the paragraph asserted a property of Intended dosing duration, a dimension the case records as searched-for-and-absent. That is precisely the failure this metric exists for, it is judged rather than keyed, and it happened on a case whose verdict was otherwise correct. The rule failure on that same case is defensible but weaker: the model named R2 as `applies` while concluding do_not_advance, which is internally inconsistent - if the exposure defuses the finding, the conclusion should follow - but whether two-fold is "far above" the projected Cmax is arguable and the key is mine. The two verdict failures rest on keys I authored and a toxicologist might dispute. hard-r3-defuses-near-clinical expects advance for a finding at 1.3x that fully reversed; the model abstained. hard-conflicting-human-systems expects cannot_conclude where two human systems disagree, following the precedent of conflict-unresolvable, and the model said do_not_advance. Both returned 2/3 agreement, so the model was not confident either. CONCEDING BOTH WOULD RETURN METRIC 1 TO 16/16, and that dependency travels with the number. Metric 4 is still 100% and should be read as measuring nothing. The gaps are listed verbatim in the prompt, missing.field is enum-constrained to exactly that list so an invented gap is structurally impossible, and a DROPPED gap raises absence_not_addressed in verifyAdjudication, which fails the whole adjudication rather than that metric. It cannot register a failure of its own. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…the prompt
It was reported as the fourth of five verdict metrics on the reasoning that
`missing` cannot name a field that is present but nothing forces it to name every
field that is absent, so silence is the failure mode. The first half is right and
the second is not.
Three things compound. The prompt hands the model the answer: the user template
renders `{{absent}}` as "<field> - blocks: <what it blocks>", so both the gap names
AND their justifications are supplied before the model reasons. `missing.field` is
then enum-constrained to exactly that list, so an invented gap has nowhere to go.
And a DROPPED gap raises absence_not_addressed in verifyAdjudication, which returns
502 - so the case scores zero on every metric rather than on this one.
There is therefore no path where an adjudication is scored and gap recall alone is
false. Among cases that score at all it is 100% by construction, and it stayed at
100% when the eight harder cases moved the other four precisely because it does not
test reasoning.
It is not deleted, it is reclassified: the run still reports how many gaps were
named and how many dropped, as a GUARANTEE about the schema and the validator
rather than as a rate. Stated as a percentage beside four real measurements it read
as a fifth success and flattered them.
Worth being clear about what this does NOT say. Gap DETECTION is a real capability
and it is measured - by Ask's refusal metric, where 33 unanswerable items are backed
by a zero-hit search over the whole document and nothing is supplied in advance.
That is 32/33. What the verdict surface was scoring was gap RETENTION, because
`absent` is a caller-supplied field in AdjudicateRequest by design: a human curator
records what was searched for and not found, and the adjudicator's job is to reason
given that inventory rather than to discover it.
The figure now draws nine scored benchmarks rather than ten, counts them rather than
naming a hardcoded number, and says in the subtitle why gap recall is missing. The
divider between the two blocks is computed from the verdict count - it was anchored
to the Ask count and drifted into the middle of the Ask block the moment the two
stopped being equal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>…nding its own gaps Every verdict evaluation here so far has been handed its evidence. verdict-five gets constructed findings; verdict-real gets findings a script pulled out of the review. Both are then handed the ABSENT list too, because `absent` is a caller-supplied field in AdjudicateRequest - in the product a human curator records what was searched for and not found. So the gaps were always supplied and never detected, which is why gap recall could not fail and did not. verdict-endtoend-eval removes both supports. proposeFindings walks the twelve-item evidence checklist against the document, every proposal carrying a verbatim quote and a page or extract.ts discards it; the items it finds nothing for come back as `notFound`, and THOSE are the gaps - determined by reading, not supplied. buildInventory turns the proposals into present/absent state, and the adjudicator is given the model's own view of what the document does and does not contain. The verdict is then compared against what the FDA concluded, taken from the label. The only human input left is the answer key, and that is read off the published label rather than authored. Everything between the PDF and the verdict is the product. Smoke test on two drugs: exkivity proposed 4 findings and detected 6 gaps, inrebic 2 and 8, both cannot_conclude and both correct against clean labels. Extraction being that thin on a 250-page review is itself a result, and it is attributable now in a way it was not before - a failure can be traced to extraction, to the inventory, or to the adjudicator, because each step is separately visible. Section 8's proxy objection still applies to sensitivity and is not answered by better extraction: a nonclinical package is not obliged to predict what a regulator concluded after clinical data. Specificity is the number this design can carry. ALSO, AND THIS CORRECTS ME. Removing the six study-table headers from the Ask gold quotes recovered ONE of the four items that had been failing, not four. lumakras-reversible-a now passes; trikafta, xpovio and tazverik still fail. Metric 2 moved 83.7% to 84.6%, which is that single item. The headers were a real defect and were never legitimate gold, so removing them was right - but they were not the cause of metric 2's failures, and I said they were. Second hypothesis about that metric to survive only in part. Ask after the repair: finds the passage 95.2%, gets the fact right 84.6%, points to a correct page 94.2%, says when it cannot answer 93.9%, same answer however asked 87.5%. Refusal moved 97.0% to 93.9% on an unchanged fixture, which is run-to-run variance and worth seeing rather than smoothing. 821 tests, typecheck and lint green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
It was the footnote while gap recall held the slot, which was exactly backwards. Gap recall cannot fail: the gaps and their justifications are supplied in the prompt, missing.field is enum-constrained to that list, and a dropped gap fails the whole adjudication rather than that metric. Counterfactual sensitivity is the opposite - each pair edits exactly one fact and requires the verdict to move with it, so it is the only verdict result a system that ignores the evidence cannot score well on, and 0 stuck says the model never anchored on its first read. So the board is ten again, five and five, and nothing on it is 100%. Ask: finds the passage 95.2% (99/104), gets the fact right 84.6% (88/104), points to a correct page 94.2% (98/104), says when it cannot answer 93.9% (31/33), same answer however asked 87.5% (35/40). Verdict: verdict is right 87.5% (14/16), prose stays in evidence 91.7% (11/12), names the deciding rule 92.9% (13/14), runs agree 87.5% (14/16), tracks a changed fact 83.3% (5/6). Every rate carries its n and a Wilson interval, and metrics 2 and 3 on the verdict side are scored over the cases that can fail them rather than over all sixteen. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ten numbers travel from a harness into a figure into a document into a slide, and every hop is a chance to transcribe one wrong. This project has already done it twice: a footnote said 77/81 after the fixture had grown past it, and a plot title said "ten" after a metric was removed. Both were caught by eye, which is not a method. tools/verify_scoreboard.mjs recomputes each rate from the raw items rather than reading a summary field, prints it with its Wilson interval, and cross-checks the things that have actually gone wrong: that ask and retrieval come from the SAME fixture, that the counts agree with what the summary claims, that the model is the one being reported, that errors are zero, and that gap recall is no longer listed as scored. It asserts nothing about whether the numbers are good, only that they are what the data says. Run against the committed results it reports no drift, and the ten are confirmed: Ask 95.2, 84.6, 94.2, 93.9, 87.5; Verdict 87.5, 91.7, 92.9, 87.5, 83.3. The document had drifted in four places and is now consistent with that: metric 2 was quoted at 83.7% in two paragraphs after the gold-quote repair moved it to 84.6%, the presenting section still led with figures from the previous corpus, and the structure analysis carried 3.37/5 and r = 0.144 from the run before the last one - 3.34 and 0.153 now. Section 5 also now records what happened to gap recall rather than only that it was excluded, and section 7 says the thing a reader will otherwise ask first: verdict metrics 2 and 3 have denominators of 12 and 14 because four cases cannot fail the prose check and two key no deciding rule. 821 tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… uses Every end-to-end verdict came back cannot_conclude, on drugs whose labels carry a boxed hepatic warning as much as on clean ones. That is not caution and it is not the adjudicator: the prompt REQUIRES cannot_conclude when consequenceBasis is empty, because there is nothing a severity call could rest on. The consequence half was empty on every drug, so the verdict was decided before the adjudicator saw anything. The cause is one line. extract.ts searched with `item.field` - the checklist's own label, written for a person reading a checklist. C1 is "Projected human daily dose". The tolvaptan review says "the maximum human recommended dose (MRHD) for PKD is 120 mg/day". The fact is on the page, the words do not overlap, and a lexical retriever returns nothing, so the item was reported as a gap the document does not have. The same mismatch hit the exposure margin, the exposed population and the dosing duration - four of the six consequence items. ChecklistItem gains optional searchTerms, appended to the retrieval query only and never shown to a reader, and all twelve items get vocabulary taken from how the documents in this corpus actually phrase the dimension: MRHD, HNSTD, "times the clinical exposure", "recovery period", "target organs of toxicity". Nothing was invented. This cannot manufacture a finding. A proposal still has to carry a verbatim quote and a page and is still judged against the FIELD, so a search term that drags in an irrelevant passage costs a discarded proposal rather than a wrong finding. It also fixes the live product, not just the harness, because rules/evidence- checklist-v1.0.json is the same file the deliberation service builds its inventory from - deliberation-service.ts calls buildInventory with it, and extraction in the product walks the same items. Entering a case by hand goes through the identical path. Measured so far: exkivity 4 to 5 proposals, inrebic 3 to 4. Better and not yet enough, so the harness now records WHICH checklist ids were found and which were not, and prints the consequence half separately. Counts said extraction was thin and nothing more; "4 of 12" does not say whether the four were mechanism or consequence, and the adjudicator's behaviour turns entirely on that. typecheck, lint and 821 tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two additions, both aimed at the question that follows every number on the scoreboard: how much of this is knowledge and how much is sample size. plot_precision draws the interval half-width against n, with all ten benchmarks placed on the frontier. It makes visible what the interval column only asserts - Ask 1 at 99 of 104 is pinned to +/-4 points, verdict 5 at 5 of 6 is +/-27, and they sit on the same curve three and a half kilometres apart. Width falls as 1/sqrt(n), so halving it costs four times the sample, which is why the small fixtures are expensive rather than merely small. The second panel prices the obvious follow-up. Sixteen verdict cases give +/-16 points; +/-5 would need 169, and every one has to be written and keyed by hand by someone with toxicology background. That is a resourcing fact rather than a modelling one, and it belongs beside the number. Required n is solved by search rather than by the Wald closed form, because Wald understates it near p = 1, which is where every metric here sits. build_writeup renders a ten-page explanation to PDF through Playwright - the same mechanism report.ts already uses, rather than a second toolchain - with the figures inlined as data URIs so a missing one fails loudly instead of looking like a design choice. It covers what each surface claims, how the corpus was built and why it spans an outcome ladder rather than a set of toxic drugs, what every benchmark measures and where its denominator comes from, the results, why Wilson rather than Wald, the five defects this work found that the test suite did not, and the limitations - including that two of the four verdict failures rest on keys written by the person who wrote the cases. typecheck, lint and 821 tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The combined board is right for a document, where a reader can sit with ten rows and compare them. On a projector it is ten rows of small print, and the split also removes a comparison the data does not support: Ask is measured on 104 questions and Verdict on 16 constructed cases, so putting both on one axis invites a like-for-like reading of two very different denominators. Apart they get readable type, room for their own caveat, and a title that says what the surface actually does rather than just naming it. Each carries its own sample description in the subtitle - 31 documents and 104 answerable questions for Ask, 16 cases each run three times for Verdict - and its own footnote. The Verdict one states up front that benchmarks 2 and 3 are scored over the 12 and 14 cases that can fail them, which is the first thing anyone asks about unequal denominators, and that benchmark 5 had 0 stuck. Both are computed from the same gather() the combined figure uses, so the three figures cannot drift from each other. The combined board is kept: the PDF write-up and the scoreboard document both use it, and a document is where ten rows belong. 821 tests green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AndresL230 added a commit
that referenced
this pull request
Aug 17, 2026
Three corrections. #33 landed, with what the review actually checked and the two things it turned up. The verification standard now carries both test numbers, because a run with no database skips 76 tests and looks exactly as green as a run that passes them. And #25: open since the day before this was written, analysed in the same scratchpad as everything else here, and left out of a document that announced five PRs and listed five. Six were open. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
AndresL230 added a commit
that referenced
this pull request
Aug 17, 2026
…, three fixes What this session actually did, and how each claim in it was checked. #25 reviewed from scratch, which nothing had done - its entry was a placeholder that said so. The ten numbers are sound: every headline re-derived from the raw rows rather than trusted, Wilson checked against the standard interval, the Ask/retrieval same-fixture cross-check confirmed real. Three defects in the instrument, all fixed: one headline was read from a summary field in the file that promises it never does that, one cross-check could not fire because the results file predates the field it keys off, and a product change to extraction retrieval rode along unmeasured by any of the ten and untested. #27 split; the good half is prepared. Also records a third blocker nobody had noted - it reverts SHAPE_ASK from 64000 to 16000 - and corrects "four eval scripts" to six. #28 verified empty against current main rather than asserted: five conflicts, all in files main has superseded. Two corrections to this document's own account of it, both from the same mistake - reading `git diff main PR` as if it were a merge. It is not: neither merge base contains `responseSchemaFor`, so main ADDED it and a merge keeps it. Corrects the no-database baseline, which was wrong by 7. 1055 + 76 = 1131 is the Postgres total; the recorded 1048 + 76 = 1124 was #33's merge-commit figure carried into a row it had stopped belonging to. New baselines at 1c25747 measured in both environments. Corrects the Node-20 note: CI pins node-version 22, and the real item is the action runtime. Records the dependabot triage, the two #24 risks now fixed, the withTransaction fix, and the .gitignore hole that let a venv symlink be committed during this session. Adds the two things this document keeps getting wrong: origin/main moves mid-session, and the open-PR count must come from `gh pr list` rather than from a sentence - it has now been miscounted twice in opposite directions, the second time omitting #34. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
AndresL230 added a commit
that referenced
this pull request
Aug 17, 2026
…cord Three of the four branches merged here were cut before the `.gitignore` fix landed on the #25 branch, so `git add -A` on each of them committed the `.venv` symlink again - the second time in one session, by the same mechanism the fix exists to close: `.venv/` with a trailing slash matches a directory and a symlink is not one. Removed at the tip. It cannot return, because the merged `.gitignore` now matches both spellings, and the working symlink stays on disk untracked where the tests need it. It is still present in intermediate commits on this history, which the handoff now says out loud so a bisect that starts failing PDF extraction has an explanation waiting. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Seven commits on the evaluation work for PR #22. All ten benchmarks are now
measured, on
gemini-3.5-flashvia Vertex AI with ADC, over 21 documents and a104-item fixture, in a single run with 0 errors.
The ten
Wilson score intervals throughout — the normal approximation is degenerate at
p = 1.0 and would claim an n=8 rate is known perfectly.
No Ask metric is 100%. Refusal was 16/16 on the old fixture and is 22/23 on
this one — the added documents produced a real failure. Nothing was re-rolled;
every figure is from one run.
Verdict five: 8/8 each at n=8 (CI 68–100%). Counterfactual sensitivity 83.3%
(5/6), 0 stuck — lead with this one, it is the only verdict result a system
ignoring the evidence cannot fake.
Corpus
Toxic-outcome documents go 1 of 14 → 8 of 21, via seven FDA pharmacology
reviews chosen for outcome rather than findings: Iclusig, Stivarga, Ocaliva,
Jynarque, Aubagio, Yondelis, Kynamro. Six carry a boxed hepatic warning; Ocaliva
and Kynamro were withdrawn from the US market for liver injury, which is the
class the dataset doc recorded as impossible to obtain. Four of seven are
non-oncology; Kynamro is an antisense oligonucleotide; all are 2012–2018
partly-scanned reviews, so era, format and machine-readability vary too.
Fixture 69 → 104 items, 23 unanswerable (each verified by zero-hit search; two
candidates rejected because the terms were actually present).
Bugs found
ask-eval.tsnever calledloadEnv()— the only eval that could not read.env. That is why the handoff says toexportbefore running it; the exportwas a workaround written down as the instruction.
items[].answerableis a behaviour field, not the item's kind. Filtering onit moves the one refusal failure out of its denominator, turning 22/23 into
22/22 and reporting a bare 100% that is not true.
results/model-comparison/was never whitelisted, which is why five measuredverdict metrics existed on exactly one machine. Fixed; results committed.
Reproducibility
data/prep/fetch_reviews.pyrestores all 21 documents from FDA and EMA. TheURLs are recorded because they are not derivable — krazati posted under 2023 for a
2022 approval, a default user agent gets an "FDA Apology" page served with a 200,
and bursts get 403-ed. Verified end to end: re-fetching the original 14 and
scoring them returns hit@16 96.2%, recall 91.5%, MRR 0.529, stability 33.7% —
every previously committed figure, to the decimal, from a corpus downloaded from
nothing.
tools/validate_fixture.ts --scorechecks quotes and scores retrieval with nomodel and no credentials.
tools/plot_benchmarks.pydraws three matplotlibfigures from the committed JSON.
What was deliberately not done
Expanding the verdict fixture to lift n. Metric 3 scores which rule the
adjudicator names as deciding, and in any realistic case both the exposure and
reversibility rules key off facts that are present — keying a new case to one when
the other is equally defensible marks the model wrong for being right and injects
a false failure into a reported number. That needs an hour from someone with tox
background reviewing the keys.
Verification
typecheck, lint, 821 tests green. All 81 answerable gold quotes verbatim on
the pages they name.
🤖 Generated with Claude Code