diff --git a/corpus/skills/principle-no-lookahead/SKILL.md b/corpus/skills/principle-no-lookahead/SKILL.md new file mode 100644 index 00000000..ea2b4c9e --- /dev/null +++ b/corpus/skills/principle-no-lookahead/SKILL.md @@ -0,0 +1,132 @@ +--- +name: principle-no-lookahead +description: "Apply to any evaluation, backtest, benchmark, grader, or training run over data with a time axis or an ordered decision sequence. Score a decision only against inputs that existed before that decision was made. Covers what the field calls look-ahead bias, target leakage, point-in-time correctness, and repainting. Reach for it on: a backtest or walk-forward run, a feature computed over a full history, a retrieval or agent eval, a label assigned after the outcome was known, or any result that looks too good." +disable-model-invocation: true +--- + +# No lookahead + +A result is a result only if the decision behind it could have been made at +the moment it is credited to. Otherwise it is a measurement of hindsight. + +The failure has a formal name: **leakage** — "the introduction of information +about the target of a data mining problem which should not be legitimately +available to mine from" (Kaufman et al., see Grounding). In time-series work +it is **look-ahead bias**; the correctness property it violates is +**point-in-time correctness**; the technical-analysis artifact that causes it +is a **repainting** indicator. One defect, four vocabularies. + +## The shape + +Two artifacts get confused because they are rendered from the same data: + +| | Display artifact | Decision artifact | +|---|---|---| +| Answers | what happened | what would I have done | +| May read | the whole history | only rows at or before `t` | +| Revised later | legitimately, yes | never | + +A chart, a labelled dataset, an annotated log, a resolved ticket, and a +post-hoc report are all display artifacts. They are drawn *after*, and are +allowed to move a marker back onto the moment the thing "really" happened. +Feed one into a decision evaluation and the decision is credited with +knowledge it did not have. + +**The leaked value can be entirely real and the leak still total.** The leak +is in *when the decision became available*, not in the number's accuracy. +A price, a label, or a document can be genuine at time `t` while the +instruction to act on it does not exist until `t + k`. + +## Must always + +- **Name the cutoff before the first measurement.** For every input, record + two timestamps: what it is *about*, and when it became *knowable*. Only + the second one may gate a decision. If a source cannot tell you the + second, it is a display artifact until proven otherwise. +- **Run the truncation test.** Recompute the decision from only the data + available at time `t`, and compare it to what the full-history computation + says at that same `t`. Any disagreement at any `t` is a leak, and the + disagreeing indices name it. Ten cut points expose a gross leak in + seconds; a replay at every step is the proof. +- **Separate knowability from actionability.** The truncation test proves the + input existed at `t`. It does not prove the action was executable at `t`. + Name the execution cutoff too: the earliest moment the decision could have + reached the world, and the price, latency, or capacity it would have met + there. A decision that observes a bar's close and fills at that same close + is zero-latency, and zero latency is a form of hindsight the prefix test + cannot see. +- **Bind the test to the artifact that produced the number**, not to a + library function beside it. A causality test that guards `compute()` + proves nothing about a script that imports `compute()` and passes it a + different mode. The gate belongs on the path the reported number came out + of — including throwaway analysis scripts, notebooks, and heredocs. +- **Treat an implausibly good result as a bug report, not an achievement.** + Fix the implausibility ceiling from the domain's own known best *before* + running, and stop on any of: a score above what the best known + practitioner achieves; an error near zero; a monotone result with no bad + period across a span long enough to contain one; a metric that improves + when an input that should be irrelevant is added; a parameter sweep where + every cell wins. The stronger the number, the earlier you stop. Stop and + find the mechanism — leakage, an inert axis that never varied the output, a + frozen constant, or a coding error — before reporting the number at all. + Report the number of configurations tried alongside the best one; without + it, a reader cannot tell skill from selection. +- **Re-run the cutoff check after any change to what an input reads** — a + new normalization, a rolling window, a resample, a join, a smoothing term, + a re-label. Each can silently widen the window a value is computed over. +- **Reject a negative offset.** Any shift, lag, or window parameter exposed + to a caller must refuse values that reach forward. An unvalidated + `shift(k)` is a lookahead switch with no label on it. + +## Must never + +- Score a decision against a label that was assigned with knowledge of the + outcome. +- Accept "it only reads past data" from a grep, a variable name, a + docstring, or a commit message. Causality is a property of the output, + measured. It is not a property of the source, read. +- Let a display mode and a decision mode share a return type. If one + function can emit both, a caller will eventually pass the wrong one and + nothing will say so. Make the display result structurally unable to reach + the scoring path. +- Report a result whose sign depends on an assumption that was never varied. + If one unswept constant flips the conclusion, that constant is the finding. +- Publish a number produced before the truncation test ran. On discovery, + void the old numbers **in place** — move them to a `voided/` path and + retract the headline — rather than deleting them, so the record shows what + was believed and for how long. + +## The cheap test, in general form + +## Grounding + +- Leakage, formalized, with the legitimacy-in-time condition and the + learn-predict separation that avoids it → Shachar Kaufman, Saharon Rosset, + Claudia Perlich, Ori Stitelman, "Leakage in Data Mining: Formulation, + Detection, and Avoidance," *ACM TKDD* 6(4), Article 15 (2012) + . This is the paper that names + this exact failure; its KDD-Cup and INFORMS case studies are both + "implausibly good result turned out to be a leak." +- Evaluating only from an origin that moves forward through the data → + Leonard J. Tashman, "Out-of-sample tests of forecasting accuracy: an + analysis and review," *International Journal of Forecasting* 16(4), + 437-450 (2000) . Rolling + origin is the truncation test as a standard evaluation design. +- Why a too-good backtest is evidence about the search process rather than + the strategy, and why the count of configurations tried must be reported → + David H. Bailey, Jonathan M. Borwein, Marcos López de Prado, Qiji Jim Zhu, + "Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest + Overfitting on Out-of-Sample Performance," *Notices of the AMS* 61(5), + 458-471 (2014) . +- Purged cross-validation and embargo, for label overlap leaking across a + split → Marcos López de Prado, *Advances in Financial Machine Learning*, + Wiley (2018), ch. 7 and ch. 11. +- "Repainting," for a display artifact that moves a marker backward: + **no known peer-reviewed prior art.** It is vendor and forum vocabulary + only. Cite the mechanism, not the word. + +Related: [[principle-assert-invariants-not-last-bug]] (the truncation test is +the class invariant, not a patch for the leak you found); +[[principle-explicit-errors]] (a suppressed or unvalidated offset is how the +leak gets in silently); [[principle-report-the-disqualifier]] (what the +report must disclose once the evaluation is clean). diff --git a/corpus/skills/principle-no-lookahead/tests/fires_example.md b/corpus/skills/principle-no-lookahead/tests/fires_example.md new file mode 100644 index 00000000..b155590f --- /dev/null +++ b/corpus/skills/principle-no-lookahead/tests/fires_example.md @@ -0,0 +1,21 @@ +`disable-model-invocation: true` means the model never reads this +skill's `description:` to decide whether to apply it -- that text isn't +even loaded into context. The only way this skill activates is an +explicit `/principle-no-lookahead` invocation. + +The agent has loaded a full price series into one dataframe and computed a +volume-spike z-score column over the whole thing -- the mean and standard +deviation in every row's denominator are taken across all bars, including +bars that come after that row. It is about to score an entry decision at +each past bar against that column and report the annualised return, which +came out at 41% with no losing month in the span. Before reporting the +number, the agent explicitly invokes `/principle-no-lookahead` to load the +full principle. + +This skill fires here specifically because of that explicit invocation +-- a feature computed over the full history and then credited to a +decision at an earlier bar is exactly the pattern the skill targets once +loaded (record what each input is about and when it became knowable, run +the truncation test at ten cut points and compare against the full-history +column, treat a no-losing-month span as a bug report rather than a result), +but no amount of matching prose alone would have triggered it. diff --git a/corpus/skills/principle-no-lookahead/tests/stays_silent_example.md b/corpus/skills/principle-no-lookahead/tests/stays_silent_example.md new file mode 100644 index 00000000..0ad0e5d5 --- /dev/null +++ b/corpus/skills/principle-no-lookahead/tests/stays_silent_example.md @@ -0,0 +1,15 @@ +The agent is asked to draw an annotated incident timeline for a postmortem +slide: read the whole quarter of alert history, mark where the outage began, +and label each marker with the cause that was only established afterwards. +The chart reads every row of history, including rows after each marker. No +explicit invocation of this skill happens anywhere in the session -- the +agent just draws the chart. + +This skill stays silent here: with `disable-model-invocation: true`, +nothing about the conversation's content could have triggered it either +way. Nothing here is a decision artifact either -- the timeline answers +what happened, not what would I have done, and no score is credited to a +choice made at any marker. A display artifact is allowed to read the whole +history and to move a label back onto the moment the thing really began, so +even if this skill somehow were invoked, its content would not apply until +that chart were fed into an evaluation. diff --git a/corpus/skills/principle-report-the-disqualifier/SKILL.md b/corpus/skills/principle-report-the-disqualifier/SKILL.md new file mode 100644 index 00000000..0cc2f158 --- /dev/null +++ b/corpus/skills/principle-report-the-disqualifier/SKILL.md @@ -0,0 +1,105 @@ +--- +name: principle-report-the-disqualifier +description: "Apply when writing or gating any report that makes a causal claim or summarises a cumulative outcome — a backtest, an experiment write-up, a capacity or cost model, a postmortem, an eval summary, a benchmark table. The report must carry what would change the reader's decision: what observation would refute the claim, whether the path crossed a state it could not return from, what the headline was before and after the fix, and which unvaried assumption flips the sign." +disable-model-invocation: true +--- + +# Report the disqualifier + +A report is not finished when its numbers are correct. It is finished when a +reader who disagrees with it knows exactly where to look. Every claim has a +disqualifier — one observation that would sink it — and a report that omits +its own disqualifier is persuasive in proportion to how wrong it can be. + +Four disqualifiers, four sections. Use the ones that apply. + +## 1. What would refute this + +Any causal claim states, in the same document, the observation that would +falsify it — a measurement, not a hedge. "This may not generalise" is not a +falsification section; "if the effect survives when X is held constant, the +mechanism is not X, and here is that run" is. + +Grade every claim in the report on one visible scale — **measured / +inferred / conjecture** — and put the grade next to the claim, not in a +preamble. A reader must be able to strip the report to its measured rows and +see what is left. + +## 2. Did the path survive + +A summary statistic computed over a path can describe a path that ended. A +positive year through an account that was liquidated in March; a mean latency +across an interval containing an outage; a mean reward over a trajectory that +terminated; an eventual success after a retry budget was exhausted. + +Whenever the process has an **absorbing state** — a barrier it cannot return +from — the report carries a survived yes/no column and the date of absorption, +beside the final value. The expected value across many runs and the outcome of +the one run you get are different quantities, and only the second one is +yours. + +## 3. Before and after the fix + +When a defect is corrected mid-analysis, report the headline metric under both +readings in the same message, with the sign. A table that quietly contains +both the pre-fix and post-fix cell has published the reversal without saying +it, and the number that travels onward will be whichever one someone quoted +first. + +## 4. Which unvaried assumption flips the sign + +Every report names its frozen constants — the cost, the threshold, the +window, the rate — and, for each one that appears in the metric expression, +the value at which the conclusion reverses. If no such value exists inside a +plausible range, say so. Disclosure is not this: repeating an assumption +twelve times is not the same as testing it once. + +An axis you swept that changed no output row is also a finding. Report it as +inert rather than presenting its cells as distinct results. + +## Must never + +- Ship a gate that checks a section's *heading* and calls that a content + check. `grep -qi "falsification"` passes a report that says no + falsification was attempted. Assert the property, then prove the gate by + deleting the section body and watching it fail — + [[principle-assert-invariants-not-last-bug]], step 5. +- Present a headline figure whose cost, fill, or capacity basis is absent. + Prefix it `PROVISIONAL —` until that basis and its sensitivity range are in + the same message. +- Let a fan-out end without a terminal aggregation step. A finding stated + independently in several sibling reports, and read by nobody across them, + is the most expensive kind of undiscovered result. + +## Grounding + +- Falsifiability as the demarcation of a real claim → Karl Popper, *The Logic + of Scientific Discovery* (1959; Ger. 1934). +- Committing to the refuting test *before* seeing the result → Chris Chambers + and Loukia Tzavella, "The past, present and future of Registered Reports," + *Nature Human Behaviour* 6, 29-42 (2022) + . +- Grading each claim on one visible evidence scale → Gordon H. Guyatt et al., + "GRADE: an emerging consensus on rating quality of evidence and strength of + recommendations," *BMJ* 336, 924-926 (2008) + . +- Why an ensemble average does not describe the single path when the path can + be absorbed → Ole Peters, "The ergodicity problem in economics," *Nature + Physics* 15, 1216-1221 (2019) ; + J. L. Kelly Jr., "A New Interpretation of Information Rate," *Bell System + Technical Journal* 35(4), 917-926 (1956) + . +- Reporting the whole space of defensible specifications rather than one → + Uri Simonsohn, Joseph P. Simmons, Leif D. Nelson, "Specification curve + analysis," *Nature Human Behaviour* 4, 1208-1214 (2020) + ; Andrea Saltelli et al., "Why so + many published sensitivity analyses are false," *Environmental Modelling & + Software* 114, 29-39 (2019) + . +- Findings that exist only across sibling reports and are never aggregated → + Robert Rosenthal, "The file drawer problem and tolerance for null results," + *Psychological Bulletin* 86(3), 638-641 (1979) + . +- The "no losing day" / too-good smell test lives in [[principle-no-lookahead]], + not here — this skill is about what a report must disclose, that one is + about what an evaluation may read. diff --git a/corpus/skills/principle-report-the-disqualifier/tests/fires_example.md b/corpus/skills/principle-report-the-disqualifier/tests/fires_example.md new file mode 100644 index 00000000..335097a6 --- /dev/null +++ b/corpus/skills/principle-report-the-disqualifier/tests/fires_example.md @@ -0,0 +1,24 @@ +`disable-model-invocation: true` means the model never reads this +skill's `description:` to decide whether to apply it -- that text isn't +even loaded into context. The only way this skill activates is an +explicit `/principle-report-the-disqualifier` invocation. + +The agent has drafted an eval write-up whose headline reads "agent success +rate rose from 61% to 88% after the retrieval change." Two facts sit in the +last paragraph, after the tables and the recommendation: a scoring bug was +found and fixed halfway through, and the pre-fix reading of the same +headline was 63%; and the 88% counts runs that only succeeded after the +retry budget was exhausted and the harness restarted them. The cost per run +is frozen at one assumed value that appears in the metric expression and was +never swept. Before sending the write-up, the agent explicitly invokes +`/principle-report-the-disqualifier` to load the full principle. + +This skill fires here specifically because of that explicit invocation +-- a causal claim carrying its own reversal, its own absorbing state, and an +unvaried constant, all of them placed after the conclusion, is exactly the +pattern the skill targets once loaded (state the observation that would +refute the claim, grade each claim measured/inferred/conjecture beside the +claim, add a survived column for the exhausted-budget runs, report the +headline under both readings with the sign, and name the cost value at which +the conclusion reverses), but no amount of matching prose alone would have +triggered it. diff --git a/corpus/skills/principle-report-the-disqualifier/tests/stays_silent_example.md b/corpus/skills/principle-report-the-disqualifier/tests/stays_silent_example.md new file mode 100644 index 00000000..ea3bfdc4 --- /dev/null +++ b/corpus/skills/principle-report-the-disqualifier/tests/stays_silent_example.md @@ -0,0 +1,13 @@ +The agent is asked how many open pull requests currently carry a given +label, runs the query, and answers with the count and the list. There is no +causal claim, nothing accumulates over a path, no defect was corrected +partway through, and no constant appears in the answer that could be varied +to reverse it. No explicit invocation of this skill happens anywhere in the +session. + +This skill stays silent here: with `disable-model-invocation: true`, +nothing about the conversation's content could have triggered it either +way. A direct count of a present-tense state has no disqualifier to lead +with -- there is no observation that would refute it short of the query +being wrong, and no summary statistic hiding a path that ended. Even if +this skill somehow were invoked, all four of its sections would be empty. diff --git a/corpus/skills/report-rendering/SKILL.md b/corpus/skills/report-rendering/SKILL.md new file mode 100644 index 00000000..291a1933 --- /dev/null +++ b/corpus/skills/report-rendering/SKILL.md @@ -0,0 +1,85 @@ +--- +name: report-rendering +description: "Apply when a report, analysis, or write-up is finished and someone other than the author will read it. Ship it as a single self-contained HTML file, and a PDF where a fixed page is wanted. Markdown split across several files in a repo is a working artifact, not a delivered report. Reach for it on: a multi-section analysis, a report with figures, anything to be handed to someone without a checkout, or any request to 'show me the report'." +disable-model-invocation: true +--- + +# Render the report + +A report that only exists as markdown in a repo has not been delivered. It has +been left where the author was working. The reader either lacks a checkout, or +has one and must open several files in the right order and imagine the figures +in place. + +Ship one file that opens by double-clicking it. + +## Must always + +- **One self-contained HTML file.** Embed every figure as a `data:` URI rather + than linking a path. A document that breaks when moved is not a deliverable. + Inline the CSS for the same reason. +- **Concatenate the sections in reading order**, with a table of contents and + each source path shown beside its section, so a reader can find the file + behind any claim. +- **Add print styles.** `page-break-before` on each top-level section, and + unstick any `position: sticky` table headers, which otherwise repeat or + vanish across page boundaries. Without this the PDF is unusable even though + the HTML looks correct. +- **Produce the PDF from the same HTML**, so the two cannot disagree. +- **Commit both**, and say where they are. A rendered report that lives only in + a temp directory has the same problem as the markdown did. + +## Must never + +- Hand back a list of markdown paths in response to "show me the report." +- Link figures by relative path in a document intended to be sent anywhere. +- Reach for pandoc, LaTeX, or a new dependency when the machine already has a + browser. The recipe below needs nothing else. +- Render a report whose numbers have not passed their own gate. Rendering makes + a claim easier to circulate, which is a reason to be more careful about what + is in it, not less. See [[principle-report-the-disqualifier]]. + +## The recipe + +Markdown to self-contained HTML to PDF, with no toolchain beyond a browser: + +```python +import base64, markdown +from pathlib import Path + +md = markdown.Markdown(extensions=["tables", "fenced_code", "toc", "sane_lists"]) +parts = [] +for title, path in SECTIONS: + md.reset() + parts.append(f"

{title}

{md.convert(Path(path).read_text())}
") + +fig = base64.b64encode(Path(FIGURE).read_bytes()).decode() +parts.append(f'
') +Path("report.html").write_text(TEMPLATE.format(body="".join(parts))) +``` + +```sh +google-chrome --headless --disable-gpu --no-sandbox \ + --print-to-pdf=report.pdf --print-to-pdf-no-header report.html +``` + +`md.reset()` between sections matters: the `toc` extension accumulates state +across calls, so without it later sections inherit earlier anchors. + +## Grounding + +- Single-file, self-contained documents as a distribution format → the MHTML + and `data:` URI approach; RFC 2397, "The 'data' URL scheme" (1998) + . +- Literate and reproducible reporting, where the document and the analysis that + produced it are one artifact → Donald E. Knuth, "Literate Programming," + *The Computer Journal* 27(2), 97-111 (1984) + ; Roger D. Peng, "Reproducible + Research in Computational Science," *Science* 334(6060), 1226-1227 (2011) + . +- Delivering a result in the form the reader can act on, rather than the form + the author worked in: **no known prior art** as a stated principle; it is + ordinary practice rather than a named result. + +Related: [[principle-report-the-disqualifier]] (what the report must contain +before it is worth rendering). diff --git a/corpus/skills/report-rendering/tests/fires_example.md b/corpus/skills/report-rendering/tests/fires_example.md new file mode 100644 index 00000000..98caf60b --- /dev/null +++ b/corpus/skills/report-rendering/tests/fires_example.md @@ -0,0 +1,21 @@ +`disable-model-invocation: true` means the model never reads this +skill's `description:` to decide whether to apply it -- that text isn't +even loaded into context. The only way this skill activates is an +explicit `/report-rendering` invocation. + +The agent has finished a five-section spend analysis. It exists as five +markdown files and three PNG charts in a scratch directory, and each section +links its figures by relative path. The user says "show me the report," and +the agent is about to reply with the list of file paths -- to a reader who +has no checkout and will forward this to two other people. Before replying, +the agent explicitly invokes `/report-rendering` to load the full principle. + +This skill fires here specifically because of that explicit invocation +-- a finished multi-section analysis with figures, about to be handed to +someone as markdown paths, is exactly the pattern the skill targets once +loaded (concatenate the sections in reading order into one self-contained +HTML file with a table of contents and each source path beside its section, +embed each chart as a data URI, add the page-break and unstick-sticky-header +print styles, print the PDF from that same HTML with headless Chrome, and +commit both rather than leaving them in the scratch directory), but no +amount of matching prose alone would have triggered it. diff --git a/corpus/skills/report-rendering/tests/stays_silent_example.md b/corpus/skills/report-rendering/tests/stays_silent_example.md new file mode 100644 index 00000000..a3b7f171 --- /dev/null +++ b/corpus/skills/report-rendering/tests/stays_silent_example.md @@ -0,0 +1,13 @@ +The agent has just changed a build script and is updating the two paragraphs +of the repo README that describe how to run it. The result is markdown in a +repo, it has no figures, and its readers are the people who already have the +checkout open in front of them. No explicit invocation of this skill happens +anywhere in the session. + +This skill stays silent here: with `disable-model-invocation: true`, +nothing about the conversation's content could have triggered it either +way. Nothing is being delivered to a reader outside the checkout, so +markdown in the repo is the finished form rather than the working one, and +there is nothing to embed or paginate. Even if this skill somehow were +invoked, rendering a README to a standalone HTML file and a PDF would leave +the copy readers actually open unchanged.