[auto] Add principle-no-lookahead, principle-report-the-disqualifier, report-rendering

corpus/skills/principle-no-lookahead/SKILL.md

@@ -0,0 +1,132 @@
+---
+name: principle-no-lookahead
+description: "Apply to any evaluation, backtest, benchmark, grader, or training run over data with a time axis or an ordered decision sequence. Score a decision only against inputs that existed before that decision was made. Covers what the field calls look-ahead bias, target leakage, point-in-time correctness, and repainting. Reach for it on: a backtest or walk-forward run, a feature computed over a full history, a retrieval or agent eval, a label assigned after the outcome was known, or any result that looks too good."
+disable-model-invocation: true
+---
+
+# No lookahead
+
+A result is a result only if the decision behind it could have been made at
+the moment it is credited to. Otherwise it is a measurement of hindsight.
+
+The failure has a formal name: **leakage** — "the introduction of information
+about the target of a data mining problem which should not be legitimately
+available to mine from" (Kaufman et al., see Grounding). In time-series work
+it is **look-ahead bias**; the correctness property it violates is
+**point-in-time correctness**; the technical-analysis artifact that causes it
+is a **repainting** indicator. One defect, four vocabularies.
+
+## The shape
+
+Two artifacts get confused because they are rendered from the same data:
+
+| | Display artifact | Decision artifact |
+|---|---|---|
+| Answers | what happened | what would I have done |
+| May read | the whole history | only rows at or before `t` |
+| Revised later | legitimately, yes | never |
+
+A chart, a labelled dataset, an annotated log, a resolved ticket, and a
+post-hoc report are all display artifacts. They are drawn *after*, and are
+allowed to move a marker back onto the moment the thing "really" happened.
+Feed one into a decision evaluation and the decision is credited with
+knowledge it did not have.
+
+**The leaked value can be entirely real and the leak still total.** The leak
+is in *when the decision became available*, not in the number's accuracy.
+A price, a label, or a document can be genuine at time `t` while the
+instruction to act on it does not exist until `t + k`.
+
+## Must always
+
+- **Name the cutoff before the first measurement.** For every input, record
+  two timestamps: what it is *about*, and when it became *knowable*. Only
+  the second one may gate a decision. If a source cannot tell you the
+  second, it is a display artifact until proven otherwise.
+- **Run the truncation test.** Recompute the decision from only the data
+  available at time `t`, and compare it to what the full-history computation
+  says at that same `t`. Any disagreement at any `t` is a leak, and the
+  disagreeing indices name it. Ten cut points expose a gross leak in
+  seconds; a replay at every step is the proof.
+- **Separate knowability from actionability.** The truncation test proves the
+  input existed at `t`. It does not prove the action was executable at `t`.
+  Name the execution cutoff too: the earliest moment the decision could have
+  reached the world, and the price, latency, or capacity it would have met
+  there. A decision that observes a bar's close and fills at that same close
+  is zero-latency, and zero latency is a form of hindsight the prefix test
+  cannot see.
+- **Bind the test to the artifact that produced the number**, not to a
+  library function beside it. A causality test that guards `compute()`
+  proves nothing about a script that imports `compute()` and passes it a
+  different mode. The gate belongs on the path the reported number came out
+  of — including throwaway analysis scripts, notebooks, and heredocs.
+- **Treat an implausibly good result as a bug report, not an achievement.**
+  Fix the implausibility ceiling from the domain's own known best *before*
+  running, and stop on any of: a score above what the best known
+  practitioner achieves; an error near zero; a monotone result with no bad
+  period across a span long enough to contain one; a metric that improves
+  when an input that should be irrelevant is added; a parameter sweep where
+  every cell wins. The stronger the number, the earlier you stop. Stop and
+  find the mechanism — leakage, an inert axis that never varied the output, a
+  frozen constant, or a coding error — before reporting the number at all.
+  Report the number of configurations tried alongside the best one; without
+  it, a reader cannot tell skill from selection.
+- **Re-run the cutoff check after any change to what an input reads** — a
+  new normalization, a rolling window, a resample, a join, a smoothing term,
+  a re-label. Each can silently widen the window a value is computed over.
+- **Reject a negative offset.** Any shift, lag, or window parameter exposed
+  to a caller must refuse values that reach forward. An unvalidated
+  `shift(k)` is a lookahead switch with no label on it.
+
+## Must never
+
+- Score a decision against a label that was assigned with knowledge of the
+  outcome.
+- Accept "it only reads past data" from a grep, a variable name, a
+  docstring, or a commit message. Causality is a property of the output,
+  measured. It is not a property of the source, read.
+- Let a display mode and a decision mode share a return type. If one
+  function can emit both, a caller will eventually pass the wrong one and
+  nothing will say so. Make the display result structurally unable to reach
+  the scoring path.
+- Report a result whose sign depends on an assumption that was never varied.
+  If one unswept constant flips the conclusion, that constant is the finding.
+- Publish a number produced before the truncation test ran. On discovery,
+  void the old numbers **in place** — move them to a `voided/` path and
+  retract the headline — rather than deleting them, so the record shows what
+  was believed and for how long.
+
+## The cheap test, in general form
+
+## Grounding
+
+- Leakage, formalized, with the legitimacy-in-time condition and the
+  learn-predict separation that avoids it → Shachar Kaufman, Saharon Rosset,
+  Claudia Perlich, Ori Stitelman, "Leakage in Data Mining: Formulation,
+  Detection, and Avoidance," *ACM TKDD* 6(4), Article 15 (2012)
+  <https://doi.org/10.1145/2382577.2382579>. This is the paper that names
+  this exact failure; its KDD-Cup and INFORMS case studies are both
+  "implausibly good result turned out to be a leak."
+- Evaluating only from an origin that moves forward through the data →
+  Leonard J. Tashman, "Out-of-sample tests of forecasting accuracy: an
+  analysis and review," *International Journal of Forecasting* 16(4),
+  437-450 (2000) <https://doi.org/10.1016/S0169-2070(00)00065-0>. Rolling
+  origin is the truncation test as a standard evaluation design.
+- Why a too-good backtest is evidence about the search process rather than
+  the strategy, and why the count of configurations tried must be reported →
+  David H. Bailey, Jonathan M. Borwein, Marcos López de Prado, Qiji Jim Zhu,
+  "Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest
+  Overfitting on Out-of-Sample Performance," *Notices of the AMS* 61(5),
+  458-471 (2014) <https://www.ams.org/notices/201405/rnoti-p458.pdf>.
+- Purged cross-validation and embargo, for label overlap leaking across a
+  split → Marcos López de Prado, *Advances in Financial Machine Learning*,
+  Wiley (2018), ch. 7 and ch. 11.
+- "Repainting," for a display artifact that moves a marker backward:
+  **no known peer-reviewed prior art.** It is vendor and forum vocabulary
+  only. Cite the mechanism, not the word.
+
+Related: [[principle-assert-invariants-not-last-bug]] (the truncation test is
+the class invariant, not a patch for the leak you found);
+[[principle-explicit-errors]] (a suppressed or unvalidated offset is how the
+leak gets in silently); [[principle-report-the-disqualifier]] (what the
+report must disclose once the evaluation is clean).

corpus/skills/principle-no-lookahead/tests/fires_example.md

@@ -0,0 +1,21 @@
+`disable-model-invocation: true` means the model never reads this
+skill's `description:` to decide whether to apply it -- that text isn't
+even loaded into context. The only way this skill activates is an
+explicit `/principle-no-lookahead` invocation.
+
+The agent has loaded a full price series into one dataframe and computed a
+volume-spike z-score column over the whole thing -- the mean and standard
+deviation in every row's denominator are taken across all bars, including
+bars that come after that row. It is about to score an entry decision at
+each past bar against that column and report the annualised return, which
+came out at 41% with no losing month in the span. Before reporting the
+number, the agent explicitly invokes `/principle-no-lookahead` to load the
+full principle.
+
+This skill fires here specifically because of that explicit invocation
+-- a feature computed over the full history and then credited to a
+decision at an earlier bar is exactly the pattern the skill targets once
+loaded (record what each input is about and when it became knowable, run
+the truncation test at ten cut points and compare against the full-history
+column, treat a no-losing-month span as a bug report rather than a result),
+but no amount of matching prose alone would have triggered it.

corpus/skills/principle-no-lookahead/tests/stays_silent_example.md

@@ -0,0 +1,15 @@
+The agent is asked to draw an annotated incident timeline for a postmortem
+slide: read the whole quarter of alert history, mark where the outage began,
+and label each marker with the cause that was only established afterwards.
+The chart reads every row of history, including rows after each marker. No
+explicit invocation of this skill happens anywhere in the session -- the
+agent just draws the chart.
+
+This skill stays silent here: with `disable-model-invocation: true`,
+nothing about the conversation's content could have triggered it either
+way. Nothing here is a decision artifact either -- the timeline answers
+what happened, not what would I have done, and no score is credited to a
+choice made at any marker. A display artifact is allowed to read the whole
+history and to move a label back onto the moment the thing really began, so
+even if this skill somehow were invoked, its content would not apply until
+that chart were fed into an evaluation.

corpus/skills/principle-report-the-disqualifier/SKILL.md

@@ -0,0 +1,105 @@
+---
+name: principle-report-the-disqualifier
+description: "Apply when writing or gating any report that makes a causal claim or summarises a cumulative outcome — a backtest, an experiment write-up, a capacity or cost model, a postmortem, an eval summary, a benchmark table. The report must carry what would change the reader's decision: what observation would refute the claim, whether the path crossed a state it could not return from, what the headline was before and after the fix, and which unvaried assumption flips the sign."
+disable-model-invocation: true
+---
+
+# Report the disqualifier
+
+A report is not finished when its numbers are correct. It is finished when a
+reader who disagrees with it knows exactly where to look. Every claim has a
+disqualifier — one observation that would sink it — and a report that omits
+its own disqualifier is persuasive in proportion to how wrong it can be.
+
+Four disqualifiers, four sections. Use the ones that apply.
+
+## 1. What would refute this
+
+Any causal claim states, in the same document, the observation that would
+falsify it — a measurement, not a hedge. "This may not generalise" is not a
+falsification section; "if the effect survives when X is held constant, the
+mechanism is not X, and here is that run" is.
+
+Grade every claim in the report on one visible scale — **measured /
+inferred / conjecture** — and put the grade next to the claim, not in a
+preamble. A reader must be able to strip the report to its measured rows and
+see what is left.
+
+## 2. Did the path survive
+
+A summary statistic computed over a path can describe a path that ended. A
+positive year through an account that was liquidated in March; a mean latency
+across an interval containing an outage; a mean reward over a trajectory that
+terminated; an eventual success after a retry budget was exhausted.
+
+Whenever the process has an **absorbing state** — a barrier it cannot return
+from — the report carries a survived yes/no column and the date of absorption,
+beside the final value. The expected value across many runs and the outcome of
+the one run you get are different quantities, and only the second one is
+yours.
+
+## 3. Before and after the fix
+
+When a defect is corrected mid-analysis, report the headline metric under both
+readings in the same message, with the sign. A table that quietly contains
+both the pre-fix and post-fix cell has published the reversal without saying
+it, and the number that travels onward will be whichever one someone quoted
+first.
+
+## 4. Which unvaried assumption flips the sign
+
+Every report names its frozen constants — the cost, the threshold, the
+window, the rate — and, for each one that appears in the metric expression,
+the value at which the conclusion reverses. If no such value exists inside a
+plausible range, say so. Disclosure is not this: repeating an assumption
+twelve times is not the same as testing it once.
+
+An axis you swept that changed no output row is also a finding. Report it as
+inert rather than presenting its cells as distinct results.
+
+## Must never
+
+- Ship a gate that checks a section's *heading* and calls that a content
+  check. `grep -qi "falsification"` passes a report that says no
+  falsification was attempted. Assert the property, then prove the gate by
+  deleting the section body and watching it fail —
+  [[principle-assert-invariants-not-last-bug]], step 5.
+- Present a headline figure whose cost, fill, or capacity basis is absent.
+  Prefix it `PROVISIONAL —` until that basis and its sensitivity range are in
+  the same message.
+- Let a fan-out end without a terminal aggregation step. A finding stated
+  independently in several sibling reports, and read by nobody across them,
+  is the most expensive kind of undiscovered result.
+
+## Grounding
+
+- Falsifiability as the demarcation of a real claim → Karl Popper, *The Logic
+  of Scientific Discovery* (1959; Ger. 1934).
+- Committing to the refuting test *before* seeing the result → Chris Chambers
+  and Loukia Tzavella, "The past, present and future of Registered Reports,"
+  *Nature Human Behaviour* 6, 29-42 (2022)
+  <https://doi.org/10.1038/s41562-021-01193-7>.
+- Grading each claim on one visible evidence scale → Gordon H. Guyatt et al.,
+  "GRADE: an emerging consensus on rating quality of evidence and strength of
+  recommendations," *BMJ* 336, 924-926 (2008)
+  <https://doi.org/10.1136/bmj.39489.470347.AD>.
+- Why an ensemble average does not describe the single path when the path can
+  be absorbed → Ole Peters, "The ergodicity problem in economics," *Nature
+  Physics* 15, 1216-1221 (2019) <https://doi.org/10.1038/s41567-019-0732-0>;
+  J. L. Kelly Jr., "A New Interpretation of Information Rate," *Bell System
+  Technical Journal* 35(4), 917-926 (1956)
+  <https://doi.org/10.1002/j.1538-7305.1956.tb03809.x>.
+- Reporting the whole space of defensible specifications rather than one →
+  Uri Simonsohn, Joseph P. Simmons, Leif D. Nelson, "Specification curve
+  analysis," *Nature Human Behaviour* 4, 1208-1214 (2020)
+  <https://doi.org/10.1038/s41562-020-0912-z>; Andrea Saltelli et al., "Why so
+  many published sensitivity analyses are false," *Environmental Modelling &
+  Software* 114, 29-39 (2019)
+  <https://doi.org/10.1016/j.envsoft.2019.01.012>.
+- Findings that exist only across sibling reports and are never aggregated →
+  Robert Rosenthal, "The file drawer problem and tolerance for null results,"
+  *Psychological Bulletin* 86(3), 638-641 (1979)
+  <https://doi.org/10.1037/0033-2909.86.3.638>.
+- The "no losing day" / too-good smell test lives in [[principle-no-lookahead]],
+  not here — this skill is about what a report must disclose, that one is
+  about what an evaluation may read.

corpus/skills/principle-report-the-disqualifier/tests/fires_example.md

@@ -0,0 +1,24 @@
+`disable-model-invocation: true` means the model never reads this
+skill's `description:` to decide whether to apply it -- that text isn't
+even loaded into context. The only way this skill activates is an
+explicit `/principle-report-the-disqualifier` invocation.
+
+The agent has drafted an eval write-up whose headline reads "agent success
+rate rose from 61% to 88% after the retrieval change." Two facts sit in the
+last paragraph, after the tables and the recommendation: a scoring bug was
+found and fixed halfway through, and the pre-fix reading of the same
+headline was 63%; and the 88% counts runs that only succeeded after the
+retry budget was exhausted and the harness restarted them. The cost per run
+is frozen at one assumed value that appears in the metric expression and was
+never swept. Before sending the write-up, the agent explicitly invokes
+`/principle-report-the-disqualifier` to load the full principle.
+
+This skill fires here specifically because of that explicit invocation
+-- a causal claim carrying its own reversal, its own absorbing state, and an
+unvaried constant, all of them placed after the conclusion, is exactly the
+pattern the skill targets once loaded (state the observation that would
+refute the claim, grade each claim measured/inferred/conjecture beside the
+claim, add a survived column for the exhausted-budget runs, report the
+headline under both readings with the sign, and name the cost value at which
+the conclusion reverses), but no amount of matching prose alone would have
+triggered it.

corpus/skills/principle-report-the-disqualifier/tests/stays_silent_example.md

@@ -0,0 +1,13 @@
+The agent is asked how many open pull requests currently carry a given
+label, runs the query, and answers with the count and the list. There is no
+causal claim, nothing accumulates over a path, no defect was corrected
+partway through, and no constant appears in the answer that could be varied
+to reverse it. No explicit invocation of this skill happens anywhere in the
+session.
+
+This skill stays silent here: with `disable-model-invocation: true`,
+nothing about the conversation's content could have triggered it either
+way. A direct count of a present-tense state has no disqualifier to lead
+with -- there is no observation that would refute it short of the query
+being wrong, and no summary statistic hiding a path that ended. Even if
+this skill somehow were invoked, all four of its sections would be empty.