From 0a690f5ee8bc015ecee3425927c402f04e127ada Mon Sep 17 00:00:00 2001 From: Edbert Chan Date: Wed, 9 Sep 2026 09:14:08 -0700 Subject: [PATCH 1/3] Add principle-no-lookahead and principle-report-the-disqualifier Problem: a backtesting session produced results for roughly three hours from a signal that could not have been known at the time it was credited to, and the test written to validate it asserted against a helper built to make that test pass rather than against the path every consumer read. Nothing in the corpus named either failure, so neither was catchable. Considerations: two skills rather than one, because the failures are separable. The first is about which inputs a decision may read; the second is about what a report must disclose once the evaluation is clean. Both carry the mechanical test rather than only prose: the truncation replay for lookahead, and the required Falsification and Confidence sections for the report. Each cites the literature that names the failure, and where none exists the entry says so rather than inventing a lineage: "repainting" is recorded as vendor and forum vocabulary with no peer-reviewed prior art. Blast radius: two new skill directories under corpus/skills. No existing skill, hook, or install path is modified. Verification: both files carry frontmatter with name and description; the lookahead skill cites Kaufman et al. TKDD 2012 for the formalisation of leakage, Tashman IJF 2000 for rolling-origin evaluation, and Bailey et al. Notices AMS 2014 for backtest overfitting; the disqualifier skill cites Popper, Chambers and Tzavella 2022, Guyatt et al. BMJ 2008, Peters 2019 and Kelly 1956 for the absorbing-barrier case, and Simonsohn et al. 2020. Prose and citations were recovered verbatim from the reflect subagent transcript rather than rewritten. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01F43CBUnsDEs6J2zEC1r8a8 --- corpus/skills/principle-no-lookahead/SKILL.md | 132 ++++++++++++++++++ .../SKILL.md | 105 ++++++++++++++ 2 files changed, 237 insertions(+) create mode 100644 corpus/skills/principle-no-lookahead/SKILL.md create mode 100644 corpus/skills/principle-report-the-disqualifier/SKILL.md diff --git a/corpus/skills/principle-no-lookahead/SKILL.md b/corpus/skills/principle-no-lookahead/SKILL.md new file mode 100644 index 00000000..ea2b4c9e --- /dev/null +++ b/corpus/skills/principle-no-lookahead/SKILL.md @@ -0,0 +1,132 @@ +--- +name: principle-no-lookahead +description: "Apply to any evaluation, backtest, benchmark, grader, or training run over data with a time axis or an ordered decision sequence. Score a decision only against inputs that existed before that decision was made. Covers what the field calls look-ahead bias, target leakage, point-in-time correctness, and repainting. Reach for it on: a backtest or walk-forward run, a feature computed over a full history, a retrieval or agent eval, a label assigned after the outcome was known, or any result that looks too good." +disable-model-invocation: true +--- + +# No lookahead + +A result is a result only if the decision behind it could have been made at +the moment it is credited to. Otherwise it is a measurement of hindsight. + +The failure has a formal name: **leakage** — "the introduction of information +about the target of a data mining problem which should not be legitimately +available to mine from" (Kaufman et al., see Grounding). In time-series work +it is **look-ahead bias**; the correctness property it violates is +**point-in-time correctness**; the technical-analysis artifact that causes it +is a **repainting** indicator. One defect, four vocabularies. + +## The shape + +Two artifacts get confused because they are rendered from the same data: + +| | Display artifact | Decision artifact | +|---|---|---| +| Answers | what happened | what would I have done | +| May read | the whole history | only rows at or before `t` | +| Revised later | legitimately, yes | never | + +A chart, a labelled dataset, an annotated log, a resolved ticket, and a +post-hoc report are all display artifacts. They are drawn *after*, and are +allowed to move a marker back onto the moment the thing "really" happened. +Feed one into a decision evaluation and the decision is credited with +knowledge it did not have. + +**The leaked value can be entirely real and the leak still total.** The leak +is in *when the decision became available*, not in the number's accuracy. +A price, a label, or a document can be genuine at time `t` while the +instruction to act on it does not exist until `t + k`. + +## Must always + +- **Name the cutoff before the first measurement.** For every input, record + two timestamps: what it is *about*, and when it became *knowable*. Only + the second one may gate a decision. If a source cannot tell you the + second, it is a display artifact until proven otherwise. +- **Run the truncation test.** Recompute the decision from only the data + available at time `t`, and compare it to what the full-history computation + says at that same `t`. Any disagreement at any `t` is a leak, and the + disagreeing indices name it. Ten cut points expose a gross leak in + seconds; a replay at every step is the proof. +- **Separate knowability from actionability.** The truncation test proves the + input existed at `t`. It does not prove the action was executable at `t`. + Name the execution cutoff too: the earliest moment the decision could have + reached the world, and the price, latency, or capacity it would have met + there. A decision that observes a bar's close and fills at that same close + is zero-latency, and zero latency is a form of hindsight the prefix test + cannot see. +- **Bind the test to the artifact that produced the number**, not to a + library function beside it. A causality test that guards `compute()` + proves nothing about a script that imports `compute()` and passes it a + different mode. The gate belongs on the path the reported number came out + of — including throwaway analysis scripts, notebooks, and heredocs. +- **Treat an implausibly good result as a bug report, not an achievement.** + Fix the implausibility ceiling from the domain's own known best *before* + running, and stop on any of: a score above what the best known + practitioner achieves; an error near zero; a monotone result with no bad + period across a span long enough to contain one; a metric that improves + when an input that should be irrelevant is added; a parameter sweep where + every cell wins. The stronger the number, the earlier you stop. Stop and + find the mechanism — leakage, an inert axis that never varied the output, a + frozen constant, or a coding error — before reporting the number at all. + Report the number of configurations tried alongside the best one; without + it, a reader cannot tell skill from selection. +- **Re-run the cutoff check after any change to what an input reads** — a + new normalization, a rolling window, a resample, a join, a smoothing term, + a re-label. Each can silently widen the window a value is computed over. +- **Reject a negative offset.** Any shift, lag, or window parameter exposed + to a caller must refuse values that reach forward. An unvalidated + `shift(k)` is a lookahead switch with no label on it. + +## Must never + +- Score a decision against a label that was assigned with knowledge of the + outcome. +- Accept "it only reads past data" from a grep, a variable name, a + docstring, or a commit message. Causality is a property of the output, + measured. It is not a property of the source, read. +- Let a display mode and a decision mode share a return type. If one + function can emit both, a caller will eventually pass the wrong one and + nothing will say so. Make the display result structurally unable to reach + the scoring path. +- Report a result whose sign depends on an assumption that was never varied. + If one unswept constant flips the conclusion, that constant is the finding. +- Publish a number produced before the truncation test ran. On discovery, + void the old numbers **in place** — move them to a `voided/` path and + retract the headline — rather than deleting them, so the record shows what + was believed and for how long. + +## The cheap test, in general form + +## Grounding + +- Leakage, formalized, with the legitimacy-in-time condition and the + learn-predict separation that avoids it → Shachar Kaufman, Saharon Rosset, + Claudia Perlich, Ori Stitelman, "Leakage in Data Mining: Formulation, + Detection, and Avoidance," *ACM TKDD* 6(4), Article 15 (2012) + . This is the paper that names + this exact failure; its KDD-Cup and INFORMS case studies are both + "implausibly good result turned out to be a leak." +- Evaluating only from an origin that moves forward through the data → + Leonard J. Tashman, "Out-of-sample tests of forecasting accuracy: an + analysis and review," *International Journal of Forecasting* 16(4), + 437-450 (2000) . Rolling + origin is the truncation test as a standard evaluation design. +- Why a too-good backtest is evidence about the search process rather than + the strategy, and why the count of configurations tried must be reported → + David H. Bailey, Jonathan M. Borwein, Marcos López de Prado, Qiji Jim Zhu, + "Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest + Overfitting on Out-of-Sample Performance," *Notices of the AMS* 61(5), + 458-471 (2014) . +- Purged cross-validation and embargo, for label overlap leaking across a + split → Marcos López de Prado, *Advances in Financial Machine Learning*, + Wiley (2018), ch. 7 and ch. 11. +- "Repainting," for a display artifact that moves a marker backward: + **no known peer-reviewed prior art.** It is vendor and forum vocabulary + only. Cite the mechanism, not the word. + +Related: [[principle-assert-invariants-not-last-bug]] (the truncation test is +the class invariant, not a patch for the leak you found); +[[principle-explicit-errors]] (a suppressed or unvalidated offset is how the +leak gets in silently); [[principle-report-the-disqualifier]] (what the +report must disclose once the evaluation is clean). diff --git a/corpus/skills/principle-report-the-disqualifier/SKILL.md b/corpus/skills/principle-report-the-disqualifier/SKILL.md new file mode 100644 index 00000000..0cc2f158 --- /dev/null +++ b/corpus/skills/principle-report-the-disqualifier/SKILL.md @@ -0,0 +1,105 @@ +--- +name: principle-report-the-disqualifier +description: "Apply when writing or gating any report that makes a causal claim or summarises a cumulative outcome — a backtest, an experiment write-up, a capacity or cost model, a postmortem, an eval summary, a benchmark table. The report must carry what would change the reader's decision: what observation would refute the claim, whether the path crossed a state it could not return from, what the headline was before and after the fix, and which unvaried assumption flips the sign." +disable-model-invocation: true +--- + +# Report the disqualifier + +A report is not finished when its numbers are correct. It is finished when a +reader who disagrees with it knows exactly where to look. Every claim has a +disqualifier — one observation that would sink it — and a report that omits +its own disqualifier is persuasive in proportion to how wrong it can be. + +Four disqualifiers, four sections. Use the ones that apply. + +## 1. What would refute this + +Any causal claim states, in the same document, the observation that would +falsify it — a measurement, not a hedge. "This may not generalise" is not a +falsification section; "if the effect survives when X is held constant, the +mechanism is not X, and here is that run" is. + +Grade every claim in the report on one visible scale — **measured / +inferred / conjecture** — and put the grade next to the claim, not in a +preamble. A reader must be able to strip the report to its measured rows and +see what is left. + +## 2. Did the path survive + +A summary statistic computed over a path can describe a path that ended. A +positive year through an account that was liquidated in March; a mean latency +across an interval containing an outage; a mean reward over a trajectory that +terminated; an eventual success after a retry budget was exhausted. + +Whenever the process has an **absorbing state** — a barrier it cannot return +from — the report carries a survived yes/no column and the date of absorption, +beside the final value. The expected value across many runs and the outcome of +the one run you get are different quantities, and only the second one is +yours. + +## 3. Before and after the fix + +When a defect is corrected mid-analysis, report the headline metric under both +readings in the same message, with the sign. A table that quietly contains +both the pre-fix and post-fix cell has published the reversal without saying +it, and the number that travels onward will be whichever one someone quoted +first. + +## 4. Which unvaried assumption flips the sign + +Every report names its frozen constants — the cost, the threshold, the +window, the rate — and, for each one that appears in the metric expression, +the value at which the conclusion reverses. If no such value exists inside a +plausible range, say so. Disclosure is not this: repeating an assumption +twelve times is not the same as testing it once. + +An axis you swept that changed no output row is also a finding. Report it as +inert rather than presenting its cells as distinct results. + +## Must never + +- Ship a gate that checks a section's *heading* and calls that a content + check. `grep -qi "falsification"` passes a report that says no + falsification was attempted. Assert the property, then prove the gate by + deleting the section body and watching it fail — + [[principle-assert-invariants-not-last-bug]], step 5. +- Present a headline figure whose cost, fill, or capacity basis is absent. + Prefix it `PROVISIONAL —` until that basis and its sensitivity range are in + the same message. +- Let a fan-out end without a terminal aggregation step. A finding stated + independently in several sibling reports, and read by nobody across them, + is the most expensive kind of undiscovered result. + +## Grounding + +- Falsifiability as the demarcation of a real claim → Karl Popper, *The Logic + of Scientific Discovery* (1959; Ger. 1934). +- Committing to the refuting test *before* seeing the result → Chris Chambers + and Loukia Tzavella, "The past, present and future of Registered Reports," + *Nature Human Behaviour* 6, 29-42 (2022) + . +- Grading each claim on one visible evidence scale → Gordon H. Guyatt et al., + "GRADE: an emerging consensus on rating quality of evidence and strength of + recommendations," *BMJ* 336, 924-926 (2008) + . +- Why an ensemble average does not describe the single path when the path can + be absorbed → Ole Peters, "The ergodicity problem in economics," *Nature + Physics* 15, 1216-1221 (2019) ; + J. L. Kelly Jr., "A New Interpretation of Information Rate," *Bell System + Technical Journal* 35(4), 917-926 (1956) + . +- Reporting the whole space of defensible specifications rather than one → + Uri Simonsohn, Joseph P. Simmons, Leif D. Nelson, "Specification curve + analysis," *Nature Human Behaviour* 4, 1208-1214 (2020) + ; Andrea Saltelli et al., "Why so + many published sensitivity analyses are false," *Environmental Modelling & + Software* 114, 29-39 (2019) + . +- Findings that exist only across sibling reports and are never aggregated → + Robert Rosenthal, "The file drawer problem and tolerance for null results," + *Psychological Bulletin* 86(3), 638-641 (1979) + . +- The "no losing day" / too-good smell test lives in [[principle-no-lookahead]], + not here — this skill is about what a report must disclose, that one is + about what an evaluation may read. From bff693d9619e6c0cca6e255fc13222bade2e2f66 Mon Sep 17 00:00:00 2001 From: Edbert Chan Date: Wed, 9 Sep 2026 11:46:10 -0700 Subject: [PATCH 2/3] Add report-rendering: ship one self-contained HTML plus PDF Problem: a three-section analysis totalling 220 KB of markdown was described as finished while it sat as separate files in a repo. Asked to see the report, the honest answer was a list of paths. The reader either lacks a checkout or must open several files in the right order and imagine the figures in place. Considerations: the recipe deliberately requires nothing beyond a browser, because a rendering step that needs pandoc or LaTeX gets skipped. Figures embed as data URIs so the file survives being moved or emailed, which a relative path does not. Print styles are called out separately because their absence is invisible in the browser and only shows up in the PDF, where sticky table headers repeat on every page. Blast radius: one new skill directory under corpus/skills. No existing skill, hook, or install path is modified. Verification: the recipe as written produced reports/master-report.html at 885,056 bytes with the distribution figure inline, and reports/master-report.pdf at 4,477,296 bytes via `google-chrome --headless --print-to-pdf`, in the trend-reversal repo. The md.reset() note comes from the toc extension carrying anchors between sections during that run. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01F43CBUnsDEs6J2zEC1r8a8 --- corpus/skills/report-rendering/SKILL.md | 85 +++++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100644 corpus/skills/report-rendering/SKILL.md diff --git a/corpus/skills/report-rendering/SKILL.md b/corpus/skills/report-rendering/SKILL.md new file mode 100644 index 00000000..291a1933 --- /dev/null +++ b/corpus/skills/report-rendering/SKILL.md @@ -0,0 +1,85 @@ +--- +name: report-rendering +description: "Apply when a report, analysis, or write-up is finished and someone other than the author will read it. Ship it as a single self-contained HTML file, and a PDF where a fixed page is wanted. Markdown split across several files in a repo is a working artifact, not a delivered report. Reach for it on: a multi-section analysis, a report with figures, anything to be handed to someone without a checkout, or any request to 'show me the report'." +disable-model-invocation: true +--- + +# Render the report + +A report that only exists as markdown in a repo has not been delivered. It has +been left where the author was working. The reader either lacks a checkout, or +has one and must open several files in the right order and imagine the figures +in place. + +Ship one file that opens by double-clicking it. + +## Must always + +- **One self-contained HTML file.** Embed every figure as a `data:` URI rather + than linking a path. A document that breaks when moved is not a deliverable. + Inline the CSS for the same reason. +- **Concatenate the sections in reading order**, with a table of contents and + each source path shown beside its section, so a reader can find the file + behind any claim. +- **Add print styles.** `page-break-before` on each top-level section, and + unstick any `position: sticky` table headers, which otherwise repeat or + vanish across page boundaries. Without this the PDF is unusable even though + the HTML looks correct. +- **Produce the PDF from the same HTML**, so the two cannot disagree. +- **Commit both**, and say where they are. A rendered report that lives only in + a temp directory has the same problem as the markdown did. + +## Must never + +- Hand back a list of markdown paths in response to "show me the report." +- Link figures by relative path in a document intended to be sent anywhere. +- Reach for pandoc, LaTeX, or a new dependency when the machine already has a + browser. The recipe below needs nothing else. +- Render a report whose numbers have not passed their own gate. Rendering makes + a claim easier to circulate, which is a reason to be more careful about what + is in it, not less. See [[principle-report-the-disqualifier]]. + +## The recipe + +Markdown to self-contained HTML to PDF, with no toolchain beyond a browser: + +```python +import base64, markdown +from pathlib import Path + +md = markdown.Markdown(extensions=["tables", "fenced_code", "toc", "sane_lists"]) +parts = [] +for title, path in SECTIONS: + md.reset() + parts.append(f"

{title}

{md.convert(Path(path).read_text())}
") + +fig = base64.b64encode(Path(FIGURE).read_bytes()).decode() +parts.append(f'
') +Path("report.html").write_text(TEMPLATE.format(body="".join(parts))) +``` + +```sh +google-chrome --headless --disable-gpu --no-sandbox \ + --print-to-pdf=report.pdf --print-to-pdf-no-header report.html +``` + +`md.reset()` between sections matters: the `toc` extension accumulates state +across calls, so without it later sections inherit earlier anchors. + +## Grounding + +- Single-file, self-contained documents as a distribution format → the MHTML + and `data:` URI approach; RFC 2397, "The 'data' URL scheme" (1998) + . +- Literate and reproducible reporting, where the document and the analysis that + produced it are one artifact → Donald E. Knuth, "Literate Programming," + *The Computer Journal* 27(2), 97-111 (1984) + ; Roger D. Peng, "Reproducible + Research in Computational Science," *Science* 334(6060), 1226-1227 (2011) + . +- Delivering a result in the form the reader can act on, rather than the form + the author worked in: **no known prior art** as a stated principle; it is + ordinary practice rather than a named result. + +Related: [[principle-report-the-disqualifier]] (what the report must contain +before it is worth rendering). From 45b6a771533286d79dc5885eb68913454cd77b5d Mon Sep 17 00:00:00 2001 From: Edbert Chan Date: Wed, 9 Sep 2026 13:03:31 -0700 Subject: [PATCH 3/3] Add trigger fixtures for the three new corpus skills principle-no-lookahead, principle-report-the-disqualifier, and report-rendering shipped without the positive/negative trigger fixtures that check_skill_test_coverage.py requires of every prose skill, so the test job failed six ways: three for the missing tests/ dir and three for changing a skill without a matching test change. Each skill now has tests/fires_example.md and tests/stays_silent_example.md. All three set disable-model-invocation: true, so each fires fixture carries the literal / invocation and each stays-silent fixture does not, which is what check_skill_trigger_mechanism.py verifies for that tier. The negatives are near misses rather than unrelated scenarios: a postmortem timeline that reads the whole history but scores no decision, a present-tense count with no causal claim or absorbing state, and a README edit read only by people who already have the checkout. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01F43CBUnsDEs6J2zEC1r8a8 --- .../tests/fires_example.md | 21 ++++++++++++++++ .../tests/stays_silent_example.md | 15 ++++++++++++ .../tests/fires_example.md | 24 +++++++++++++++++++ .../tests/stays_silent_example.md | 13 ++++++++++ .../report-rendering/tests/fires_example.md | 21 ++++++++++++++++ .../tests/stays_silent_example.md | 13 ++++++++++ 6 files changed, 107 insertions(+) create mode 100644 corpus/skills/principle-no-lookahead/tests/fires_example.md create mode 100644 corpus/skills/principle-no-lookahead/tests/stays_silent_example.md create mode 100644 corpus/skills/principle-report-the-disqualifier/tests/fires_example.md create mode 100644 corpus/skills/principle-report-the-disqualifier/tests/stays_silent_example.md create mode 100644 corpus/skills/report-rendering/tests/fires_example.md create mode 100644 corpus/skills/report-rendering/tests/stays_silent_example.md diff --git a/corpus/skills/principle-no-lookahead/tests/fires_example.md b/corpus/skills/principle-no-lookahead/tests/fires_example.md new file mode 100644 index 00000000..b155590f --- /dev/null +++ b/corpus/skills/principle-no-lookahead/tests/fires_example.md @@ -0,0 +1,21 @@ +`disable-model-invocation: true` means the model never reads this +skill's `description:` to decide whether to apply it -- that text isn't +even loaded into context. The only way this skill activates is an +explicit `/principle-no-lookahead` invocation. + +The agent has loaded a full price series into one dataframe and computed a +volume-spike z-score column over the whole thing -- the mean and standard +deviation in every row's denominator are taken across all bars, including +bars that come after that row. It is about to score an entry decision at +each past bar against that column and report the annualised return, which +came out at 41% with no losing month in the span. Before reporting the +number, the agent explicitly invokes `/principle-no-lookahead` to load the +full principle. + +This skill fires here specifically because of that explicit invocation +-- a feature computed over the full history and then credited to a +decision at an earlier bar is exactly the pattern the skill targets once +loaded (record what each input is about and when it became knowable, run +the truncation test at ten cut points and compare against the full-history +column, treat a no-losing-month span as a bug report rather than a result), +but no amount of matching prose alone would have triggered it. diff --git a/corpus/skills/principle-no-lookahead/tests/stays_silent_example.md b/corpus/skills/principle-no-lookahead/tests/stays_silent_example.md new file mode 100644 index 00000000..0ad0e5d5 --- /dev/null +++ b/corpus/skills/principle-no-lookahead/tests/stays_silent_example.md @@ -0,0 +1,15 @@ +The agent is asked to draw an annotated incident timeline for a postmortem +slide: read the whole quarter of alert history, mark where the outage began, +and label each marker with the cause that was only established afterwards. +The chart reads every row of history, including rows after each marker. No +explicit invocation of this skill happens anywhere in the session -- the +agent just draws the chart. + +This skill stays silent here: with `disable-model-invocation: true`, +nothing about the conversation's content could have triggered it either +way. Nothing here is a decision artifact either -- the timeline answers +what happened, not what would I have done, and no score is credited to a +choice made at any marker. A display artifact is allowed to read the whole +history and to move a label back onto the moment the thing really began, so +even if this skill somehow were invoked, its content would not apply until +that chart were fed into an evaluation. diff --git a/corpus/skills/principle-report-the-disqualifier/tests/fires_example.md b/corpus/skills/principle-report-the-disqualifier/tests/fires_example.md new file mode 100644 index 00000000..335097a6 --- /dev/null +++ b/corpus/skills/principle-report-the-disqualifier/tests/fires_example.md @@ -0,0 +1,24 @@ +`disable-model-invocation: true` means the model never reads this +skill's `description:` to decide whether to apply it -- that text isn't +even loaded into context. The only way this skill activates is an +explicit `/principle-report-the-disqualifier` invocation. + +The agent has drafted an eval write-up whose headline reads "agent success +rate rose from 61% to 88% after the retrieval change." Two facts sit in the +last paragraph, after the tables and the recommendation: a scoring bug was +found and fixed halfway through, and the pre-fix reading of the same +headline was 63%; and the 88% counts runs that only succeeded after the +retry budget was exhausted and the harness restarted them. The cost per run +is frozen at one assumed value that appears in the metric expression and was +never swept. Before sending the write-up, the agent explicitly invokes +`/principle-report-the-disqualifier` to load the full principle. + +This skill fires here specifically because of that explicit invocation +-- a causal claim carrying its own reversal, its own absorbing state, and an +unvaried constant, all of them placed after the conclusion, is exactly the +pattern the skill targets once loaded (state the observation that would +refute the claim, grade each claim measured/inferred/conjecture beside the +claim, add a survived column for the exhausted-budget runs, report the +headline under both readings with the sign, and name the cost value at which +the conclusion reverses), but no amount of matching prose alone would have +triggered it. diff --git a/corpus/skills/principle-report-the-disqualifier/tests/stays_silent_example.md b/corpus/skills/principle-report-the-disqualifier/tests/stays_silent_example.md new file mode 100644 index 00000000..ea3bfdc4 --- /dev/null +++ b/corpus/skills/principle-report-the-disqualifier/tests/stays_silent_example.md @@ -0,0 +1,13 @@ +The agent is asked how many open pull requests currently carry a given +label, runs the query, and answers with the count and the list. There is no +causal claim, nothing accumulates over a path, no defect was corrected +partway through, and no constant appears in the answer that could be varied +to reverse it. No explicit invocation of this skill happens anywhere in the +session. + +This skill stays silent here: with `disable-model-invocation: true`, +nothing about the conversation's content could have triggered it either +way. A direct count of a present-tense state has no disqualifier to lead +with -- there is no observation that would refute it short of the query +being wrong, and no summary statistic hiding a path that ended. Even if +this skill somehow were invoked, all four of its sections would be empty. diff --git a/corpus/skills/report-rendering/tests/fires_example.md b/corpus/skills/report-rendering/tests/fires_example.md new file mode 100644 index 00000000..98caf60b --- /dev/null +++ b/corpus/skills/report-rendering/tests/fires_example.md @@ -0,0 +1,21 @@ +`disable-model-invocation: true` means the model never reads this +skill's `description:` to decide whether to apply it -- that text isn't +even loaded into context. The only way this skill activates is an +explicit `/report-rendering` invocation. + +The agent has finished a five-section spend analysis. It exists as five +markdown files and three PNG charts in a scratch directory, and each section +links its figures by relative path. The user says "show me the report," and +the agent is about to reply with the list of file paths -- to a reader who +has no checkout and will forward this to two other people. Before replying, +the agent explicitly invokes `/report-rendering` to load the full principle. + +This skill fires here specifically because of that explicit invocation +-- a finished multi-section analysis with figures, about to be handed to +someone as markdown paths, is exactly the pattern the skill targets once +loaded (concatenate the sections in reading order into one self-contained +HTML file with a table of contents and each source path beside its section, +embed each chart as a data URI, add the page-break and unstick-sticky-header +print styles, print the PDF from that same HTML with headless Chrome, and +commit both rather than leaving them in the scratch directory), but no +amount of matching prose alone would have triggered it. diff --git a/corpus/skills/report-rendering/tests/stays_silent_example.md b/corpus/skills/report-rendering/tests/stays_silent_example.md new file mode 100644 index 00000000..a3b7f171 --- /dev/null +++ b/corpus/skills/report-rendering/tests/stays_silent_example.md @@ -0,0 +1,13 @@ +The agent has just changed a build script and is updating the two paragraphs +of the repo README that describe how to run it. The result is markdown in a +repo, it has no figures, and its readers are the people who already have the +checkout open in front of them. No explicit invocation of this skill happens +anywhere in the session. + +This skill stays silent here: with `disable-model-invocation: true`, +nothing about the conversation's content could have triggered it either +way. Nothing is being delivered to a reader outside the checkout, so +markdown in the repo is the finished form rather than the working one, and +there is nothing to embed or paginate. Even if this skill somehow were +invoked, rendering a README to a standalone HTML file and a PDF would leave +the copy readers actually open unchanged.