[auto] Add principle-no-lookahead, principle-report-the-disqualifier, report-rendering - #300
Open
EdbertChan wants to merge 2 commits into
Open
[auto] Add principle-no-lookahead, principle-report-the-disqualifier, report-rendering#300EdbertChan wants to merge 2 commits into
EdbertChan wants to merge 2 commits into
Conversation
Problem: a backtesting session produced results for roughly three hours from a signal that could not have been known at the time it was credited to, and the test written to validate it asserted against a helper built to make that test pass rather than against the path every consumer read. Nothing in the corpus named either failure, so neither was catchable. Considerations: two skills rather than one, because the failures are separable. The first is about which inputs a decision may read; the second is about what a report must disclose once the evaluation is clean. Both carry the mechanical test rather than only prose: the truncation replay for lookahead, and the required Falsification and Confidence sections for the report. Each cites the literature that names the failure, and where none exists the entry says so rather than inventing a lineage: "repainting" is recorded as vendor and forum vocabulary with no peer-reviewed prior art. Blast radius: two new skill directories under corpus/skills. No existing skill, hook, or install path is modified. Verification: both files carry frontmatter with name and description; the lookahead skill cites Kaufman et al. TKDD 2012 for the formalisation of leakage, Tashman IJF 2000 for rolling-origin evaluation, and Bailey et al. Notices AMS 2014 for backtest overfitting; the disqualifier skill cites Popper, Chambers and Tzavella 2022, Guyatt et al. BMJ 2008, Peters 2019 and Kelly 1956 for the absorbing-barrier case, and Simonsohn et al. 2020. Prose and citations were recovered verbatim from the reflect subagent transcript rather than rewritten. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F43CBUnsDEs6J2zEC1r8a8
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_460a78d6-0366-4542-b122-182afa359448) |
Problem: a three-section analysis totalling 220 KB of markdown was described as finished while it sat as separate files in a repo. Asked to see the report, the honest answer was a list of paths. The reader either lacks a checkout or must open several files in the right order and imagine the figures in place. Considerations: the recipe deliberately requires nothing beyond a browser, because a rendering step that needs pandoc or LaTeX gets skipped. Figures embed as data URIs so the file survives being moved or emailed, which a relative path does not. Print styles are called out separately because their absence is invisible in the browser and only shows up in the PDF, where sticky table headers repeat on every page. Blast radius: one new skill directory under corpus/skills. No existing skill, hook, or install path is modified. Verification: the recipe as written produced reports/master-report.html at 885,056 bytes with the distribution figure inline, and reports/master-report.pdf at 4,477,296 bytes via `google-chrome --headless --print-to-pdf`, in the trend-reversal repo. The md.reset() note comes from the toc extension carrying anchors between sections during that run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F43CBUnsDEs6J2zEC1r8a8
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_c05823a5-0592-482b-9cf1-b516cb3b7784) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two principles from a backtesting session that spent roughly three hours producing results from a signal that could not have been known at the time it was credited to.
principle-no-lookahead— score a decision only against inputs that existed before that decision was made. Names the one defect across its four vocabularies (leakage, look-ahead bias, point-in-time correctness, repainting), and carries the cheap mechanical test: recompute from only the data available at timetand compare against the full-history computation; any difference is a leak.principle-report-the-disqualifier— a report is finished when a reader who disagrees knows where to look. Requires a falsification observation, a measured/inferred/conjecture grade per claim, ruin reporting where the path can cross an absorbing barrier, and the count of configurations tried.Review Claim
Both skills state a general principle, name the established prior art with DOIs, and carry a mechanical test rather than only prose.
Safety Invariant
Where no prior art exists it is recorded as such rather than invented: "repainting" is marked vendor and forum vocabulary with no peer-reviewed source.
Motivation
The incident: a study's chart view placed a pivot on the bar it occurred; the live view only from its confirmation bar, 15 to 50 minutes later. A truncation test showed the signal at 00:05 does not exist when the data ends at 00:05, 00:30 or 00:45, and appears when it ends at 00:55. Backtesting the chart view produced 23 green days out of 23 and 38,970% on a 10,000 USD account; the live view liquidated the same account. Neither the too-good result nor the unvalidated code path had a name in the corpus.
Non-goals
No existing skill, hook, or install path is modified. No always-on file is touched.
Test Plan
Manual review. Both files carry valid frontmatter; every claim of prior art resolves to a DOI or a stated absence.
Revert Plan
git revertthe commit; two new directories, no existing file changed.🤖 Generated with Claude Code
https://claude.ai/code/session_01F43CBUnsDEs6J2zEC1r8a8
Note
Low Risk
Adds markdown skill docs under
corpus/skills/only; no runtime, auth, or data-path changes.Overview
Adds three new corpus skills (documentation-only; no hooks or install paths changed).
principle-no-lookaheadcodifies point-in-time evaluation: separate display vs decision artifacts, require cutoffs and a truncation test (prefix recompute vs full history), treat implausibly good metrics as bugs, and cite leakage/rolling-origin literature—with explicit “no peer-reviewed prior art” for repainting vocabulary.principle-report-the-disqualifierdefines what causal or outcome reports must expose: falsifying observations, measured / inferred / conjecture grades, absorbing-state (“did the path survive”) columns, before/after fix headlines, and sensitivity to frozen constants; it cross-links to existing assert-invariants and no-lookahead skills.report-renderingrequires delivering analyses as one self-contained HTML (embeddeddata:figures, print CSS) plus PDF from the same HTML via headless Chrome, with a minimal Python + shell recipe and a gate to not render unvetted numbers.Reviewed by Cursor Bugbot for commit bff693d. Bugbot is set up for automated code reviews on this repo. Configure here.
Added:
report-renderingThird skill on this branch, from the same session. A three-section analysis was
called finished while it sat as separate markdown files in a repo; asked to see
the report, the only answer was a list of paths.
Ship one self-contained HTML file with figures embedded as
data:URIs, plus aPDF printed from that same HTML so the two cannot disagree. The recipe needs
nothing beyond a browser:
markdownto HTML, thengoogle-chrome --headless --print-to-pdf. Print styles are called outexplicitly because their absence is invisible in the browser and only breaks in
the PDF.
Verified by producing
reports/master-report.html(885,056 bytes, figureinline) and
reports/master-report.pdf(4,477,296 bytes).Grounding: RFC 2397 for
data:URIs, Knuth 1984 and Peng 2011 for thedocument-and-analysis-as-one-artifact lineage, and an explicit "no known prior
art" for the delivery-format principle itself.