docs(main): ATR-only baseline on Garak + PINT (per #66)

[eeee2345]

@hiskudin — as you asked in #66, here it is as a PR rather than a comment: the reproducible ATR-only baseline on Garak and PINT.

benchmarks/atr-baseline.md contains:

• Garak — in-scope recall 74.4% (1,321/1,775 across 21 families), with the two out-of-scope families (snowball, harmbench) called out and excluded from the headline number.
• PINT — precision 99.7% / recall 63.6% / F1 77.7% (TP=287, FP=1, TN=398, FN=164); PINT is the only corpus here with a real measured precision, since it carries benign samples.
• Reproduce — pinned to agent-threat-rules commit 1831d0d5, with the exact scripts to regenerate both reports.

The Defender-side column is intentionally left blank — this is the ATR half of a like-for-like table, for your team to fill in on the same corpora, per your offer to review on your side.

Happy to adjust the file's location or format, or to align on the tool-result-focused subset you raised in #66 as a starting point for a bundled pack.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[Copilot]

Pull request overview

Adds a reproducible benchmark note capturing an ATR-only baseline on two corpora (Garak and Lakera PINT) to support the ongoing discussion in issue #66 about potential ATR rule bundling and like-for-like comparison against Defender.

Changes:

• Documented pinned ATR baseline results for Garak (recall) and PINT (precision/recall/F1), including scope caveats.
• Added step-by-step reproduction commands pinned to a specific agent-threat-rules commit for repeatability.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[hiskudin]

Defender column (same corpora as this baseline)

Ran locally against the same Garak + PINT inputs pinned in this PR (agent-threat-rules@1831d0d5 corpora). Criterion: @stackone/defenderdefendToolResult with blockHighRisk: true → count as positive when allowed === false.

Engine	PINT P	PINT R	PINT F1	PINT confusion	Garak in-scope R (1,775)	Garak overall R (3,475)
ATR (this PR)	99.7%	63.6%	77.7%	TP=287 FP=1 TN=398 FN=164	74.4% (1,321/1,775)	38.2% (1,329/3,475)
Defender Tier 1	100%	12.9%	22.8%	TP=58 FP=0 TN=399 FN=393	33.5% (595/1,775)	17.1% (595/3,475)
Defender Tier 1+2	82.1%	87.6%	84.8%	TP=395 FP=86 TN=313 FN=56	76.8% (1,363/1,775)	41.7% (1,448/3,475)

Notes for reading this honestly:

• Reporting both T1 and T1+T2 — a single “Defender” cell would be misleading (T1 is a small curated regex pack; T2 carries most recall on these chat/jailbreak sets).
• ATR still leads on PINT precision; Defender T1+T2 leads on PINT recall/F1 and is roughly tied with ATR on Garak in-scope recall.
• These corpora are jailbreak/PINT-shaped, not tool-result traffic — still want an FP check on tool-like benign data before deciding on an opt-in ATR pack (ATR rule-pack as a Tier 1 supplement (462 patterns, MIT) + cross-benchmark offer #66).

Happy to fold this into benchmarks/atr-baseline.md once this PR lands (or as a follow-up docs commit).

[eeee2345]

@hiskudin — thank you for running it rather than taking the table at face value, and for reporting T1 and T1+T2 separately. A single "Defender" cell would have been the easier and worse choice.

Two things I owe you, both about the ATR column you compared against.

The PINT row was mislabelled, and I have corrected it in this PR. I described that corpus as "Lakera's public Prompt Injection Test set". It is not. It is a self-built 850-sample corpus in PINT's format, assembled from deepset/prompt-injections and Lakera/gandalf_ignore_instructions; Lakera's actual benchmark is private and roughly five times larger. The measurements are unchanged — the label was wrong. It also carries a scope caveat worth knowing before drawing a conclusion from the precision figure: only a minority of the ruleset fires on that corpus at all, and a single rule accounts for most of the detections. So "ATR leads on PINT precision" is true of this corpus and says less about the ruleset as a whole than the number suggests.

The ATR figures you compared against are pinned to an older commit, and the newer ones moved more than I expected. Everything in this document is 1831d0d5 (v3.5.7, 714 rules), which is the right way to do it — but the upstream table has since been re-measured on 3.5.11, and the all-families garak recall is materially higher than the 38.2% recorded here. That is the row your "roughly tied on Garak in-scope recall" reading rests on, so it may not survive a re-run. I have made the pinning explicit in the doc rather than quietly refreshing the numbers, because a comparison across two different ATR versions would be worse than a stale one.

If you want the comparison to stand as a decision input for #66, the clean version is both engines re-run on one newer commit. I am happy to produce the ATR column for whichever commit you pick, and to have the result be whatever it is — including it going against ATR.

On folding this into benchmarks/atr-baseline.md: yes, please, and your table belongs in it rather than beside it. Your point about these corpora being jailbreak-shaped rather than tool-result traffic is the one I would most want kept in the text — an FP check on tool-like benign data is a different measurement and this document does not stand in for it.