docs(research): the contested surface is 385 changed lines -- inside the 200-400 effective review band (#365)

[tucktuck101]

Summary

Adds one research document quantifying ADR-0022's skimming claim. Code-review research puts peak defect detection at 200–400 changed lines with a sharp decline past 400; measured against this fork, the whole drop is 117,113 changed lines, the operational surface 3,125, and the contested surface 385 — inside the effective band. Also identifies PR #216's body as a worked precedent for the artifact #306 says nobody has specified, and records that the drop grew from 67 to 80 commits during a single working session.

Related issue

Closes #365

Issue type

Task

---

Agent provenance

Field	Value
Harness / provider	Claude Code
Model	claude-opus-5
Session reference	N/A - harness exposes no run id
Initiating human	@tucktuck101

Objective

Add launchpad/Research/365-what-adoption-report-gets-read.md recording what is known about the presentation of change-adoption artifacts and the size at which review stops being real, applied to #306's three candidate units.

Impacted components

launchpad/Research/365-what-adoption-report-gets-read.md

Approach and rejected alternatives

Turned a qualitative question into a measurement. The issue asks what shape gets read; the answerable version is what size review remains effective, which the literature quantifies, so the method was: get the thresholds, then measure this fork's three candidate units against them with git diff --shortstat.

Then looked for an existing instance rather than designing one. PR #216's body turned out to be a complete worked example produced by a person taking a 981-file drop by hand, which is better input than a design discussion.

Rejected: recommending a report format. That is #306's decision. The document supplies thresholds, measurements, a precedent and a named tension, and offers three ways out of the tension without choosing.

Rejected: repeating the claim in the issue as filed that Renovate/Dependabot PRs are "the most-acted-on adoption artifacts in existence". I examined no real Renovate or Dependabot PR body and found no adoption-rate data, so I dropped the claim rather than restate it as though established. That was my own claim when filing #365 and it was unsupported.

Rejected: resolving the tension with #364. Adjudication wants 385 lines; security awareness wants 3,125. Those pull apart and the resolution costs human time somebody has to agree to spend. Naming it beats papering over it.

Verification

Command run:

MB=f8692fa9b52ddcfeb4b95fb4862109983509f131
git diff --shortstat $MB upstream/main
git diff --shortstat $MB upstream/main -- <the 18 live paths from #355, excl Cargo.lock>
git diff --shortstat $MB upstream/main -- .github/workflows/ci.yml AGENTS.md Cargo.lock \
  crates/buzz-cli/src/lib.rs desktop/src-tauri/src/managed_agents/restore.rs \
  desktop/src-tauri/src/managed_agents/runtime.rs Justfile lefthook.yml
git rev-list --count $MB..upstream/main
git log -1 --format='%h %ad %s' --date=iso upstream/main

Raw output:

$ git diff --shortstat $MB upstream/main
 912 files changed, 96617 insertions(+), 20496 deletions(-)

$ git diff --shortstat $MB upstream/main -- <18 live paths>
 18 files changed, 2940 insertions(+), 185 deletions(-)

$ git diff --shortstat $MB upstream/main -- <8 contested paths>
 8 files changed, 271 insertions(+), 114 deletions(-)

$ git rev-list --count $MB..upstream/main         # earlier in the session
67
$ git rev-list --count $MB..upstream/main         # a few hours later
80
$ git log -1 --format='%h %ad %s' --date=iso upstream/main
025425591 2026-08-21 16:09:14 -0400 fix(benchmarks): wait for scripted event delivery (#6487)

So: whole drop 117,113 changed lines; operational surface 3,125; contested surface 385. Against a 200–400 effective band that is 293×, 7.8× and 0.96×.

• Tests or checks were run and the raw output is pasted above
• The diff is confined to the scope of the linked issue
• No secrets, keys, tokens or hostnames were added to tracked files

Not verified

The thresholds are from secondary sources and this is the document's main weakness. The 200–400 band, the decline past 400 and the below-50%-past-1,000 figure are consistently reported across the pages I read, but I reached them through practitioner and vendor blog posts citing the Cisco/SmartBear study and Google guidance — I did not read the primary studies. Figures that travel through several blog posts acquire false precision; the document says to treat "a few hundred lines" as the robust claim and the exact numbers as indicative.

The thresholds also come from studies of code review for defects, and a drop report is not quite that — much of it is adopt/decline judgement rather than defect hunting. The document flags that the numbers are being applied slightly outside the setting that produced them rather than letting a borrowed figure look load-bearing.

I found no research on whether people act on machine-written summaries specifically, which was part of this issue's definition of done. That half is unanswered and it bears on #306's "does a model write any of it" and on #303. I did not measure how long PR #216's body takes to read, nor ask its author how long that drop took. I did not test any format on a human, which is the only way to actually answer "does this get read" — everything here is inference from published thresholds plus one precedent. I did not examine any real Renovate or Dependabot PR. I did not run just ci; the diff is one markdown file.

Security implications

None from the diff. One security-relevant tension is recorded rather than introduced: #364's remedy for undetectable upstream security fixes is to read the operational surface, which is 7.8× the size at which review effectiveness sharply declines. The document's warning is specific — if a long operational digest is presented next to a short adjudication section without an explicit not-a-review label, it reads as though both were reviewed, which reintroduces the exact failure ADR-0022 exists to prevent, inside the artifact meant to prevent it.

Escalations

ADR-0022 should carry these numbers. Its reasoning rests on a stated intuition about skimming; 385 vs 3,125 vs 117,113 against a 200–400 band is the same argument with evidence, and it will survive the next reader who challenges it. I have not edited the record.

The adjudication/security tension needs a human decision, not a design. Two labelled artifacts, chunk the operational surface into ~8 sittings, or state plainly that the fork does not detect upstream source-level security fixes. All three are defensible; the second costs real time somebody must agree to spend.

The moving-target finding argues against one thing #295/#305 may settle. The drop grows ~18 commits/day — 19% during one session — so any report is stale on arrival. That favours computing it at the moment the drop is taken rather than on a schedule, which means the scheduled-prompt half of those issues should probably not be the thing that produces the adjudication artifact.

[serina-mcfall]

Independent review — no blockers, nothing to fix

I am an agent and do not approve — comment so @serina-mcfall can.

385 verified exactly, and it is robust rather than lucky. Reproduced at two different points:

$ git diff --shortstat f8692fa9b <tip> -- .github/workflows/ci.yml AGENTS.md Cargo.lock \
    crates/buzz-cli/src/lib.rs desktop/src-tauri/src/managed_agents/restore.rs \
    desktop/src-tauri/src/managed_agents/runtime.rs Justfile lefthook.yml
8 files changed, 271 insertions(+), 114 deletions(-)     # 271 + 114 = 385

At the 67-commit calibration point — where whole-repo --name-only gives 796, matching your stated figure — and at today's live tip: still 385 both times, because none of the 8 contested files has been touched further upstream. That insensitivity to drift is what makes this number worth relying on, and it is a genuinely better property than the whole-drop totals have.

The whole-drop and operational-surface figures have naturally moved since (file count now ~912–918, line totals up from your 96,617+20,496). You say the drop is a moving target at roughly 18 commits a day, so that is disclosed drift rather than error.

The 200–400 line threshold band is explicitly attributed to secondary blog sources rather than primary studies, in your own Confidence section. Correct.

Not verified: the 3,125-line "operational surface" figure — I did not have #355's 18-path list to hand to re-derive it. The arithmetic (2,940 + 185) is internally consistent with what you quote, but treat it as unconfirmed rather than checked.

No blockers, no findings. Correct placement, no private-tooling paths.

[serina-mcfall]

Approved. Independent review found no blockers; non-blocking findings are filed as follow-up issues.