Skip to content

test(uat): land the acceptance suite that found this week's four defects - #353

Merged
IanFrelinger merged 2 commits into
masterfrom
test/uat-suite
Aug 17, 2026
Merged

test(uat): land the acceptance suite that found this week's four defects#353
IanFrelinger merged 2 commits into
masterfrom
test/uat-suite

Conversation

@IanFrelinger

@IanFrelingerIanFrelinger commented Aug 17, 2026

Copy link
Copy Markdown
Owner

The acceptance suite that found this week's four defects, landed so those claims cannot rot again.

Why this and not more testing

These scripts test claims, not code. A unit test asks whether a function does what its author meant; these ask whether the product does what the documentation promises a user.

Every defect found on 2026-08-17 was a claim that used to be true, and in every case the code was fine:

PRthe claim that had rotted
#349a test convention required a timeout xunit cannot enforce on a void test
#350the quickstart's hero command could not start the API after Nexo.API began multi-targeting
#351the autonomy lane's "shortest working path" reported FAILED on correct behaviour
#352sandbox attestation crashed the loop on a CLI/daemon version gap

None were reachable by reading the source. Continuing to new tiers finds more defects; landing this protects the ones already fixed.

What runs

tierasks
0Can a tester clone, build and doctor from TesterQuickstart.md verbatim?
1Does a submitted task leave a retrievable record and a trust-log entry under its own id?
2Does the certification gate admit correct code and reject a weak witness?
4Does the reference brick build, and does the API actually fail closed?

The strongest single check greps the API command out ofdocs/TesterQuickstart.md and runs it. That is what catches the page going stale — no amount of testing the API itself would have caught #350.

There is no path filter, deliberately. The change that broke the quickstart was a .csproj edit; the set of files that can invalidate these claims is "most of them", and a filter would have let the motivating defect straight through.

What is excluded, and why

Tier 3, the autonomy loop. It needs a container engine and a local model server. A gate that depends on that infrastructure goes red for reasons unrelated to the code and trains people to ignore it. It stays a deliberate run, documented in tests/uat/README.md.

This complements rather than duplicates onboarding-quickstart-gate, which covers GettingStarted.md and the CLI but not TesterQuickstart.md and never starts the API — part of why #350 went unnoticed.

Honesty about the harness

tests/uat/README.md records the four ways this suite lied before it stopped, because each is a trap the next author will otherwise re-enter:

  1. A hardcoded verdict.time-to-first-audited-job was an unconditional PASS and reported "167s to a CopilotTask entry" during a run where no entry existed.
  2. Evidence scraped from a log that never contained it. Two gate tests scored "did not run" because dotnet test prints names only for failures.
  3. An unanchored parse. A "first http:// in the log" grep pointed curl at the Ollama probe URL, turning three security checks into 000 false negatives — on exactly the checks where a false negative is least acceptable.
  4. Environment mistaken for product. The SDK image's ASPNETCORE_HTTP_PORTS=8080 made the documented :5000 look wrong.

It also insists on pairing negatives with positive controls: "POST without a key returns 401" proves nothing alone, since a broken endpoint also fails to return 200. The paired "and with the key returns 200" is what makes the first mean authentication.

Verification

Tiers 0–2 at 23/23 and tier 4 at 6/6 against merged master 173ea9a2, in a clean mcr.microsoft.com/dotnet/sdk:10.0 container with a cold package cache. This PR's own run is the first exercise of the CI path (UAT_REPO_DIR against the workspace checkout), so the gate proves itself here — watch uat-gate on this PR.

Full scorecard: https://claude.ai/code/artifact/6e71b25f-b449-4d1b-8c12-0063325bfdc7

🤖 Generated with Claude Code

@PlzTouchGrass@claude
test(uat): land the acceptance suite that found this week's four defects
48d34be
These scripts test claims rather than code: not "does this function do what the
author meant" but "does the product do what the documentation promises a user".
Every defect found on 2026-08-17 (#349, #350, #351, #352) was a claim that had
quietly stopped being true, and in each case the code itself was fine -- none
were reachable by reading the source, and none would have been caught by a
unit test.
Tiers 0-2 (quickstart verbatim, audit trail, certification teeth) and tier 4
(reference brick, fail-closed exposure and API keys) run in CI on every PR.
The strongest check greps the API command OUT of docs/TesterQuickstart.md and
runs it, which is precisely what would have caught #350: a csproj change broke
a documented command, and nothing tested the page.
No path filter, for the same reason. The set of files that can invalidate these
claims is "most of them", and a filter would have let exactly the motivating
change through.
Tier 3 (the autonomy loop) is deliberately excluded: it needs a container engine
and a local model server, and a gate that depends on that infrastructure trains
people to ignore red. It stays a deliberate, documented run.
UAT_REPO_DIR runs the scripts against the workspace checkout so the gate tests
the commit under review; without it they clone fresh, which is the point of
tier 0 -- a tester's cold first fifteen minutes.
The README records the four ways this harness lied before it stopped: a
hardcoded PASS that reported a time for an audit entry that did not exist,
evidence scraped from a log that never contained it, an unanchored URL grep
that turned three security checks into false negatives, and an image
environment variable mistaken for a product defect.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@cursor

cursorBot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@PlzTouchGrass@claude
test(uat): the doctor check asserts truthfulness, not the absence of …
a77b561
…docker
The gate's first run on its own PR failed, and the failure was the check rather
than the product: doctor-container-warn asserted "container smoke: warn"
unconditionally. That is only the documented behaviour when Docker is ABSENT --
section 2 says so -- and the CI runner has Docker, so doctor correctly reported
"container smoke: pass" and the check called it a defect. The assertion had
encoded a property of one test environment as though it were a product claim.
The honest form is also the stronger one, and it is what section 5 actually
asks ("does the doctor tell the truth?"): probe the box, then require doctor to
report the state that matches it. It now fails if doctor says "warn" where
Docker is reachable, which the old check could never have caught.
The workflow had the same flaw the scripts were written to avoid. Tier 4 was
skipped when tiers 0-2 failed, so one bad claim hid every claim after it. Both
tiers now run to completion and a final Verdict step reads the recorded results
-- so the step summary is always published, and the job is red exactly when a
check is.
22 of 23 passed in CI on the first run, including the audit trail end to end and
the guard that greps the API command out of TesterQuickstart.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@IanFrelinger
IanFrelinger merged commit 12e9eae into masterAug 17, 2026
4 checks passed
@IanFrelinger
IanFrelinger deleted the test/uat-suite branch August 17, 2026 19:37
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@IanFrelinger@PlzTouchGrass