Skip to content

test: make the reset-during-processing case actually test its race - #14

Merged
FHoffarth merged 1 commit into
mainfrom
fix/reset-during-processing-stability
Sep 7, 2026
Merged

FHoffarth merged 1 commit into
mainfrom
fix/reset-during-processing-stability

Conversation

@FHoffarth

Copy link
Copy Markdown
Owner

Closes #11. Test-only: git diff -- src/ is empty.

1. The named test never produced the race it claims

This is the finding, and it is larger than the flakiness that started the investigation.

Removing every runRef guard from App.tsx left the test green:

case in describe B catches the removed guard?
the slower first file cannot overwrite the newer one yes
a reset during processing is not undone when the old run finishes no — passed with the guard gone
an older run that fails cannot replace the newer result with an error yes
a file chosen during cleaning wins over the finishing clean yes

dirty.jpg and clean.jpg are both JPEGs, so a delay keyed on the format slowed them equally.
The abandoned run therefore finished before the newer one and could not have overwritten it
whatever the product did. The test carried the name of a race without ever arranging one.

The delay now keys on which call it is rather than on the format, so the abandoned run finishes
last — the ordering the other three cases already arranged for themselves.

2. Completion-based synchronisation instead of a 900 ms wall clock

All four cases asserted at a fixed point 900 ms after the second upload. Measured on an idle
machine, that left roughly 400 ms of slack. They now wait for the runs themselves to settle.

That is stabler, and it is also stricter: a wall-clock wait that expires early does not
fail, it quietly stops testing the race. The old form could pass without ever exercising what
it was written for — which is exactly what case 2 turned out to be doing.

3. Validated by mutation, not by hope

With the runRef guards removed from App.tsx:

before this change   3 of 4 cases fail
after  this change   4 of 4 cases fail

And under the load that reproduced the flake — CPU pressure plus a running vite preview
server:

before   2 failures in 3 full-suite runs
after    5 green runs out of 5

What actually flaked

A line that asserted nothing: a click on the drop zone, commented in the source as a no-op. It
opened nothing under jsdom and was where the failure landed. It has been removed rather than
made tolerant.

Stated plainly because it is a loose end: the micro-race behind that click is not fully
explained.
A probe immediately before it still found the button present. Since the line
carried no assertion and the scenario reads the same without it, the surface was removed
instead of the cause pursued further. If a getByRole in a busy state ever flakes again, that
is the thread to pull.

Two hypotheses were tested and dropped along the way: CPU starvation of the 500 ms timer (24
busy processes on 4 cores moved the slack from 402 ms to 420 ms — no effect), and a real race
in the shipped UI (the guard holds; it was verified in Chromium during #8, where a second file
dropped 150 ms into a slow run took over and the abandoned result never appeared).

Scope

Beyond the single test named in #11. All four cases in describe B share the same wall-clock
construction, and leaving one stabilised case beside three identical ones would be the same
mistake as leaving a stale paragraph beside a corrected one. Say the word and it trims to one.

Gates

Gate Result
Product tests 5 files / 43 tests passed
Audit tests 23 files / 316 tests passed
tsc --noEmit clean
vite build 933 ms
npm audit --audit-level=high 0 vulnerabilities
git diff --check clean
git diff -- src/ empty — no product code
tracked evidence files untouched

describe B also runs faster: no case waits out a fixed 900 ms block any more.

No tag, release or deploy proposed.

🤖 Generated with Claude Code

The flaky test turned out to be the smaller of two problems. Removing every
runRef guard from App.tsx left it green: dirty.jpg and clean.jpg are both JPEGs
and shared one delay, so the abandoned run finished before the newer one and
could not have overwritten it whatever the product did. The test carried the
name of a race without ever arranging one. The delay now keys on which call it
is rather than on the format, so the abandoned run finishes last, and the same
mutation turns it red along with its three siblings.

All four cases in describe B asserted at a fixed point on the wall clock, 900 ms
after the second upload. On an idle machine that left about 400 ms of slack.
They now wait for the runs themselves to settle, which is stabler and also
stricter: a wall-clock wait that expires early does not fail, it quietly stops
testing the race.

What actually flaked was a line that asserted nothing - a click on the drop zone
commented as a no-op, which is where the failure landed under load. It is gone
rather than made tolerant. The micro-race behind it is not fully explained; a
probe immediately before it still found the button. Since the line carried no
assertion and the scenario reads the same without it, the surface was removed
instead of the cause pursued further.

No product code. Verified by mutation rather than by hope: with the runRef
guards removed, three of the four cases failed before this change and all four
fail after it. Under the load that reproduced the flake - CPU pressure plus a
running preview server - the suite went from two failures in three runs to five
green runs out of five.

Closes #11.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@FHoffarth
FHoffarth merged commit 013178d into main Sep 7, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stabilize reset-during-processing coverage under load

1 participant