Uh oh!
There was an error while loading. Please reload this page.
Drive the circuit breaker's clock from tests, not sleep() - #619
Merged
Conversation
The circuit breaker already had the seam — every time read goes through CircuitBreaker.now() — it just hardcoded time.Now(). Give it an unexported nowFn defaulting to time.Now, so in-package tests can substitute a clock they advance by hand. TestCircuitBreakerResetsStaleHalfOpenAttempts is the test that flaked: it slept against a 100ms StaleAttemptTimeout, and under -race on a loaded runner the stale reset could fire early. Sleeping to cross a timeout makes the assertion a bet on scheduler latency. Convert every sleeping circuit-breaker test, not just that one, and use realistic production timeouts now that crossing them is free: - ClosesAfterSuccesses, StateTransitionsCorrectly, ResetsStaleHalfOpenAttempts - SetsHalfOpenLastAttemptAt, whose real-time before/after window assertion is meaningless once the clock is fake — it becomes an exact equality against the Allow() that reserved the slot - The hooks integration test, which needs no new seam: NewGatingHooks already takes the breaker, so it builds its own and passes it in `go test -race -count=20 ./internal/resilience/` is stable. The package is not sleep-free — rate_limiter_test.go has three sleeps of its own against a primitive with no now() seam, and those are untouched here. Closes#586
Uh oh!
There was an error while loading. Please reload this page.
jeremy added a commit
that referenced
this pull request
Aug 22, 2026
…or-roundtrip * origin/main: (96 commits) ci: bump the github-actions group with 6 updates (#639) Reject three more doomed invocations before draining stdin (#645) Stdin `-` support everywhere sensible; usage error for stray `-` elsewhere (#641) Add hey-cli Windows signing secrets to the release env manifest (#642) deps: bump the go-dependencies group with 5 updates (#638) Update nix flake and plugin version for v0.9.1 ci: bump the github-actions group with 4 updates (#633) Add basecamp files replace: publish a new version of an uploaded file (#634) Add basecamp files versions — HELD, blocked on the SDK (#622) Update nix flake and plugin version for v0.9.0 Make the Codex probe's timeout actually bound doctor (#629) Make the lockstep check catch stale agreement and .yaml workflows (#628) Keep refreshing opencode's other spelling (#627) Lint the release the same way we lint everything else (#625) Install the skill where opencode actually looks (#624) Take the communiques out of the source tree (#623) Correct the API coverage claim: 183/184, not 100% (#621) Stop echoing back step fields the caller never changed (#620) Drive the circuit breaker's clock from tests, not sleep() (#619) Tell agents the truth about card column moves (#618) ...
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The circuit breaker already had the seam — every time read goes through
CircuitBreaker.now()— it just hardcodedtime.Now(). This gives it anunexported
nowFndefaulting totime.Now, so in-package tests can substitutea clock they advance by hand. The constructor signature is unchanged; the only
non-test caller (
hooks.go:36) is untouched.TestCircuitBreakerResetsStaleHalfOpenAttemptsis the reported flake: it sleptagainst a 100ms
StaleAttemptTimeout, so under-raceon a loaded runner thestale reset could fire early. Sleeping to cross a timeout makes the assertion a
bet on scheduler latency.
Every sleeping circuit-breaker test is converted, not just that one — a fake
clock next to four other tests still racing real time would just relocate the
flake. With crossing a timeout now free, they also use realistic production
timeouts (30s open, 2min stale) instead of the millisecond values that only
existed to keep the sleeps short:
ClosesAfterSuccesses,StateTransitionsCorrectly,ResetsStaleHalfOpenAttemptsSetsHalfOpenLastAttemptAt— its real-time.Now()before/after windowassertion becomes meaningless (and separately flaky) once the clock is fake,
so it becomes an exact equality against the
Allow()that reserved the slothooks_test.go's integration test needs no new seam:NewGatingHooksalreadytakes the breaker, so the test builds its own, sets
nowFn, and passes it inVerified:
go test -race -count=20 ./internal/resilience/is stable, andbin/ciis green.Not claimed: the package is not sleep-free.
rate_limiter_test.gohasthree sleeps of its own against a separate primitive with no
now()seam. Theyare untouched, so package runtime does not drop to zero — the claim here is
about the converted tests only.
Closes#586
Summary by cubic
Inject a test-driven clock into the circuit breaker via an unexported
nowFn, removing sleeps from tests and fixing the half-open stale attempt flake. No public API changes.nowFntoCircuitBreaker(defaults totime.Now) and routed all time reads through it.SetsHalfOpenLastAttemptAtto assert the exact timestamp set byAllow().go test -race -count=20 ./internal/resilience/;rate_limiter_test.gosleeps remain. ClosesFlaky under -race: TestCircuitBreakerResetsStaleHalfOpenAttempts #586.Written for commit 069a5d6. Summary will update on new commits.