Skip to content

test(runtime): withhold and assert the expected error noise in two passing pins - #10630

Merged
os-elon merged 1 commit into
mainfrom
claude/issue-10380-quiet-expected-test-error-noise
Aug 21, 2026
Merged

test(runtime): withhold and assert the expected error noise in two passing pins#10630
os-elon merged 1 commit into
mainfrom
claude/issue-10380-quiet-expected-test-error-noise

Conversation

@os-elon

Copy link
Copy Markdown
Collaborator

Fixes#10380

Two passing @objectstack/runtime tests printed ERROR-shaped lines into the shared Test Core log, where turbo interleaves package output without attribution. They were lifted verbatim into a p1 flake signature and cost a full dispatch cycle aimed at the wrong mechanism. This withholds that output — and asserts it, so the pins get stronger, not quieter.

Not a mute — the hazard this is built against

Silencing console output is one step from swallowing a real error: a naive capture makes the log quiet and the test blind, and a live pin becomes a decoration. Every sink here does two things instead of one.

  • Scoped to the expected fault only. Each predicate names both the table and the reason. The ambient pin withholds a refusal only when it names sys_metadataand carries knex's Timeout acquiring a connection; the conformance suite withholds one only when the envelope and the dialect detail name the same absent table (no such table: X). Any other table, any other reason, any other driver, any other logger method — forwarded to the console untouched.
  • Counted and asserted. BARE asserts all three channels fired; AMBIENT asserts that none did, which is a new false-negative guard in its own right — an ambient read that silently started stalling would otherwise have had its evidence swallowed by the sinks installed for the bare case. The conformance suite asserts in afterAll that each of the five per-request authz reads still failed.
  • The engine frame is gated on the driver refusal. An engine error is withheld only when it sits directly above a driver refusal the capture already recognised. A DATABASE_ERROR on one of those tables from any other cause is not recognised by the driver sink, so its frame is not withheld either — it reaches the log with both halves intact.

The #7842 pin's assertions are untouched. No expect was edited, none relaxed, no threshold moved, nothing skipped or retry-wrapped. The added assertions are additions only.

Three channels, not the two the card named

The card named Acquire connection error and refused a read on. Re-deriving both paths at head found a third emitter on the same fault in the same two files: the engine's ERROR Find operation failed frame (objectql/src/engine.ts), which carries the same error as a full stack. It is the loudest of the three and the only one that bypasses the reporter, so quieting only the two named features would have left the worst line in place. Named here rather than folded in silently.

channelemitterhow it is scoped
Acquire connection errorknex Logger.warnconsole.logknex log.warn via the driver config; debug/error/deprecate keep their defaults
refused a read onSqlDriver.backendStatementFaultconsole.warnthe logger field's own documented "tests inject a spy" idiom, mirroring the default { warn, error } shape
Find operation failedObjectQL engine logger → raw stdouta Proxy on error alone, the idiom engine-readonly-when-parent.test.ts established

Measured, not asserted

Counts are matching log lines per suite, taken with --reporter=default. That flag is load-bearing: vitest 4 picks MinimalReporter when stdout is not a TTY and it defaults to silent: 'passed-only', so a piped local run drops console.* from passing tests entirely — while turbo gives CI's task a pty and prints every line. Measuring without it yields a false zero in both directions. Filed as part of #10629.

suiteAcquire connection errorrefused a read onFind operation failedlog lines
metadata-list-ambient-vs-bare-transaction before11145
metadata-list-ambient-vs-bare-transaction after00033
notification-schema-conformance before06363380
notification-schema-conformance after000128

The second leg: the assertions fail when the expected error is prevented

A green run alone is uninformative — it cannot tell a pin from a mute. Both were ablated.

Conformance suitetryFind short-circuited for the six tables, so the expected reads never fail. @objectstack/core is aliased to source in this package's vitest config, so the mutation took effect without a rebuild, demonstrated by the verdict changing:

AssertionError: the expected 'sys_user' read failure was never emitted — resolveUserAuthzGrants
no longer reads it, or it no longer fails, and this capture has stopped measuring it
Test Files 1 failed (1)
Tests 8 passed (8)

All eight conformance tests still pass. Nothing but the new assertion notices — which is exactly the blindness the card was written against.

Ambient pin — the this.logger.warn call dropped from SqlDriver.backendStatementFault, so the fault still occurs but one channel goes silent. @objectstack/driver-sql resolves through dist here, so this leg was rebuilt and proven to reach the artifact both ways (scripts/ablation-dist-preflight.mjs, marker present after the mutation, absent after restore — the first marker spelling was tree-shaken into the sourcemaps only and the preflight correctly voided that run):

× BARE — ... the loader read stalls out connection acquisition
AssertionError: the driver's read refusal for 'sys_metadata' was never emitted: expected 0 to be greater than 0
Test Files 1 failed (1)
Tests 1 failed | 1 passed (2)

Every original pin assertion passed; only the new one failed.

Verification at 2779695860

Gate set re-derived with node scripts/pm/dispatch-gates.mjs (no path arguments) after the final commit; every family it named was run, each verdict line quoted.

  • pnpm --filter @objectstack/runtime exec vitest runTest Files 178 passed (178) · Tests 2670 passed (2670)
  • pnpm --filter @objectstack/runtime typechecktsc --noEmit, exit 0
  • check-cross-package-test-inputsOK: 12 package(s) read outside themselves, all declared
  • check-affected-docs — exit 0
  • check:query-options-erasure✓ query-options-erasure ratchet holds: 67 unswept non-test site(s) ... none new
  • check:engine-double-contractOK — 350 pinned, 133 in the DEBT ledger, 2 exempt
  • check:where-matcher✓ where-matcher conformance holds: 271 matcher(s) discovered ... none new
  • check:nul-bytesOK (scanned 6203 text file(s) ... no raw ASCII control bytes)
  • check:type-check-coverageOK — 64/77 workspace packages type-checked
  • check:type-check-debt✓ ... OK — 33 ledger entr(ies) re-measured in 237.1s, 1924 raw tsc error(s) total, none above its recorded number

check:type-check-debt first refused (exit 1, not measured) on an unbuilt worktree; re-run after turbo run build --filter='./packages/*' --filter='./packages/*/*', which is the verdict quoted.

skip-changeset

Test-only. Two *.test.ts files, no production source touched, nothing published changes and no request is answered differently.

Out of scope, filed not fixed

#10293 is deliberately not addressed here — this removes the cause of that misreading; re-grading it is not this card's.


Generated by Claude Code

…ssing pins
Both suites PASS while printing ERROR-shaped lines into the shared Test Core
log, where turbo interleaves package output without attribution. Those lines
were adopted verbatim as a p1 flake signature (#10293) and sent a full dispatch
cycle at the wrong mechanism.
Each expected line is now withheld by a sink scoped to the named table AND the
named reason, and COUNTED — the tests assert the counts, so a capture that went
quiet because the emission disappeared fails instead of passing silently.
Anything unexpected still reaches the console untouched.
Fixes#10380
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019bmVFqoQPq63zhKrxdYG1r
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

Nothing in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from 0 changed package(s)), so this run has no opinion about the docs.

@os-elonos-elon added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Aug 21, 2026 — with Claude
@os-elon
os-elon marked this pull request as ready for review August 21, 2026 08:23
@os-elon
os-elon added this pull request to the merge queueAug 21, 2026
Merged via the queue into main with commit dd8172eAug 21, 2026
35 checks passed
@os-elon
os-elon deleted the claude/issue-10380-quiet-expected-test-error-noise branch August 21, 2026 08:37
os-elon pushed a commit that referenced this pull request Aug 22, 2026
…oise in 16 passing pins (#10629)
Sixteen passing `@objectstack/runtime` fixtures printed expected read
failures into the shared shard log: 134 `[sql-driver] DATABASE_ERROR — the
backend refused a read on '<table>'` envelopes plus their 134 matching
`ERROR Find operation failed` engine frames, 268 lines in all. Turbo
interleaves package logs without attribution, so an ERROR-shaped line from a
green test is indistinguishable from a real failure — lines of this shape
were once lifted verbatim into a p1 flake signature (#10293).
Every one is a fail-soft probe whose missing-table answer the runtime
swallows by design: `resolveUserAuthzGrants`' six `sys_*` `tryFind`s,
`ObjectQL.probeInstallOrganizations`,
`SeedLoaderService.resolveSoleOrganizationId`, the lifecycle governance
snapshot, `runBuildProbes`' view read, and the boot metadata load. Each was
judged expected rather than diagnostic; none was silenced on the strength of
looking like noise.
Applies PR #10630's ruled shape verbatim, factored into one shared test-only
module: withhold only a line naming an expected table AND carrying that same
table's `no such table` reason, count what was withheld, and assert the
counts. The engine frame is withheld only when it sits directly above a
driver refusal the capture already recognised, so an identically-shaped
fault from another cause still reaches the log with both halves intact.
No test skipped, quarantined, retried or removed: 179 files / 2679 tests
pass before and after.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019bmVFqoQPq63zhKrxdYG1r
os-elon pushed a commit that referenced this pull request Aug 23, 2026
extend expected-read-refusal-noise.ts with a second predicate keyed on
the cross-field refusal's own message identity, since the refusal never
reaches the driver's backendStatementFault and so has no table to key
on. Counts per object instead of per table; the caller asserts the
counts in its own afterAll, same withhold-and-assert discipline as
#10629/#10630.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/mskip-changesetPR has no user-facing published change; bypasses the changeset gatetests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@os-elon@claude