fix(types,runtime): log every 5xx at error level instead of answering it silently - #14654

Merged
hotlong merged 4 commits into
mainfrom
claude/issue-14310-log-every-5xx
Sep 2, 2026
Merged

fix(types,runtime): log every 5xx at error level instead of answering it silently#14654
hotlong merged 4 commits into
mainfrom
claude/issue-14310-log-every-5xx

Conversation

@hotlong

Copy link
Copy Markdown
Contributor

Fixes#14310

A 500 that leaves no server-side line is diagnosed from the browser or not at all. Reproduced on main @ ca48cf377, driving the real dispatcher plugin and the real route handlers: a plain Error thrown out of a route answered 500 INTERNAL_ERROR with zero log records at any level.

MEASURED status = 500
MEASURED logger.error calls = 0
MEASURED body = {"success":false,"error":{"code":"INTERNAL_ERROR","message":"boom-plain-error","httpStatus":500}}

The single logger.warn in that run was unrelated boot chatter (a fake transport with no setFallbackHandler), so the fault itself reached nobody at any level.

Why the existing machinery did not already cover this

Two independent reasons, both measured rather than reasoned:

  1. ErrorReporter.captureException defaults to NoopErrorReporter. A dev server — the surface an operator actually watches — wires no APM, so the capture was a no-op every time. A log line is the operator's floor; APM is opt-in telemetry on top of it.
  2. It is fed by res.__obsRecordedError, which only the THROWN exit sets. A route that catches its own fault and RETURNS a 5xx envelope — how every /packages handler answers, via deps.errorFromThrown — recorded nothing, so even a wired reporter never saw those. Confirmed in the same run: the thrown path had __obsRecordedError set = true, the returned path false.

Where it lands, and why that is the single conversion point

logServerFault (new, packages/types/src/server-fault-log.ts) emits exactly one error-level record carrying method, path, request id, the message and — where the door still holds the throw — the stack.

It shares a home with resolveThrownHttpError on that rule's own argument: a rule two doors must agree on cannot live inside one of them, because @objectstack/runtime depends on @objectstack/rest and an import could only ever point one way. That is why #8016 moved the "what status does this throw mean" rule to @objectstack/types; "is this answer worth an operator's attention" is the same kind of rule read by the same two doors.

Wired at each transport's own single exit, so a fault costs one line and never two:

ExitFileCovers
sendErrorpackages/types/src/response-envelope.tsevery nested-envelope 5xx in the repo, incl. the REST direct-mount /api/v1/packages registrar
errorResponseBasepackages/runtime/src/dispatcher-plugin.tsthe dispatcher's THROWN exit (stack-bearing)
sendResultBasepackages/runtime/src/dispatcher-plugin.tsthe dispatcher's RETURNED exit — the /packages shape the card was filed on
AI-route mountpackages/runtime/src/dispatcher-plugin.tsthe family that writes its own result inline

sendError is the load-bearing choice for the REST side: it is the one writer every catch in the package registrar already ends in, so those doors become loud from the producer side with no per-door call — and a door added later cannot forget one. packages/rest's /data doors were already loud via logUnexpectedRouteError and are untouched.

Request coordinates reach the two dispatcher exits through a res.__obsRequest side-channel parked by instrumentRouteHandler — the same shape and lifetime as the __obsRecordedError channel that exit has used since #3867. That avoids threading a third argument through the ~50 route handlers that call the locally-shadowed sendResult / errorResponse wrappers.

Evidence that 4xx is unaffected

The 5xx test lives once, inside the helper, rather than at each call site — so no call site can drift from it.

  • Unit: isServerFault pinned at the 499/500 boundary; logServerFault returns false and emits on no channel for a 404.
  • Funnel: sendError at 409 DESTRUCTIVE_CHANGE and 403 FORBIDDEN — the coded refusals that door exists to carry — emit zero lines.
  • End-to-end: an anonymous caller on an auth-gated dispatcher route (401) produces zero fault lines, measured through the real plugin.
  • The wire body is byte-identical at every door; this adds a side effect, never a field (pinned).

error level is load-bearing rather than incidental: the CLI default is warn (DEFAULT_LOG_LEVEL, packages/cli/src/utils/log-level.ts) and error (40) outranks warn (30) in LEVEL_PRIORITY, so the record clears --log-level's default without bypassing the level system. The pins assert the level, not merely that output happened.

Reverse verification

Both landings were ablated from a committed tree, each mutation confirmed on disk by anchor count and object hash before the run, and restored by git checkout HEAD -- ABSOLUTE_PATH verified by empty git diff HEAD plus a worktree/HEAD blob-hash match:

  • removing the call from errorResponseBase → the two thrown-path pins fail (expected [] to have a length of 1), the returned-path and 4xx pins stay green;
  • removing the call from sendError → only the funnel pin fails (1 failed / 12 passed).

Each exit's pin is falsifiable by that exit alone.

Serial fence

Re-derived against the domain seat's fence on the card. packages/rest/src/package-routes.ts is held by the open PR #14499, so this PR does not touch it — the REST doors are covered from the producer side instead. Verified byte-identical to origin/main at this head: package-routes.ts, error-response.ts (PR #14544, merged), dispatcher-error-vocabulary.ts (PR #14625, open).

The #7898 H17 rider does not fire: this diff touches none of packages/core/src/security/auth-gate.ts, packages/runtime/src/http-dispatcher.ts, or packages/adapters/**.

Verification (all at b4bc2ab47)

  • pnpm --filter @objectstack/types exec vitest run — 19 files, 548 passed
  • runtime sweep (dispatcher-plugin*, dispatcher-5xx*, instrument, validation-error, withhold, package-door parity, error-envelope conformance) — 21 files, 285 passed
  • typecheck on types + runtime + rest — clean
  • Gate union re-derived with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack (no paths passed) — 35 families, 31 exit 0. The other four exit 3 = PREREQUISITE NOT MET, not findings: check-test-completeness, check-half-states, check:dual-build-cjs-loads, check:type-check-debt each need a full-repo build or GitHub API access and print "nothing was measured". CI runs them.

⚠️NOT MEASURED, stated rather than implied:packages/runtime/tsconfig.json excludes **/*.test.ts, so pnpm --filter @objectstack/runtime typecheck says nothing about the new test file (tsc --listFiles → 0 matches). Both new test files were type-checked explicitly under a temporary config (0 errors) and the temp configs removed.

Declared narrowing — pnpm lint (whole-repo ESLint) was not run. Targeted instead: eslint --no-inline-config --format json over all 7 changed .ts files — 0 errors, 0 warnings. The three facts that make this a measurement rather than a gap: (a) the file count is read from ESLint's own --format json output, not estimated; (b) the population is read from the repo's config, which never enables type-aware linting for any file (no parserOptions.project, no typed @typescript-eslint rules — eslint.config.mjs, with its own recorded positive control); (c) therefore no untouched file's verdict can move as a result of this diff.

Declared narrowing — verification ran UNLOCKED.scripts/pm/os-verify-lock.sh
could not take the shared verify lock on this host: no usable flock. The shared
verify lock is declared Linux-only (flock is util-linux, and a stock macOS does
not ship it), so the command below was run directly, without the lock —
a declared narrowing, not a silent one. No serialization guarantee held for this
run, nor for any sibling agent in this container while it ran.

One consequence worth a maintainer's eye

A deployment answering a declared 5xx on a polled route — 501 NOT_IMPLEMENTED from an uninstalled optional service, e.g. /api/v1/notifications without service-messaging — now prints one error line per request where it previously printed none. This PR implements the band the issue specifies verbatim ("4xx may stay quiet; 5xx never") and deliberately invents no carve-out: narrowing it for declared capability-absence would be a contract decision, not an implementer's call.

Generated by Claude Code

hotlongand others added 4 commits September 3, 2026 00:57
…ad of answering it silently
A 500 that leaves no server-side line is diagnosed from the browser or not at
all. Measured on `main` @ ca48cf3, through the real plugin and the real route
handlers: a plain `Error` thrown out of a dispatcher route answered
`500 INTERNAL_ERROR` with zero log records at any level.
The reporting that existed was not a substitute: `ErrorReporter` defaults to
`NoopErrorReporter` (so a dev server captured nothing), and it is fed by
`res.__obsRecordedError`, which only the THROWN exit sets — a route that
catches its own fault and RETURNS a 5xx envelope recorded nothing at all.
`logServerFault` (new, `@objectstack/observability`) is the one definition of
the rule, emitting exactly one `error`-level record with method, path, request
id, message and stack. It is wired at each transport's own single exit so a
fault costs one line and never two. 4xx stays quiet, decided inside the helper.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…utes import
`check:check-system-context-census` went red on pure line rot: the
`logServerFault` import added one line to `packages/rest/src/package-routes.ts`,
shifting both anchored elevation reads by one (97 -> 98, 102 -> 103). Repaired
with the gate's own `--fix`; no prose changed and no row added or removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…d land it on sendError
Re-homed under the domain seat's serial fence: `packages/rest/src/package-routes.ts`
is held by the open PR #14499, so this no longer edits that file. The REST
direct-mount doors are covered from the producer side instead.
`sendError` (`@objectstack/types`) is the single writer for every
nested-envelope error in the repo, and every catch in the package registrar
ends there — so wiring the rule at that one exit covers those doors with no
per-door call, and covers any door added later by construction. That also puts
the helper in the same package as `resolveThrownHttpError`, on the same
argument: a rule two doors must agree on cannot live inside one of them.
`@objectstack/observability` and `packages/rest` are back to origin/main
byte-for-byte, as is the system-context census page (its line rot was caused
by the package-routes import this drops).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/runtime, @objectstack/types, touching 22 documentable anchor(s). ⚠️1 changed file(s) yielded no anchor (packages/types/src/index.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

15 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/api/client-sdk.mdx(via requestId (symbol, a field of interface ServerFaultRequest), packages.list (sdk, the route ledger binds it to GET /api/v1/packages))
  • content/docs/api/environment-routing.mdx(via packages.list (sdk, the route ledger binds it to GET /api/v1/packages))
  • content/docs/api/error-catalog.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/api/error-handling-client.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/automation/approvals.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/deployment/production-readiness.mdx(via createDispatcherPlugin (symbol, a top-level function), requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/getting-started/examples.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/kernel/contracts/metadata-service.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/kernel/runtime-services/audit-service.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/kernel/services-checklist.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/permissions/permission-sets.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/plugins/packages.mdx(via createDispatcherPlugin (symbol, a top-level function))
  • content/docs/protocol/kernel/error-handling.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/protocol/kernel/http-protocol.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/ui/apps.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))

1 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/v17.mdx(via requestId (symbol, a field of interface ServerFaultRequest), /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/types/src/index.ts) — pages documenting those are invisible to this run
  • 2 name(s) were too generic to anchor anything (single lowercase words)
  • the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 24 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006fpackageMentionDocs.

Which tree this was computed on

This run read content/docs from 25beb23d6384a4a100200ed6882330e0b71ff003 — the merge of head b4bc2ab473e5f8b9d95326e9a191f4ded2ac98e5 into base 20b883918aa787e65299e72b85c2b3396aa2006f, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 25beb23d6384a4a100200ed6882330e0b71ff003 && git checkout 25beb23d6384a4a100200ed6882330e0b71ff003
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 20b883918aa787e65299e72b85c2b3396aa2006f b4bc2ab473e5f8b9d95326e9a191f4ded2ac98e5 && git checkout -B drift-repro 20b883918aa787e65299e72b85c2b3396aa2006f && git merge --no-ff b4bc2ab473e5f8b9d95326e9a191f4ded2ac98e5
node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006f

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 20b883918aa787e65299e72b85c2b3396aa2006f → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@github-actionsgithub-actionsBot added documentation Improvements or additions to documentation tests tooling labels Sep 2, 2026
@hotlong
hotlong marked this pull request as ready for review September 2, 2026 17:41
@hotlong
hotlong added this pull request to the merge queueSep 2, 2026
Merged via the queue into main with commit 22e5236Sep 2, 2026
35 checks passed
@hotlong
hotlong deleted the claude/issue-14310-log-every-5xx branch September 2, 2026 19:33
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/lteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A 500 from GET /api/v1/packages (and /meta/package/:name) leaves no server-side log line at all

1 participant

@hotlong
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

fix(types,runtime): log every 5xx at error level instead of answering it silently - #14654

Merged
hotlong merged 4 commits into
mainfrom
claude/issue-14310-log-every-5xx
Sep 2, 2026
Merged

fix(types,runtime): log every 5xx at error level instead of answering it silently#14654
hotlong merged 4 commits into
mainfrom
claude/issue-14310-log-every-5xx

Conversation

@hotlong

Copy link
Copy Markdown
Contributor

Fixes#14310

A 500 that leaves no server-side line is diagnosed from the browser or not at all. Reproduced on main @ ca48cf377, driving the real dispatcher plugin and the real route handlers: a plain Error thrown out of a route answered 500 INTERNAL_ERROR with zero log records at any level.

MEASURED status = 500
MEASURED logger.error calls = 0
MEASURED body = {"success":false,"error":{"code":"INTERNAL_ERROR","message":"boom-plain-error","httpStatus":500}}

The single logger.warn in that run was unrelated boot chatter (a fake transport with no setFallbackHandler), so the fault itself reached nobody at any level.

Why the existing machinery did not already cover this

Two independent reasons, both measured rather than reasoned:

  1. ErrorReporter.captureException defaults to NoopErrorReporter. A dev server — the surface an operator actually watches — wires no APM, so the capture was a no-op every time. A log line is the operator's floor; APM is opt-in telemetry on top of it.
  2. It is fed by res.__obsRecordedError, which only the THROWN exit sets. A route that catches its own fault and RETURNS a 5xx envelope — how every /packages handler answers, via deps.errorFromThrown — recorded nothing, so even a wired reporter never saw those. Confirmed in the same run: the thrown path had __obsRecordedError set = true, the returned path false.

Where it lands, and why that is the single conversion point

logServerFault (new, packages/types/src/server-fault-log.ts) emits exactly one error-level record carrying method, path, request id, the message and — where the door still holds the throw — the stack.

It shares a home with resolveThrownHttpError on that rule's own argument: a rule two doors must agree on cannot live inside one of them, because @objectstack/runtime depends on @objectstack/rest and an import could only ever point one way. That is why #8016 moved the "what status does this throw mean" rule to @objectstack/types; "is this answer worth an operator's attention" is the same kind of rule read by the same two doors.

Wired at each transport's own single exit, so a fault costs one line and never two:

ExitFileCovers
sendErrorpackages/types/src/response-envelope.tsevery nested-envelope 5xx in the repo, incl. the REST direct-mount /api/v1/packages registrar
errorResponseBasepackages/runtime/src/dispatcher-plugin.tsthe dispatcher's THROWN exit (stack-bearing)
sendResultBasepackages/runtime/src/dispatcher-plugin.tsthe dispatcher's RETURNED exit — the /packages shape the card was filed on
AI-route mountpackages/runtime/src/dispatcher-plugin.tsthe family that writes its own result inline

sendError is the load-bearing choice for the REST side: it is the one writer every catch in the package registrar already ends in, so those doors become loud from the producer side with no per-door call — and a door added later cannot forget one. packages/rest's /data doors were already loud via logUnexpectedRouteError and are untouched.

Request coordinates reach the two dispatcher exits through a res.__obsRequest side-channel parked by instrumentRouteHandler — the same shape and lifetime as the __obsRecordedError channel that exit has used since #3867. That avoids threading a third argument through the ~50 route handlers that call the locally-shadowed sendResult / errorResponse wrappers.

Evidence that 4xx is unaffected

The 5xx test lives once, inside the helper, rather than at each call site — so no call site can drift from it.

  • Unit: isServerFault pinned at the 499/500 boundary; logServerFault returns false and emits on no channel for a 404.
  • Funnel: sendError at 409 DESTRUCTIVE_CHANGE and 403 FORBIDDEN — the coded refusals that door exists to carry — emit zero lines.
  • End-to-end: an anonymous caller on an auth-gated dispatcher route (401) produces zero fault lines, measured through the real plugin.
  • The wire body is byte-identical at every door; this adds a side effect, never a field (pinned).

error level is load-bearing rather than incidental: the CLI default is warn (DEFAULT_LOG_LEVEL, packages/cli/src/utils/log-level.ts) and error (40) outranks warn (30) in LEVEL_PRIORITY, so the record clears --log-level's default without bypassing the level system. The pins assert the level, not merely that output happened.

Reverse verification

Both landings were ablated from a committed tree, each mutation confirmed on disk by anchor count and object hash before the run, and restored by git checkout HEAD -- ABSOLUTE_PATH verified by empty git diff HEAD plus a worktree/HEAD blob-hash match:

  • removing the call from errorResponseBase → the two thrown-path pins fail (expected [] to have a length of 1), the returned-path and 4xx pins stay green;
  • removing the call from sendError → only the funnel pin fails (1 failed / 12 passed).

Each exit's pin is falsifiable by that exit alone.

Serial fence

Re-derived against the domain seat's fence on the card. packages/rest/src/package-routes.ts is held by the open PR #14499, so this PR does not touch it — the REST doors are covered from the producer side instead. Verified byte-identical to origin/main at this head: package-routes.ts, error-response.ts (PR #14544, merged), dispatcher-error-vocabulary.ts (PR #14625, open).

The #7898 H17 rider does not fire: this diff touches none of packages/core/src/security/auth-gate.ts, packages/runtime/src/http-dispatcher.ts, or packages/adapters/**.

Verification (all at b4bc2ab47)

  • pnpm --filter @objectstack/types exec vitest run — 19 files, 548 passed
  • runtime sweep (dispatcher-plugin*, dispatcher-5xx*, instrument, validation-error, withhold, package-door parity, error-envelope conformance) — 21 files, 285 passed
  • typecheck on types + runtime + rest — clean
  • Gate union re-derived with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack (no paths passed) — 35 families, 31 exit 0. The other four exit 3 = PREREQUISITE NOT MET, not findings: check-test-completeness, check-half-states, check:dual-build-cjs-loads, check:type-check-debt each need a full-repo build or GitHub API access and print "nothing was measured". CI runs them.

⚠️NOT MEASURED, stated rather than implied:packages/runtime/tsconfig.json excludes **/*.test.ts, so pnpm --filter @objectstack/runtime typecheck says nothing about the new test file (tsc --listFiles → 0 matches). Both new test files were type-checked explicitly under a temporary config (0 errors) and the temp configs removed.

Declared narrowing — pnpm lint (whole-repo ESLint) was not run. Targeted instead: eslint --no-inline-config --format json over all 7 changed .ts files — 0 errors, 0 warnings. The three facts that make this a measurement rather than a gap: (a) the file count is read from ESLint's own --format json output, not estimated; (b) the population is read from the repo's config, which never enables type-aware linting for any file (no parserOptions.project, no typed @typescript-eslint rules — eslint.config.mjs, with its own recorded positive control); (c) therefore no untouched file's verdict can move as a result of this diff.

Declared narrowing — verification ran UNLOCKED.scripts/pm/os-verify-lock.sh
could not take the shared verify lock on this host: no usable flock. The shared
verify lock is declared Linux-only (flock is util-linux, and a stock macOS does
not ship it), so the command below was run directly, without the lock —
a declared narrowing, not a silent one. No serialization guarantee held for this
run, nor for any sibling agent in this container while it ran.

One consequence worth a maintainer's eye

A deployment answering a declared 5xx on a polled route — 501 NOT_IMPLEMENTED from an uninstalled optional service, e.g. /api/v1/notifications without service-messaging — now prints one error line per request where it previously printed none. This PR implements the band the issue specifies verbatim ("4xx may stay quiet; 5xx never") and deliberately invents no carve-out: narrowing it for declared capability-absence would be a contract decision, not an implementer's call.

Generated by Claude Code

hotlongand others added 4 commits September 3, 2026 00:57
…ad of answering it silently
A 500 that leaves no server-side line is diagnosed from the browser or not at
all. Measured on `main` @ ca48cf3, through the real plugin and the real route
handlers: a plain `Error` thrown out of a dispatcher route answered
`500 INTERNAL_ERROR` with zero log records at any level.
The reporting that existed was not a substitute: `ErrorReporter` defaults to
`NoopErrorReporter` (so a dev server captured nothing), and it is fed by
`res.__obsRecordedError`, which only the THROWN exit sets — a route that
catches its own fault and RETURNS a 5xx envelope recorded nothing at all.
`logServerFault` (new, `@objectstack/observability`) is the one definition of
the rule, emitting exactly one `error`-level record with method, path, request
id, message and stack. It is wired at each transport's own single exit so a
fault costs one line and never two. 4xx stays quiet, decided inside the helper.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…utes import
`check:check-system-context-census` went red on pure line rot: the
`logServerFault` import added one line to `packages/rest/src/package-routes.ts`,
shifting both anchored elevation reads by one (97 -> 98, 102 -> 103). Repaired
with the gate's own `--fix`; no prose changed and no row added or removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…d land it on sendError
Re-homed under the domain seat's serial fence: `packages/rest/src/package-routes.ts`
is held by the open PR #14499, so this no longer edits that file. The REST
direct-mount doors are covered from the producer side instead.
`sendError` (`@objectstack/types`) is the single writer for every
nested-envelope error in the repo, and every catch in the package registrar
ends there — so wiring the rule at that one exit covers those doors with no
per-door call, and covers any door added later by construction. That also puts
the helper in the same package as `resolveThrownHttpError`, on the same
argument: a rule two doors must agree on cannot live inside one of them.
`@objectstack/observability` and `packages/rest` are back to origin/main
byte-for-byte, as is the system-context census page (its line rot was caused
by the package-routes import this drops).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/runtime, @objectstack/types, touching 22 documentable anchor(s). ⚠️1 changed file(s) yielded no anchor (packages/types/src/index.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

15 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/api/client-sdk.mdx(via requestId (symbol, a field of interface ServerFaultRequest), packages.list (sdk, the route ledger binds it to GET /api/v1/packages))
  • content/docs/api/environment-routing.mdx(via packages.list (sdk, the route ledger binds it to GET /api/v1/packages))
  • content/docs/api/error-catalog.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/api/error-handling-client.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/automation/approvals.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/deployment/production-readiness.mdx(via createDispatcherPlugin (symbol, a top-level function), requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/getting-started/examples.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/kernel/contracts/metadata-service.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/kernel/runtime-services/audit-service.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/kernel/services-checklist.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/permissions/permission-sets.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/plugins/packages.mdx(via createDispatcherPlugin (symbol, a top-level function))
  • content/docs/protocol/kernel/error-handling.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/protocol/kernel/http-protocol.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/ui/apps.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))

1 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/v17.mdx(via requestId (symbol, a field of interface ServerFaultRequest), /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/types/src/index.ts) — pages documenting those are invisible to this run
  • 2 name(s) were too generic to anchor anything (single lowercase words)
  • the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 24 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006fpackageMentionDocs.

Which tree this was computed on

This run read content/docs from 25beb23d6384a4a100200ed6882330e0b71ff003 — the merge of head b4bc2ab473e5f8b9d95326e9a191f4ded2ac98e5 into base 20b883918aa787e65299e72b85c2b3396aa2006f, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 25beb23d6384a4a100200ed6882330e0b71ff003 && git checkout 25beb23d6384a4a100200ed6882330e0b71ff003
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 20b883918aa787e65299e72b85c2b3396aa2006f b4bc2ab473e5f8b9d95326e9a191f4ded2ac98e5 && git checkout -B drift-repro 20b883918aa787e65299e72b85c2b3396aa2006f && git merge --no-ff b4bc2ab473e5f8b9d95326e9a191f4ded2ac98e5
node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006f

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 20b883918aa787e65299e72b85c2b3396aa2006f → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@github-actionsgithub-actionsBot added documentation Improvements or additions to documentation tests tooling labels Sep 2, 2026
@hotlong
hotlong marked this pull request as ready for review September 2, 2026 17:41
@hotlong
hotlong added this pull request to the merge queueSep 2, 2026
Merged via the queue into main with commit 22e5236Sep 2, 2026
35 checks passed
@hotlong
hotlong deleted the claude/issue-14310-log-every-5xx branch September 2, 2026 19:33
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/lteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A 500 from GET /api/v1/packages (and /meta/package/:name) leaves no server-side log line at all

1 participant

@hotlong
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

fix(types,runtime): log every 5xx at error level instead of answering it silently - #14654

Merged
hotlong merged 4 commits into
mainfrom
claude/issue-14310-log-every-5xx
Sep 2, 2026
Merged

fix(types,runtime): log every 5xx at error level instead of answering it silently#14654
hotlong merged 4 commits into
mainfrom
claude/issue-14310-log-every-5xx

Conversation

@hotlong

Copy link
Copy Markdown
Contributor

Fixes#14310

A 500 that leaves no server-side line is diagnosed from the browser or not at all. Reproduced on main @ ca48cf377, driving the real dispatcher plugin and the real route handlers: a plain Error thrown out of a route answered 500 INTERNAL_ERROR with zero log records at any level.

MEASURED status = 500
MEASURED logger.error calls = 0
MEASURED body = {"success":false,"error":{"code":"INTERNAL_ERROR","message":"boom-plain-error","httpStatus":500}}

The single logger.warn in that run was unrelated boot chatter (a fake transport with no setFallbackHandler), so the fault itself reached nobody at any level.

Why the existing machinery did not already cover this

Two independent reasons, both measured rather than reasoned:

  1. ErrorReporter.captureException defaults to NoopErrorReporter. A dev server — the surface an operator actually watches — wires no APM, so the capture was a no-op every time. A log line is the operator's floor; APM is opt-in telemetry on top of it.
  2. It is fed by res.__obsRecordedError, which only the THROWN exit sets. A route that catches its own fault and RETURNS a 5xx envelope — how every /packages handler answers, via deps.errorFromThrown — recorded nothing, so even a wired reporter never saw those. Confirmed in the same run: the thrown path had __obsRecordedError set = true, the returned path false.

Where it lands, and why that is the single conversion point

logServerFault (new, packages/types/src/server-fault-log.ts) emits exactly one error-level record carrying method, path, request id, the message and — where the door still holds the throw — the stack.

It shares a home with resolveThrownHttpError on that rule's own argument: a rule two doors must agree on cannot live inside one of them, because @objectstack/runtime depends on @objectstack/rest and an import could only ever point one way. That is why #8016 moved the "what status does this throw mean" rule to @objectstack/types; "is this answer worth an operator's attention" is the same kind of rule read by the same two doors.

Wired at each transport's own single exit, so a fault costs one line and never two:

ExitFileCovers
sendErrorpackages/types/src/response-envelope.tsevery nested-envelope 5xx in the repo, incl. the REST direct-mount /api/v1/packages registrar
errorResponseBasepackages/runtime/src/dispatcher-plugin.tsthe dispatcher's THROWN exit (stack-bearing)
sendResultBasepackages/runtime/src/dispatcher-plugin.tsthe dispatcher's RETURNED exit — the /packages shape the card was filed on
AI-route mountpackages/runtime/src/dispatcher-plugin.tsthe family that writes its own result inline

sendError is the load-bearing choice for the REST side: it is the one writer every catch in the package registrar already ends in, so those doors become loud from the producer side with no per-door call — and a door added later cannot forget one. packages/rest's /data doors were already loud via logUnexpectedRouteError and are untouched.

Request coordinates reach the two dispatcher exits through a res.__obsRequest side-channel parked by instrumentRouteHandler — the same shape and lifetime as the __obsRecordedError channel that exit has used since #3867. That avoids threading a third argument through the ~50 route handlers that call the locally-shadowed sendResult / errorResponse wrappers.

Evidence that 4xx is unaffected

The 5xx test lives once, inside the helper, rather than at each call site — so no call site can drift from it.

  • Unit: isServerFault pinned at the 499/500 boundary; logServerFault returns false and emits on no channel for a 404.
  • Funnel: sendError at 409 DESTRUCTIVE_CHANGE and 403 FORBIDDEN — the coded refusals that door exists to carry — emit zero lines.
  • End-to-end: an anonymous caller on an auth-gated dispatcher route (401) produces zero fault lines, measured through the real plugin.
  • The wire body is byte-identical at every door; this adds a side effect, never a field (pinned).

error level is load-bearing rather than incidental: the CLI default is warn (DEFAULT_LOG_LEVEL, packages/cli/src/utils/log-level.ts) and error (40) outranks warn (30) in LEVEL_PRIORITY, so the record clears --log-level's default without bypassing the level system. The pins assert the level, not merely that output happened.

Reverse verification

Both landings were ablated from a committed tree, each mutation confirmed on disk by anchor count and object hash before the run, and restored by git checkout HEAD -- ABSOLUTE_PATH verified by empty git diff HEAD plus a worktree/HEAD blob-hash match:

  • removing the call from errorResponseBase → the two thrown-path pins fail (expected [] to have a length of 1), the returned-path and 4xx pins stay green;
  • removing the call from sendError → only the funnel pin fails (1 failed / 12 passed).

Each exit's pin is falsifiable by that exit alone.

Serial fence

Re-derived against the domain seat's fence on the card. packages/rest/src/package-routes.ts is held by the open PR #14499, so this PR does not touch it — the REST doors are covered from the producer side instead. Verified byte-identical to origin/main at this head: package-routes.ts, error-response.ts (PR #14544, merged), dispatcher-error-vocabulary.ts (PR #14625, open).

The #7898 H17 rider does not fire: this diff touches none of packages/core/src/security/auth-gate.ts, packages/runtime/src/http-dispatcher.ts, or packages/adapters/**.

Verification (all at b4bc2ab47)

  • pnpm --filter @objectstack/types exec vitest run — 19 files, 548 passed
  • runtime sweep (dispatcher-plugin*, dispatcher-5xx*, instrument, validation-error, withhold, package-door parity, error-envelope conformance) — 21 files, 285 passed
  • typecheck on types + runtime + rest — clean
  • Gate union re-derived with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack (no paths passed) — 35 families, 31 exit 0. The other four exit 3 = PREREQUISITE NOT MET, not findings: check-test-completeness, check-half-states, check:dual-build-cjs-loads, check:type-check-debt each need a full-repo build or GitHub API access and print "nothing was measured". CI runs them.

⚠️NOT MEASURED, stated rather than implied:packages/runtime/tsconfig.json excludes **/*.test.ts, so pnpm --filter @objectstack/runtime typecheck says nothing about the new test file (tsc --listFiles → 0 matches). Both new test files were type-checked explicitly under a temporary config (0 errors) and the temp configs removed.

Declared narrowing — pnpm lint (whole-repo ESLint) was not run. Targeted instead: eslint --no-inline-config --format json over all 7 changed .ts files — 0 errors, 0 warnings. The three facts that make this a measurement rather than a gap: (a) the file count is read from ESLint's own --format json output, not estimated; (b) the population is read from the repo's config, which never enables type-aware linting for any file (no parserOptions.project, no typed @typescript-eslint rules — eslint.config.mjs, with its own recorded positive control); (c) therefore no untouched file's verdict can move as a result of this diff.

Declared narrowing — verification ran UNLOCKED.scripts/pm/os-verify-lock.sh
could not take the shared verify lock on this host: no usable flock. The shared
verify lock is declared Linux-only (flock is util-linux, and a stock macOS does
not ship it), so the command below was run directly, without the lock —
a declared narrowing, not a silent one. No serialization guarantee held for this
run, nor for any sibling agent in this container while it ran.

One consequence worth a maintainer's eye

A deployment answering a declared 5xx on a polled route — 501 NOT_IMPLEMENTED from an uninstalled optional service, e.g. /api/v1/notifications without service-messaging — now prints one error line per request where it previously printed none. This PR implements the band the issue specifies verbatim ("4xx may stay quiet; 5xx never") and deliberately invents no carve-out: narrowing it for declared capability-absence would be a contract decision, not an implementer's call.

Generated by Claude Code

hotlongand others added 4 commits September 3, 2026 00:57
…ad of answering it silently
A 500 that leaves no server-side line is diagnosed from the browser or not at
all. Measured on `main` @ ca48cf3, through the real plugin and the real route
handlers: a plain `Error` thrown out of a dispatcher route answered
`500 INTERNAL_ERROR` with zero log records at any level.
The reporting that existed was not a substitute: `ErrorReporter` defaults to
`NoopErrorReporter` (so a dev server captured nothing), and it is fed by
`res.__obsRecordedError`, which only the THROWN exit sets — a route that
catches its own fault and RETURNS a 5xx envelope recorded nothing at all.
`logServerFault` (new, `@objectstack/observability`) is the one definition of
the rule, emitting exactly one `error`-level record with method, path, request
id, message and stack. It is wired at each transport's own single exit so a
fault costs one line and never two. 4xx stays quiet, decided inside the helper.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…utes import
`check:check-system-context-census` went red on pure line rot: the
`logServerFault` import added one line to `packages/rest/src/package-routes.ts`,
shifting both anchored elevation reads by one (97 -> 98, 102 -> 103). Repaired
with the gate's own `--fix`; no prose changed and no row added or removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…d land it on sendError
Re-homed under the domain seat's serial fence: `packages/rest/src/package-routes.ts`
is held by the open PR #14499, so this no longer edits that file. The REST
direct-mount doors are covered from the producer side instead.
`sendError` (`@objectstack/types`) is the single writer for every
nested-envelope error in the repo, and every catch in the package registrar
ends there — so wiring the rule at that one exit covers those doors with no
per-door call, and covers any door added later by construction. That also puts
the helper in the same package as `resolveThrownHttpError`, on the same
argument: a rule two doors must agree on cannot live inside one of them.
`@objectstack/observability` and `packages/rest` are back to origin/main
byte-for-byte, as is the system-context census page (its line rot was caused
by the package-routes import this drops).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/runtime, @objectstack/types, touching 22 documentable anchor(s). ⚠️1 changed file(s) yielded no anchor (packages/types/src/index.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

15 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/api/client-sdk.mdx(via requestId (symbol, a field of interface ServerFaultRequest), packages.list (sdk, the route ledger binds it to GET /api/v1/packages))
  • content/docs/api/environment-routing.mdx(via packages.list (sdk, the route ledger binds it to GET /api/v1/packages))
  • content/docs/api/error-catalog.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/api/error-handling-client.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/automation/approvals.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/deployment/production-readiness.mdx(via createDispatcherPlugin (symbol, a top-level function), requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/getting-started/examples.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/kernel/contracts/metadata-service.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/kernel/runtime-services/audit-service.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/kernel/services-checklist.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/permissions/permission-sets.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/plugins/packages.mdx(via createDispatcherPlugin (symbol, a top-level function))
  • content/docs/protocol/kernel/error-handling.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/protocol/kernel/http-protocol.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/ui/apps.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))

1 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/v17.mdx(via requestId (symbol, a field of interface ServerFaultRequest), /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/types/src/index.ts) — pages documenting those are invisible to this run
  • 2 name(s) were too generic to anchor anything (single lowercase words)
  • the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 24 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006fpackageMentionDocs.

Which tree this was computed on

This run read content/docs from 25beb23d6384a4a100200ed6882330e0b71ff003 — the merge of head b4bc2ab473e5f8b9d95326e9a191f4ded2ac98e5 into base 20b883918aa787e65299e72b85c2b3396aa2006f, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 25beb23d6384a4a100200ed6882330e0b71ff003 && git checkout 25beb23d6384a4a100200ed6882330e0b71ff003
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 20b883918aa787e65299e72b85c2b3396aa2006f b4bc2ab473e5f8b9d95326e9a191f4ded2ac98e5 && git checkout -B drift-repro 20b883918aa787e65299e72b85c2b3396aa2006f && git merge --no-ff b4bc2ab473e5f8b9d95326e9a191f4ded2ac98e5
node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006f

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 20b883918aa787e65299e72b85c2b3396aa2006f → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@github-actionsgithub-actionsBot added documentation Improvements or additions to documentation tests tooling labels Sep 2, 2026
@hotlong
hotlong marked this pull request as ready for review September 2, 2026 17:41
@hotlong
hotlong added this pull request to the merge queueSep 2, 2026
Merged via the queue into main with commit 22e5236Sep 2, 2026
35 checks passed
@hotlong
hotlong deleted the claude/issue-14310-log-every-5xx branch September 2, 2026 19:33
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/lteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A 500 from GET /api/v1/packages (and /meta/package/:name) leaves no server-side log line at all

1 participant

@hotlong
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

fix(types,runtime): log every 5xx at error level instead of answering it silently - #14654

Merged
hotlong merged 4 commits into
mainfrom
claude/issue-14310-log-every-5xx
Sep 2, 2026
Merged

fix(types,runtime): log every 5xx at error level instead of answering it silently#14654
hotlong merged 4 commits into
mainfrom
claude/issue-14310-log-every-5xx

Conversation

@hotlong

Copy link
Copy Markdown
Contributor

Fixes#14310

A 500 that leaves no server-side line is diagnosed from the browser or not at all. Reproduced on main @ ca48cf377, driving the real dispatcher plugin and the real route handlers: a plain Error thrown out of a route answered 500 INTERNAL_ERROR with zero log records at any level.

MEASURED status = 500
MEASURED logger.error calls = 0
MEASURED body = {"success":false,"error":{"code":"INTERNAL_ERROR","message":"boom-plain-error","httpStatus":500}}

The single logger.warn in that run was unrelated boot chatter (a fake transport with no setFallbackHandler), so the fault itself reached nobody at any level.

Why the existing machinery did not already cover this

Two independent reasons, both measured rather than reasoned:

  1. ErrorReporter.captureException defaults to NoopErrorReporter. A dev server — the surface an operator actually watches — wires no APM, so the capture was a no-op every time. A log line is the operator's floor; APM is opt-in telemetry on top of it.
  2. It is fed by res.__obsRecordedError, which only the THROWN exit sets. A route that catches its own fault and RETURNS a 5xx envelope — how every /packages handler answers, via deps.errorFromThrown — recorded nothing, so even a wired reporter never saw those. Confirmed in the same run: the thrown path had __obsRecordedError set = true, the returned path false.

Where it lands, and why that is the single conversion point

logServerFault (new, packages/types/src/server-fault-log.ts) emits exactly one error-level record carrying method, path, request id, the message and — where the door still holds the throw — the stack.

It shares a home with resolveThrownHttpError on that rule's own argument: a rule two doors must agree on cannot live inside one of them, because @objectstack/runtime depends on @objectstack/rest and an import could only ever point one way. That is why #8016 moved the "what status does this throw mean" rule to @objectstack/types; "is this answer worth an operator's attention" is the same kind of rule read by the same two doors.

Wired at each transport's own single exit, so a fault costs one line and never two:

ExitFileCovers
sendErrorpackages/types/src/response-envelope.tsevery nested-envelope 5xx in the repo, incl. the REST direct-mount /api/v1/packages registrar
errorResponseBasepackages/runtime/src/dispatcher-plugin.tsthe dispatcher's THROWN exit (stack-bearing)
sendResultBasepackages/runtime/src/dispatcher-plugin.tsthe dispatcher's RETURNED exit — the /packages shape the card was filed on
AI-route mountpackages/runtime/src/dispatcher-plugin.tsthe family that writes its own result inline

sendError is the load-bearing choice for the REST side: it is the one writer every catch in the package registrar already ends in, so those doors become loud from the producer side with no per-door call — and a door added later cannot forget one. packages/rest's /data doors were already loud via logUnexpectedRouteError and are untouched.

Request coordinates reach the two dispatcher exits through a res.__obsRequest side-channel parked by instrumentRouteHandler — the same shape and lifetime as the __obsRecordedError channel that exit has used since #3867. That avoids threading a third argument through the ~50 route handlers that call the locally-shadowed sendResult / errorResponse wrappers.

Evidence that 4xx is unaffected

The 5xx test lives once, inside the helper, rather than at each call site — so no call site can drift from it.

  • Unit: isServerFault pinned at the 499/500 boundary; logServerFault returns false and emits on no channel for a 404.
  • Funnel: sendError at 409 DESTRUCTIVE_CHANGE and 403 FORBIDDEN — the coded refusals that door exists to carry — emit zero lines.
  • End-to-end: an anonymous caller on an auth-gated dispatcher route (401) produces zero fault lines, measured through the real plugin.
  • The wire body is byte-identical at every door; this adds a side effect, never a field (pinned).

error level is load-bearing rather than incidental: the CLI default is warn (DEFAULT_LOG_LEVEL, packages/cli/src/utils/log-level.ts) and error (40) outranks warn (30) in LEVEL_PRIORITY, so the record clears --log-level's default without bypassing the level system. The pins assert the level, not merely that output happened.

Reverse verification

Both landings were ablated from a committed tree, each mutation confirmed on disk by anchor count and object hash before the run, and restored by git checkout HEAD -- ABSOLUTE_PATH verified by empty git diff HEAD plus a worktree/HEAD blob-hash match:

  • removing the call from errorResponseBase → the two thrown-path pins fail (expected [] to have a length of 1), the returned-path and 4xx pins stay green;
  • removing the call from sendError → only the funnel pin fails (1 failed / 12 passed).

Each exit's pin is falsifiable by that exit alone.

Serial fence

Re-derived against the domain seat's fence on the card. packages/rest/src/package-routes.ts is held by the open PR #14499, so this PR does not touch it — the REST doors are covered from the producer side instead. Verified byte-identical to origin/main at this head: package-routes.ts, error-response.ts (PR #14544, merged), dispatcher-error-vocabulary.ts (PR #14625, open).

The #7898 H17 rider does not fire: this diff touches none of packages/core/src/security/auth-gate.ts, packages/runtime/src/http-dispatcher.ts, or packages/adapters/**.

Verification (all at b4bc2ab47)

  • pnpm --filter @objectstack/types exec vitest run — 19 files, 548 passed
  • runtime sweep (dispatcher-plugin*, dispatcher-5xx*, instrument, validation-error, withhold, package-door parity, error-envelope conformance) — 21 files, 285 passed
  • typecheck on types + runtime + rest — clean
  • Gate union re-derived with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack (no paths passed) — 35 families, 31 exit 0. The other four exit 3 = PREREQUISITE NOT MET, not findings: check-test-completeness, check-half-states, check:dual-build-cjs-loads, check:type-check-debt each need a full-repo build or GitHub API access and print "nothing was measured". CI runs them.

⚠️NOT MEASURED, stated rather than implied:packages/runtime/tsconfig.json excludes **/*.test.ts, so pnpm --filter @objectstack/runtime typecheck says nothing about the new test file (tsc --listFiles → 0 matches). Both new test files were type-checked explicitly under a temporary config (0 errors) and the temp configs removed.

Declared narrowing — pnpm lint (whole-repo ESLint) was not run. Targeted instead: eslint --no-inline-config --format json over all 7 changed .ts files — 0 errors, 0 warnings. The three facts that make this a measurement rather than a gap: (a) the file count is read from ESLint's own --format json output, not estimated; (b) the population is read from the repo's config, which never enables type-aware linting for any file (no parserOptions.project, no typed @typescript-eslint rules — eslint.config.mjs, with its own recorded positive control); (c) therefore no untouched file's verdict can move as a result of this diff.

Declared narrowing — verification ran UNLOCKED.scripts/pm/os-verify-lock.sh
could not take the shared verify lock on this host: no usable flock. The shared
verify lock is declared Linux-only (flock is util-linux, and a stock macOS does
not ship it), so the command below was run directly, without the lock —
a declared narrowing, not a silent one. No serialization guarantee held for this
run, nor for any sibling agent in this container while it ran.

One consequence worth a maintainer's eye

A deployment answering a declared 5xx on a polled route — 501 NOT_IMPLEMENTED from an uninstalled optional service, e.g. /api/v1/notifications without service-messaging — now prints one error line per request where it previously printed none. This PR implements the band the issue specifies verbatim ("4xx may stay quiet; 5xx never") and deliberately invents no carve-out: narrowing it for declared capability-absence would be a contract decision, not an implementer's call.

Generated by Claude Code

hotlongand others added 4 commits September 3, 2026 00:57
…ad of answering it silently
A 500 that leaves no server-side line is diagnosed from the browser or not at
all. Measured on `main` @ ca48cf3, through the real plugin and the real route
handlers: a plain `Error` thrown out of a dispatcher route answered
`500 INTERNAL_ERROR` with zero log records at any level.
The reporting that existed was not a substitute: `ErrorReporter` defaults to
`NoopErrorReporter` (so a dev server captured nothing), and it is fed by
`res.__obsRecordedError`, which only the THROWN exit sets — a route that
catches its own fault and RETURNS a 5xx envelope recorded nothing at all.
`logServerFault` (new, `@objectstack/observability`) is the one definition of
the rule, emitting exactly one `error`-level record with method, path, request
id, message and stack. It is wired at each transport's own single exit so a
fault costs one line and never two. 4xx stays quiet, decided inside the helper.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…utes import
`check:check-system-context-census` went red on pure line rot: the
`logServerFault` import added one line to `packages/rest/src/package-routes.ts`,
shifting both anchored elevation reads by one (97 -> 98, 102 -> 103). Repaired
with the gate's own `--fix`; no prose changed and no row added or removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…d land it on sendError
Re-homed under the domain seat's serial fence: `packages/rest/src/package-routes.ts`
is held by the open PR #14499, so this no longer edits that file. The REST
direct-mount doors are covered from the producer side instead.
`sendError` (`@objectstack/types`) is the single writer for every
nested-envelope error in the repo, and every catch in the package registrar
ends there — so wiring the rule at that one exit covers those doors with no
per-door call, and covers any door added later by construction. That also puts
the helper in the same package as `resolveThrownHttpError`, on the same
argument: a rule two doors must agree on cannot live inside one of them.
`@objectstack/observability` and `packages/rest` are back to origin/main
byte-for-byte, as is the system-context census page (its line rot was caused
by the package-routes import this drops).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/runtime, @objectstack/types, touching 22 documentable anchor(s). ⚠️1 changed file(s) yielded no anchor (packages/types/src/index.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

15 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/api/client-sdk.mdx(via requestId (symbol, a field of interface ServerFaultRequest), packages.list (sdk, the route ledger binds it to GET /api/v1/packages))
  • content/docs/api/environment-routing.mdx(via packages.list (sdk, the route ledger binds it to GET /api/v1/packages))
  • content/docs/api/error-catalog.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/api/error-handling-client.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/automation/approvals.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/deployment/production-readiness.mdx(via createDispatcherPlugin (symbol, a top-level function), requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/getting-started/examples.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/kernel/contracts/metadata-service.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/kernel/runtime-services/audit-service.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/kernel/services-checklist.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/permissions/permission-sets.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/plugins/packages.mdx(via createDispatcherPlugin (symbol, a top-level function))
  • content/docs/protocol/kernel/error-handling.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/protocol/kernel/http-protocol.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/ui/apps.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))

1 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/v17.mdx(via requestId (symbol, a field of interface ServerFaultRequest), /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/types/src/index.ts) — pages documenting those are invisible to this run
  • 2 name(s) were too generic to anchor anything (single lowercase words)
  • the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 24 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006fpackageMentionDocs.

Which tree this was computed on

This run read content/docs from 25beb23d6384a4a100200ed6882330e0b71ff003 — the merge of head b4bc2ab473e5f8b9d95326e9a191f4ded2ac98e5 into base 20b883918aa787e65299e72b85c2b3396aa2006f, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 25beb23d6384a4a100200ed6882330e0b71ff003 && git checkout 25beb23d6384a4a100200ed6882330e0b71ff003
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 20b883918aa787e65299e72b85c2b3396aa2006f b4bc2ab473e5f8b9d95326e9a191f4ded2ac98e5 && git checkout -B drift-repro 20b883918aa787e65299e72b85c2b3396aa2006f && git merge --no-ff b4bc2ab473e5f8b9d95326e9a191f4ded2ac98e5
node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006f

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 20b883918aa787e65299e72b85c2b3396aa2006f → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@github-actionsgithub-actionsBot added documentation Improvements or additions to documentation tests tooling labels Sep 2, 2026
@hotlong
hotlong marked this pull request as ready for review September 2, 2026 17:41
@hotlong
hotlong added this pull request to the merge queueSep 2, 2026
Merged via the queue into main with commit 22e5236Sep 2, 2026
35 checks passed
@hotlong
hotlong deleted the claude/issue-14310-log-every-5xx branch September 2, 2026 19:33
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/lteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A 500 from GET /api/v1/packages (and /meta/package/:name) leaves no server-side log line at all

1 participant

@hotlong
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

fix(types,runtime): log every 5xx at error level instead of answering it silently - #14654

Merged
hotlong merged 4 commits into
mainfrom
claude/issue-14310-log-every-5xx
Sep 2, 2026
Merged

fix(types,runtime): log every 5xx at error level instead of answering it silently#14654
hotlong merged 4 commits into
mainfrom
claude/issue-14310-log-every-5xx

Conversation

@hotlong

Copy link
Copy Markdown
Contributor

Fixes#14310

A 500 that leaves no server-side line is diagnosed from the browser or not at all. Reproduced on main @ ca48cf377, driving the real dispatcher plugin and the real route handlers: a plain Error thrown out of a route answered 500 INTERNAL_ERROR with zero log records at any level.

MEASURED status = 500
MEASURED logger.error calls = 0
MEASURED body = {"success":false,"error":{"code":"INTERNAL_ERROR","message":"boom-plain-error","httpStatus":500}}

The single logger.warn in that run was unrelated boot chatter (a fake transport with no setFallbackHandler), so the fault itself reached nobody at any level.

Why the existing machinery did not already cover this

Two independent reasons, both measured rather than reasoned:

  1. ErrorReporter.captureException defaults to NoopErrorReporter. A dev server — the surface an operator actually watches — wires no APM, so the capture was a no-op every time. A log line is the operator's floor; APM is opt-in telemetry on top of it.
  2. It is fed by res.__obsRecordedError, which only the THROWN exit sets. A route that catches its own fault and RETURNS a 5xx envelope — how every /packages handler answers, via deps.errorFromThrown — recorded nothing, so even a wired reporter never saw those. Confirmed in the same run: the thrown path had __obsRecordedError set = true, the returned path false.

Where it lands, and why that is the single conversion point

logServerFault (new, packages/types/src/server-fault-log.ts) emits exactly one error-level record carrying method, path, request id, the message and — where the door still holds the throw — the stack.

It shares a home with resolveThrownHttpError on that rule's own argument: a rule two doors must agree on cannot live inside one of them, because @objectstack/runtime depends on @objectstack/rest and an import could only ever point one way. That is why #8016 moved the "what status does this throw mean" rule to @objectstack/types; "is this answer worth an operator's attention" is the same kind of rule read by the same two doors.

Wired at each transport's own single exit, so a fault costs one line and never two:

ExitFileCovers
sendErrorpackages/types/src/response-envelope.tsevery nested-envelope 5xx in the repo, incl. the REST direct-mount /api/v1/packages registrar
errorResponseBasepackages/runtime/src/dispatcher-plugin.tsthe dispatcher's THROWN exit (stack-bearing)
sendResultBasepackages/runtime/src/dispatcher-plugin.tsthe dispatcher's RETURNED exit — the /packages shape the card was filed on
AI-route mountpackages/runtime/src/dispatcher-plugin.tsthe family that writes its own result inline

sendError is the load-bearing choice for the REST side: it is the one writer every catch in the package registrar already ends in, so those doors become loud from the producer side with no per-door call — and a door added later cannot forget one. packages/rest's /data doors were already loud via logUnexpectedRouteError and are untouched.

Request coordinates reach the two dispatcher exits through a res.__obsRequest side-channel parked by instrumentRouteHandler — the same shape and lifetime as the __obsRecordedError channel that exit has used since #3867. That avoids threading a third argument through the ~50 route handlers that call the locally-shadowed sendResult / errorResponse wrappers.

Evidence that 4xx is unaffected

The 5xx test lives once, inside the helper, rather than at each call site — so no call site can drift from it.

  • Unit: isServerFault pinned at the 499/500 boundary; logServerFault returns false and emits on no channel for a 404.
  • Funnel: sendError at 409 DESTRUCTIVE_CHANGE and 403 FORBIDDEN — the coded refusals that door exists to carry — emit zero lines.
  • End-to-end: an anonymous caller on an auth-gated dispatcher route (401) produces zero fault lines, measured through the real plugin.
  • The wire body is byte-identical at every door; this adds a side effect, never a field (pinned).

error level is load-bearing rather than incidental: the CLI default is warn (DEFAULT_LOG_LEVEL, packages/cli/src/utils/log-level.ts) and error (40) outranks warn (30) in LEVEL_PRIORITY, so the record clears --log-level's default without bypassing the level system. The pins assert the level, not merely that output happened.

Reverse verification

Both landings were ablated from a committed tree, each mutation confirmed on disk by anchor count and object hash before the run, and restored by git checkout HEAD -- ABSOLUTE_PATH verified by empty git diff HEAD plus a worktree/HEAD blob-hash match:

  • removing the call from errorResponseBase → the two thrown-path pins fail (expected [] to have a length of 1), the returned-path and 4xx pins stay green;
  • removing the call from sendError → only the funnel pin fails (1 failed / 12 passed).

Each exit's pin is falsifiable by that exit alone.

Serial fence

Re-derived against the domain seat's fence on the card. packages/rest/src/package-routes.ts is held by the open PR #14499, so this PR does not touch it — the REST doors are covered from the producer side instead. Verified byte-identical to origin/main at this head: package-routes.ts, error-response.ts (PR #14544, merged), dispatcher-error-vocabulary.ts (PR #14625, open).

The #7898 H17 rider does not fire: this diff touches none of packages/core/src/security/auth-gate.ts, packages/runtime/src/http-dispatcher.ts, or packages/adapters/**.

Verification (all at b4bc2ab47)

  • pnpm --filter @objectstack/types exec vitest run — 19 files, 548 passed
  • runtime sweep (dispatcher-plugin*, dispatcher-5xx*, instrument, validation-error, withhold, package-door parity, error-envelope conformance) — 21 files, 285 passed
  • typecheck on types + runtime + rest — clean
  • Gate union re-derived with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack (no paths passed) — 35 families, 31 exit 0. The other four exit 3 = PREREQUISITE NOT MET, not findings: check-test-completeness, check-half-states, check:dual-build-cjs-loads, check:type-check-debt each need a full-repo build or GitHub API access and print "nothing was measured". CI runs them.

⚠️NOT MEASURED, stated rather than implied:packages/runtime/tsconfig.json excludes **/*.test.ts, so pnpm --filter @objectstack/runtime typecheck says nothing about the new test file (tsc --listFiles → 0 matches). Both new test files were type-checked explicitly under a temporary config (0 errors) and the temp configs removed.

Declared narrowing — pnpm lint (whole-repo ESLint) was not run. Targeted instead: eslint --no-inline-config --format json over all 7 changed .ts files — 0 errors, 0 warnings. The three facts that make this a measurement rather than a gap: (a) the file count is read from ESLint's own --format json output, not estimated; (b) the population is read from the repo's config, which never enables type-aware linting for any file (no parserOptions.project, no typed @typescript-eslint rules — eslint.config.mjs, with its own recorded positive control); (c) therefore no untouched file's verdict can move as a result of this diff.

Declared narrowing — verification ran UNLOCKED.scripts/pm/os-verify-lock.sh
could not take the shared verify lock on this host: no usable flock. The shared
verify lock is declared Linux-only (flock is util-linux, and a stock macOS does
not ship it), so the command below was run directly, without the lock —
a declared narrowing, not a silent one. No serialization guarantee held for this
run, nor for any sibling agent in this container while it ran.

One consequence worth a maintainer's eye

A deployment answering a declared 5xx on a polled route — 501 NOT_IMPLEMENTED from an uninstalled optional service, e.g. /api/v1/notifications without service-messaging — now prints one error line per request where it previously printed none. This PR implements the band the issue specifies verbatim ("4xx may stay quiet; 5xx never") and deliberately invents no carve-out: narrowing it for declared capability-absence would be a contract decision, not an implementer's call.

Generated by Claude Code

hotlongand others added 4 commits September 3, 2026 00:57
…ad of answering it silently
A 500 that leaves no server-side line is diagnosed from the browser or not at
all. Measured on `main` @ ca48cf3, through the real plugin and the real route
handlers: a plain `Error` thrown out of a dispatcher route answered
`500 INTERNAL_ERROR` with zero log records at any level.
The reporting that existed was not a substitute: `ErrorReporter` defaults to
`NoopErrorReporter` (so a dev server captured nothing), and it is fed by
`res.__obsRecordedError`, which only the THROWN exit sets — a route that
catches its own fault and RETURNS a 5xx envelope recorded nothing at all.
`logServerFault` (new, `@objectstack/observability`) is the one definition of
the rule, emitting exactly one `error`-level record with method, path, request
id, message and stack. It is wired at each transport's own single exit so a
fault costs one line and never two. 4xx stays quiet, decided inside the helper.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…utes import
`check:check-system-context-census` went red on pure line rot: the
`logServerFault` import added one line to `packages/rest/src/package-routes.ts`,
shifting both anchored elevation reads by one (97 -> 98, 102 -> 103). Repaired
with the gate's own `--fix`; no prose changed and no row added or removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…d land it on sendError
Re-homed under the domain seat's serial fence: `packages/rest/src/package-routes.ts`
is held by the open PR #14499, so this no longer edits that file. The REST
direct-mount doors are covered from the producer side instead.
`sendError` (`@objectstack/types`) is the single writer for every
nested-envelope error in the repo, and every catch in the package registrar
ends there — so wiring the rule at that one exit covers those doors with no
per-door call, and covers any door added later by construction. That also puts
the helper in the same package as `resolveThrownHttpError`, on the same
argument: a rule two doors must agree on cannot live inside one of them.
`@objectstack/observability` and `packages/rest` are back to origin/main
byte-for-byte, as is the system-context census page (its line rot was caused
by the package-routes import this drops).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/runtime, @objectstack/types, touching 22 documentable anchor(s). ⚠️1 changed file(s) yielded no anchor (packages/types/src/index.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

15 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/api/client-sdk.mdx(via requestId (symbol, a field of interface ServerFaultRequest), packages.list (sdk, the route ledger binds it to GET /api/v1/packages))
  • content/docs/api/environment-routing.mdx(via packages.list (sdk, the route ledger binds it to GET /api/v1/packages))
  • content/docs/api/error-catalog.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/api/error-handling-client.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/automation/approvals.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/deployment/production-readiness.mdx(via createDispatcherPlugin (symbol, a top-level function), requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/getting-started/examples.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/kernel/contracts/metadata-service.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/kernel/runtime-services/audit-service.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/kernel/services-checklist.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/permissions/permission-sets.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/plugins/packages.mdx(via createDispatcherPlugin (symbol, a top-level function))
  • content/docs/protocol/kernel/error-handling.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/protocol/kernel/http-protocol.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/ui/apps.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))

1 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/v17.mdx(via requestId (symbol, a field of interface ServerFaultRequest), /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/types/src/index.ts) — pages documenting those are invisible to this run
  • 2 name(s) were too generic to anchor anything (single lowercase words)
  • the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 24 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006fpackageMentionDocs.

Which tree this was computed on

This run read content/docs from 25beb23d6384a4a100200ed6882330e0b71ff003 — the merge of head b4bc2ab473e5f8b9d95326e9a191f4ded2ac98e5 into base 20b883918aa787e65299e72b85c2b3396aa2006f, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 25beb23d6384a4a100200ed6882330e0b71ff003 && git checkout 25beb23d6384a4a100200ed6882330e0b71ff003
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 20b883918aa787e65299e72b85c2b3396aa2006f b4bc2ab473e5f8b9d95326e9a191f4ded2ac98e5 && git checkout -B drift-repro 20b883918aa787e65299e72b85c2b3396aa2006f && git merge --no-ff b4bc2ab473e5f8b9d95326e9a191f4ded2ac98e5
node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006f

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 20b883918aa787e65299e72b85c2b3396aa2006f → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@github-actionsgithub-actionsBot added documentation Improvements or additions to documentation tests tooling labels Sep 2, 2026
@hotlong
hotlong marked this pull request as ready for review September 2, 2026 17:41
@hotlong
hotlong added this pull request to the merge queueSep 2, 2026
Merged via the queue into main with commit 22e5236Sep 2, 2026
35 checks passed
@hotlong
hotlong deleted the claude/issue-14310-log-every-5xx branch September 2, 2026 19:33
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/lteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A 500 from GET /api/v1/packages (and /meta/package/:name) leaves no server-side log line at all

1 participant

@hotlong
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

fix(types,runtime): log every 5xx at error level instead of answering it silently - #14654

Merged
hotlong merged 4 commits into
mainfrom
claude/issue-14310-log-every-5xx
Sep 2, 2026
Merged

fix(types,runtime): log every 5xx at error level instead of answering it silently#14654
hotlong merged 4 commits into
mainfrom
claude/issue-14310-log-every-5xx

Conversation

@hotlong

Copy link
Copy Markdown
Contributor

Fixes#14310

A 500 that leaves no server-side line is diagnosed from the browser or not at all. Reproduced on main @ ca48cf377, driving the real dispatcher plugin and the real route handlers: a plain Error thrown out of a route answered 500 INTERNAL_ERROR with zero log records at any level.

MEASURED status = 500
MEASURED logger.error calls = 0
MEASURED body = {"success":false,"error":{"code":"INTERNAL_ERROR","message":"boom-plain-error","httpStatus":500}}

The single logger.warn in that run was unrelated boot chatter (a fake transport with no setFallbackHandler), so the fault itself reached nobody at any level.

Why the existing machinery did not already cover this

Two independent reasons, both measured rather than reasoned:

  1. ErrorReporter.captureException defaults to NoopErrorReporter. A dev server — the surface an operator actually watches — wires no APM, so the capture was a no-op every time. A log line is the operator's floor; APM is opt-in telemetry on top of it.
  2. It is fed by res.__obsRecordedError, which only the THROWN exit sets. A route that catches its own fault and RETURNS a 5xx envelope — how every /packages handler answers, via deps.errorFromThrown — recorded nothing, so even a wired reporter never saw those. Confirmed in the same run: the thrown path had __obsRecordedError set = true, the returned path false.

Where it lands, and why that is the single conversion point

logServerFault (new, packages/types/src/server-fault-log.ts) emits exactly one error-level record carrying method, path, request id, the message and — where the door still holds the throw — the stack.

It shares a home with resolveThrownHttpError on that rule's own argument: a rule two doors must agree on cannot live inside one of them, because @objectstack/runtime depends on @objectstack/rest and an import could only ever point one way. That is why #8016 moved the "what status does this throw mean" rule to @objectstack/types; "is this answer worth an operator's attention" is the same kind of rule read by the same two doors.

Wired at each transport's own single exit, so a fault costs one line and never two:

ExitFileCovers
sendErrorpackages/types/src/response-envelope.tsevery nested-envelope 5xx in the repo, incl. the REST direct-mount /api/v1/packages registrar
errorResponseBasepackages/runtime/src/dispatcher-plugin.tsthe dispatcher's THROWN exit (stack-bearing)
sendResultBasepackages/runtime/src/dispatcher-plugin.tsthe dispatcher's RETURNED exit — the /packages shape the card was filed on
AI-route mountpackages/runtime/src/dispatcher-plugin.tsthe family that writes its own result inline

sendError is the load-bearing choice for the REST side: it is the one writer every catch in the package registrar already ends in, so those doors become loud from the producer side with no per-door call — and a door added later cannot forget one. packages/rest's /data doors were already loud via logUnexpectedRouteError and are untouched.

Request coordinates reach the two dispatcher exits through a res.__obsRequest side-channel parked by instrumentRouteHandler — the same shape and lifetime as the __obsRecordedError channel that exit has used since #3867. That avoids threading a third argument through the ~50 route handlers that call the locally-shadowed sendResult / errorResponse wrappers.

Evidence that 4xx is unaffected

The 5xx test lives once, inside the helper, rather than at each call site — so no call site can drift from it.

  • Unit: isServerFault pinned at the 499/500 boundary; logServerFault returns false and emits on no channel for a 404.
  • Funnel: sendError at 409 DESTRUCTIVE_CHANGE and 403 FORBIDDEN — the coded refusals that door exists to carry — emit zero lines.
  • End-to-end: an anonymous caller on an auth-gated dispatcher route (401) produces zero fault lines, measured through the real plugin.
  • The wire body is byte-identical at every door; this adds a side effect, never a field (pinned).

error level is load-bearing rather than incidental: the CLI default is warn (DEFAULT_LOG_LEVEL, packages/cli/src/utils/log-level.ts) and error (40) outranks warn (30) in LEVEL_PRIORITY, so the record clears --log-level's default without bypassing the level system. The pins assert the level, not merely that output happened.

Reverse verification

Both landings were ablated from a committed tree, each mutation confirmed on disk by anchor count and object hash before the run, and restored by git checkout HEAD -- ABSOLUTE_PATH verified by empty git diff HEAD plus a worktree/HEAD blob-hash match:

  • removing the call from errorResponseBase → the two thrown-path pins fail (expected [] to have a length of 1), the returned-path and 4xx pins stay green;
  • removing the call from sendError → only the funnel pin fails (1 failed / 12 passed).

Each exit's pin is falsifiable by that exit alone.

Serial fence

Re-derived against the domain seat's fence on the card. packages/rest/src/package-routes.ts is held by the open PR #14499, so this PR does not touch it — the REST doors are covered from the producer side instead. Verified byte-identical to origin/main at this head: package-routes.ts, error-response.ts (PR #14544, merged), dispatcher-error-vocabulary.ts (PR #14625, open).

The #7898 H17 rider does not fire: this diff touches none of packages/core/src/security/auth-gate.ts, packages/runtime/src/http-dispatcher.ts, or packages/adapters/**.

Verification (all at b4bc2ab47)

  • pnpm --filter @objectstack/types exec vitest run — 19 files, 548 passed
  • runtime sweep (dispatcher-plugin*, dispatcher-5xx*, instrument, validation-error, withhold, package-door parity, error-envelope conformance) — 21 files, 285 passed
  • typecheck on types + runtime + rest — clean
  • Gate union re-derived with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack (no paths passed) — 35 families, 31 exit 0. The other four exit 3 = PREREQUISITE NOT MET, not findings: check-test-completeness, check-half-states, check:dual-build-cjs-loads, check:type-check-debt each need a full-repo build or GitHub API access and print "nothing was measured". CI runs them.

⚠️NOT MEASURED, stated rather than implied:packages/runtime/tsconfig.json excludes **/*.test.ts, so pnpm --filter @objectstack/runtime typecheck says nothing about the new test file (tsc --listFiles → 0 matches). Both new test files were type-checked explicitly under a temporary config (0 errors) and the temp configs removed.

Declared narrowing — pnpm lint (whole-repo ESLint) was not run. Targeted instead: eslint --no-inline-config --format json over all 7 changed .ts files — 0 errors, 0 warnings. The three facts that make this a measurement rather than a gap: (a) the file count is read from ESLint's own --format json output, not estimated; (b) the population is read from the repo's config, which never enables type-aware linting for any file (no parserOptions.project, no typed @typescript-eslint rules — eslint.config.mjs, with its own recorded positive control); (c) therefore no untouched file's verdict can move as a result of this diff.

Declared narrowing — verification ran UNLOCKED.scripts/pm/os-verify-lock.sh
could not take the shared verify lock on this host: no usable flock. The shared
verify lock is declared Linux-only (flock is util-linux, and a stock macOS does
not ship it), so the command below was run directly, without the lock —
a declared narrowing, not a silent one. No serialization guarantee held for this
run, nor for any sibling agent in this container while it ran.

One consequence worth a maintainer's eye

A deployment answering a declared 5xx on a polled route — 501 NOT_IMPLEMENTED from an uninstalled optional service, e.g. /api/v1/notifications without service-messaging — now prints one error line per request where it previously printed none. This PR implements the band the issue specifies verbatim ("4xx may stay quiet; 5xx never") and deliberately invents no carve-out: narrowing it for declared capability-absence would be a contract decision, not an implementer's call.

Generated by Claude Code

hotlongand others added 4 commits September 3, 2026 00:57
…ad of answering it silently
A 500 that leaves no server-side line is diagnosed from the browser or not at
all. Measured on `main` @ ca48cf3, through the real plugin and the real route
handlers: a plain `Error` thrown out of a dispatcher route answered
`500 INTERNAL_ERROR` with zero log records at any level.
The reporting that existed was not a substitute: `ErrorReporter` defaults to
`NoopErrorReporter` (so a dev server captured nothing), and it is fed by
`res.__obsRecordedError`, which only the THROWN exit sets — a route that
catches its own fault and RETURNS a 5xx envelope recorded nothing at all.
`logServerFault` (new, `@objectstack/observability`) is the one definition of
the rule, emitting exactly one `error`-level record with method, path, request
id, message and stack. It is wired at each transport's own single exit so a
fault costs one line and never two. 4xx stays quiet, decided inside the helper.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…utes import
`check:check-system-context-census` went red on pure line rot: the
`logServerFault` import added one line to `packages/rest/src/package-routes.ts`,
shifting both anchored elevation reads by one (97 -> 98, 102 -> 103). Repaired
with the gate's own `--fix`; no prose changed and no row added or removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…d land it on sendError
Re-homed under the domain seat's serial fence: `packages/rest/src/package-routes.ts`
is held by the open PR #14499, so this no longer edits that file. The REST
direct-mount doors are covered from the producer side instead.
`sendError` (`@objectstack/types`) is the single writer for every
nested-envelope error in the repo, and every catch in the package registrar
ends there — so wiring the rule at that one exit covers those doors with no
per-door call, and covers any door added later by construction. That also puts
the helper in the same package as `resolveThrownHttpError`, on the same
argument: a rule two doors must agree on cannot live inside one of them.
`@objectstack/observability` and `packages/rest` are back to origin/main
byte-for-byte, as is the system-context census page (its line rot was caused
by the package-routes import this drops).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/runtime, @objectstack/types, touching 22 documentable anchor(s). ⚠️1 changed file(s) yielded no anchor (packages/types/src/index.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

15 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/api/client-sdk.mdx(via requestId (symbol, a field of interface ServerFaultRequest), packages.list (sdk, the route ledger binds it to GET /api/v1/packages))
  • content/docs/api/environment-routing.mdx(via packages.list (sdk, the route ledger binds it to GET /api/v1/packages))
  • content/docs/api/error-catalog.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/api/error-handling-client.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/automation/approvals.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/deployment/production-readiness.mdx(via createDispatcherPlugin (symbol, a top-level function), requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/getting-started/examples.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/kernel/contracts/metadata-service.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/kernel/runtime-services/audit-service.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/kernel/services-checklist.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/permissions/permission-sets.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/plugins/packages.mdx(via createDispatcherPlugin (symbol, a top-level function))
  • content/docs/protocol/kernel/error-handling.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/protocol/kernel/http-protocol.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/ui/apps.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))

1 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/v17.mdx(via requestId (symbol, a field of interface ServerFaultRequest), /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/types/src/index.ts) — pages documenting those are invisible to this run
  • 2 name(s) were too generic to anchor anything (single lowercase words)
  • the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 24 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006fpackageMentionDocs.

Which tree this was computed on

This run read content/docs from 25beb23d6384a4a100200ed6882330e0b71ff003 — the merge of head b4bc2ab473e5f8b9d95326e9a191f4ded2ac98e5 into base 20b883918aa787e65299e72b85c2b3396aa2006f, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 25beb23d6384a4a100200ed6882330e0b71ff003 && git checkout 25beb23d6384a4a100200ed6882330e0b71ff003
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 20b883918aa787e65299e72b85c2b3396aa2006f b4bc2ab473e5f8b9d95326e9a191f4ded2ac98e5 && git checkout -B drift-repro 20b883918aa787e65299e72b85c2b3396aa2006f && git merge --no-ff b4bc2ab473e5f8b9d95326e9a191f4ded2ac98e5
node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006f

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 20b883918aa787e65299e72b85c2b3396aa2006f → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@github-actionsgithub-actionsBot added documentation Improvements or additions to documentation tests tooling labels Sep 2, 2026
@hotlong
hotlong marked this pull request as ready for review September 2, 2026 17:41
@hotlong
hotlong added this pull request to the merge queueSep 2, 2026
Merged via the queue into main with commit 22e5236Sep 2, 2026
35 checks passed
@hotlong
hotlong deleted the claude/issue-14310-log-every-5xx branch September 2, 2026 19:33
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/lteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A 500 from GET /api/v1/packages (and /meta/package/:name) leaves no server-side log line at all

1 participant

@hotlong
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

fix(types,runtime): log every 5xx at error level instead of answering it silently - #14654

Merged
hotlong merged 4 commits into
mainfrom
claude/issue-14310-log-every-5xx
Sep 2, 2026
Merged

fix(types,runtime): log every 5xx at error level instead of answering it silently#14654
hotlong merged 4 commits into
mainfrom
claude/issue-14310-log-every-5xx

Conversation

@hotlong

Copy link
Copy Markdown
Contributor

Fixes#14310

A 500 that leaves no server-side line is diagnosed from the browser or not at all. Reproduced on main @ ca48cf377, driving the real dispatcher plugin and the real route handlers: a plain Error thrown out of a route answered 500 INTERNAL_ERROR with zero log records at any level.

MEASURED status = 500
MEASURED logger.error calls = 0
MEASURED body = {"success":false,"error":{"code":"INTERNAL_ERROR","message":"boom-plain-error","httpStatus":500}}

The single logger.warn in that run was unrelated boot chatter (a fake transport with no setFallbackHandler), so the fault itself reached nobody at any level.

Why the existing machinery did not already cover this

Two independent reasons, both measured rather than reasoned:

  1. ErrorReporter.captureException defaults to NoopErrorReporter. A dev server — the surface an operator actually watches — wires no APM, so the capture was a no-op every time. A log line is the operator's floor; APM is opt-in telemetry on top of it.
  2. It is fed by res.__obsRecordedError, which only the THROWN exit sets. A route that catches its own fault and RETURNS a 5xx envelope — how every /packages handler answers, via deps.errorFromThrown — recorded nothing, so even a wired reporter never saw those. Confirmed in the same run: the thrown path had __obsRecordedError set = true, the returned path false.

Where it lands, and why that is the single conversion point

logServerFault (new, packages/types/src/server-fault-log.ts) emits exactly one error-level record carrying method, path, request id, the message and — where the door still holds the throw — the stack.

It shares a home with resolveThrownHttpError on that rule's own argument: a rule two doors must agree on cannot live inside one of them, because @objectstack/runtime depends on @objectstack/rest and an import could only ever point one way. That is why #8016 moved the "what status does this throw mean" rule to @objectstack/types; "is this answer worth an operator's attention" is the same kind of rule read by the same two doors.

Wired at each transport's own single exit, so a fault costs one line and never two:

ExitFileCovers
sendErrorpackages/types/src/response-envelope.tsevery nested-envelope 5xx in the repo, incl. the REST direct-mount /api/v1/packages registrar
errorResponseBasepackages/runtime/src/dispatcher-plugin.tsthe dispatcher's THROWN exit (stack-bearing)
sendResultBasepackages/runtime/src/dispatcher-plugin.tsthe dispatcher's RETURNED exit — the /packages shape the card was filed on
AI-route mountpackages/runtime/src/dispatcher-plugin.tsthe family that writes its own result inline

sendError is the load-bearing choice for the REST side: it is the one writer every catch in the package registrar already ends in, so those doors become loud from the producer side with no per-door call — and a door added later cannot forget one. packages/rest's /data doors were already loud via logUnexpectedRouteError and are untouched.

Request coordinates reach the two dispatcher exits through a res.__obsRequest side-channel parked by instrumentRouteHandler — the same shape and lifetime as the __obsRecordedError channel that exit has used since #3867. That avoids threading a third argument through the ~50 route handlers that call the locally-shadowed sendResult / errorResponse wrappers.

Evidence that 4xx is unaffected

The 5xx test lives once, inside the helper, rather than at each call site — so no call site can drift from it.

  • Unit: isServerFault pinned at the 499/500 boundary; logServerFault returns false and emits on no channel for a 404.
  • Funnel: sendError at 409 DESTRUCTIVE_CHANGE and 403 FORBIDDEN — the coded refusals that door exists to carry — emit zero lines.
  • End-to-end: an anonymous caller on an auth-gated dispatcher route (401) produces zero fault lines, measured through the real plugin.
  • The wire body is byte-identical at every door; this adds a side effect, never a field (pinned).

error level is load-bearing rather than incidental: the CLI default is warn (DEFAULT_LOG_LEVEL, packages/cli/src/utils/log-level.ts) and error (40) outranks warn (30) in LEVEL_PRIORITY, so the record clears --log-level's default without bypassing the level system. The pins assert the level, not merely that output happened.

Reverse verification

Both landings were ablated from a committed tree, each mutation confirmed on disk by anchor count and object hash before the run, and restored by git checkout HEAD -- ABSOLUTE_PATH verified by empty git diff HEAD plus a worktree/HEAD blob-hash match:

  • removing the call from errorResponseBase → the two thrown-path pins fail (expected [] to have a length of 1), the returned-path and 4xx pins stay green;
  • removing the call from sendError → only the funnel pin fails (1 failed / 12 passed).

Each exit's pin is falsifiable by that exit alone.

Serial fence

Re-derived against the domain seat's fence on the card. packages/rest/src/package-routes.ts is held by the open PR #14499, so this PR does not touch it — the REST doors are covered from the producer side instead. Verified byte-identical to origin/main at this head: package-routes.ts, error-response.ts (PR #14544, merged), dispatcher-error-vocabulary.ts (PR #14625, open).

The #7898 H17 rider does not fire: this diff touches none of packages/core/src/security/auth-gate.ts, packages/runtime/src/http-dispatcher.ts, or packages/adapters/**.

Verification (all at b4bc2ab47)

  • pnpm --filter @objectstack/types exec vitest run — 19 files, 548 passed
  • runtime sweep (dispatcher-plugin*, dispatcher-5xx*, instrument, validation-error, withhold, package-door parity, error-envelope conformance) — 21 files, 285 passed
  • typecheck on types + runtime + rest — clean
  • Gate union re-derived with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack (no paths passed) — 35 families, 31 exit 0. The other four exit 3 = PREREQUISITE NOT MET, not findings: check-test-completeness, check-half-states, check:dual-build-cjs-loads, check:type-check-debt each need a full-repo build or GitHub API access and print "nothing was measured". CI runs them.

⚠️NOT MEASURED, stated rather than implied:packages/runtime/tsconfig.json excludes **/*.test.ts, so pnpm --filter @objectstack/runtime typecheck says nothing about the new test file (tsc --listFiles → 0 matches). Both new test files were type-checked explicitly under a temporary config (0 errors) and the temp configs removed.

Declared narrowing — pnpm lint (whole-repo ESLint) was not run. Targeted instead: eslint --no-inline-config --format json over all 7 changed .ts files — 0 errors, 0 warnings. The three facts that make this a measurement rather than a gap: (a) the file count is read from ESLint's own --format json output, not estimated; (b) the population is read from the repo's config, which never enables type-aware linting for any file (no parserOptions.project, no typed @typescript-eslint rules — eslint.config.mjs, with its own recorded positive control); (c) therefore no untouched file's verdict can move as a result of this diff.

Declared narrowing — verification ran UNLOCKED.scripts/pm/os-verify-lock.sh
could not take the shared verify lock on this host: no usable flock. The shared
verify lock is declared Linux-only (flock is util-linux, and a stock macOS does
not ship it), so the command below was run directly, without the lock —
a declared narrowing, not a silent one. No serialization guarantee held for this
run, nor for any sibling agent in this container while it ran.

One consequence worth a maintainer's eye

A deployment answering a declared 5xx on a polled route — 501 NOT_IMPLEMENTED from an uninstalled optional service, e.g. /api/v1/notifications without service-messaging — now prints one error line per request where it previously printed none. This PR implements the band the issue specifies verbatim ("4xx may stay quiet; 5xx never") and deliberately invents no carve-out: narrowing it for declared capability-absence would be a contract decision, not an implementer's call.

Generated by Claude Code

hotlongand others added 4 commits September 3, 2026 00:57
…ad of answering it silently
A 500 that leaves no server-side line is diagnosed from the browser or not at
all. Measured on `main` @ ca48cf3, through the real plugin and the real route
handlers: a plain `Error` thrown out of a dispatcher route answered
`500 INTERNAL_ERROR` with zero log records at any level.
The reporting that existed was not a substitute: `ErrorReporter` defaults to
`NoopErrorReporter` (so a dev server captured nothing), and it is fed by
`res.__obsRecordedError`, which only the THROWN exit sets — a route that
catches its own fault and RETURNS a 5xx envelope recorded nothing at all.
`logServerFault` (new, `@objectstack/observability`) is the one definition of
the rule, emitting exactly one `error`-level record with method, path, request
id, message and stack. It is wired at each transport's own single exit so a
fault costs one line and never two. 4xx stays quiet, decided inside the helper.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…utes import
`check:check-system-context-census` went red on pure line rot: the
`logServerFault` import added one line to `packages/rest/src/package-routes.ts`,
shifting both anchored elevation reads by one (97 -> 98, 102 -> 103). Repaired
with the gate's own `--fix`; no prose changed and no row added or removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…d land it on sendError
Re-homed under the domain seat's serial fence: `packages/rest/src/package-routes.ts`
is held by the open PR #14499, so this no longer edits that file. The REST
direct-mount doors are covered from the producer side instead.
`sendError` (`@objectstack/types`) is the single writer for every
nested-envelope error in the repo, and every catch in the package registrar
ends there — so wiring the rule at that one exit covers those doors with no
per-door call, and covers any door added later by construction. That also puts
the helper in the same package as `resolveThrownHttpError`, on the same
argument: a rule two doors must agree on cannot live inside one of them.
`@objectstack/observability` and `packages/rest` are back to origin/main
byte-for-byte, as is the system-context census page (its line rot was caused
by the package-routes import this drops).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/runtime, @objectstack/types, touching 22 documentable anchor(s). ⚠️1 changed file(s) yielded no anchor (packages/types/src/index.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

15 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/api/client-sdk.mdx(via requestId (symbol, a field of interface ServerFaultRequest), packages.list (sdk, the route ledger binds it to GET /api/v1/packages))
  • content/docs/api/environment-routing.mdx(via packages.list (sdk, the route ledger binds it to GET /api/v1/packages))
  • content/docs/api/error-catalog.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/api/error-handling-client.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/automation/approvals.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/deployment/production-readiness.mdx(via createDispatcherPlugin (symbol, a top-level function), requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/getting-started/examples.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/kernel/contracts/metadata-service.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/kernel/runtime-services/audit-service.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/kernel/services-checklist.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/permissions/permission-sets.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/plugins/packages.mdx(via createDispatcherPlugin (symbol, a top-level function))
  • content/docs/protocol/kernel/error-handling.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/protocol/kernel/http-protocol.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/ui/apps.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))

1 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/v17.mdx(via requestId (symbol, a field of interface ServerFaultRequest), /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/types/src/index.ts) — pages documenting those are invisible to this run
  • 2 name(s) were too generic to anchor anything (single lowercase words)
  • the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 24 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006fpackageMentionDocs.

Which tree this was computed on

This run read content/docs from 25beb23d6384a4a100200ed6882330e0b71ff003 — the merge of head b4bc2ab473e5f8b9d95326e9a191f4ded2ac98e5 into base 20b883918aa787e65299e72b85c2b3396aa2006f, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 25beb23d6384a4a100200ed6882330e0b71ff003 && git checkout 25beb23d6384a4a100200ed6882330e0b71ff003
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 20b883918aa787e65299e72b85c2b3396aa2006f b4bc2ab473e5f8b9d95326e9a191f4ded2ac98e5 && git checkout -B drift-repro 20b883918aa787e65299e72b85c2b3396aa2006f && git merge --no-ff b4bc2ab473e5f8b9d95326e9a191f4ded2ac98e5
node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006f

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 20b883918aa787e65299e72b85c2b3396aa2006f → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@github-actionsgithub-actionsBot added documentation Improvements or additions to documentation tests tooling labels Sep 2, 2026
@hotlong
hotlong marked this pull request as ready for review September 2, 2026 17:41
@hotlong
hotlong added this pull request to the merge queueSep 2, 2026
Merged via the queue into main with commit 22e5236Sep 2, 2026
35 checks passed
@hotlong
hotlong deleted the claude/issue-14310-log-every-5xx branch September 2, 2026 19:33
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/lteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A 500 from GET /api/v1/packages (and /meta/package/:name) leaves no server-side log line at all

1 participant

@hotlong
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

fix(types,runtime): log every 5xx at error level instead of answering it silently - #14654

Merged
hotlong merged 4 commits into
mainfrom
claude/issue-14310-log-every-5xx
Sep 2, 2026
Merged

fix(types,runtime): log every 5xx at error level instead of answering it silently#14654
hotlong merged 4 commits into
mainfrom
claude/issue-14310-log-every-5xx

Conversation

@hotlong

Copy link
Copy Markdown
Contributor

Fixes#14310

A 500 that leaves no server-side line is diagnosed from the browser or not at all. Reproduced on main @ ca48cf377, driving the real dispatcher plugin and the real route handlers: a plain Error thrown out of a route answered 500 INTERNAL_ERROR with zero log records at any level.

MEASURED status = 500
MEASURED logger.error calls = 0
MEASURED body = {"success":false,"error":{"code":"INTERNAL_ERROR","message":"boom-plain-error","httpStatus":500}}

The single logger.warn in that run was unrelated boot chatter (a fake transport with no setFallbackHandler), so the fault itself reached nobody at any level.

Why the existing machinery did not already cover this

Two independent reasons, both measured rather than reasoned:

  1. ErrorReporter.captureException defaults to NoopErrorReporter. A dev server — the surface an operator actually watches — wires no APM, so the capture was a no-op every time. A log line is the operator's floor; APM is opt-in telemetry on top of it.
  2. It is fed by res.__obsRecordedError, which only the THROWN exit sets. A route that catches its own fault and RETURNS a 5xx envelope — how every /packages handler answers, via deps.errorFromThrown — recorded nothing, so even a wired reporter never saw those. Confirmed in the same run: the thrown path had __obsRecordedError set = true, the returned path false.

Where it lands, and why that is the single conversion point

logServerFault (new, packages/types/src/server-fault-log.ts) emits exactly one error-level record carrying method, path, request id, the message and — where the door still holds the throw — the stack.

It shares a home with resolveThrownHttpError on that rule's own argument: a rule two doors must agree on cannot live inside one of them, because @objectstack/runtime depends on @objectstack/rest and an import could only ever point one way. That is why #8016 moved the "what status does this throw mean" rule to @objectstack/types; "is this answer worth an operator's attention" is the same kind of rule read by the same two doors.

Wired at each transport's own single exit, so a fault costs one line and never two:

ExitFileCovers
sendErrorpackages/types/src/response-envelope.tsevery nested-envelope 5xx in the repo, incl. the REST direct-mount /api/v1/packages registrar
errorResponseBasepackages/runtime/src/dispatcher-plugin.tsthe dispatcher's THROWN exit (stack-bearing)
sendResultBasepackages/runtime/src/dispatcher-plugin.tsthe dispatcher's RETURNED exit — the /packages shape the card was filed on
AI-route mountpackages/runtime/src/dispatcher-plugin.tsthe family that writes its own result inline

sendError is the load-bearing choice for the REST side: it is the one writer every catch in the package registrar already ends in, so those doors become loud from the producer side with no per-door call — and a door added later cannot forget one. packages/rest's /data doors were already loud via logUnexpectedRouteError and are untouched.

Request coordinates reach the two dispatcher exits through a res.__obsRequest side-channel parked by instrumentRouteHandler — the same shape and lifetime as the __obsRecordedError channel that exit has used since #3867. That avoids threading a third argument through the ~50 route handlers that call the locally-shadowed sendResult / errorResponse wrappers.

Evidence that 4xx is unaffected

The 5xx test lives once, inside the helper, rather than at each call site — so no call site can drift from it.

  • Unit: isServerFault pinned at the 499/500 boundary; logServerFault returns false and emits on no channel for a 404.
  • Funnel: sendError at 409 DESTRUCTIVE_CHANGE and 403 FORBIDDEN — the coded refusals that door exists to carry — emit zero lines.
  • End-to-end: an anonymous caller on an auth-gated dispatcher route (401) produces zero fault lines, measured through the real plugin.
  • The wire body is byte-identical at every door; this adds a side effect, never a field (pinned).

error level is load-bearing rather than incidental: the CLI default is warn (DEFAULT_LOG_LEVEL, packages/cli/src/utils/log-level.ts) and error (40) outranks warn (30) in LEVEL_PRIORITY, so the record clears --log-level's default without bypassing the level system. The pins assert the level, not merely that output happened.

Reverse verification

Both landings were ablated from a committed tree, each mutation confirmed on disk by anchor count and object hash before the run, and restored by git checkout HEAD -- ABSOLUTE_PATH verified by empty git diff HEAD plus a worktree/HEAD blob-hash match:

  • removing the call from errorResponseBase → the two thrown-path pins fail (expected [] to have a length of 1), the returned-path and 4xx pins stay green;
  • removing the call from sendError → only the funnel pin fails (1 failed / 12 passed).

Each exit's pin is falsifiable by that exit alone.

Serial fence

Re-derived against the domain seat's fence on the card. packages/rest/src/package-routes.ts is held by the open PR #14499, so this PR does not touch it — the REST doors are covered from the producer side instead. Verified byte-identical to origin/main at this head: package-routes.ts, error-response.ts (PR #14544, merged), dispatcher-error-vocabulary.ts (PR #14625, open).

The #7898 H17 rider does not fire: this diff touches none of packages/core/src/security/auth-gate.ts, packages/runtime/src/http-dispatcher.ts, or packages/adapters/**.

Verification (all at b4bc2ab47)

  • pnpm --filter @objectstack/types exec vitest run — 19 files, 548 passed
  • runtime sweep (dispatcher-plugin*, dispatcher-5xx*, instrument, validation-error, withhold, package-door parity, error-envelope conformance) — 21 files, 285 passed
  • typecheck on types + runtime + rest — clean
  • Gate union re-derived with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack (no paths passed) — 35 families, 31 exit 0. The other four exit 3 = PREREQUISITE NOT MET, not findings: check-test-completeness, check-half-states, check:dual-build-cjs-loads, check:type-check-debt each need a full-repo build or GitHub API access and print "nothing was measured". CI runs them.

⚠️NOT MEASURED, stated rather than implied:packages/runtime/tsconfig.json excludes **/*.test.ts, so pnpm --filter @objectstack/runtime typecheck says nothing about the new test file (tsc --listFiles → 0 matches). Both new test files were type-checked explicitly under a temporary config (0 errors) and the temp configs removed.

Declared narrowing — pnpm lint (whole-repo ESLint) was not run. Targeted instead: eslint --no-inline-config --format json over all 7 changed .ts files — 0 errors, 0 warnings. The three facts that make this a measurement rather than a gap: (a) the file count is read from ESLint's own --format json output, not estimated; (b) the population is read from the repo's config, which never enables type-aware linting for any file (no parserOptions.project, no typed @typescript-eslint rules — eslint.config.mjs, with its own recorded positive control); (c) therefore no untouched file's verdict can move as a result of this diff.

Declared narrowing — verification ran UNLOCKED.scripts/pm/os-verify-lock.sh
could not take the shared verify lock on this host: no usable flock. The shared
verify lock is declared Linux-only (flock is util-linux, and a stock macOS does
not ship it), so the command below was run directly, without the lock —
a declared narrowing, not a silent one. No serialization guarantee held for this
run, nor for any sibling agent in this container while it ran.

One consequence worth a maintainer's eye

A deployment answering a declared 5xx on a polled route — 501 NOT_IMPLEMENTED from an uninstalled optional service, e.g. /api/v1/notifications without service-messaging — now prints one error line per request where it previously printed none. This PR implements the band the issue specifies verbatim ("4xx may stay quiet; 5xx never") and deliberately invents no carve-out: narrowing it for declared capability-absence would be a contract decision, not an implementer's call.

Generated by Claude Code

hotlongand others added 4 commits September 3, 2026 00:57
…ad of answering it silently
A 500 that leaves no server-side line is diagnosed from the browser or not at
all. Measured on `main` @ ca48cf3, through the real plugin and the real route
handlers: a plain `Error` thrown out of a dispatcher route answered
`500 INTERNAL_ERROR` with zero log records at any level.
The reporting that existed was not a substitute: `ErrorReporter` defaults to
`NoopErrorReporter` (so a dev server captured nothing), and it is fed by
`res.__obsRecordedError`, which only the THROWN exit sets — a route that
catches its own fault and RETURNS a 5xx envelope recorded nothing at all.
`logServerFault` (new, `@objectstack/observability`) is the one definition of
the rule, emitting exactly one `error`-level record with method, path, request
id, message and stack. It is wired at each transport's own single exit so a
fault costs one line and never two. 4xx stays quiet, decided inside the helper.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…utes import
`check:check-system-context-census` went red on pure line rot: the
`logServerFault` import added one line to `packages/rest/src/package-routes.ts`,
shifting both anchored elevation reads by one (97 -> 98, 102 -> 103). Repaired
with the gate's own `--fix`; no prose changed and no row added or removed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…d land it on sendError
Re-homed under the domain seat's serial fence: `packages/rest/src/package-routes.ts`
is held by the open PR #14499, so this no longer edits that file. The REST
direct-mount doors are covered from the producer side instead.
`sendError` (`@objectstack/types`) is the single writer for every
nested-envelope error in the repo, and every catch in the package registrar
ends there — so wiring the rule at that one exit covers those doors with no
per-door call, and covers any door added later by construction. That also puts
the helper in the same package as `resolveThrownHttpError`, on the same
argument: a rule two doors must agree on cannot live inside one of them.
`@objectstack/observability` and `packages/rest` are back to origin/main
byte-for-byte, as is the system-context census page (its line rot was caused
by the package-routes import this drops).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/runtime, @objectstack/types, touching 22 documentable anchor(s). ⚠️1 changed file(s) yielded no anchor (packages/types/src/index.ts), so the pages documenting them are NOT COVERED by this run — this is not a clean bill of health for those files.

15 hand-written doc(s) NAME something this change touched and may need an implementation-accuracy re-verification:

  • content/docs/api/client-sdk.mdx(via requestId (symbol, a field of interface ServerFaultRequest), packages.list (sdk, the route ledger binds it to GET /api/v1/packages))
  • content/docs/api/environment-routing.mdx(via packages.list (sdk, the route ledger binds it to GET /api/v1/packages))
  • content/docs/api/error-catalog.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/api/error-handling-client.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/automation/approvals.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/deployment/production-readiness.mdx(via createDispatcherPlugin (symbol, a top-level function), requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/getting-started/examples.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/kernel/contracts/metadata-service.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/kernel/runtime-services/audit-service.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/kernel/services-checklist.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/permissions/permission-sets.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/plugins/packages.mdx(via createDispatcherPlugin (symbol, a top-level function))
  • content/docs/protocol/kernel/error-handling.mdx(via requestId (symbol, a field of interface ServerFaultRequest))
  • content/docs/protocol/kernel/http-protocol.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))
  • content/docs/ui/apps.mdx(via /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))

1 release-owned page(s) also name something this change touched. These are read-only:

  • content/docs/releases/v17.mdx(via requestId (symbol, a field of interface ServerFaultRequest), /api/v1/packages (route, a path literal in ServerFaultRequest; a path literal in sendError; a path literal on a changed line))

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

What this run could not see
  • 1 changed file(s) yielded no anchor (packages/types/src/index.ts) — pages documenting those are invisible to this run
  • 2 name(s) were too generic to anchor anything (single lowercase words)
  • the SDK route bridge reached 47 of 219 client-bound route-ledger rows — the other 172 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 172: 14 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 56 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 102 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
  • a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.

Coarse fallback — 24 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006fpackageMentionDocs.

Which tree this was computed on

This run read content/docs from 25beb23d6384a4a100200ed6882330e0b71ff003 — the merge of head b4bc2ab473e5f8b9d95326e9a191f4ded2ac98e5 into base 20b883918aa787e65299e72b85c2b3396aa2006f, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 25beb23d6384a4a100200ed6882330e0b71ff003 && git checkout 25beb23d6384a4a100200ed6882330e0b71ff003
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 20b883918aa787e65299e72b85c2b3396aa2006f b4bc2ab473e5f8b9d95326e9a191f4ded2ac98e5 && git checkout -B drift-repro 20b883918aa787e65299e72b85c2b3396aa2006f && git merge --no-ff b4bc2ab473e5f8b9d95326e9a191f4ded2ac98e5
node scripts/docs-audit/affected-docs.mjs --json 20b883918aa787e65299e72b85c2b3396aa2006f

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 20b883918aa787e65299e72b85c2b3396aa2006f → pass the list as
args.docs, on the commit named under Which tree this was computed on.

@github-actionsgithub-actionsBot added documentation Improvements or additions to documentation tests tooling labels Sep 2, 2026
@hotlong
hotlong marked this pull request as ready for review September 2, 2026 17:41
@hotlong
hotlong added this pull request to the merge queueSep 2, 2026
Merged via the queue into main with commit 22e5236Sep 2, 2026
35 checks passed
@hotlong
hotlong deleted the claude/issue-14310-log-every-5xx branch September 2, 2026 19:33
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/lteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A 500 from GET /api/v1/packages (and /meta/package/:name) leaves no server-side log line at all

1 participant

@hotlong