Skip to content

REST routes pay the host kernel-waiter window TWICE per request: resolveRequestEnvironmentId buys a kernel it discards, then resolveProtocol buys it again (42s vs 21s measured) #10988

Description

@os-zhuang

One REST request on a multi-tenant host pays the host's kernel-waiter window twice: once inside RestServer.resolveRequestEnvironmentId (which asks the ADR-0006 kernel-resolver seam for an environment id and gets a kernel acquisition as a side effect), and again in RestServer.resolveProtocol, which throws the first result away and calls kernelManager.getOrCreate(envId) itself.

Measured on a live deployment

ObjectStack Cloud's KernelManager.waiterTimeoutMs is 20s. Against a wedged environment, same client, within one minute (downstream report: objectstack-ai/cloud#1548):

/health 200 2.23s (static route, no kernel)
/api/v1/health 200 0.92s (host kernel, no tenant kernel)
/api/v1/i18n/locales 503 21.45s ← ONE waiter window (dispatcher-owned route)
/api/v1/metadata 503 22.46s ← ONE waiter window
/api/v1/discovery 503 42.49s ← TWO waiter windows (REST-owned route)
/api/v1/data/sys_user 503 42.71s ← TWO waiter windows (REST-owned route)

The split follows route ownership: the routes that pay once are the runtime dispatcher's; the two that pay twice are the ones @objectstack/rest owns (registerDiscoveryEndpoints — ceded to REST by dispatcher-plugin.ts when both plugins are mounted — and registerCrudEndpoints).

Root cause — one resolution is bought and thrown away

packages/rest/src/rest-api-plugin.ts:187-198 wraps the host's kernel-resolver so the REST server can ask "which environment is this request in?":

asyncresolveRequestEnvironmentId(req: unknown): Promise<string|undefined>{constcontext: {request: unknown;environmentId?: string}={request: req};awaitkernelResolver.resolveKernel(context,hostKernelFacade);// ← acquires a KERNELreturncontext.environmentId;// ← keeps only the ID}

resolveKernel is a kernel-acquisition API. A host implementation resolves the environment, writes context.environmentId, and then awaits the kernel for that environment before returning. The wrapper wants only the id — the kernel it just paid for is discarded on the success path and lost with the rejection on the cold path.

packages/rest/src/rest-server.ts:1057-1069 then swallows the rejection and falls through to the legacy chain:

if(req&&this.requestEnvResolver){try{returnawaitthis.requestEnvResolver.resolveRequestEnvironmentId(req);}catch{/* resolver failure → legacy chain */}}// … legacy hostname / X-Environment-Id chain (cached, cheap) …

and resolveProtocol (rest-server.ts:1112-1118) pays for the kernel a second time:

privateasyncresolveProtocol(environmentId?: string,req?: any): Promise<RestProtocol>{if(environmentId==='platform')returnthis.protocol;constenvId=awaitthis.resolveRequestEnvironmentId(environmentId,req);// ← window #1if(!envId||!this.kernelManager)returnthis.protocol;constkernel=awaitthis.kernelManager.getOrCreate(envId);// ← window #2returnkernel.getServiceAsync<RestProtocol>('protocol');}

On a warm environment window #2 is a cache hit and costs nothing, which is why this has been invisible. On a cold or wedged environment each getOrCreate opens its own bounded wait, so one request serially burns two.

The same "resolve the env (→ acquire a kernel), then acquire the kernel again" shape repeats at rest-server.ts lines 934, 971, 1141 and 3107 (resolveEndpointMatchAuthority, resolveMetadataService, resolveI18nService, probeMcpServeable), so routes that reach several of them can stack more than two windows.

Evidence — instrumented, with stacks

Harness: the realRestServer (built packages/rest/dist) wired to a host kernel-resolver implementation and a kernel manager whose build stays in flight, waiterTimeoutMs scaled to 300ms. Every getOrCreate records a stack.

=== GET /api/v1/discovery [GET /api/v1/discovery] ===
wall time : 768 ms (ONE window = 300 ms)
windows paid : 2.56 x
getOrCreate : 2 call(s)
response : status=503 code=SERVICE_UNAVAILABLE declaredCode=kernel_warming
--- getOrCreate #1 t+2ms
at <host>/kernel-resolver.ts (resolveKernel → kernelFor)
at async _RestServer.resolveRequestEnvironmentId (rest/dist/index.js:57017) ← src rest-server.ts:1067
at async _RestServer.resolveProtocol (rest/dist/index.js:57052) ← src rest-server.ts:1114
at async discoveryHandler (rest/dist/index.js:58586) ← src rest-server.ts:3165
--- getOrCreate #2 t+460ms
at _RestServer.resolveProtocol (rest/dist/index.js:57054) ← src rest-server.ts:1116
at async discoveryHandler (rest/dist/index.js:58586)
=== GET /api/v1/data/:object [GET /api/v1/data/:object] ===
wall time : 605 ms (2.02 x) getOrCreate: 2 call(s) status=503
--- getOrCreate #1 at rest-server.ts:1067 (via resolveProtocol → list handler, rest-server.ts:6639)
--- getOrCreate #2 at rest-server.ts:1116

Self-contained framework-only reproduction (no host distribution needed) — a resolver that behaves like a real one, i.e. writes the id and then awaits a kernel:

constNEVER=()=>newPromise<any>((r)=>setTimeout(r,60_000));constkm={getOrCreate: async(_id: string)=>{calls++;returnboundedWait(NEVER(),300);}};constkernelResolver={asyncresolveKernel(context: any){context.environmentId='env_probe';// resolved BEFORE the kernel is awaitedreturnkm.getOrCreate('env_probe');// ← rejects after one 300ms window},};// wire as rest-api-plugin does, hand RestServer `km` + that requestEnvResolver,// then invoke the GET /api/v1/data/:object handler and time it.// observed: ~2 x window, calls === 2

The obvious fix is NOT the fix — also measured

"Swallow the resolver's throw but keep the environmentId it already wrote on the context" removes the legacy-chain detour but changes nothing: measured 2.38x / 2.02x, still 2 getOrCreate calls. The wasted window is spent inside the resolver call, before any id is returned, so no amount of catching at the REST layer reclaims it. The waste is on the success path too — it is simply free there.

Proposed fix — give the seam its own question

The seam asks a kernel-acquisition API an environment-resolution question. Add an environment-only capability to the ADR-0006 contract (KernelResolver, packages/runtime/src/http-dispatcher.ts), have rest-api-plugin's wrapper prefer it, and let resolveProtocol remain the single place that acquires a kernel:

exportinterfaceKernelResolver{resolveKernel(context,defaultKernel): Promise<ObjectKernel|undefined>|ObjectKernel|undefined;/** Resolve ONLY the request's environment onto `context`. No kernel acquisition. */resolveEnvironment?(context,defaultKernel): Promise<void>|void;}

Measured with the resolver's env-resolution chain run kernel-acquisition-free (--envonly in the same harness):

mode/api/v1/discovery/api/v1/data/:objectgetOrCreateresponse
today768 ms (2.56x)605 ms (2.02x)2503 kernel_warming
catch-and-keep-the-id714 ms (2.38x)607 ms (2.02x)2503 kernel_warming
resolveEnvironment (proposed)413 ms (1.38x)301 ms (1.00x)1503 kernel_warming

Two properties the fix must keep, and does:

  • Fail closed. The surviving getOrCreate in resolveProtocol still rejects, so a genuinely unavailable kernel is still a 503 — collapsing the windows must not turn "waited 40s then 503" into "served against no kernel". All three rows above answer 503.
  • The window itself is untouched.waiterTimeoutMs is a host setting and stays as it is; the defect is waiting twice, not waiting wrong.

An ?.-optional method keeps every existing resolver working unchanged (it just keeps paying twice on cold builds), so this can land before any host implements the new half.

Filed from objectstack-ai/cloud#1548, where the production measurement was taken. The consumer side is deliberately not patched there — the resolution seam and both call sites are framework code.

Metadata

Metadata

Assignees

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions