You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
lead_finder cannot sweep a capital city: the raw addressComponents arrays make the flatten step's journal entry exceed 1MiB
Three pipeline-run errors, all on instance 933ebec5…, read back today via list_errors:
2026-08-03 08:08 pipeline "lead_finder" crashed: WorkflowInternalError: Step s3-flatten-1 output is too large. Maximum allowed size is 1MiB.
2026-08-03 06:49 pipeline "lead_finder" crashed: WorkflowInternalError: Step s4-map-1 output is too large. Maximum allowed size is 1MiB.
2026-08-03 06:32 pipeline "lead-finder" crashed: WorkflowInternalError: Step s3-map-1 output is too large. Maximum allowed size is 1MiB.
list_pipeline_runs names the cities: the two failures with a city param are Sydney, NSW and Brisbane, QLD. Every completed sweep is Hobart (116 places) or Launceston (83), both at radius: 900. This is the one catalog card promising a general capability — "Search by city and
business type" — that fails on the first city a visitor would try.
Root cause, confirmed against the code
Not "the payload grows with the city" in general. It is one field, and one property of the runner.
A StepResult carries its payload twice. Every step handler returns pretty-printed JSON
(steps.ts:510 — ok(JSON.stringify({ items: out, count: out.length }, null, 2))), and executePipelineStep ALSO stores the parsed value (pipeline.ts:436, output: parseOutput(r.content)). capStepOutput measures JSON.stringify(result) (pipeline.ts:88), so a step's journal entry is
roughly 3× the records it holds. A forEach step is the exception: its content is "25 item(s)"
(pipeline.ts:431), so the fan-out carries ~1×. That is exactly why flatten — which holds the same
data as the fan-out that fed it — is the first step to fail, and why the production error names s3-flatten-1 rather than the http_request above it.
The size is structural, not unlucky.extentKm: 2 / stepKm: 1 → gridCells yields 25 cells
(steps.ts:295), each requesting maxResultCount: 20, so a dense city returns 500 places and
cannot return more. Hobart's 116 fit; 500 do not.
addressComponents is 73% of the bytes. The responseMap projected Google's raw typed-component
array onto every record so the map step could extract city/state/suburb/country from it. Measured
on a 500-place fixture with the component arrays Google actually returns: the flatten step's StepResult is 1681KB with them and 459KB without. capStepOutput's bound is 900KB.
Where the audit's prescription was wrong
#64 diagnosed the mechanism correctly (unbounded payload, slice unused) but prescribed "add slice
after flatten". That would not have fixed the reported failure: flatten is the step that
fails, and a later step cannot shrink an earlier step's journal entry. A slice there would have
bounded map, enrich, filter and the sink while the run still died at s3-flatten-1 on Sydney.
The fix has to move the bulk before the first non-forEach step — i.e. into the connector's own responseMap, which runs inside the fan-out where the payload is only counted once.
What it does not claim
I did not read the live instance's pipeline definition. No read-only MCP tool returns one. Two
facts say the live copy differs from workers/api/src/lib/pipelines/lead-finder.json: the completed
runs report "8 step(s)" where the reference has 9, and the 06:49 run failed at s4-map-1after s3-flatten succeeded, which is only possible if its map output is larger than its flatten
output — i.e. that map does not narrow with keep. The mechanism above is asserted for the
reference definition, which I ran; the live one is inferred from step indices and counts.
I have not measured a real Sydney response. The 500-place fixture uses the component array shape
Google documents and returns, not a captured body.
Also true, and separate: since #381 the live pipeline cannot start at all
lead-finder.json names http_request, which is connector-provided (connectors/registry.ts:99, HTTP_TOOLS), so it is gated by the declared-tools rule. declaredToolsFor maps an agent that
resolves but declares nothing to an empty array — a real assertion, not an absent one
(pipeline-tool-policy.ts) — and undeclaredToolRefusal then refuses (tool-refusal.ts:55). startPipelineRun applies it before the run opens.
list_instance_tools(933ebec5…, allowed_only: true) returns seven tools today — start_work, check_work, get_behaviour, set_behaviour, get_stats, set_stats_card, run_pipeline — and no http_request. So lead_finder is refused at kick regardless of city.
This is live data and I have not touched it. The owner must add http_request to the small-business-website-lead-finder agent's capabilities.tools:
Everything else the pipeline dispatches — geocode, fan_out, flatten, slice, map, enrich, filter, dedupe_upsert, http_reachable — is step-library and connector-less, so it is exempt.
Note that a declared tools list is authoritative and replaces the surface default, so it must also
name any other connector tool the agent uses.
Fix
In workers/api/src/lib/pipelines/lead-finder.json:
Project the four geo strings in the responseMap (country:addressComponents[types~=country].longText,
and the three siblings) instead of passing the raw arrays down the chain; drop the now-redundant extract from the map step. The getPath type-predicate from map/responseMap: type-predicate selection for Google addressComponents (child of #94) #116 already works inside a responseMap projection, so this is a config change, not a new primitive.
Add slice after flatten and before enrich — bounding the payload is only half of it; the enrich step is one HTTP probe per place, so a cap after it would leave the spend unbounded.
Give the cap a home and a default. A slice whose limit is an unsupplied $param resolves
to undefined, and an absent limit means keep everything — so the one step that bounds the run
silently stops bounding it whenever the wiring omits the param.
Say what the cap left. A run that quietly keeps the first N closes completed with tidy
counts over a city it only partly looked at, and seen counts what was examined — so a capped
Sydney and a complete Hobart are indistinguishable without a line that says so.
Source: the #64 catalog audit, finding 2. Verified against main at 7332a5e.
Seeding the definition onto the agent row (#64 finding 1) is deliberately not part of this — it is a
change to live catalog data, and the crash is fixable in the reference definition on its own.
lead_findercannot sweep a capital city: the rawaddressComponentsarrays make the flatten step's journal entry exceed 1MiBThree
pipeline-runerrors, all on instance933ebec5…, read back today vialist_errors:list_pipeline_runsnames the cities: the two failures with acityparam are Sydney, NSW andBrisbane, QLD. Every completed sweep is Hobart (116 places) or Launceston (83), both at
radius: 900. This is the one catalog card promising a general capability — "Search by city andbusiness type" — that fails on the first city a visitor would try.
Root cause, confirmed against the code
Not "the payload grows with the city" in general. It is one field, and one property of the runner.
A
StepResultcarries its payload twice. Every step handler returns pretty-printed JSON(
steps.ts:510—ok(JSON.stringify({ items: out, count: out.length }, null, 2))), andexecutePipelineStepALSO stores the parsed value (pipeline.ts:436,output: parseOutput(r.content)).capStepOutputmeasuresJSON.stringify(result)(pipeline.ts:88), so a step's journal entry isroughly 3× the records it holds. A
forEachstep is the exception: its content is"25 item(s)"(
pipeline.ts:431), so the fan-out carries ~1×. That is exactly whyflatten— which holds the samedata as the fan-out that fed it — is the first step to fail, and why the production error names
s3-flatten-1rather than thehttp_requestabove it.The size is structural, not unlucky.
extentKm: 2 / stepKm: 1→gridCellsyields 25 cells(
steps.ts:295), each requestingmaxResultCount: 20, so a dense city returns 500 places andcannot return more. Hobart's 116 fit; 500 do not.
addressComponentsis 73% of the bytes. TheresponseMapprojected Google's raw typed-componentarray onto every record so the
mapstep couldextractcity/state/suburb/country from it. Measuredon a 500-place fixture with the component arrays Google actually returns: the flatten step's
StepResultis 1681KB with them and 459KB without.capStepOutput's bound is 900KB.Where the audit's prescription was wrong
#64 diagnosed the mechanism correctly (unbounded payload,
sliceunused) but prescribed "addsliceafter
flatten". That would not have fixed the reported failure:flattenis the step thatfails, and a later step cannot shrink an earlier step's journal entry. A
slicethere would havebounded
map,enrich,filterand the sink while the run still died ats3-flatten-1on Sydney.The fix has to move the bulk before the first non-
forEachstep — i.e. into the connector's ownresponseMap, which runs inside the fan-out where the payload is only counted once.What it does not claim
facts say the live copy differs from
workers/api/src/lib/pipelines/lead-finder.json: the completedruns report
"8 step(s)"where the reference has 9, and the 06:49 run failed ats4-map-1afters3-flattensucceeded, which is only possible if itsmapoutput is larger than itsflattenoutput — i.e. that
mapdoes not narrow withkeep. The mechanism above is asserted for thereference definition, which I ran; the live one is inferred from step indices and counts.
Google documents and returns, not a captured body.
Also true, and separate: since #381 the live pipeline cannot start at all
lead-finder.jsonnameshttp_request, which is connector-provided (connectors/registry.ts:99,HTTP_TOOLS), so it is gated by the declared-tools rule.declaredToolsFormaps an agent thatresolves but declares nothing to an empty array — a real assertion, not an absent one
(
pipeline-tool-policy.ts) — andundeclaredToolRefusalthen refuses (tool-refusal.ts:55).startPipelineRunapplies it before the run opens.list_instance_tools(933ebec5…, allowed_only: true)returns seven tools today —start_work,check_work,get_behaviour,set_behaviour,get_stats,set_stats_card,run_pipeline— and nohttp_request. Solead_finderis refused at kick regardless of city.This is live data and I have not touched it. The owner must add
http_requestto thesmall-business-website-lead-finderagent'scapabilities.tools:Everything else the pipeline dispatches —
geocode,fan_out,flatten,slice,map,enrich,filter,dedupe_upsert,http_reachable— is step-library and connector-less, so it is exempt.Note that a declared
toolslist is authoritative and replaces the surface default, so it must alsoname any other connector tool the agent uses.
Fix
In
workers/api/src/lib/pipelines/lead-finder.json:responseMap(country:addressComponents[types~=country].longText,and the three siblings) instead of passing the raw arrays down the chain; drop the now-redundant
extractfrom themapstep. ThegetPathtype-predicate from map/responseMap: type-predicate selection for Google addressComponents (child of #94) #116 already works inside aresponseMapprojection, so this is a config change, not a new primitive.sliceafterflattenand beforeenrich— bounding the payload is only half of it; theenrichstep is one HTTP probe per place, so a cap after it would leave the spend unbounded.slicewhoselimitis an unsupplied$paramresolvesto
undefined, and an absent limit means keep everything — so the one step that bounds the runsilently stops bounding it whenever the wiring omits the param.
completedwith tidycounts over a city it only partly looked at, and
seencounts what was examined — so a cappedSydney and a complete Hobart are indistinguishable without a line that says so.
Source: the #64 catalog audit, finding 2. Verified against
mainat7332a5e.Seeding the definition onto the agent row (#64 finding 1) is deliberately not part of this — it is a
change to live catalog data, and the crash is fixable in the reference definition on its own.