Skip to content

[bug] lead_finder cannot sweep a capital city: raw addressComponents push the flatten step's journal entry over the 1MiB Workflow limit #394

Description

@serge-ivo

lead_finder cannot sweep a capital city: the raw addressComponents arrays make the flatten step's journal entry exceed 1MiB

Three pipeline-run errors, all on instance 933ebec5…, read back today via list_errors:

2026-08-03 08:08 pipeline "lead_finder" crashed: WorkflowInternalError: Step s3-flatten-1 output is too large. Maximum allowed size is 1MiB.
2026-08-03 06:49 pipeline "lead_finder" crashed: WorkflowInternalError: Step s4-map-1 output is too large. Maximum allowed size is 1MiB.
2026-08-03 06:32 pipeline "lead-finder" crashed: WorkflowInternalError: Step s3-map-1 output is too large. Maximum allowed size is 1MiB.

list_pipeline_runs names the cities: the two failures with a city param are Sydney, NSW and
Brisbane, QLD. Every completed sweep is Hobart (116 places) or Launceston (83), both at
radius: 900. This is the one catalog card promising a general capability — "Search by city and
business type" — that fails on the first city a visitor would try.

Root cause, confirmed against the code

Not "the payload grows with the city" in general. It is one field, and one property of the runner.

A StepResult carries its payload twice. Every step handler returns pretty-printed JSON
(steps.ts:510ok(JSON.stringify({ items: out, count: out.length }, null, 2))), and
executePipelineStep ALSO stores the parsed value (pipeline.ts:436, output: parseOutput(r.content)).
capStepOutput measures JSON.stringify(result) (pipeline.ts:88), so a step's journal entry is
roughly 3× the records it holds. A forEach step is the exception: its content is "25 item(s)"
(pipeline.ts:431), so the fan-out carries ~1×. That is exactly why flatten — which holds the same
data as the fan-out that fed it — is the first step to fail, and why the production error names
s3-flatten-1 rather than the http_request above it.

The size is structural, not unlucky.extentKm: 2 / stepKm: 1gridCells yields 25 cells
(steps.ts:295), each requesting maxResultCount: 20, so a dense city returns 500 places and
cannot return more. Hobart's 116 fit; 500 do not.

addressComponents is 73% of the bytes. The responseMap projected Google's raw typed-component
array onto every record so the map step could extract city/state/suburb/country from it. Measured
on a 500-place fixture with the component arrays Google actually returns: the flatten step's
StepResult is 1681KB with them and 459KB without. capStepOutput's bound is 900KB.

Where the audit's prescription was wrong

#64 diagnosed the mechanism correctly (unbounded payload, slice unused) but prescribed "add slice
after flatten"
. That would not have fixed the reported failure: flatten is the step that
fails
, and a later step cannot shrink an earlier step's journal entry. A slice there would have
bounded map, enrich, filter and the sink while the run still died at s3-flatten-1 on Sydney.

The fix has to move the bulk before the first non-forEach step — i.e. into the connector's own
responseMap, which runs inside the fan-out where the payload is only counted once.

What it does not claim

  • I did not read the live instance's pipeline definition. No read-only MCP tool returns one. Two
    facts say the live copy differs from workers/api/src/lib/pipelines/lead-finder.json: the completed
    runs report "8 step(s)" where the reference has 9, and the 06:49 run failed at s4-map-1after
    s3-flatten succeeded, which is only possible if its map output is larger than its flatten
    output — i.e. that map does not narrow with keep. The mechanism above is asserted for the
    reference definition, which I ran; the live one is inferred from step indices and counts.
  • I have not measured a real Sydney response. The 500-place fixture uses the component array shape
    Google documents and returns, not a captured body.

Also true, and separate: since #381 the live pipeline cannot start at all

lead-finder.json names http_request, which is connector-provided (connectors/registry.ts:99,
HTTP_TOOLS), so it is gated by the declared-tools rule. declaredToolsFor maps an agent that
resolves but declares nothing to an empty array — a real assertion, not an absent one
(pipeline-tool-policy.ts) — and undeclaredToolRefusal then refuses (tool-refusal.ts:55).
startPipelineRun applies it before the run opens.

list_instance_tools(933ebec5…, allowed_only: true) returns seven tools today —
start_work, check_work, get_behaviour, set_behaviour, get_stats, set_stats_card,
run_pipeline — and no http_request. So lead_finder is refused at kick regardless of city.

This is live data and I have not touched it. The owner must add http_request to the
small-business-website-lead-finder agent's capabilities.tools:

PUT /v1/agents/<id> { "config": { … , "capabilities": { … , "tools": ["http_request"] } } }

Everything else the pipeline dispatches — geocode, fan_out, flatten, slice, map, enrich,
filter, dedupe_upsert, http_reachable — is step-library and connector-less, so it is exempt.
Note that a declared tools list is authoritative and replaces the surface default, so it must also
name any other connector tool the agent uses.

Fix

In workers/api/src/lib/pipelines/lead-finder.json:

  1. Project the four geo strings in the responseMap (country:addressComponents[types~=country].longText,
    and the three siblings) instead of passing the raw arrays down the chain; drop the now-redundant
    extract from the map step. The getPath type-predicate from map/responseMap: type-predicate selection for Google addressComponents (child of #94) #116 already works inside a
    responseMap projection, so this is a config change, not a new primitive.
  2. Add slice after flatten and before enrich — bounding the payload is only half of it; the
    enrich step is one HTTP probe per place, so a cap after it would leave the spend unbounded.
  3. Give the cap a home and a default. A slice whose limit is an unsupplied $param resolves
    to undefined, and an absent limit means keep everything — so the one step that bounds the run
    silently stops bounding it whenever the wiring omits the param.
  4. Say what the cap left. A run that quietly keeps the first N closes completed with tidy
    counts over a city it only partly looked at, and seen counts what was examined — so a capped
    Sydney and a complete Hobart are indistinguishable without a line that says so.

Source: the #64 catalog audit, finding 2. Verified against main at 7332a5e.
Seeding the definition onto the agent row (#64 finding 1) is deliberately not part of this — it is a
change to live catalog data, and the crash is fixable in the reference definition on its own.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0: wrong todayProduces a wrong result, loses work, or spends money in production right nowbackendBackend / Worker / API workbugSomething isn't workingpipelinesPipelines, steps, triggers and the event pump — what CALLS a connector

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions