Skip to content

feat(service-automation,qa): surface a failed try-region's steps so a caught failure leaves a trace (#7546)#7699

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
os-help merged 1 commit into from
Aug 11, 2026
Merged

feat(service-automation,qa): surface a failed try-region's steps so a caught failure leaves a trace (#7546) #7699

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions .changeset/try-catch-failed-attempt-steps.md
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
---
"@objectstack/service-automation": minor
---

feat(service-automation): a caught `try_catch` failure now records what failed, how many attempts ran, and which node threw (#7546)

A caught failure used to leave **no forensic trace**. The whole run log was:

```
[ start, guarded_push (try_catch, success), record_failure (catch) ]
```

Nothing carried `regionKind: 'try'`. Nothing carried `status: 'failure'`. The
container's own step read `success`. From the log alone a caught failure was
indistinguishable from a clean run that happened to also touch the catch path —
the only evidence a failure had occurred was the catch region's side effects,
which is nothing at all when the catch is a bare notification, and worse than
nothing when the catch's own write is the thing you are trying to explain. An
operator (or an agent) reading such a log was not merely under-informed: the
most natural reading was that the try region had never run, which points at
"fix" work on a region that was behaving exactly as designed.

The steps were never missing for a structural reason. A failing node pushes its
own `failure` step into the region's step array *before* it throws, and the
`childSteps` splice that folds region steps into the parent log has existed
since #1479 and works for every region kind that succeeds. The failed attempt's
array was simply dropped on the floor as the region unwound.

**What changes.** `runRegion()` now hands a failed region's partial steps to the
caller through an opt-in sink before the throw propagates, tagged exactly as a
successful region's are, and `try_catch` accumulates every failed attempt across
the retry ladder and folds them into `childSteps` **ahead of** the steps of
whichever region finally succeeded. So a caught failure's log now contains, in
execution order, each failed try attempt (the throwing node's `failure` step
with its error, plus whatever the attempt got through before it) followed by the
catch handler's steps. The same applies to a ladder that recovers on a retry:
the attempts it burned are recorded rather than erased.

Where a retry policy is declared, those steps also carry `retryAttempt` — the
zero-based attempt index — so the number of attempts is a **count** in the log
rather than something inferred from elapsed wall time. `retryAttempt` is not new
vocabulary: it has been declared on the spec's `ExecutionStepLogSchema` since
that schema was written, with exactly this meaning, and had no producer anywhere
in the engine until now.

**What does not change.** The retry and throw semantics of `try_catch` are
untouched: the same number of attempts, the same fall-through to the catch
region, the same node-level outcome. A container that recovers still reports
`success` — giving it a distinct status such as `recovered` was considered as
part of this decision and deliberately not adopted, because the container's
contract is "the error was handled" and the forensic detail belongs in the step
log underneath it, which is what this change delivers.

**Log volume.** A try region that retries N times now emits up to N times its
body's steps, and a retry ladder nested in a loop multiplies. Durable run
history is unaffected in shape: `compactStepLogForHistory` already caps
persisted steps and already prioritises failures and their container chains, so
the extra records land inside the existing budget rather than growing it.

Run summaries need no special case and get more accurate. A try node that failed
twice before succeeding folds to `runs: 3, failures: 2` — all three numbers
true, and the same "worst outcome wins, `runs`/`failures` carry the nuance" rule
a loop body has always folded under. Records written by an attempt that then
threw now reach the run's `selected`/`acted` totals instead of vanishing, which
is what those counters are for.
48 changes: 33 additions & 15 deletions docs/qa/platform-checklist/areas/automation.json
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -517,10 +517,10 @@
},
{
"id": "automation.flow-error-handling",
"title": "A failing node inside try_catch is handled (catch region, $error binding); outside it fails the run loudly",
"title": "A failing node inside try_catch is handled (catch region, $error binding) and the failed attempt is still recorded; outside it fails the run loudly",
"since": "v16",
"status": "active",
"revision": 1,
"revision": 2,
"priority": "P1",
"surface": "mixed",
"personas": [
Expand All@@ -536,7 +536,7 @@
"steps": [
"boot showcase isolated (dogfood §0); sign in as the dev admin",
"handled case: POST a showcase_task and PATCH status→'done' — showcase_resilient_sync runs, its try-region http node fails against the unroutable host, retries, then the catch region runs",
"GET /api/v1/automation/showcase_resilient_sync/runs/:runId — record overall status, the try step's status/error/regionKind, the catch step's status/regionKind, and the run duration",
"GET /api/v1/automation/showcase_resilient_sync/runs/:runId — record the overall status, the try_catch container step's own status, EVERY step carrying regionKind='try' (nodeId, status, error, retryAttempt) in log order, the catch step's status/regionKind, and the run duration",
"re-read the task over /api/v1/data/showcase_task/:id — sync_status and sync_error",
"unhandled case: POST /api/v1/automation a minimal autolaunched probe flow (start → http POST to the same unroutable URL, NOT wrapped in try_catch → end) in the scratch package; trigger it via POST /api/v1/automation/<probe>/trigger",
"GET the probe run: overall status and the run-level error",
Expand All@@ -545,22 +545,28 @@
],
"acceptance": [
{
"clause": "handled: the run completes (status=completed) — the failure was absorbed by the container, not the run",
"clause": "handled: the run completes (status=completed) — the failure was absorbed by the container, not the run — and the try_catch container's OWN step reads status='success'",
"oracle": "api",
"verify": "run detail: status=completed; the http step inside the try region records status=failure with regionKind='try'; the catch-region update_record step records status=success with regionKind='catch'",
"verify": "run detail: status=completed; the step whose nodeType is 'try_catch' carries status='success'. ⛔ That 'success' is BY DESIGN and is never on its own a finding: #7546 weighed giving a recovered container a distinct status (a 'recovered' vocabulary, the card's Option C) and did NOT adopt it — the container's contract is 'the error was handled', and the forensic detail lives in the step log underneath it, which is what the next clause asserts",
"evidence": "run-detail read"
},
{
"clause": "the caught error binds to $error and lands in data: the task carries sync_status='failed' and a non-empty sync_error message",
"clause": "the failed try attempt is RECORDED, not discarded: every attempt of the try region that failed contributes its steps to the same flat run log, tagged parentNodeId=<the try_catch node id> and regionKind='try', with the throwing node's own step carrying status='failure' and a populated error {code,message} — and all of them ordered BEFORE the catch region's steps. An operator reading the run log alone can therefore answer all three of WHAT failed (the error on the failing step), WHICH node threw (that step's nodeId), and HOW MANY attempts ran (the next clause). Maintainer ruling on #7546, 2026-08-11: 'surface the failed try-region's steps … so an operator can see what failed, how many attempts ran, and which node threw', implemented over the existing #1479 childSteps splice with the retry/throw semantics of try_catch unchanged",
"oracle": "api",
"verify": "task read after the run: sync_status='failed', sync_error interpolated from {$error.message}",
"evidence": "task read"
"verify": "in the run detail, the steps carrying regionKind='try' are present and non-empty; the one for the unroutable http node has status='failure' with error.message naming the connection failure; its index in the step array is lower than the catch step's. NOTE the version boundary: this contract begins with #7546 — a run captured against a framework build that predates it will legitimately show NO regionKind='try' step on a caught failure, which is the old behaviour being replaced, not a fresh defect. Record the framework revision with the result",
"evidence": "ordered step-log excerpt showing each regionKind='try' failure step and the following regionKind='catch' step"
},
{
"clause": "the retry policy actually ran before the catch: the failure is not instantaneous",
"clause": "the retry ladder is countable from the log, not merely inferable from elapsed time: with a retry policy declared, each try attempt's steps carry a zero-based retryAttempt, so the number of distinct retryAttempt values equals maxRetries+1 when every attempt failed",
"oracle": "api",
"verify": "the run/step duration is at least the first backoff delay (>= ~1s per backoffMs:1000), evidencing at least one retry before the catch — note this oracle's weakness (duration, not a retry counter) in the evidence",
"evidence": "durationMs from the run detail"
"verify": "collect retryAttempt across the regionKind='try' steps: for showcase_resilient_sync (retry.maxRetries=3) expect the four values 0,1,2,3, each with status='failure'. Corroborate — do not substitute — with the duration oracle: the run/step duration is at least the first backoff delay (>= ~1s per backoffMs:1000). The duration alone was this item's only retry evidence before #7546 and was explicitly recorded as a weak oracle; the counter is now the primary one and the duration is the cross-check",
"evidence": "retryAttempt values from the run detail + durationMs"
},
{
"clause": "the caught error binds to $error and lands in data: the task carries sync_status='failed' and a non-empty sync_error message",
"oracle": "api",
"verify": "task read after the run: sync_status='failed', sync_error interpolated from {$error.message}",
"evidence": "task read"
},
{
"clause": "unhandled: the probe run terminates status=failed with the run-level error populated",
Expand All@@ -571,7 +577,7 @@
{
"clause": "both failures surface in the designer Runs panel: the failed step marked with its error message, catch-body steps nested under the container",
"oracle": "screenshot",
"verify": "Runs panel screenshots for both runs — the panel renders run/step errors (string run-level, {code,message} step-level) and nests region steps",
"verify": "Runs panel screenshots for both runs — the panel renders run/step errors (string run-level, {code,message} step-level) and nests region steps; the handled run's panel now also shows the failed try attempts nested under the container alongside the catch body",
"evidence": "two Runs panel screenshots"
},
{
Expand All@@ -583,7 +589,9 @@
],
"negative": [
"an unhandled node failure that leaves its run status=completed, or leaves the run-level error empty, is a FAIL — a dead outbound call reporting success is the inert-automation failure shape (#1887)",
"a catch region that runs when the try did NOT fail is a FAIL of the container semantics — check the catch steps are absent from a successful run"
"a catch region that runs when the try did NOT fail is a FAIL of the container semantics — check the catch steps are absent from a successful run",
"a caught failure that leaves NO forensic trace is the FAIL this item's core clause guards, and it is the exact shape #7546 fixed: a run log of only [start, <container>(success), <catch step>] — nothing carrying regionKind='try', nothing carrying status='failure' — makes a caught failure indistinguishable from a clean run that merely touched the catch path, so the only evidence of failure is the catch's own side effects. Silence here is a regression of #7546, not a cosmetic gap",
"⛔ the try_catch container step reading status='success' after it caught a failure is NOT a finding — #7546 ruled on exactly that and kept it; re-filing it re-litigates a closed decision"
],
"traps": [
"wrong-panel",
Expand All@@ -592,16 +600,26 @@
"source": [
"examples/app-showcase/src/automation/flows/index.ts (ResilientSyncFlow, ADR-0031 try/catch/retry; canonical retry keys #4661)",
"packages/spec/src/automation/control-flow.zod.ts (TryCatchConfigSchema)",
"packages/spec/src/automation/execution.zod.ts (ExecutionStatus 'failed'; step status/error; regionKind)",
"packages/spec/src/automation/execution.zod.ts (ExecutionStatus 'failed'; step status/error; regionKind; retryAttempt — declared since the schema was written, given its first producer by #7546)",
"packages/services/service-automation/src/engine.ts (runRegion's partialSteps sink — the failed attempt's tagged steps handed to the caller before the throw propagates; StepLogEntry.retryAttempt)",
"packages/services/service-automation/src/builtin/try-catch-node.ts (failed attempts accumulated across the retry ladder and folded into childSteps ahead of the surviving region's steps)",
"packages/services/service-automation/src/builtin/try-catch-failed-attempt-steps.test.ts (the #7546 unit pins, including 'a recovered container still reports success')",
"objectui packages/app-shell/src/views/metadata-admin/previews/FlowRunsPanel.tsx (run-level string error vs step-level {code,message})",
"packages/runtime/src/route-ledger.ts (POST /automation — automation.create)"
"packages/runtime/src/route-ledger.ts (POST /automation — automation.create)",
"#7546 (maintainer ruling 2026-08-11 — surface the failed try-region's steps; Option C's 'recovered' container status NOT adopted)"
],
"history": [
{
"revision": 1,
"date": "2026-08-07",
"change": "initial — splits handled (try_catch) vs unhandled failure into one contrast item with API + panel + log oracles",
"ref": "claude/platform-test-checklist-ocwugl"
},
{
"revision": 2,
"date": "2026-08-11",
"change": "recorded the #7546 ruling. The handled-case clause asserted that the try-region's failing step appears with regionKind='try' — a contract the engine had never promised, so QA run #7516 read a by-design silence as a FAIL. Rather than weaken the clause to match the engine, the ruling implemented it: failed try attempts are now surfaced. The clause set is re-cut accordingly — container 'success' pinned as by-design (Option C's 'recovered' status was considered and rejected), the failed-attempt steps promoted to their own clause with the ordering requirement, and the retry evidence upgraded from the weak duration oracle to a retryAttempt count with duration as cross-check. The negative now names post-#7546 silence as the regression shape and forbids re-filing the container's 'success'",
"ref": "#7546 (ruling 2026-08-11; QA run #7516; #1479 splice plumbing)"
}
]
},
Expand Down
Loading
Loading