Skip to content

feat(automation): durable run history with failure reasons (run observability) - #2581

Merged
os-zhuang merged 1 commit into
mainfrom
feat/automation-run-history
Jul 4, 2026
Merged

feat(automation): durable run history with failure reasons (run observability)#2581
os-zhuang merged 1 commit into
mainfrom
feat/automation-run-history

Conversation

@os-zhuang

Copy link
Copy Markdown
Contributor

Why

Automation runs were observable only in memory. The engine kept the last N runs in a ring buffer, so after a process restart (or once the buffer evicted the entry) you could not answer "did this flow run, and why did it fail?" — and a failed run surfaced no reason at all. This was the single biggest silent-trust gap for anyone authoring automations: a flow could stop firing or start failing with nothing durable to inspect.

This is the #1 UX recommendation from the online-automation dogfood: run observability — durable, queryable runs with status + failure reason.

What

sys_automation_run — previously the ADR-0019 store for live suspended runs only — becomes a durable run-history table.

  • enginerecordLog mirrors every terminal run (completed / failed) to the store via recordTerminal (fire-and-forget). listRuns() merges durable history with the in-memory buffer (in-memory wins on id, newest-first) so the Studio "Runs" surface shows runs that predate the current process.
  • storerecordTerminal / listHistory on both SuspendedRunStore impls. Terminal history rows use a run_-prefixed id, disjoint from live suspended runs (raw runId, status: 'paused', deleted on completion), so the suspend save/load/delete/list path is untouched and resume sweeps (list() filters status: 'paused') never see history rows.
  • schemafinished_at / duration_ms / error (all optional) + the status enum gains running / completed / failed alongside paused.

Safe & additive

  • Distinct id namespace ⇒ zero collision with the ADR-0019 suspend/resume lifecycle.
  • Persisting is best-effort + fire-and-forget — a history write/read failure is logged and swallowed, never breaking the run that produced it.
  • New fields optional; new store methods optional (in-memory/test defaults still work).

Verification

Unit — new run-history.test.ts: completed persistence, failed persistence with error, read-across-restart (fresh engine over the same store), best-effort isolation (a throwing store never breaks the run). Full suite green (231 passed), DTS typecheck clean.

Runtime, end-to-end on a clean showcase instance (the standard that caught the earlier no-op #2560 fix — a fake-service unit test alone can mask a runtime-broken change):

  • schedule-triggered showcase_scheduled_digest + 7 task-completion flows → durable completed rows with finished_at;
  • showcase_resilient_sync → a failed row carrying its try_catch failure reason;
  • a live showcase_task_follow_uppaused suspended run coexisted without collision;
  • after a full process restart, the failed row — reason intact — was still queryable via /api/v1/data/sys_automation_run.

🤖 Generated with Claude Code

…vability)
Mirror every terminal automation run (completed / failed) to the
sys_automation_run table so "did it run, and why did it fail?" survives a
process restart and the in-memory ring-buffer eviction — previously runs were
observable only in memory and failures surfaced no reason.
- engine: `recordLog` fire-and-forget `recordTerminal` on terminal runs;
`listRuns` merges durable history with the in-memory buffer (freshest wins).
- store: `recordTerminal` / `listHistory` on both SuspendedRunStore impls,
keyed by a `run_`-prefixed id disjoint from live suspended runs so the
ADR-0019 suspend/resume path is untouched and resume sweeps skip history.
- schema: sys_automation_run gains `finished_at` / `duration_ms` / `error`
(all optional) and a `completed` / `failed` / `running` status alongside
`paused`; becomes a durable run-history table, not suspend-only.
- best-effort throughout: a history write/read failure is logged and swallowed,
never breaking the run that produced it.
Verified end-to-end on a clean showcase: completed + failed runs persisted
durable rows (failure reason intact), a live paused run coexisted without
collision, and the failed row survived a full process restart. New
run-history.test.ts covers persistence, read-across-restart, best-effort isolation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@vercel

vercelBot commented Jul 4, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

ProjectDeploymentActionsUpdated (UTC)
specReadyReadyPreview, CommentJul 4, 2026 1:56pm

Request Review

@github-actionsgithub-actionsBot added documentation Improvements or additions to documentation tests tooling size/m labels Jul 4, 2026
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 1 package(s): packages/services.

5 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:

  • content/docs/guides/packages.mdx(via packages/services)
  • content/docs/guides/runtime-services/audit-service.mdx(via packages/services)
  • content/docs/guides/runtime-services/index.mdx(via packages/services)
  • content/docs/guides/runtime-services/settings-service.mdx(via packages/services)
  • content/docs/protocol/objectos/i18n-standard.mdx(via packages/services)

Advisory only. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs origin/main → pass the list as args.docs.

@os-zhuang
os-zhuang merged commit ffafb30 into mainJul 4, 2026
16 checks passed
@os-zhuang
os-zhuang deleted the feat/automation-run-history branch July 4, 2026 14:01
This was referenced Jul 5, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentationImprovements or additions to documentationsize/mteststooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@os-zhuang