Skip to content

bug: a workflow run can get stuck at Running forever if the terminal DB write fails #7093

Description

@serina-mcfall

Summary

A workflow run's database record can be left permanently at RunStatus::Running if the terminal status write fails, with no reconciliation path to detect or recover it. Separately, the record is not updated per-step, so any read of a running run shows a stale snapshot by construction.

Evidence

Stale snapshot during execution

  • crates/buzz-workflow/src/executor.rs (execute_steps) never writes to the database as it moves through steps. The run row is only written at creation, at the Pending → Running transition, and once more at finalize_run. A read of the run record mid-execution therefore always reflects an earlier point in the run, not current progress.

Fire-and-forget terminal write

  • crates/buzz-workflow/src/lib.rs (finalize_run) performs the terminal (Completed/Failed) database write as fire-and-forget: on write failure it only logs, it does not retry. This is asymmetric with the Pending → Running write, which is propagated. If the terminal write fails, the run's persisted status stays Running forever — there is no background reconciliation job that revisits runs stuck in that state.

Impact

An operator or the buzz-cli workflows surface has no reliable way to tell a genuinely long-running workflow apart from one whose completion was silently lost to a transient DB write failure. Both look identical: RunStatus::Running indefinitely.

How this was found

Surfaced while authoring corpus documentation for launchpad-26/buzz (Feature #613, issue #841) — verified by reading execute_steps, finalize_run, and the run-record write call sites directly.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions