improvement(logs): move per-block progress markers to Redis to cut write amplification by waleedlatif1 · Pull Request #5248 · simstudioai/sim

waleedlatif1 · 2026-06-27T22:35:40Z

Summary

Per-block lastStartedBlock/lastCompletedBlock progress markers were persisted with a jsonb_set UPDATE on workflow_execution_logs on every block start and complete (~2N row UPDATEs per run) — the heaviest write query on the table by volume.
These are live progress breadcrumbs with no DB-polling consumer (live progress streams from the executor over WebSocket); their only durable value is a breadcrumb folded into the final record.
Behind the redis-progress-markers flag, markers now live in Redis during the run and are folded into the single terminal UPDATE at completion — dropping per-run row UPDATEs from ~2N+1 to 1.

How it works

New progress-markers.ts: one Redis HASH per execution (execution:progress:{id}), atomic Lua monotonic-guard writes that preserve the existing <= ordering, a reservation-aligned TTL backstop, and graceful no-op when Redis is unavailable.
Deterministic GC: markers are cleared at every terminal/pause boundary (complete/error/cancel/pause via completeWorkflowExecution, plus markExecutionAsFailed); the TTL only covers crashed runs. Paused executions hold zero Redis keys.
The write path is resolved once per logging session, so a run never mixes Redis and SQL paths.
Completion folds markers into the record (Redis wins, falls back to markers already on the row); in-flight detail reads merge live markers for parity.
Extracted getExecutionReservationTtlMs() so the marker TTL and the admission-slot TTL share one source of truth.

Rollout

Flag-gated and off by default — shipping is a no-op (legacy jsonb_set path unchanged). Enable to cut the write amplification; flip off for instant rollback.
Falls back to the legacy SQL path when the flag is off, Redis is unavailable, or flag resolution fails — safe for self-hosted deployments without Redis.

Type of Change

Improvement (performance / DB write reduction)

Testing

New unit tests for the marker store (monotonic guard, get/clear, malformed-JSON safety, graceful no-op without Redis).
New logging-session tests: flag-on routes to Redis (not the row), flag-off uses jsonb_set, flag-resolution failure falls back to SQL, and markExecutionAsFailed clears markers.
Existing logger/execution-core/usage-reservation suites pass unchanged (carry-forward fallback preserved). tsc clean, biome clean.

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

…ite amplification Per-block lastStartedBlock/lastCompletedBlock markers were persisted via a jsonb_set UPDATE on workflow_execution_logs on every block start and complete (~2N UPDATEs per run) — the heaviest write query in the DB. These are live progress breadcrumbs with no DB-polling consumer (live progress comes from the executor over WebSocket); their only durable value is a breadcrumb folded into the final record. Behind the redis-progress-markers flag, markers now live in Redis during the run and are folded into the single terminal UPDATE at completion, dropping per-run row UPDATEs from ~2N+1 to 1. - New progress-markers module: HASH execution:progress:{id}, atomic Lua monotonic-guard writes preserving the existing <= ordering, reservation-aligned TTL backstop, graceful no-op when Redis is unavailable - Deterministic GC: cleared at every terminal/pause boundary; TTL covers crashes - Flag resolved once per logging session so a run never mixes write paths - Fold markers into the completion record (Redis wins, falls back to row markers) - Merge live markers for in-flight detail reads - Extract shared getExecutionReservationTtlMs so marker and admission-slot TTLs share one source of truth

vercel · 2026-06-27T22:35:47Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Jun 27, 2026 11:52pm

cursor · 2026-06-27T22:35:50Z

PR Summary

Medium Risk
Touches hot execution logging and Redis availability paths; flag-off behavior is unchanged, but enabled runs depend on correct marker fold/clear at every terminal boundary to avoid stale or missing breadcrumbs.

Overview
Adds an opt-in redis-progress-markers flag (off by default) that stores live lastStartedBlock / lastCompletedBlock breadcrumbs in Redis during a run instead of issuing jsonb_set row UPDATEs on every block start/complete, folding them into the single terminal workflow_execution_logs write at completion.

New progress-markers module uses a per-execution Redis HASH with Lua monotonic writes, validated reads, and TTL aligned via shared getExecutionReservationTtlMs() (also wired into usage admission reservations). LoggingSession resolves the write path once at start, falls back to SQL when Redis or flag resolution fails, and passes readProgressMarkers into completion only when Redis mode is active. completeWorkflowExecution merges Redis vs row markers with timestamp “latest wins”; markExecutionAsFailed folds Redis markers before clear and skips clear when reads return null. fetchLogDetail merges live Redis markers for running/pending executions.

^{Reviewed by Cursor Bugbot for commit 6669170. Configure here.}

greptile-apps · 2026-06-27T22:39:08Z

Greptile Summary

This PR moves live execution progress markers from per-block database updates into Redis. The main changes are:

Adds Redis-backed progress marker storage with monotonic timestamp guards.
Folds live markers into the terminal execution log update.
Falls back to SQL marker updates when Redis writes are unavailable.
Merges live markers into in-flight log detail reads.
Shares the execution reservation TTL with progress marker expiry.

Confidence Score: 5/5

This looks safe to merge.

No blocking issues found in the changed code.
The Redis write path now falls back to SQL when Redis writes fail.
Failed-run handling now folds readable live markers before clearing them.

Important Files Changed

Filename	Overview
apps/sim/lib/logs/execution/progress-markers.ts	Adds Redis hash helpers for writing, reading, validating, and clearing live progress markers.
apps/sim/lib/logs/execution/logging-session.ts	Routes per-block marker writes through Redis when enabled, with SQL fallback and failed-run marker folding.
apps/sim/lib/logs/execution/logger.ts	Merges Redis progress markers with row data during terminal execution log updates.
apps/sim/lib/logs/fetch-log-detail.ts	Adds live Redis marker merging for running and pending execution detail reads.

_{Reviews (6): Last reviewed commit: "perf(logs): skip completion Redis read/c..." | Re-trigger Greptile}

…n force-fail, validate marker shape Addresses review feedback on the redis-progress-markers PR: - persistLast* now falls back to the jsonb_set UPDATE when Redis is unavailable or the write fails (setLast* returns whether it persisted), so a marker is never dropped when the flag is on without a healthy Redis. - markExecutionAsFailed folds live Redis markers into execution_data before clearing, so the last-started/last-completed breadcrumb survives the force-fail path. - getProgressMarkers validates marker shape (rebuilds from typed fields), so a stale or wrong-shaped Redis value can never reach API consumers.

waleedlatif1 · 2026-06-27T22:54:19Z

@greptile review

waleedlatif1 · 2026-06-27T22:54:20Z

@cursor review

getProgressMarkers now returns null on a Redis read error (vs {} for genuinely empty). completeWorkflowExecution and markExecutionAsFailed skip clearProgressMarkers when the read returns null, so a transient read error at completion no longer wipes markers that are still durably in Redis — the TTL reclaims them instead.

waleedlatif1 · 2026-06-27T23:03:38Z

@greptile review

waleedlatif1 · 2026-06-27T23:03:39Z

@cursor review

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 08f54ef. Configure here.}

waleedlatif1 · 2026-06-27T23:09:25Z

@greptile review

waleedlatif1 · 2026-06-27T23:09:26Z

@cursor review

…+ drain on force-fail - When a Redis marker write falls back to SQL, Redis and the row can each hold a marker for a different block; reads/folds previously preferred Redis unconditionally and could pick a stale value. Now the completion fold, the in-flight detail read, and the force-fail fold all pick the marker with the later timestamp (pickLatestStartedMarker/pickLatestCompletedMarker; markExecutionAsFailed uses a monotonic SQL guard). - markAsFailed now drains pending per-block marker writes (not just the completion promise) before folding, so a force-fail racing onBlockStart/onBlockComplete still captures the latest breadcrumb.

waleedlatif1 · 2026-06-27T23:20:21Z

@greptile review

waleedlatif1 · 2026-06-27T23:20:22Z

@cursor review

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 6c4c289. Configure here.}

Guard the monotonic-check index with type(decoded) == 'table' so a corrupted Redis field that decodes to a non-table (e.g. a number) can't error the eval; our write path only ever stores JSON objects, so this is defense-in-depth.

completeWorkflowExecution now takes readProgressMarkers (the session's resolved marker mode); when the flag is off it skips the per-completion HGETALL+DEL entirely instead of probing a key that was never written. Sticky to the session so it stays flip-safe (an execution that wrote to Redis always folds+clears Redis). Non-session callers default to true (safe read-and-fold). Also hardened the Lua guard with type(decoded)=='table'.

waleedlatif1 · 2026-06-27T23:52:46Z

@greptile review

waleedlatif1 · 2026-06-27T23:52:48Z

@cursor review

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 6669170. Configure here.}

cursorBot reviewed Jun 27, 2026
View reviewed changes

Comment threadapps/sim/lib/logs/execution/logging-session.ts Outdated
Comment threadapps/sim/lib/logs/execution/logging-session.ts

greptile-appsBot reviewed Jun 27, 2026
View reviewed changes

Comment threadapps/sim/lib/logs/execution/logging-session.ts Outdated
Comment threadapps/sim/lib/logs/execution/logging-session.ts Outdated
Comment threadapps/sim/lib/logs/execution/progress-markers.ts Outdated

vercelBottemporarily deployed to Preview June 27, 2026 22:49 Inactive

chore(logs): convert inline marker comments to TSDoc
27afbce

vercelBottemporarily deployed to Preview June 27, 2026 22:53 Inactive

cursorBot reviewed Jun 27, 2026
View reviewed changes

Comment threadapps/sim/lib/logs/execution/progress-markers.ts

vercelBottemporarily deployed to Preview June 27, 2026 23:03 Inactive

cursorBot reviewed Jun 27, 2026
View reviewed changes

greptile-appsBot reviewed Jun 27, 2026
View reviewed changes

Comment threadapps/sim/lib/logs/execution/logger.ts Outdated

cursorBot reviewed Jun 27, 2026
View reviewed changes

Comment threadapps/sim/lib/logs/execution/logger.ts Outdated
Comment threadapps/sim/lib/logs/execution/logging-session.ts

vercelBottemporarily deployed to Preview June 27, 2026 23:20 Inactive

cursorBot reviewed Jun 27, 2026
View reviewed changes

waleedlatif1 mentioned this pull request Jun 27, 2026
v0.7.16: security hardening, db o11y and profiling, settings UI #5245
Merged

vercelBottemporarily deployed to Preview June 27, 2026 23:42 Inactive

vercelBottemporarily deployed to Preview June 27, 2026 23:52 Inactive

cursorBot reviewed Jun 27, 2026
View reviewed changes

waleedlatif1 merged commit 7a2103e into stagingJun 27, 2026
11 checks passed

waleedlatif1 deleted the redis-progress-markers branch June 27, 2026 23:56

Uh oh!

Conversation

waleedlatif1 commented Jun 27, 2026

Summary

How it works

Rollout

Type of Change

Testing

Checklist

Uh oh!

vercelBot commented Jun 27, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursorBot commented Jun 27, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

Uh oh!

Uh oh!

greptile-appsBot commented Jun 27, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Jun 27, 2026

Uh oh!

waleedlatif1 commented Jun 27, 2026

Uh oh!

Uh oh!

waleedlatif1 commented Jun 27, 2026

Uh oh!

waleedlatif1 commented Jun 27, 2026

Uh oh!

cursorBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Jun 27, 2026

Uh oh!

waleedlatif1 commented Jun 27, 2026

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Jun 27, 2026

Uh oh!

waleedlatif1 commented Jun 27, 2026

Uh oh!

cursorBot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Jun 27, 2026

Uh oh!

waleedlatif1 commented Jun 27, 2026

Uh oh!

cursorBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercelBot commented Jun 27, 2026•
edited
Loading

cursorBot commented Jun 27, 2026•
edited
Loading

greptile-appsBot commented Jun 27, 2026•
edited
Loading