Skip to content

emrg: scheduler workspace-heal async offload + clone backoff (rant 2026-08-19T00:54:32) - #858

Merged
argszero merged 1 commit into
masterfrom
feature/scheduler-async-git-backoff
Aug 18, 2026
Merged

emrg: scheduler workspace-heal async offload + clone backoff (rant 2026-08-19T00:54:32)#858
argszero merged 1 commit into
masterfrom
feature/scheduler-async-git-backoff

Conversation

@argszero

Copy link
Copy Markdown
Owner

Summary

Per host rant 2026-08-19T00:54:32 (reproduced on Windows v0.2.51): the scheduler's workspace self-heal ran synchronous git subprocesses (clone / ls-remote / config) directly on the asyncio event loop. A slow/failing clone (28s on a blocked network) wedged every websocket client (TUI/GUI) for the whole duration → server connection lost — reconnecting every 60s, on a loop.

Changes (emrg/server/scheduler.py)

  1. Event-loop offload_run_evolution_cycle now awaits asyncio.to_thread(self._ensure_evolution_workspace): the entire self-heal (all sync git calls inside _clone_workspace / _ensure_origin_reachable / _align_to_installed_version / _ensure_git_identity / _is_usable_git_repo) runs in a worker thread, never blocking the loop.
  2. Clone-failure backoff — after a failed self-heal (network down), exponential backoff 5min → 10min → 20min → 30min cap before retrying (_workspace_heal_failures + _workspace_heal_next_retry_at); reset on success. A network-down daemon stops hammering every 60s.
  3. Shorter connect timeouts — https clone http.connectTimeout 10s → 5s; SSH retry now carries GIT_SSH_COMMAND="ssh -o ConnectTimeout=5" so a blocked SSH port also fails fast instead of eating the full timeout.

Verification

  • test_workspace_heal_backoff_skips_retries (new): clone failure arms backoff; cycles inside the window skip with zero git calls; expiry → retry re-arms with 2× delay.
  • test_run_evolution_cycle_offloads_workspace_heal_to_thread (new): static source assertion on the asyncio.to_thread wiring.
  • test_ensure_evolution_workspace_clone_falls_back_to_ssh extended: http.connectTimeout=5 on the https clone + GIT_SSH_COMMAND=ssh -o ConnectTimeout=5 on the SSH retry; success resets the backoff counters.
  • All workspace-heal tests run directly (mock git subprocess, no real clone, no pytest runner) — PASS; existing tests unchanged (sync function signatures preserved).
  • Import + CLI checks pass. Agent.md count 959 → 961 (2 new test functions).
  • Full CI (ubuntu + windows, incl. actionlint + doc-count guard) validates.

@argszeroargszero left a comment

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle 788 (Committer). CI test + test-windows both PASS on head 8d2f7b6 (doc-count guard validated 961). Offload + backoff + timeout changes verified against the workspace-heal tests (all green when run directly with mocked git).

@argszeroargszero left a comment

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle 789 (Committer). Independent re-verify: head unchanged (8d2f7b6), CI test + test-windows PASS. Diff re-inspected — to_thread offload, backoff gate/skip, and connectTimeout=5 (https) + GIT_SSH_COMMAND ConnectTimeout=5 (SSH) all correct. 2/3.

@argszeroargszero left a comment

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle 790 (Committer). 3rd consecutive approval. Head unchanged (8d2f7b6), CI green, MERGEABLE. Merging.

@argszero
argszero merged commit faf12c0 into masterAug 18, 2026
2 checks passed
@argszero
argszero deleted the feature/scheduler-async-git-backoff branch August 18, 2026 18:10
argszero added a commit that referenced this pull request Aug 19, 2026
Co-authored-by: EMRG Evolution <emrg@argszero.dev>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@argszero