injection retries

.trajectories/completed/2026-01/traj_q6n7i3r1xik1.json

@@ -0,0 +1,49 @@
+{
+ "id": "traj_q6n7i3r1xik1",
+ "version": 1,
+ "task": {
+ "title": "Add activity-based verification for task injection"
+ },
+ "status": "completed",
+ "startedAt": "2026-01-30T12:20:59.974Z",
+ "agents": [
+ {
+ "name": "default",
+ "role": "lead",
+ "joinedAt": "2026-01-30T12:21:13.109Z"
+ }
+ ],
+ "chapters": [
+ {
+ "id": "chap_aofbp322rook",
+ "title": "Work",
+ "agentName": "default",
+ "startedAt": "2026-01-30T12:21:13.109Z",
+ "events": [
+ {
+ "ts": 1769775673110,
+ "type": "decision",
+ "content": "Used activity pattern matching to verify CLI processed the task, not just PTY delivery: Used activity pattern matching to verify CLI processed the task, not just PTY delivery",
+ "raw": {
+ "question": "Used activity pattern matching to verify CLI processed the task, not just PTY delivery",
+ "chosen": "Used activity pattern matching to verify CLI processed the task, not just PTY delivery",
+ "alternatives": [],
+ "reasoning": "T-003 failure showed PTY delivery succeeded but CLI wasn't ready. Tested Claude Code, Codex, and Droid - found universal patterns: [Pasted text], Relay message from, thinking spinners"
+ },
+ "significance": "high"
+ }
+ ],
+ "endedAt": "2026-01-30T12:21:19.584Z"
+ }
+ ],
+ "commits": [],
+ "filesChanged": [],
+ "projectId": "/Users/khaliqgant/Projects/agent-workforce/relay",
+ "tags": [],
+ "completedAt": "2026-01-30T12:21:19.584Z",
+ "retrospective": {
+ "summary": "Added activity verification to injectTask() in relay-pty-orchestrator.ts. After socket confirms delivery, verifies CLI shows activity (task received patterns, thinking spinners, tool execution) within 5s. Retries up to 3 times if no activity detected. Prevents T-003-style failures where PTY delivery succeeded but CLI wasn't ready.",
+ "approach": "Standard approach",
+ "confidence": 0.8
+ }
+}

.trajectories/completed/2026-01/traj_q6n7i3r1xik1.md

@@ -0,0 +1,31 @@
+# Trajectory: Add activity-based verification for task injection
+
+> **Status:** ✅ Completed
+> **Confidence:** 80%
+> **Started:** January 30, 2026 at 01:20 PM
+> **Completed:** January 30, 2026 at 01:21 PM
+
+---
+
+## Summary
+
+Added activity verification to injectTask() in relay-pty-orchestrator.ts. After socket confirms delivery, verifies CLI shows activity (task received patterns, thinking spinners, tool execution) within 5s. Retries up to 3 times if no activity detected. Prevents T-003-style failures where PTY delivery succeeded but CLI wasn't ready.
+
+**Approach:** Standard approach
+
+---
+
+## Key Decisions
+
+### Used activity pattern matching to verify CLI processed the task, not just PTY delivery
+- **Chose:** Used activity pattern matching to verify CLI processed the task, not just PTY delivery
+- **Reasoning:** T-003 failure showed PTY delivery succeeded but CLI wasn't ready. Tested Claude Code, Codex, and Droid - found universal patterns: [Pasted text], Relay message from, thinking spinners
+
+---
+
+## Chapters
+
+### 1. Work
+*Agent: default*
+
+- Used activity pattern matching to verify CLI processed the task, not just PTY delivery: Used activity pattern matching to verify CLI processed the task, not just PTY delivery

.trajectories/index.json

@@ -1,6 +1,6 @@
 {
  "version": 1,
- "lastUpdated": "2026-01-29T10:12:48.153Z",
+ "lastUpdated": "2026-01-30T12:21:19.597Z",
  "trajectories": {
  "traj_1b1dj40sl6jl": {
  "title": "Revert aggressive retry logic in relay-pty-orchestrator",
@@ -183,6 +183,13 @@
  "startedAt": "2026-01-29T10:03:49.538Z",
  "completedAt": "2026-01-29T10:12:48.143Z",
  "path": "/Users/khaliqgant/Projects/agent-workforce/relay/.trajectories/completed/2026-01/traj_xlvah6igh9it.json"
+ },
+ "traj_q6n7i3r1xik1": {
+ "title": "Add activity-based verification for task injection",
+ "status": "completed",
+ "startedAt": "2026-01-30T12:20:59.974Z",
+ "completedAt": "2026-01-30T12:21:19.584Z",
+ "path": "/Users/khaliqgant/Projects/agent-workforce/relay/.trajectories/completed/2026-01/traj_q6n7i3r1xik1.json"
  }
  }
 }

CHANGELOG.md

@@ -7,6 +7,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Breaking Changes
+- **Task injection failures now return errors**: When spawning an agent with a task, if delivery fails after 3 retry attempts, spawn returns `success: false` and kills the agent. Previously, spawn would return `success: true` even if task injection silently failed, leaving zombie agents.
+
+### Migration Guidance
+- Callers of `relay.spawn()` with a `task` parameter should handle `success: false` results (though automatic retries make failures rare).
+- Spawning without a task is unaffected and always succeeds.
+- See the updated [Worker Orchestration guide](/docs/guides/worker-orchestration) for retry patterns.
+
+### Fixed
+- Task injection failures are no longer silent - spawn properly returns an error when delivery fails.
+- Zombie agents (spawned but never received their task) are now cleaned up automatically.
+- Automatic retry (3 attempts with 2s delays) for task injection improves reliability.
+
 ## [2.0.20] - 2026-01-26
 
 ### Overview

docs/guides/worker-orchestration.mdx

@@ -234,6 +234,96 @@ Waiting for:
 - [ ] TestRunner - Running tests
 ```
 
+## Spawn Failures and Retries
+
+When spawning a worker with a task, the system:
+
+1. **Attempts delivery** of the task to the CLI via socket (or stdin fallback)
+2. **Retries delivery** up to 3 times (with 2s delays) if the write fails
+3. **Returns result** - succeeds if delivery was confirmed by relay-pty
+
+A periodic sweep monitors for stuck injections as an additional safety net.
+
+### Why injection can fail
+
+**Delivery failures** (retried automatically):
+- Socket disconnected
+- Write error to stdin
+- relay-pty process issue
+
+### Automatic retry behavior
+
+```
+Delivery attempt 1 → Failed (socket error)
+ ↓ (2s delay)
+Delivery attempt 2 → Failed (write error)
+ ↓ (2s delay)
+Delivery attempt 3 → Success ✓
+ ↓
+Return true
+```
+
+### Handling spawn failures
+
+Even with automatic retries, spawns can still fail. Always check the result:
+
+```typescript
+const result = await relay.spawn({
+ name: 'Worker-1',
+ cli: 'claude',
+ task: 'Implement the login feature',
+});
+
+if (!result.success) {
+ console.error(`Spawn failed after 3 attempts: ${result.error}`);
+ // Additional retry at caller level (optional)
+ await sleep(5000);
+ const retryResult = await relay.spawn({ ... });
+}
+```
+
+### Additional retry strategies
+
+<CardGroup cols={2}>
+
+<Card title="Exponential backoff" icon="clock-rotate-left">
+Wait longer between each caller-level retry: 5s, 10s, 20s...
+</Card>
+
+<Card title="Max retries" icon="rotate">
+Limit total retries (e.g., 2 caller-level attempts) before escalating.
+</Card>
+
+<Card title="Health check first" icon="heart-pulse">
+Verify relay daemon is healthy before retry.
+</Card>
+
+<Card title="Alternative CLI" icon="arrows-rotate">
+Try a different CLI variant if one keeps failing.
+</Card>
+
+</CardGroup>
+
+### Spawn without task
+
+Spawning an agent **without** a task always succeeds (no verification needed):
+
+```typescript
+// This always succeeds - agent waits for messages
+const result = await relay.spawn({
+ name: 'Listener',
+ cli: 'claude',
+ // No task - agent will receive work via messages
+});
+```
+
+Use this pattern when you want to send the task via relay message instead:
+
+```typescript
+await relay.spawn({ name: 'Worker', cli: 'claude' });
+await relay.sendMessage('Worker', 'Your task here...');
+```
+
 ## Error Handling
 
 ### Worker reports failure

package.json

@@ -148,7 +148,7 @@
  "@agent-relay/utils": "2.1.4",
  "@agent-relay/wrapper": "2.1.4",
  "@modelcontextprotocol/sdk": "^1.0.0",
- "agent-trajectories": "^0.2.3",
+ "agent-trajectories": "^0.3.0",
  "chokidar": "^5.0.0",
  "commander": "^12.1.0",
  "compare-versions": "^6.1.1",

packages/bridge/src/spawner.ts

@@ -1195,8 +1195,9 @@ export class AgentSpawner {
  // Send task to the newly spawned agent if provided
  if (task && task.trim()) {
  const ready = await openCodeWrapper.waitUntilReadyForMessages(20000, 100);
+ let taskSent = false;
  if (ready) {
-const taskSent = await openCodeWrapper.injectTask(task, spawnerName || 'spawner');
+ taskSent = await openCodeWrapper.injectTask(task, spawnerName || 'spawner');
  if (!taskSent) {
  log.warn(`Failed to inject task for ${name} via OpenCodeWrapper`);
  } else if (debug) {
@@ -1205,6 +1206,28 @@ export class AgentSpawner {
  } else {
  log.warn(`OpenCodeWrapper ${name} not ready for task injection`);
  }
+
+ // If task injection failed, kill the agent and return error
+ // An agent without its task is useless and will just sit idle
+ if (!taskSent) {
+ const tracedError = createTraceableError('Task injection failed', {
+ agentName: name,
+ cli,
+ taskLength: task.length,
+ ready,
+ });
+ log.error(`CRITICAL: ${tracedError.logMessage}`);
+ await openCodeWrapper.stop();
+ if (this.onClearSpawning) {
+ this.onClearSpawning(name);
+ }
+ return {
+ success: false,
+ name,
+ error: tracedError.userMessage,
+ errorId: tracedError.errorId,
+ };
+ }
  }
 
  // Track the worker (cast to AgentWrapper for type compatibility)
@@ -1371,14 +1394,15 @@ export class AgentSpawner {
  await (pty as RelayPtyOrchestrator).waitUntilCliReady(15000, 100);
  }
 
- // Inject task via socket (with verification and retries)
+ // Inject task via socket (relay-pty confirms write)
  const success = await pty.injectTask(task, spawnerName || 'spawner');
  if (success) {
  taskSent = true;
  if (debug) log.debug(`Task injected to ${name} (attempt ${attempt})`);
  break;
  } else {
- throw new Error('Task injection returned false');
+ // Delivery failed - safe to retry
+ throw new Error('Task injection returned false - delivery failed');
  }
  } catch (err: any) {
  // Log retry attempts at DEBUG level to avoid terminal noise
@@ -1398,8 +1422,19 @@ export class AgentSpawner {
  taskLength: task.length,
  });
  log.error(`CRITICAL: ${tracedError.logMessage}`);
- // Note: We don't return an error here because the agent is running,
- // but we track the errorId so support can investigate if user reports it
+ // Kill the agent since it's useless without its task - it will just sit idle
+ // Return an error so the caller knows the spawn effectively failed
+ await pty.stop();
+ this.activeWorkers.delete(name);
+ if (this.onClearSpawning) {
+ this.onClearSpawning(name);
+ }
+ return {
+ success: false,
+ name,
+ error: tracedError.userMessage,
+ errorId: tracedError.errorId,
+ };
  }
  }