pags up should attach newly configured runner agents without restart

[serge-ivo]

Problem

When pags up is already running, newly subscribed/configured runtime agents do not get attached to that runner. The user has to restart pags up before the new agent can use the local runner.

This showed up while configuring terminal test agents: the machine was online, but the new instances showed nodeOnline: true and connected: false until the runner was restarted with the new instance ids.

This is codebase-confirmed behavior, not just an observed UI glitch: the API route explicitly documents the state where a machine is online but a newly created agent is not attached because pags up started before that agent existed.

Current behavior from code

pags up snapshots runtime-capable instances once at startup:

• packages/cli/src/commands/up.ts:64-83 fetches /v1/instances/my/instances and filters active runtime agents.
• packages/cli/src/commands/up.ts:99-109 updates the TUI state from that fixed list.
• packages/cli/src/commands/up.ts:117-119 spawns runner connect <fixed instance ids...>.
• packages/cli/src/commands/up.ts:232-251 handles r, but if the child is still alive it only redraws status; it does not refetch instances or attach new ones.

runner connect also assumes a fixed instance set:

• packages/cli/src/commands/runner/command.ts:55-66 declares connect <instanceIds...> and receives the list from argv.
• packages/cli/src/commands/runner/command.ts:96-100 waits for the local runner, calls connectViaRelay(instanceIds, ...), then waits for runner exit.
• packages/cli/src/commands/runner/relay.ts:14-19 accepts instanceIds: string[] as a captured array.
• packages/cli/src/commands/runner/relay.ts:25-43 registers runtime metadata once for each id in that array.
• packages/cli/src/commands/runner/relay.ts:45-51 opens one relay socket per id in that array.
• packages/cli/src/commands/runner/relay.ts:64-73 heartbeats only that same fixed array.
• packages/cli/src/commands/runner/relay.ts:76-170 manages reconnect for one specific instance socket; there is no account/node-level socket or attach/detach manager.

Server-side evidence:

• workers/api/src/routes/instances.ts:414-418 explains the exact failure mode: machine online, but this agent is not attached because pags up was started before the agent existed and never opened that instance's socket.
• workers/api/src/relay-do.ts:69-90 rejects a second live runner for the same instance with 4409 unless --force is used, so auto-discovery must avoid noisy multi-machine takeover loops.

Desired behavior

A running unscoped pags up should attach newly configured runtime agents in near-realtime, without requiring a manual restart.

Expected flow:

1. User runs pags up once.
2. User subscribes to/configures a new runtime agent in the UI.
3. The active runner detects the new eligible instance.
4. It registers runtime metadata for that instance, opens its relay socket, and starts heartbeating it.
5. UI changes from nodeOnline: true, connected: false to connected: true automatically after the discovery interval.

Use the term near-realtime for the first implementation. True realtime would require a new account/node-level control channel or server push path; the current relay architecture only has per-instance sockets, so a brand-new instance has no socket to push over.

Proposed implementation

Start with CLI polling. Do not add server push for this ticket; issue #83 already tracks real-time status push instead of polling.

Runner membership manager

Refactor connectViaRelay(instanceIds, ...) so the runner owns a mutable membership set instead of a captured immutable array.

The manager should track attached instances, for example:

Map<instanceId, {
  name?: string;
  socket/control handle;
  reconnect/backoff state;
  conflict/backoff state;
}>

Add an attachInstance(instance) path that:

• POSTs /v1/instances/:id/runtime with the same local URL, token, capabilities, runner version, runner node, and force flag used today.
• Mints a relay token.
• Opens the relay socket.
• Adds the instance to the heartbeat set.
• Emits a useful CLI/TUI log line.

Add a detachInstance(instanceId) path that:

• Closes the local relay socket/control handle.
• Removes the instance from heartbeat.
• Emits a useful CLI/TUI log line.
• Avoids aggressive DB deletion unless the server contract is clarified; routing already checks RelayDO socket liveness, not only DB status.

Discovery loop

Add a watch/discovery mode to runner connect, for example --watch-instances, and have normal unscoped pags up use it.

When watch mode is enabled, poll /v1/instances/my/instances every 15-30 seconds and compute eligible instances:

• status === "active"
• capabilities.runtime != null

Attach newly eligible ids and detach ids that disappear, become inactive, or no longer need a local runtime.

pags up --instance X must remain fixed-scope and must not auto-attach other agents. This is already called out by the existing scoped restart comment in packages/cli/src/commands/up.ts:236-238 and should remain protected by tests.

Multi-machine and pinning constraints

This is the main correctness risk.

Auto-discovery must not blindly attach every eligible runtime agent from every online machine:

• If an instance is pinned to another runner node, do not attach it from this node unless --force/takeover semantics explicitly say to.
• If another live runner already owns the relay and RelayDO returns 4409, stop or heavily back off for that instance instead of retrying forever.
• If unpinned, attaching from the current node is acceptable, but avoid flapping if another node won the race.
• Preserve existing --force behavior for deliberate takeover.

Relevant code:

• workers/api/src/relay-do.ts:69-90 emits 4409 for another live runner.
• workers/api/src/lib/runner-client.ts checks relay liveness and honors runner-node pinning during routing.
• workers/api/src/lib/runtime-nodes.ts includes runner node in the relay object name.

Manual refresh improvement

As part of this ticket, make the r key useful while the child is alive:

• Either trigger an immediate discovery pass in watch mode, or clearly show that discovery is already active and when the last scan ran.
• Do not require the child to be dead before a newly configured agent can be picked up.

TUI updates

Update visible state when membership changes:

• Agent count should reflect the current attached/discovered set.
• Logs should include events such as Attached new agent: <name> (<id prefix>...) and Detached agent: <name> (<id prefix>...).
• Errors such as 4409 should surface as actionable conflict/takeover messages rather than endless generic reconnect noise.

Acceptance criteria

• Starting pags up, then subscribing to/configuring a new coding/browser/terminal runtime agent, attaches that agent without restarting the CLI.
• The Settings runner-node panel for the new instance changes to connected automatically after the poll interval.
• Existing connected instances remain connected while the membership set changes.
• pags up --instance X remains scoped to X and does not auto-attach other agents.
• Pressing r while the runner is alive triggers or reflects a discovery refresh instead of only redrawing stale state.
• Heartbeats include newly attached instances and stop for detached instances.
• A runner pinned to another machine is not auto-attached by this machine unless forced.
• A 4409 conflict does not create an infinite noisy reconnect loop for auto-discovered instances.
• Tests cover add/remove membership, scoped mode behavior, manual refresh, pinned-other-node behavior, and relay conflict/backoff.

Related issues

• CODER-007 — (Stretch) Real-time status push instead of polling (Durable Object / relay) #83 covers a later server-push/realtime status architecture. Do not block this polling MVP on that.
• [bug] Task cancel/delete/read routes still dispatch through stale default runtime rows #218 covers stale default runtime rows and task routing. This ticket should avoid making that worse, but does not need to solve it.
• [docs] Update Coder docs for runner-node picker, engines, and package paths #211 covers docs updates for runner-node behavior.

[serge-ivo]

Another occurrence today (2026-08-06), on an instance that had previously been attached — so this is not only about newly created agents.

An FWS Repo Coder (5219a03a, registered 2026-08-04) sat amber in Settings → Runs on with pags up running on RLs-MacBook-Air.local, CLI/runner 0.4.29. Every runner-backed tool failed (repo_git → "No runner is connected") while the machine read online. A plain pags up restart turned it green immediately.

Worth noting for the fix: the two failure modes described here and in #237 are indistinguishable in the UI — both render as "machine online, agent not attached" with no cause and no remedy. Whatever discovery loop lands, the amber state needs to say which one it is, or users cannot tell a not-yet-discovered instance from one being rejected 4409.

Server-side confirmation that a runner-less instance still has real reach, which is why this is more than cosmetic: the Repo Coder's github_* tools work fine (GitHub App, cloud-side), but all four repo-local tools (repo_tree, repo_read_file, repo_git, repo_remote) go over the relay, so the agent can read and file tickets while being completely unable to see the code.

[serge-ivo]

Shipped in e274f27, published as CLI 0.4.30.

The two blockers were structural: connectViaRelay captured the ids as an immutable array, and openRelaySocket returned nothing — so there was no way to add a socket and no way to close one. The runner now owns a mutable membership set with attach/detach, and the heartbeat walks the live set rather than the startup array (heartbeating a detached instance kept it looking online; a newly attached one looked offline).

runner connect --watch-instances polls every 20s and diffs eligible against attached; pags up passes it only when unscoped. Polling not push, as the issue asked — a brand-new instance has no socket to push over. #83 still tracks the push path.

Both multi-machine constraints are pure and tested in membership.ts rather than buried in socket code:

• Pinning: an instance pinned to another node is not eligible here. Losing that race is bad; winning it is worse — it would silently relocate the agent away from where its owner pinned it.
• 4409: now terminal for that socket instead of reconnecting forever, with the id held in a blocked set so the next pass doesn't immediately re-attach. Blocked ≠ ineligible: when the other machine disconnects or you run --force, the next pass attaches normally.

pags up --instance X stays scoped — that invariant has its own test, since auto-attach would otherwise have reintroduced the exact fan-out bug the restart path already guards.

Not done from the proposed scope: the TUI agent-count/log updates and the r-key immediate-rescan. Attach/detach emit CLI log lines, but the TUI still shows the startup count. Worth a small follow-up if you want the panel to reflect membership live.