tower-cron: conditions can't see exitCode (ReferenceError every run) and failure runs can never deliver alerts

[waleedkadous]

Symptom

~/.agent-farm/tower.log fills with a recurring WARN (every 15 min for months, survives Tower restarts):

[WARN] Condition evaluation failed for 'Service Health Check': ReferenceError: exitCode is not defined

The triggering task is a user health-check cron (.af-cron/*.yaml) with:

condition: "exitCode != 0"
message: "Service Health Alert:\n${output}"
target: architect

Root cause (two layers, both in packages/codev/src/agent-farm/servers/tower-cron.ts)

1. evaluateCondition() (line ~293) builds new Function('output', 'return ' + condition) — only output is in scope. Any condition referencing exitCode throws ReferenceError, which is caught at line ~251, logged as WARN, and forces shouldNotify = false. The docs/UX imply exit-code awareness; the evaluator doesn't provide it.

2. Delivery gate (line ~256) is if (shouldNotify && result === 'success'). A monitoring script that detects a problem and exits non-zero produces result = 'failure', whose branch only logs a WARN ("infrastructure noise"). Consequence: "alert me when this command fails" is inexpressible — the alert is suppressed on exactly the runs that matter. (In the wild, task authors work around it by pushing notifications from inside the script.)

Prescribed fix (design decided — carry it in BUGFIX)

1. Capture the exit code in runCommand(): in the exec callback, error.code (number) is the exit code when the process ran and exited non-zero. Resolve with { output, exitCode } (0 on success) instead of rejecting for plain non-zero exits. Keep rejecting/flagging genuinely infrastructural errors (spawn failure, timeout/kill — error.killed/error.signal) as today's failure class.

2. Expose exitCode to conditions: new Function('output', 'exitCode', 'return ' + condition), invoked with both. Conditions using only output are unchanged (backward compatible).

3. Rework the delivery gate: deliver when the condition (now able to see exitCode) evaluates true, regardless of the old success/failure split:

   • Task with a condition: delivery is decided by the condition alone (a non-zero exit is data, not noise).
   • Task without a condition: keep today's behavior (deliver only on exit 0) so existing no-condition tasks don't start alert-spamming on flaky commands.
   • True infrastructure errors (spawn fail/timeout) keep the existing WARN-only path, with exitCode reported as non-zero (e.g. 124/-1) so exitCode != 0 conditions still fire if the author wants them to.

4. State row semantics: last_result in cron_tasks should record success/failure by exit code as before — only the delivery logic changes.

5. Docs: document the condition environment (output: string, exitCode: number) wherever .af-cron YAML is documented (afx skill + agent-farm.md), BOTH trees.

Acceptance

• A task with condition: "exitCode != 0" produces no ReferenceError; delivers its message exactly when the command exits non-zero.
• A task with condition: "output.includes('FAIL')" behaves exactly as today.
• A task with no condition delivers only on exit 0 (unchanged).
• Unit tests cover: exitCode condition true/false, output-only condition regression, no-condition regression, timeout path.

[waleedkadous]

On it! Working on a fix now.

[amrmelsayed]

Adjacent false-alarm mode observed 2026-08-06 10:00Z: the CI-alert cron fired 'CI Alert: currently failing on main: error connecting to api.github.com / check your internet connection' — a transient network failure of the cron's own gh invocation was pasted into the currently-failing-on-main alert template and delivered as if CI were red. Actual state at investigation: main fully green, api.github.com reachable in 23ms. The alert builder should distinguish 'the check command itself failed to execute' (network/gh-auth error → either retry silently or alert as MONITORING failure) from 'the check ran and found failing runs' — same class as this issue's conditions-can't-see-exitCode gap: the pipeline can't currently tell command failure from check-result failure.