fix: heartbeat timing bugs — reset on sell, missing activation, ConfigMap race - #281

Closed
bussyjd wants to merge 7 commits into
mainfrom
fix/timing
Closed

fix: heartbeat timing bugs — reset on sell, missing activation, ConfigMap race#281
bussyjd wants to merge 7 commits into
mainfrom
fix/timing

Conversation

@bussyjd

@bussyjdbussyjd commented Mar 19, 2026

Copy link
Copy Markdown
Contributor

Closes#280

Summary

Fixes 4 interrelated heartbeat timing bugs discovered by pi-autoresearch automated user-flow validation (44 → 90/90 steps passing).

Changes

Bug 1: SyncAgentBaseURL resets heartbeat on every obol sell http

obol sell http → EnsureTunnelForSell → SyncAgentBaseURL → helmfile sync
↓
ConfigMap re-rendered WITHOUT heartbeat
↓
Agent falls back to 30m default

Added patchHeartbeatAfterSync() + idempotency (skip sync when URL unchanged).

Bug 2: Heartbeat not activated after obol agent init

Added ensureHeartbeatActive() — reads ConfigMap, patches every: "5m" if missing.

Bug 3: Chokidar misses ConfigMap symlink swaps

K8s updates ConfigMaps via symlink swap. Chokidar inotify doesn't detect this. Added rollout restart after patch for deterministic behavior.

Bug 4: Unnecessary pod restart removed

Heartbeat patch was restarting the pod when hot-reload should handle it. Removed.

Test plan

  • pi-autoresearch: 90/90 steps, 39 runs, 37 kept
  • flow-06: obol sell http → heartbeat fires within 5 min → ServiceOffer Ready
  • flow-09: full lifecycle (sell → stop → delete → cleanup verified)
  • Manual: obol stack up && obol sell http test --wallet ... --chain base-sepolia --per-request 0.001 --namespace llm --upstream ollama --port 11434 → check obol sell status test -n llm within 5 min

bussyjd added 7 commits March 19, 2026 02:56
bussyjd
fix(flow-scripts): initial committed flow scripts with known fixes
6cac784
- flow-01: remove unnecessary sudo check
- flow-02: poll for pods ready (60x5s) + longer frontend poll
- flow-03: exec into litellm pod for in-cluster test (not kubectl run)
poll for port-forward ready before inference
- flow-04: poll for port-forward ready before agent inference
- flow-05: skip local node deploy; test eRPC RPC gateway instead
correct URL format: /rpc/evm/<chainId>
- flow-06: wait for obol-agent heartbeat to reconcile (96x5s=8min)
use poll_step_grep on 'obol sell list' for READY=True
- flow-07: 402 via local Traefik + tunnel, verifier metrics
- flow-08: buy flow with blockrun-llm + Foundry balance checks
- flow-09: lifecycle (list, status, stop, delete, verify cleanup)
- flow-10: Anvil + facilitator setup for paid flows
- lib.sh: add poll_step_grep helper
bussyjd
Baseline: 44/57 steps passed. Failures: exec-in-container (flow-03), …
f1bbe63
…LiteLLM inference timeout (flow-03), ServiceOffer not reconciled (flow-06), 404 on /services (flow-07/08), x402 metrics missing (flow-07), false passes on cast balance checks (flow-10/08).
Result: {"status":"keep","steps_passed":44,"total_steps":57}
bussyjd
fix(timing): ensure heartbeat activates after obol agent init
2000389
patchHeartbeatConfig in doSync patches the openclaw-config ConfigMap
AFTER helmfile sync starts the pod, so the pod loads config without
heartbeat and cron/jobs.json stays empty. Two fixes:
1. doSync/patchHeartbeatConfig: rollout-restart openclaw deployment
after patching the ConfigMap, so the pod re-reads it on startup.
2. agent.Init/ensureHeartbeatActive: new idempotent helper that
- reads the live ConfigMap and checks for agents.defaults.heartbeat
- patches it if missing (every: 5m, target: none)
- rollout-restarts the deployment + waits for rollout
This covers the 'already running' case where flow-02 skips stack
init, doSync is never called, and the pod was never restarted with
heartbeat config. obol agent init is called every run (flow-04),
so ensureHeartbeatActive fires on every iteration.
bussyjd
fix(timing): re-patch heartbeat ConfigMap after SyncAgentBaseURL helm…
9ccc5ca
…file sync
SyncAgentBaseURL (called on every tunnel start/obol sell http) runs a raw
helmfile sync that renders openclaw-config from the chart template, which
does not include agents.defaults.heartbeat. This silently resets the
interval back to the 30m default, preventing the heartbeat from firing
within the 8-minute flow-06 poll window.
Add patchHeartbeatAfterSync() which mirrors openclaw.patchHeartbeatConfig()
but lives in the tunnel package to avoid a circular import. It reads
values-obol.yaml for the heartbeat every/target, reads the live ConfigMap,
injects agents.defaults.heartbeat, and applies via server-side kubectl.
OpenClaw hot-reloads the change (~30-60s) — no pod restart needed.
bussyjd
fix(timing): SyncAgentBaseURL idempotency — skip helm sync when URL u…
c67153f
…nchanged
Every obol sell pricing + obol sell http triggers EnsureTunnelForSell
which calls SyncAgentBaseURL unconditionally, resetting the ConfigMap
(removing heartbeat config) on EVERY sell command, even when URL unchanged.
Add readCurrentAgentBaseURL() to read the current value from overlay.
Skip sync if URL matches: avoids unnecessary ConfigMap resets and prevents
the heartbeat interval from reverting to 30m default on each sell command.
bussyjd
fix(timing): remove pod restart from heartbeat patch — use ConfigMap …
2815d08
…hot reload
openclaw.go patchHeartbeatConfig: remove pod rollout-restart that was
incorrectly added. OpenClaw hot-reloads ConfigMap file changes within
~30-60s via its built-in file watcher, no restart needed.
agent.go ensureHeartbeatActive: new idempotent helper that patches the
openclaw-config ConfigMap if agents.defaults.heartbeat is missing.
Called by obol agent init to handle 'already running' clusters where
doSync was never called this session. Only patches if missing, then
lets OpenClaw hot reload handle the interval change.
bussyjd
chore: sync autoresearch state to fix/timing
95d2ea7
@bussyjd

bussyjd commented Mar 19, 2026

Copy link
Copy Markdown
ContributorAuthor

Superseded by #282 which includes these fixes in a clean single commit alongside the autoresearch flow scripts.

@bussyjdbussyjd closed this Mar 19, 2026
@OisinKyne
OisinKyne deleted the fix/timing branch April 23, 2026 19:26
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

fix: heartbeat timing bugs — reset on sell, missing activation, ConfigMap race

1 participant

@bussyjd
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
fix: heartbeat timing bugs — reset on sell, missing activation, ConfigMap race - #281
Closed
bussyjd wants to merge 7 commits into
mainfrom 
fix/timing
Closed
fix: heartbeat timing bugs — reset on sell, missing activation, ConfigMap race#281
bussyjd wants to merge 7 commits into
mainfrom 
fix/timing
Conversation
@bussyjd
@bussyjdbussyjd
commented
Mar 19, 2026

Copy link


Copy Markdown

Contributor
Closes#280
Summary
Fixes 4 interrelated heartbeat timing bugs discovered by pi-autoresearch automated user-flow validation (44 → 90/90 steps passing).
Changes
Bug 1: SyncAgentBaseURL resets heartbeat on every obol sell http
obol sell http → EnsureTunnelForSell → SyncAgentBaseURL → helmfile sync
↓
ConfigMap re-rendered WITHOUT heartbeat
↓
Agent falls back to 30m default

Added patchHeartbeatAfterSync() + idempotency (skip sync when URL unchanged).
Bug 2: Heartbeat not activated after obol agent init
Added ensureHeartbeatActive() — reads ConfigMap, patches every: "5m" if missing.
Bug 3: Chokidar misses ConfigMap symlink swaps
K8s updates ConfigMaps via symlink swap. Chokidar inotify doesn't detect this. Added rollout restart after patch for deterministic behavior.
Bug 4: Unnecessary pod restart removed
Heartbeat patch was restarting the pod when hot-reload should handle it. Removed.
Test plan
  •  pi-autoresearch: 90/90 steps, 39 runs, 37 kept
  •  flow-06: obol sell http → heartbeat fires within 5 min → ServiceOffer Ready
  •  flow-09: full lifecycle (sell → stop → delete → cleanup verified)
  •  Manual: obol stack up && obol sell http test --wallet ... --chain base-sepolia --per-request 0.001 --namespace llm --upstream ollama --port 11434 → check obol sell status test -n llm within 5 min
bussyjd
added 7 commits
March 19, 2026 02:56
bussyjd
fix(flow-scripts): initial committed flow scripts with known fixes
6cac784
- flow-01: remove unnecessary sudo check
- flow-02: poll for pods ready (60x5s) + longer frontend poll
- flow-03: exec into litellm pod for in-cluster test (not kubectl run)
poll for port-forward ready before inference
- flow-04: poll for port-forward ready before agent inference
- flow-05: skip local node deploy; test eRPC RPC gateway instead
correct URL format: /rpc/evm/<chainId>
- flow-06: wait for obol-agent heartbeat to reconcile (96x5s=8min)
use poll_step_grep on 'obol sell list' for READY=True
- flow-07: 402 via local Traefik + tunnel, verifier metrics
- flow-08: buy flow with blockrun-llm + Foundry balance checks
- flow-09: lifecycle (list, status, stop, delete, verify cleanup)
- flow-10: Anvil + facilitator setup for paid flows
- lib.sh: add poll_step_grep helper
bussyjd
Baseline: 44/57 steps passed. Failures: exec-in-container (flow-03), …
f1bbe63
…LiteLLM inference timeout (flow-03), ServiceOffer not reconciled (flow-06), 404 on /services (flow-07/08), x402 metrics missing (flow-07), false passes on cast balance checks (flow-10/08).
Result: {"status":"keep","steps_passed":44,"total_steps":57}
bussyjd
fix(timing): ensure heartbeat activates after obol agent init
2000389
patchHeartbeatConfig in doSync patches the openclaw-config ConfigMap
AFTER helmfile sync starts the pod, so the pod loads config without
heartbeat and cron/jobs.json stays empty. Two fixes:
1. doSync/patchHeartbeatConfig: rollout-restart openclaw deployment
after patching the ConfigMap, so the pod re-reads it on startup.
2. agent.Init/ensureHeartbeatActive: new idempotent helper that
- reads the live ConfigMap and checks for agents.defaults.heartbeat
- patches it if missing (every: 5m, target: none)
- rollout-restarts the deployment + waits for rollout
This covers the 'already running' case where flow-02 skips stack
init, doSync is never called, and the pod was never restarted with
heartbeat config. obol agent init is called every run (flow-04),
so ensureHeartbeatActive fires on every iteration.
bussyjd
fix(timing): re-patch heartbeat ConfigMap after SyncAgentBaseURL helm…
9ccc5ca
…file sync
SyncAgentBaseURL (called on every tunnel start/obol sell http) runs a raw
helmfile sync that renders openclaw-config from the chart template, which
does not include agents.defaults.heartbeat. This silently resets the
interval back to the 30m default, preventing the heartbeat from firing
within the 8-minute flow-06 poll window.
Add patchHeartbeatAfterSync() which mirrors openclaw.patchHeartbeatConfig()
but lives in the tunnel package to avoid a circular import. It reads
values-obol.yaml for the heartbeat every/target, reads the live ConfigMap,
injects agents.defaults.heartbeat, and applies via server-side kubectl.
OpenClaw hot-reloads the change (~30-60s) — no pod restart needed.
bussyjd
fix(timing): SyncAgentBaseURL idempotency — skip helm sync when URL u…
c67153f
…nchanged
Every obol sell pricing + obol sell http triggers EnsureTunnelForSell
which calls SyncAgentBaseURL unconditionally, resetting the ConfigMap
(removing heartbeat config) on EVERY sell command, even when URL unchanged.
Add readCurrentAgentBaseURL() to read the current value from overlay.
Skip sync if URL matches: avoids unnecessary ConfigMap resets and prevents
the heartbeat interval from reverting to 30m default on each sell command.
bussyjd
fix(timing): remove pod restart from heartbeat patch — use ConfigMap …
2815d08
…hot reload
openclaw.go patchHeartbeatConfig: remove pod rollout-restart that was
incorrectly added. OpenClaw hot-reloads ConfigMap file changes within
~30-60s via its built-in file watcher, no restart needed.
agent.go ensureHeartbeatActive: new idempotent helper that patches the
openclaw-config ConfigMap if agents.defaults.heartbeat is missing.
Called by obol agent init to handle 'already running' clusters where
doSync was never called this session. Only patches if missing, then
lets OpenClaw hot reload handle the interval change.
bussyjd
chore: sync autoresearch state to fix/timing
95d2ea7
@bussyjd
bussyjd
commented
Mar 19, 2026

Copy link


Copy Markdown

ContributorAuthor
Superseded by #282 which includes these fixes in a clean single commit alongside the autoresearch flow scripts.
@bussyjdbussyjd
closed this
Mar 19, 2026
@OisinKyne
OisinKyne
deleted the

fix/timing

branch
April 23, 2026 19:26
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

fix: heartbeat timing bugs — reset on sell, missing activation, ConfigMap race


1 participant

@bussyjd

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
fix: heartbeat timing bugs — reset on sell, missing activation, ConfigMap race - #281
Closed
bussyjd wants to merge 7 commits into
mainfrom 
fix/timing
Closed
fix: heartbeat timing bugs — reset on sell, missing activation, ConfigMap race#281
bussyjd wants to merge 7 commits into
mainfrom 
fix/timing
Conversation
@bussyjd
@bussyjdbussyjd
commented
Mar 19, 2026

Copy link


Copy Markdown

Contributor
Closes#280
Summary
Fixes 4 interrelated heartbeat timing bugs discovered by pi-autoresearch automated user-flow validation (44 → 90/90 steps passing).
Changes
Bug 1: SyncAgentBaseURL resets heartbeat on every obol sell http
obol sell http → EnsureTunnelForSell → SyncAgentBaseURL → helmfile sync
↓
ConfigMap re-rendered WITHOUT heartbeat
↓
Agent falls back to 30m default

Added patchHeartbeatAfterSync() + idempotency (skip sync when URL unchanged).
Bug 2: Heartbeat not activated after obol agent init
Added ensureHeartbeatActive() — reads ConfigMap, patches every: "5m" if missing.
Bug 3: Chokidar misses ConfigMap symlink swaps
K8s updates ConfigMaps via symlink swap. Chokidar inotify doesn't detect this. Added rollout restart after patch for deterministic behavior.
Bug 4: Unnecessary pod restart removed
Heartbeat patch was restarting the pod when hot-reload should handle it. Removed.
Test plan
  •  pi-autoresearch: 90/90 steps, 39 runs, 37 kept
  •  flow-06: obol sell http → heartbeat fires within 5 min → ServiceOffer Ready
  •  flow-09: full lifecycle (sell → stop → delete → cleanup verified)
  •  Manual: obol stack up && obol sell http test --wallet ... --chain base-sepolia --per-request 0.001 --namespace llm --upstream ollama --port 11434 → check obol sell status test -n llm within 5 min
bussyjd
added 7 commits
March 19, 2026 02:56
bussyjd
fix(flow-scripts): initial committed flow scripts with known fixes
6cac784
- flow-01: remove unnecessary sudo check
- flow-02: poll for pods ready (60x5s) + longer frontend poll
- flow-03: exec into litellm pod for in-cluster test (not kubectl run)
poll for port-forward ready before inference
- flow-04: poll for port-forward ready before agent inference
- flow-05: skip local node deploy; test eRPC RPC gateway instead
correct URL format: /rpc/evm/<chainId>
- flow-06: wait for obol-agent heartbeat to reconcile (96x5s=8min)
use poll_step_grep on 'obol sell list' for READY=True
- flow-07: 402 via local Traefik + tunnel, verifier metrics
- flow-08: buy flow with blockrun-llm + Foundry balance checks
- flow-09: lifecycle (list, status, stop, delete, verify cleanup)
- flow-10: Anvil + facilitator setup for paid flows
- lib.sh: add poll_step_grep helper
bussyjd
Baseline: 44/57 steps passed. Failures: exec-in-container (flow-03), …
f1bbe63
…LiteLLM inference timeout (flow-03), ServiceOffer not reconciled (flow-06), 404 on /services (flow-07/08), x402 metrics missing (flow-07), false passes on cast balance checks (flow-10/08).
Result: {"status":"keep","steps_passed":44,"total_steps":57}
bussyjd
fix(timing): ensure heartbeat activates after obol agent init
2000389
patchHeartbeatConfig in doSync patches the openclaw-config ConfigMap
AFTER helmfile sync starts the pod, so the pod loads config without
heartbeat and cron/jobs.json stays empty. Two fixes:
1. doSync/patchHeartbeatConfig: rollout-restart openclaw deployment
after patching the ConfigMap, so the pod re-reads it on startup.
2. agent.Init/ensureHeartbeatActive: new idempotent helper that
- reads the live ConfigMap and checks for agents.defaults.heartbeat
- patches it if missing (every: 5m, target: none)
- rollout-restarts the deployment + waits for rollout
This covers the 'already running' case where flow-02 skips stack
init, doSync is never called, and the pod was never restarted with
heartbeat config. obol agent init is called every run (flow-04),
so ensureHeartbeatActive fires on every iteration.
bussyjd
fix(timing): re-patch heartbeat ConfigMap after SyncAgentBaseURL helm…
9ccc5ca
…file sync
SyncAgentBaseURL (called on every tunnel start/obol sell http) runs a raw
helmfile sync that renders openclaw-config from the chart template, which
does not include agents.defaults.heartbeat. This silently resets the
interval back to the 30m default, preventing the heartbeat from firing
within the 8-minute flow-06 poll window.
Add patchHeartbeatAfterSync() which mirrors openclaw.patchHeartbeatConfig()
but lives in the tunnel package to avoid a circular import. It reads
values-obol.yaml for the heartbeat every/target, reads the live ConfigMap,
injects agents.defaults.heartbeat, and applies via server-side kubectl.
OpenClaw hot-reloads the change (~30-60s) — no pod restart needed.
bussyjd
fix(timing): SyncAgentBaseURL idempotency — skip helm sync when URL u…
c67153f
…nchanged
Every obol sell pricing + obol sell http triggers EnsureTunnelForSell
which calls SyncAgentBaseURL unconditionally, resetting the ConfigMap
(removing heartbeat config) on EVERY sell command, even when URL unchanged.
Add readCurrentAgentBaseURL() to read the current value from overlay.
Skip sync if URL matches: avoids unnecessary ConfigMap resets and prevents
the heartbeat interval from reverting to 30m default on each sell command.
bussyjd
fix(timing): remove pod restart from heartbeat patch — use ConfigMap …
2815d08
…hot reload
openclaw.go patchHeartbeatConfig: remove pod rollout-restart that was
incorrectly added. OpenClaw hot-reloads ConfigMap file changes within
~30-60s via its built-in file watcher, no restart needed.
agent.go ensureHeartbeatActive: new idempotent helper that patches the
openclaw-config ConfigMap if agents.defaults.heartbeat is missing.
Called by obol agent init to handle 'already running' clusters where
doSync was never called this session. Only patches if missing, then
lets OpenClaw hot reload handle the interval change.
bussyjd
chore: sync autoresearch state to fix/timing
95d2ea7
@bussyjd
bussyjd
commented
Mar 19, 2026

Copy link


Copy Markdown

ContributorAuthor
Superseded by #282 which includes these fixes in a clean single commit alongside the autoresearch flow scripts.
@bussyjdbussyjd
closed this
Mar 19, 2026
@OisinKyne
OisinKyne
deleted the

fix/timing

branch
April 23, 2026 19:26
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

fix: heartbeat timing bugs — reset on sell, missing activation, ConfigMap race


1 participant

@bussyjd

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
fix: heartbeat timing bugs — reset on sell, missing activation, ConfigMap race - #281
Closed
bussyjd wants to merge 7 commits into
mainfrom 
fix/timing
Closed
fix: heartbeat timing bugs — reset on sell, missing activation, ConfigMap race#281
bussyjd wants to merge 7 commits into
mainfrom 
fix/timing
Conversation
@bussyjd
@bussyjdbussyjd
commented
Mar 19, 2026

Copy link


Copy Markdown

Contributor
Closes#280
Summary
Fixes 4 interrelated heartbeat timing bugs discovered by pi-autoresearch automated user-flow validation (44 → 90/90 steps passing).
Changes
Bug 1: SyncAgentBaseURL resets heartbeat on every obol sell http
obol sell http → EnsureTunnelForSell → SyncAgentBaseURL → helmfile sync
↓
ConfigMap re-rendered WITHOUT heartbeat
↓
Agent falls back to 30m default

Added patchHeartbeatAfterSync() + idempotency (skip sync when URL unchanged).
Bug 2: Heartbeat not activated after obol agent init
Added ensureHeartbeatActive() — reads ConfigMap, patches every: "5m" if missing.
Bug 3: Chokidar misses ConfigMap symlink swaps
K8s updates ConfigMaps via symlink swap. Chokidar inotify doesn't detect this. Added rollout restart after patch for deterministic behavior.
Bug 4: Unnecessary pod restart removed
Heartbeat patch was restarting the pod when hot-reload should handle it. Removed.
Test plan
  •  pi-autoresearch: 90/90 steps, 39 runs, 37 kept
  •  flow-06: obol sell http → heartbeat fires within 5 min → ServiceOffer Ready
  •  flow-09: full lifecycle (sell → stop → delete → cleanup verified)
  •  Manual: obol stack up && obol sell http test --wallet ... --chain base-sepolia --per-request 0.001 --namespace llm --upstream ollama --port 11434 → check obol sell status test -n llm within 5 min
bussyjd
added 7 commits
March 19, 2026 02:56
bussyjd
fix(flow-scripts): initial committed flow scripts with known fixes
6cac784
- flow-01: remove unnecessary sudo check
- flow-02: poll for pods ready (60x5s) + longer frontend poll
- flow-03: exec into litellm pod for in-cluster test (not kubectl run)
poll for port-forward ready before inference
- flow-04: poll for port-forward ready before agent inference
- flow-05: skip local node deploy; test eRPC RPC gateway instead
correct URL format: /rpc/evm/<chainId>
- flow-06: wait for obol-agent heartbeat to reconcile (96x5s=8min)
use poll_step_grep on 'obol sell list' for READY=True
- flow-07: 402 via local Traefik + tunnel, verifier metrics
- flow-08: buy flow with blockrun-llm + Foundry balance checks
- flow-09: lifecycle (list, status, stop, delete, verify cleanup)
- flow-10: Anvil + facilitator setup for paid flows
- lib.sh: add poll_step_grep helper
bussyjd
Baseline: 44/57 steps passed. Failures: exec-in-container (flow-03), …
f1bbe63
…LiteLLM inference timeout (flow-03), ServiceOffer not reconciled (flow-06), 404 on /services (flow-07/08), x402 metrics missing (flow-07), false passes on cast balance checks (flow-10/08).
Result: {"status":"keep","steps_passed":44,"total_steps":57}
bussyjd
fix(timing): ensure heartbeat activates after obol agent init
2000389
patchHeartbeatConfig in doSync patches the openclaw-config ConfigMap
AFTER helmfile sync starts the pod, so the pod loads config without
heartbeat and cron/jobs.json stays empty. Two fixes:
1. doSync/patchHeartbeatConfig: rollout-restart openclaw deployment
after patching the ConfigMap, so the pod re-reads it on startup.
2. agent.Init/ensureHeartbeatActive: new idempotent helper that
- reads the live ConfigMap and checks for agents.defaults.heartbeat
- patches it if missing (every: 5m, target: none)
- rollout-restarts the deployment + waits for rollout
This covers the 'already running' case where flow-02 skips stack
init, doSync is never called, and the pod was never restarted with
heartbeat config. obol agent init is called every run (flow-04),
so ensureHeartbeatActive fires on every iteration.
bussyjd
fix(timing): re-patch heartbeat ConfigMap after SyncAgentBaseURL helm…
9ccc5ca
…file sync
SyncAgentBaseURL (called on every tunnel start/obol sell http) runs a raw
helmfile sync that renders openclaw-config from the chart template, which
does not include agents.defaults.heartbeat. This silently resets the
interval back to the 30m default, preventing the heartbeat from firing
within the 8-minute flow-06 poll window.
Add patchHeartbeatAfterSync() which mirrors openclaw.patchHeartbeatConfig()
but lives in the tunnel package to avoid a circular import. It reads
values-obol.yaml for the heartbeat every/target, reads the live ConfigMap,
injects agents.defaults.heartbeat, and applies via server-side kubectl.
OpenClaw hot-reloads the change (~30-60s) — no pod restart needed.
bussyjd
fix(timing): SyncAgentBaseURL idempotency — skip helm sync when URL u…
c67153f
…nchanged
Every obol sell pricing + obol sell http triggers EnsureTunnelForSell
which calls SyncAgentBaseURL unconditionally, resetting the ConfigMap
(removing heartbeat config) on EVERY sell command, even when URL unchanged.
Add readCurrentAgentBaseURL() to read the current value from overlay.
Skip sync if URL matches: avoids unnecessary ConfigMap resets and prevents
the heartbeat interval from reverting to 30m default on each sell command.
bussyjd
fix(timing): remove pod restart from heartbeat patch — use ConfigMap …
2815d08
…hot reload
openclaw.go patchHeartbeatConfig: remove pod rollout-restart that was
incorrectly added. OpenClaw hot-reloads ConfigMap file changes within
~30-60s via its built-in file watcher, no restart needed.
agent.go ensureHeartbeatActive: new idempotent helper that patches the
openclaw-config ConfigMap if agents.defaults.heartbeat is missing.
Called by obol agent init to handle 'already running' clusters where
doSync was never called this session. Only patches if missing, then
lets OpenClaw hot reload handle the interval change.
bussyjd
chore: sync autoresearch state to fix/timing
95d2ea7
@bussyjd
bussyjd
commented
Mar 19, 2026

Copy link


Copy Markdown

ContributorAuthor
Superseded by #282 which includes these fixes in a clean single commit alongside the autoresearch flow scripts.
@bussyjdbussyjd
closed this
Mar 19, 2026
@OisinKyne
OisinKyne
deleted the

fix/timing

branch
April 23, 2026 19:26
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

fix: heartbeat timing bugs — reset on sell, missing activation, ConfigMap race


1 participant

@bussyjd

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
fix: heartbeat timing bugs — reset on sell, missing activation, ConfigMap race - #281
Closed
bussyjd wants to merge 7 commits into
mainfrom 
fix/timing
Closed
fix: heartbeat timing bugs — reset on sell, missing activation, ConfigMap race#281
bussyjd wants to merge 7 commits into
mainfrom 
fix/timing
Conversation
@bussyjd
@bussyjdbussyjd
commented
Mar 19, 2026

Copy link


Copy Markdown

Contributor
Closes#280
Summary
Fixes 4 interrelated heartbeat timing bugs discovered by pi-autoresearch automated user-flow validation (44 → 90/90 steps passing).
Changes
Bug 1: SyncAgentBaseURL resets heartbeat on every obol sell http
obol sell http → EnsureTunnelForSell → SyncAgentBaseURL → helmfile sync
↓
ConfigMap re-rendered WITHOUT heartbeat
↓
Agent falls back to 30m default

Added patchHeartbeatAfterSync() + idempotency (skip sync when URL unchanged).
Bug 2: Heartbeat not activated after obol agent init
Added ensureHeartbeatActive() — reads ConfigMap, patches every: "5m" if missing.
Bug 3: Chokidar misses ConfigMap symlink swaps
K8s updates ConfigMaps via symlink swap. Chokidar inotify doesn't detect this. Added rollout restart after patch for deterministic behavior.
Bug 4: Unnecessary pod restart removed
Heartbeat patch was restarting the pod when hot-reload should handle it. Removed.
Test plan
  •  pi-autoresearch: 90/90 steps, 39 runs, 37 kept
  •  flow-06: obol sell http → heartbeat fires within 5 min → ServiceOffer Ready
  •  flow-09: full lifecycle (sell → stop → delete → cleanup verified)
  •  Manual: obol stack up && obol sell http test --wallet ... --chain base-sepolia --per-request 0.001 --namespace llm --upstream ollama --port 11434 → check obol sell status test -n llm within 5 min
bussyjd
added 7 commits
March 19, 2026 02:56
bussyjd
fix(flow-scripts): initial committed flow scripts with known fixes
6cac784
- flow-01: remove unnecessary sudo check
- flow-02: poll for pods ready (60x5s) + longer frontend poll
- flow-03: exec into litellm pod for in-cluster test (not kubectl run)
poll for port-forward ready before inference
- flow-04: poll for port-forward ready before agent inference
- flow-05: skip local node deploy; test eRPC RPC gateway instead
correct URL format: /rpc/evm/<chainId>
- flow-06: wait for obol-agent heartbeat to reconcile (96x5s=8min)
use poll_step_grep on 'obol sell list' for READY=True
- flow-07: 402 via local Traefik + tunnel, verifier metrics
- flow-08: buy flow with blockrun-llm + Foundry balance checks
- flow-09: lifecycle (list, status, stop, delete, verify cleanup)
- flow-10: Anvil + facilitator setup for paid flows
- lib.sh: add poll_step_grep helper
bussyjd
Baseline: 44/57 steps passed. Failures: exec-in-container (flow-03), …
f1bbe63
…LiteLLM inference timeout (flow-03), ServiceOffer not reconciled (flow-06), 404 on /services (flow-07/08), x402 metrics missing (flow-07), false passes on cast balance checks (flow-10/08).
Result: {"status":"keep","steps_passed":44,"total_steps":57}
bussyjd
fix(timing): ensure heartbeat activates after obol agent init
2000389
patchHeartbeatConfig in doSync patches the openclaw-config ConfigMap
AFTER helmfile sync starts the pod, so the pod loads config without
heartbeat and cron/jobs.json stays empty. Two fixes:
1. doSync/patchHeartbeatConfig: rollout-restart openclaw deployment
after patching the ConfigMap, so the pod re-reads it on startup.
2. agent.Init/ensureHeartbeatActive: new idempotent helper that
- reads the live ConfigMap and checks for agents.defaults.heartbeat
- patches it if missing (every: 5m, target: none)
- rollout-restarts the deployment + waits for rollout
This covers the 'already running' case where flow-02 skips stack
init, doSync is never called, and the pod was never restarted with
heartbeat config. obol agent init is called every run (flow-04),
so ensureHeartbeatActive fires on every iteration.
bussyjd
fix(timing): re-patch heartbeat ConfigMap after SyncAgentBaseURL helm…
9ccc5ca
…file sync
SyncAgentBaseURL (called on every tunnel start/obol sell http) runs a raw
helmfile sync that renders openclaw-config from the chart template, which
does not include agents.defaults.heartbeat. This silently resets the
interval back to the 30m default, preventing the heartbeat from firing
within the 8-minute flow-06 poll window.
Add patchHeartbeatAfterSync() which mirrors openclaw.patchHeartbeatConfig()
but lives in the tunnel package to avoid a circular import. It reads
values-obol.yaml for the heartbeat every/target, reads the live ConfigMap,
injects agents.defaults.heartbeat, and applies via server-side kubectl.
OpenClaw hot-reloads the change (~30-60s) — no pod restart needed.
bussyjd
fix(timing): SyncAgentBaseURL idempotency — skip helm sync when URL u…
c67153f
…nchanged
Every obol sell pricing + obol sell http triggers EnsureTunnelForSell
which calls SyncAgentBaseURL unconditionally, resetting the ConfigMap
(removing heartbeat config) on EVERY sell command, even when URL unchanged.
Add readCurrentAgentBaseURL() to read the current value from overlay.
Skip sync if URL matches: avoids unnecessary ConfigMap resets and prevents
the heartbeat interval from reverting to 30m default on each sell command.
bussyjd
fix(timing): remove pod restart from heartbeat patch — use ConfigMap …
2815d08
…hot reload
openclaw.go patchHeartbeatConfig: remove pod rollout-restart that was
incorrectly added. OpenClaw hot-reloads ConfigMap file changes within
~30-60s via its built-in file watcher, no restart needed.
agent.go ensureHeartbeatActive: new idempotent helper that patches the
openclaw-config ConfigMap if agents.defaults.heartbeat is missing.
Called by obol agent init to handle 'already running' clusters where
doSync was never called this session. Only patches if missing, then
lets OpenClaw hot reload handle the interval change.
bussyjd
chore: sync autoresearch state to fix/timing
95d2ea7
@bussyjd
bussyjd
commented
Mar 19, 2026

Copy link


Copy Markdown

ContributorAuthor
Superseded by #282 which includes these fixes in a clean single commit alongside the autoresearch flow scripts.
@bussyjdbussyjd
closed this
Mar 19, 2026
@OisinKyne
OisinKyne
deleted the

fix/timing

branch
April 23, 2026 19:26
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

fix: heartbeat timing bugs — reset on sell, missing activation, ConfigMap race


1 participant

@bussyjd

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
fix: heartbeat timing bugs — reset on sell, missing activation, ConfigMap race - #281
Closed
bussyjd wants to merge 7 commits into
mainfrom 
fix/timing
Closed
fix: heartbeat timing bugs — reset on sell, missing activation, ConfigMap race#281
bussyjd wants to merge 7 commits into
mainfrom 
fix/timing
Conversation
@bussyjd
@bussyjdbussyjd
commented
Mar 19, 2026

Copy link


Copy Markdown

Contributor
Closes#280
Summary
Fixes 4 interrelated heartbeat timing bugs discovered by pi-autoresearch automated user-flow validation (44 → 90/90 steps passing).
Changes
Bug 1: SyncAgentBaseURL resets heartbeat on every obol sell http
obol sell http → EnsureTunnelForSell → SyncAgentBaseURL → helmfile sync
↓
ConfigMap re-rendered WITHOUT heartbeat
↓
Agent falls back to 30m default

Added patchHeartbeatAfterSync() + idempotency (skip sync when URL unchanged).
Bug 2: Heartbeat not activated after obol agent init
Added ensureHeartbeatActive() — reads ConfigMap, patches every: "5m" if missing.
Bug 3: Chokidar misses ConfigMap symlink swaps
K8s updates ConfigMaps via symlink swap. Chokidar inotify doesn't detect this. Added rollout restart after patch for deterministic behavior.
Bug 4: Unnecessary pod restart removed
Heartbeat patch was restarting the pod when hot-reload should handle it. Removed.
Test plan
  •  pi-autoresearch: 90/90 steps, 39 runs, 37 kept
  •  flow-06: obol sell http → heartbeat fires within 5 min → ServiceOffer Ready
  •  flow-09: full lifecycle (sell → stop → delete → cleanup verified)
  •  Manual: obol stack up && obol sell http test --wallet ... --chain base-sepolia --per-request 0.001 --namespace llm --upstream ollama --port 11434 → check obol sell status test -n llm within 5 min
bussyjd
added 7 commits
March 19, 2026 02:56
bussyjd
fix(flow-scripts): initial committed flow scripts with known fixes
6cac784
- flow-01: remove unnecessary sudo check
- flow-02: poll for pods ready (60x5s) + longer frontend poll
- flow-03: exec into litellm pod for in-cluster test (not kubectl run)
poll for port-forward ready before inference
- flow-04: poll for port-forward ready before agent inference
- flow-05: skip local node deploy; test eRPC RPC gateway instead
correct URL format: /rpc/evm/<chainId>
- flow-06: wait for obol-agent heartbeat to reconcile (96x5s=8min)
use poll_step_grep on 'obol sell list' for READY=True
- flow-07: 402 via local Traefik + tunnel, verifier metrics
- flow-08: buy flow with blockrun-llm + Foundry balance checks
- flow-09: lifecycle (list, status, stop, delete, verify cleanup)
- flow-10: Anvil + facilitator setup for paid flows
- lib.sh: add poll_step_grep helper
bussyjd
Baseline: 44/57 steps passed. Failures: exec-in-container (flow-03), …
f1bbe63
…LiteLLM inference timeout (flow-03), ServiceOffer not reconciled (flow-06), 404 on /services (flow-07/08), x402 metrics missing (flow-07), false passes on cast balance checks (flow-10/08).
Result: {"status":"keep","steps_passed":44,"total_steps":57}
bussyjd
fix(timing): ensure heartbeat activates after obol agent init
2000389
patchHeartbeatConfig in doSync patches the openclaw-config ConfigMap
AFTER helmfile sync starts the pod, so the pod loads config without
heartbeat and cron/jobs.json stays empty. Two fixes:
1. doSync/patchHeartbeatConfig: rollout-restart openclaw deployment
after patching the ConfigMap, so the pod re-reads it on startup.
2. agent.Init/ensureHeartbeatActive: new idempotent helper that
- reads the live ConfigMap and checks for agents.defaults.heartbeat
- patches it if missing (every: 5m, target: none)
- rollout-restarts the deployment + waits for rollout
This covers the 'already running' case where flow-02 skips stack
init, doSync is never called, and the pod was never restarted with
heartbeat config. obol agent init is called every run (flow-04),
so ensureHeartbeatActive fires on every iteration.
bussyjd
fix(timing): re-patch heartbeat ConfigMap after SyncAgentBaseURL helm…
9ccc5ca
…file sync
SyncAgentBaseURL (called on every tunnel start/obol sell http) runs a raw
helmfile sync that renders openclaw-config from the chart template, which
does not include agents.defaults.heartbeat. This silently resets the
interval back to the 30m default, preventing the heartbeat from firing
within the 8-minute flow-06 poll window.
Add patchHeartbeatAfterSync() which mirrors openclaw.patchHeartbeatConfig()
but lives in the tunnel package to avoid a circular import. It reads
values-obol.yaml for the heartbeat every/target, reads the live ConfigMap,
injects agents.defaults.heartbeat, and applies via server-side kubectl.
OpenClaw hot-reloads the change (~30-60s) — no pod restart needed.
bussyjd
fix(timing): SyncAgentBaseURL idempotency — skip helm sync when URL u…
c67153f
…nchanged
Every obol sell pricing + obol sell http triggers EnsureTunnelForSell
which calls SyncAgentBaseURL unconditionally, resetting the ConfigMap
(removing heartbeat config) on EVERY sell command, even when URL unchanged.
Add readCurrentAgentBaseURL() to read the current value from overlay.
Skip sync if URL matches: avoids unnecessary ConfigMap resets and prevents
the heartbeat interval from reverting to 30m default on each sell command.
bussyjd
fix(timing): remove pod restart from heartbeat patch — use ConfigMap …
2815d08
…hot reload
openclaw.go patchHeartbeatConfig: remove pod rollout-restart that was
incorrectly added. OpenClaw hot-reloads ConfigMap file changes within
~30-60s via its built-in file watcher, no restart needed.
agent.go ensureHeartbeatActive: new idempotent helper that patches the
openclaw-config ConfigMap if agents.defaults.heartbeat is missing.
Called by obol agent init to handle 'already running' clusters where
doSync was never called this session. Only patches if missing, then
lets OpenClaw hot reload handle the interval change.
bussyjd
chore: sync autoresearch state to fix/timing
95d2ea7
@bussyjd
bussyjd
commented
Mar 19, 2026

Copy link


Copy Markdown

ContributorAuthor
Superseded by #282 which includes these fixes in a clean single commit alongside the autoresearch flow scripts.
@bussyjdbussyjd
closed this
Mar 19, 2026
@OisinKyne
OisinKyne
deleted the

fix/timing

branch
April 23, 2026 19:26
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

fix: heartbeat timing bugs — reset on sell, missing activation, ConfigMap race


1 participant

@bussyjd

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
fix: heartbeat timing bugs — reset on sell, missing activation, ConfigMap race - #281
Closed
bussyjd wants to merge 7 commits into
mainfrom 
fix/timing
Closed
fix: heartbeat timing bugs — reset on sell, missing activation, ConfigMap race#281
bussyjd wants to merge 7 commits into
mainfrom 
fix/timing
Conversation
@bussyjd
@bussyjdbussyjd
commented
Mar 19, 2026

Copy link


Copy Markdown

Contributor
Closes#280
Summary
Fixes 4 interrelated heartbeat timing bugs discovered by pi-autoresearch automated user-flow validation (44 → 90/90 steps passing).
Changes
Bug 1: SyncAgentBaseURL resets heartbeat on every obol sell http
obol sell http → EnsureTunnelForSell → SyncAgentBaseURL → helmfile sync
↓
ConfigMap re-rendered WITHOUT heartbeat
↓
Agent falls back to 30m default

Added patchHeartbeatAfterSync() + idempotency (skip sync when URL unchanged).
Bug 2: Heartbeat not activated after obol agent init
Added ensureHeartbeatActive() — reads ConfigMap, patches every: "5m" if missing.
Bug 3: Chokidar misses ConfigMap symlink swaps
K8s updates ConfigMaps via symlink swap. Chokidar inotify doesn't detect this. Added rollout restart after patch for deterministic behavior.
Bug 4: Unnecessary pod restart removed
Heartbeat patch was restarting the pod when hot-reload should handle it. Removed.
Test plan
  •  pi-autoresearch: 90/90 steps, 39 runs, 37 kept
  •  flow-06: obol sell http → heartbeat fires within 5 min → ServiceOffer Ready
  •  flow-09: full lifecycle (sell → stop → delete → cleanup verified)
  •  Manual: obol stack up && obol sell http test --wallet ... --chain base-sepolia --per-request 0.001 --namespace llm --upstream ollama --port 11434 → check obol sell status test -n llm within 5 min
bussyjd
added 7 commits
March 19, 2026 02:56
bussyjd
fix(flow-scripts): initial committed flow scripts with known fixes
6cac784
- flow-01: remove unnecessary sudo check
- flow-02: poll for pods ready (60x5s) + longer frontend poll
- flow-03: exec into litellm pod for in-cluster test (not kubectl run)
poll for port-forward ready before inference
- flow-04: poll for port-forward ready before agent inference
- flow-05: skip local node deploy; test eRPC RPC gateway instead
correct URL format: /rpc/evm/<chainId>
- flow-06: wait for obol-agent heartbeat to reconcile (96x5s=8min)
use poll_step_grep on 'obol sell list' for READY=True
- flow-07: 402 via local Traefik + tunnel, verifier metrics
- flow-08: buy flow with blockrun-llm + Foundry balance checks
- flow-09: lifecycle (list, status, stop, delete, verify cleanup)
- flow-10: Anvil + facilitator setup for paid flows
- lib.sh: add poll_step_grep helper
bussyjd
Baseline: 44/57 steps passed. Failures: exec-in-container (flow-03), …
f1bbe63
…LiteLLM inference timeout (flow-03), ServiceOffer not reconciled (flow-06), 404 on /services (flow-07/08), x402 metrics missing (flow-07), false passes on cast balance checks (flow-10/08).
Result: {"status":"keep","steps_passed":44,"total_steps":57}
bussyjd
fix(timing): ensure heartbeat activates after obol agent init
2000389
patchHeartbeatConfig in doSync patches the openclaw-config ConfigMap
AFTER helmfile sync starts the pod, so the pod loads config without
heartbeat and cron/jobs.json stays empty. Two fixes:
1. doSync/patchHeartbeatConfig: rollout-restart openclaw deployment
after patching the ConfigMap, so the pod re-reads it on startup.
2. agent.Init/ensureHeartbeatActive: new idempotent helper that
- reads the live ConfigMap and checks for agents.defaults.heartbeat
- patches it if missing (every: 5m, target: none)
- rollout-restarts the deployment + waits for rollout
This covers the 'already running' case where flow-02 skips stack
init, doSync is never called, and the pod was never restarted with
heartbeat config. obol agent init is called every run (flow-04),
so ensureHeartbeatActive fires on every iteration.
bussyjd
fix(timing): re-patch heartbeat ConfigMap after SyncAgentBaseURL helm…
9ccc5ca
…file sync
SyncAgentBaseURL (called on every tunnel start/obol sell http) runs a raw
helmfile sync that renders openclaw-config from the chart template, which
does not include agents.defaults.heartbeat. This silently resets the
interval back to the 30m default, preventing the heartbeat from firing
within the 8-minute flow-06 poll window.
Add patchHeartbeatAfterSync() which mirrors openclaw.patchHeartbeatConfig()
but lives in the tunnel package to avoid a circular import. It reads
values-obol.yaml for the heartbeat every/target, reads the live ConfigMap,
injects agents.defaults.heartbeat, and applies via server-side kubectl.
OpenClaw hot-reloads the change (~30-60s) — no pod restart needed.
bussyjd
fix(timing): SyncAgentBaseURL idempotency — skip helm sync when URL u…
c67153f
…nchanged
Every obol sell pricing + obol sell http triggers EnsureTunnelForSell
which calls SyncAgentBaseURL unconditionally, resetting the ConfigMap
(removing heartbeat config) on EVERY sell command, even when URL unchanged.
Add readCurrentAgentBaseURL() to read the current value from overlay.
Skip sync if URL matches: avoids unnecessary ConfigMap resets and prevents
the heartbeat interval from reverting to 30m default on each sell command.
bussyjd
fix(timing): remove pod restart from heartbeat patch — use ConfigMap …
2815d08
…hot reload
openclaw.go patchHeartbeatConfig: remove pod rollout-restart that was
incorrectly added. OpenClaw hot-reloads ConfigMap file changes within
~30-60s via its built-in file watcher, no restart needed.
agent.go ensureHeartbeatActive: new idempotent helper that patches the
openclaw-config ConfigMap if agents.defaults.heartbeat is missing.
Called by obol agent init to handle 'already running' clusters where
doSync was never called this session. Only patches if missing, then
lets OpenClaw hot reload handle the interval change.
bussyjd
chore: sync autoresearch state to fix/timing
95d2ea7
@bussyjd
bussyjd
commented
Mar 19, 2026

Copy link


Copy Markdown

ContributorAuthor
Superseded by #282 which includes these fixes in a clean single commit alongside the autoresearch flow scripts.
@bussyjdbussyjd
closed this
Mar 19, 2026
@OisinKyne
OisinKyne
deleted the

fix/timing

branch
April 23, 2026 19:26
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

fix: heartbeat timing bugs — reset on sell, missing activation, ConfigMap race


1 participant

@bussyjd

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
fix: heartbeat timing bugs — reset on sell, missing activation, ConfigMap race - #281
Closed
bussyjd wants to merge 7 commits into
mainfrom 
fix/timing
Closed
fix: heartbeat timing bugs — reset on sell, missing activation, ConfigMap race#281
bussyjd wants to merge 7 commits into
mainfrom 
fix/timing
Conversation
@bussyjd
@bussyjdbussyjd
commented
Mar 19, 2026

Copy link


Copy Markdown

Contributor
Closes#280
Summary
Fixes 4 interrelated heartbeat timing bugs discovered by pi-autoresearch automated user-flow validation (44 → 90/90 steps passing).
Changes
Bug 1: SyncAgentBaseURL resets heartbeat on every obol sell http
obol sell http → EnsureTunnelForSell → SyncAgentBaseURL → helmfile sync
↓
ConfigMap re-rendered WITHOUT heartbeat
↓
Agent falls back to 30m default

Added patchHeartbeatAfterSync() + idempotency (skip sync when URL unchanged).
Bug 2: Heartbeat not activated after obol agent init
Added ensureHeartbeatActive() — reads ConfigMap, patches every: "5m" if missing.
Bug 3: Chokidar misses ConfigMap symlink swaps
K8s updates ConfigMaps via symlink swap. Chokidar inotify doesn't detect this. Added rollout restart after patch for deterministic behavior.
Bug 4: Unnecessary pod restart removed
Heartbeat patch was restarting the pod when hot-reload should handle it. Removed.
Test plan
  •  pi-autoresearch: 90/90 steps, 39 runs, 37 kept
  •  flow-06: obol sell http → heartbeat fires within 5 min → ServiceOffer Ready
  •  flow-09: full lifecycle (sell → stop → delete → cleanup verified)
  •  Manual: obol stack up && obol sell http test --wallet ... --chain base-sepolia --per-request 0.001 --namespace llm --upstream ollama --port 11434 → check obol sell status test -n llm within 5 min
bussyjd
added 7 commits
March 19, 2026 02:56
bussyjd
fix(flow-scripts): initial committed flow scripts with known fixes
6cac784
- flow-01: remove unnecessary sudo check
- flow-02: poll for pods ready (60x5s) + longer frontend poll
- flow-03: exec into litellm pod for in-cluster test (not kubectl run)
poll for port-forward ready before inference
- flow-04: poll for port-forward ready before agent inference
- flow-05: skip local node deploy; test eRPC RPC gateway instead
correct URL format: /rpc/evm/<chainId>
- flow-06: wait for obol-agent heartbeat to reconcile (96x5s=8min)
use poll_step_grep on 'obol sell list' for READY=True
- flow-07: 402 via local Traefik + tunnel, verifier metrics
- flow-08: buy flow with blockrun-llm + Foundry balance checks
- flow-09: lifecycle (list, status, stop, delete, verify cleanup)
- flow-10: Anvil + facilitator setup for paid flows
- lib.sh: add poll_step_grep helper
bussyjd
Baseline: 44/57 steps passed. Failures: exec-in-container (flow-03), …
f1bbe63
…LiteLLM inference timeout (flow-03), ServiceOffer not reconciled (flow-06), 404 on /services (flow-07/08), x402 metrics missing (flow-07), false passes on cast balance checks (flow-10/08).
Result: {"status":"keep","steps_passed":44,"total_steps":57}
bussyjd
fix(timing): ensure heartbeat activates after obol agent init
2000389
patchHeartbeatConfig in doSync patches the openclaw-config ConfigMap
AFTER helmfile sync starts the pod, so the pod loads config without
heartbeat and cron/jobs.json stays empty. Two fixes:
1. doSync/patchHeartbeatConfig: rollout-restart openclaw deployment
after patching the ConfigMap, so the pod re-reads it on startup.
2. agent.Init/ensureHeartbeatActive: new idempotent helper that
- reads the live ConfigMap and checks for agents.defaults.heartbeat
- patches it if missing (every: 5m, target: none)
- rollout-restarts the deployment + waits for rollout
This covers the 'already running' case where flow-02 skips stack
init, doSync is never called, and the pod was never restarted with
heartbeat config. obol agent init is called every run (flow-04),
so ensureHeartbeatActive fires on every iteration.
bussyjd
fix(timing): re-patch heartbeat ConfigMap after SyncAgentBaseURL helm…
9ccc5ca
…file sync
SyncAgentBaseURL (called on every tunnel start/obol sell http) runs a raw
helmfile sync that renders openclaw-config from the chart template, which
does not include agents.defaults.heartbeat. This silently resets the
interval back to the 30m default, preventing the heartbeat from firing
within the 8-minute flow-06 poll window.
Add patchHeartbeatAfterSync() which mirrors openclaw.patchHeartbeatConfig()
but lives in the tunnel package to avoid a circular import. It reads
values-obol.yaml for the heartbeat every/target, reads the live ConfigMap,
injects agents.defaults.heartbeat, and applies via server-side kubectl.
OpenClaw hot-reloads the change (~30-60s) — no pod restart needed.
bussyjd
fix(timing): SyncAgentBaseURL idempotency — skip helm sync when URL u…
c67153f
…nchanged
Every obol sell pricing + obol sell http triggers EnsureTunnelForSell
which calls SyncAgentBaseURL unconditionally, resetting the ConfigMap
(removing heartbeat config) on EVERY sell command, even when URL unchanged.
Add readCurrentAgentBaseURL() to read the current value from overlay.
Skip sync if URL matches: avoids unnecessary ConfigMap resets and prevents
the heartbeat interval from reverting to 30m default on each sell command.
bussyjd
fix(timing): remove pod restart from heartbeat patch — use ConfigMap …
2815d08
…hot reload
openclaw.go patchHeartbeatConfig: remove pod rollout-restart that was
incorrectly added. OpenClaw hot-reloads ConfigMap file changes within
~30-60s via its built-in file watcher, no restart needed.
agent.go ensureHeartbeatActive: new idempotent helper that patches the
openclaw-config ConfigMap if agents.defaults.heartbeat is missing.
Called by obol agent init to handle 'already running' clusters where
doSync was never called this session. Only patches if missing, then
lets OpenClaw hot reload handle the interval change.
bussyjd
chore: sync autoresearch state to fix/timing
95d2ea7
@bussyjd
bussyjd
commented
Mar 19, 2026

Copy link


Copy Markdown

ContributorAuthor
Superseded by #282 which includes these fixes in a clean single commit alongside the autoresearch flow scripts.
@bussyjdbussyjd
closed this
Mar 19, 2026
@OisinKyne
OisinKyne
deleted the

fix/timing

branch
April 23, 2026 19:26
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

fix: heartbeat timing bugs — reset on sell, missing activation, ConfigMap race


1 participant

@bussyjd