fix(infra): PDBs for payment/RPC gates + remote-signer Recreate strategy

[bussyjd]

Replica-stance hardening: PDBs for payment/RPC gates + remote-signer Recreate

Follow-up to #707, from the replica-stance review of the remaining services. Targets multi-node clusters, where drains and RWO volume attach are real.

Stacked on #707 (fix/litellm-zero-downtime) — merge that first; this PR's own diff is the last commit.

Changes

PodDisruptionBudgets (minAvailable: 1)

• x402-verifier (base/templates/x402.yaml) — the ForwardAuth gate for every paid route. On multi-node, a kubectl drain would silently evict the only pod and 5xx all /services/* traffic; the PDB makes that require explicit operator intent (delete the pod), same stance as the existing litellm PDB.
• eRPC (base/templates/erpc.yaml) — the RPC front door for agents and the frontend. Lives in the base chart because the ethereum/erpc chart (0.0.4) has no PDB template; its podDisruptionBudget value in our values file is unwired. Selector pinned to the chart's live selectorLabels.
• cloudflared — already ships a PDB (gated on active tunnel + >1 replica); now pinned by a test so it doesn't regress.

Rollouts are unaffected in all cases: at 1 replica the default RollingUpdate rounds to maxUnavailable 0 / maxSurge 1, and both verifier (/readyz gates on config + routes loaded) and eRPC surge gaplessly behind readiness probes.

remote-signer → strategy: Recreate

The obol/remote-signer chart (0.3.3) cannot express spec.strategy, so signer deployments defaulted to RollingUpdate — a singleton over a ReadWriteOnce keystore PVC. On multi-node, a surge pod can land on another node and wedge forever on the volume attach; on any cluster it briefly double-runs the signer over the same keystore. Same doctrine as hermes and x402-buyer: RWO-backed singletons run Recreate.

Implemented as a post-helmfile-sync kubectl patch (agentruntime.EnforceRemoteSignerRecreate) from both the hermes and openclaw sync paths, mirroring the existing hermes strategy-migration pattern — including the load-bearing explicit rollingUpdate: null that clears the k8s-defaulted block the API would otherwise reject the type flip over. Patching spec.strategy doesn't touch the pod template, so the pin never rolls the signer itself. Helm leaves it alone on later upgrades because the chart never renders the field.

Upstream follow-up: add strategy/updateStrategy support to the remote-signer chart in ObolNetwork/helm-charts, then delete this patch.

Reviewed and deliberately unchanged

• serviceoffer-controller — flagged in the review as "RollingUpdate without leader election", but the controller already has Lease-based leader election (cmd/serviceoffer-controller/main.go:84, on by default), so its surge is safe: the new pod blocks on the Lease until the old leader releases it. No change; the x402.yaml comment already documents this.
• Frontend readiness probe (cosmetic local-UI 502s during rollouts) — left for a UI-side change in obol-stack-front-end/chart.

Tests

• internal/embed/embed_pdb_test.go — table-driven pins for both PDBs (namespace, minAvailable, exact selector — extra labels would silently match no pods) + cloudflared PDB template gating.
• internal/agentruntime/signer_test.go — patch-args shape incl. the explicit rollingUpdate: null.

Live validation (multi-service k3d cluster from the #707 run)

• obol stack up created both PDBs: erpc/erpc and x402/x402-verifier, MIN AVAILABLE 1, ALLOWED DISRUPTIONS 0.
• The stack-up agent re-sync flipped remote-signer from RollingUpdate to {"type":"Recreate"}without rolling the pod (pod age preserved), confirming the patch is non-disruptive.
• helm lint clean; full go test green on touched packages.

https://claude.ai/code/session_01YLuXwN1A4tG3xmEMKvAzeD

[OisinKyne]

Upstream follow-up: add strategy/updateStrategy support to the remote-signer chart in ObolNetwork/helm-charts, then delete this patch.

We control this chart, lets just fix this there rather than a post-install hook? I assume that would shorten install time

[bussyjd]

Closing as superseded by integration/v0.13.0-rc0 @ a29c826 (rolled up in #716 / release v0.13.0-rc0). All commits of this PR are contained in the RC tip (verified by ancestry/patch-id audit).

[bussyjd]

Replica-stance hardening: PDBs for payment/RPC gates + remote-signer Recreate

Follow-up to #707, from the replica-stance review of the remaining services. Targets multi-node clusters, where drains and RWO volume attach are real.

Stacked on #707 (fix/litellm-zero-downtime) — merge that first; this PR's own diff is the last commit.

Changes

PodDisruptionBudgets (minAvailable: 1)

• x402-verifier (base/templates/x402.yaml) — the ForwardAuth gate for every paid route. On multi-node, a kubectl drain would silently evict the only pod and 5xx all /services/* traffic; the PDB makes that require explicit operator intent (delete the pod), same stance as the existing litellm PDB.
• eRPC (base/templates/erpc.yaml) — the RPC front door for agents and the frontend. Lives in the base chart because the ethereum/erpc chart (0.0.4) has no PDB template; its podDisruptionBudget value in our values file is unwired. Selector pinned to the chart's live selectorLabels.
• cloudflared — already ships a PDB (gated on active tunnel + >1 replica); now pinned by a test so it doesn't regress.

Rollouts are unaffected in all cases: at 1 replica the default RollingUpdate rounds to maxUnavailable 0 / maxSurge 1, and both verifier (/readyz gates on config + routes loaded) and eRPC surge gaplessly behind readiness probes.

remote-signer → strategy: Recreate

The obol/remote-signer chart (0.3.3) cannot express spec.strategy, so signer deployments defaulted to RollingUpdate — a singleton over a ReadWriteOnce keystore PVC. On multi-node, a surge pod can land on another node and wedge forever on the volume attach; on any cluster it briefly double-runs the signer over the same keystore. Same doctrine as hermes and x402-buyer: RWO-backed singletons run Recreate.

Implemented as a post-helmfile-sync kubectl patch (agentruntime.EnforceRemoteSignerRecreate) from both the hermes and openclaw sync paths, mirroring the existing hermes strategy-migration pattern — including the load-bearing explicit rollingUpdate: null that clears the k8s-defaulted block the API would otherwise reject the type flip over. Patching spec.strategy doesn't touch the pod template, so the pin never rolls the signer itself. Helm leaves it alone on later upgrades because the chart never renders the field.

Upstream follow-up: add strategy/updateStrategy support to the remote-signer chart in ObolNetwork/helm-charts, then delete this patch.

Reviewed and deliberately unchanged

• serviceoffer-controller — flagged in the review as "RollingUpdate without leader election", but the controller already has Lease-based leader election (cmd/serviceoffer-controller/main.go:84, on by default), so its surge is safe: the new pod blocks on the Lease until the old leader releases it. No change; the x402.yaml comment already documents this.
• Frontend readiness probe (cosmetic local-UI 502s during rollouts) — left for a UI-side change in obol-stack-front-end/chart.

Tests

• internal/embed/embed_pdb_test.go — table-driven pins for both PDBs (namespace, minAvailable, exact selector — extra labels would silently match no pods) + cloudflared PDB template gating.
• internal/agentruntime/signer_test.go — patch-args shape incl. the explicit rollingUpdate: null.

Live validation (multi-service k3d cluster from the #707 run)

• obol stack up created both PDBs: erpc/erpc and x402/x402-verifier, MIN AVAILABLE 1, ALLOWED DISRUPTIONS 0.
• The stack-up agent re-sync flipped remote-signer from RollingUpdate to {"type":"Recreate"}without rolling the pod (pod age preserved), confirming the patch is non-disruptive.
• helm lint clean; full go test green on touched packages.

https://claude.ai/code/session_01YLuXwN1A4tG3xmEMKvAzeD

[OisinKyne]

Upstream follow-up: add strategy/updateStrategy support to the remote-signer chart in ObolNetwork/helm-charts, then delete this patch.

We control this chart, lets just fix this there rather than a post-install hook? I assume that would shorten install time

[bussyjd]

Closing as superseded by integration/v0.13.0-rc0 @ a29c826 (rolled up in #716 / release v0.13.0-rc0). All commits of this PR are contained in the RC tip (verified by ancestry/patch-id audit).

[bussyjd]

Replica-stance hardening: PDBs for payment/RPC gates + remote-signer Recreate

Follow-up to #707, from the replica-stance review of the remaining services. Targets multi-node clusters, where drains and RWO volume attach are real.

Stacked on #707 (fix/litellm-zero-downtime) — merge that first; this PR's own diff is the last commit.

Changes

PodDisruptionBudgets (minAvailable: 1)

• x402-verifier (base/templates/x402.yaml) — the ForwardAuth gate for every paid route. On multi-node, a kubectl drain would silently evict the only pod and 5xx all /services/* traffic; the PDB makes that require explicit operator intent (delete the pod), same stance as the existing litellm PDB.
• eRPC (base/templates/erpc.yaml) — the RPC front door for agents and the frontend. Lives in the base chart because the ethereum/erpc chart (0.0.4) has no PDB template; its podDisruptionBudget value in our values file is unwired. Selector pinned to the chart's live selectorLabels.
• cloudflared — already ships a PDB (gated on active tunnel + >1 replica); now pinned by a test so it doesn't regress.

Rollouts are unaffected in all cases: at 1 replica the default RollingUpdate rounds to maxUnavailable 0 / maxSurge 1, and both verifier (/readyz gates on config + routes loaded) and eRPC surge gaplessly behind readiness probes.

remote-signer → strategy: Recreate

The obol/remote-signer chart (0.3.3) cannot express spec.strategy, so signer deployments defaulted to RollingUpdate — a singleton over a ReadWriteOnce keystore PVC. On multi-node, a surge pod can land on another node and wedge forever on the volume attach; on any cluster it briefly double-runs the signer over the same keystore. Same doctrine as hermes and x402-buyer: RWO-backed singletons run Recreate.

Implemented as a post-helmfile-sync kubectl patch (agentruntime.EnforceRemoteSignerRecreate) from both the hermes and openclaw sync paths, mirroring the existing hermes strategy-migration pattern — including the load-bearing explicit rollingUpdate: null that clears the k8s-defaulted block the API would otherwise reject the type flip over. Patching spec.strategy doesn't touch the pod template, so the pin never rolls the signer itself. Helm leaves it alone on later upgrades because the chart never renders the field.

Upstream follow-up: add strategy/updateStrategy support to the remote-signer chart in ObolNetwork/helm-charts, then delete this patch.

Reviewed and deliberately unchanged

• serviceoffer-controller — flagged in the review as "RollingUpdate without leader election", but the controller already has Lease-based leader election (cmd/serviceoffer-controller/main.go:84, on by default), so its surge is safe: the new pod blocks on the Lease until the old leader releases it. No change; the x402.yaml comment already documents this.
• Frontend readiness probe (cosmetic local-UI 502s during rollouts) — left for a UI-side change in obol-stack-front-end/chart.

Tests

• internal/embed/embed_pdb_test.go — table-driven pins for both PDBs (namespace, minAvailable, exact selector — extra labels would silently match no pods) + cloudflared PDB template gating.
• internal/agentruntime/signer_test.go — patch-args shape incl. the explicit rollingUpdate: null.

Live validation (multi-service k3d cluster from the #707 run)

• obol stack up created both PDBs: erpc/erpc and x402/x402-verifier, MIN AVAILABLE 1, ALLOWED DISRUPTIONS 0.
• The stack-up agent re-sync flipped remote-signer from RollingUpdate to {"type":"Recreate"}without rolling the pod (pod age preserved), confirming the patch is non-disruptive.
• helm lint clean; full go test green on touched packages.

https://claude.ai/code/session_01YLuXwN1A4tG3xmEMKvAzeD

[OisinKyne]

Upstream follow-up: add strategy/updateStrategy support to the remote-signer chart in ObolNetwork/helm-charts, then delete this patch.

We control this chart, lets just fix this there rather than a post-install hook? I assume that would shorten install time

[bussyjd]

Closing as superseded by integration/v0.13.0-rc0 @ a29c826 (rolled up in #716 / release v0.13.0-rc0). All commits of this PR are contained in the RC tip (verified by ancestry/patch-id audit).

[bussyjd]

Replica-stance hardening: PDBs for payment/RPC gates + remote-signer Recreate

Follow-up to #707, from the replica-stance review of the remaining services. Targets multi-node clusters, where drains and RWO volume attach are real.

Stacked on #707 (fix/litellm-zero-downtime) — merge that first; this PR's own diff is the last commit.

Changes

PodDisruptionBudgets (minAvailable: 1)

• x402-verifier (base/templates/x402.yaml) — the ForwardAuth gate for every paid route. On multi-node, a kubectl drain would silently evict the only pod and 5xx all /services/* traffic; the PDB makes that require explicit operator intent (delete the pod), same stance as the existing litellm PDB.
• eRPC (base/templates/erpc.yaml) — the RPC front door for agents and the frontend. Lives in the base chart because the ethereum/erpc chart (0.0.4) has no PDB template; its podDisruptionBudget value in our values file is unwired. Selector pinned to the chart's live selectorLabels.
• cloudflared — already ships a PDB (gated on active tunnel + >1 replica); now pinned by a test so it doesn't regress.

Rollouts are unaffected in all cases: at 1 replica the default RollingUpdate rounds to maxUnavailable 0 / maxSurge 1, and both verifier (/readyz gates on config + routes loaded) and eRPC surge gaplessly behind readiness probes.

remote-signer → strategy: Recreate

The obol/remote-signer chart (0.3.3) cannot express spec.strategy, so signer deployments defaulted to RollingUpdate — a singleton over a ReadWriteOnce keystore PVC. On multi-node, a surge pod can land on another node and wedge forever on the volume attach; on any cluster it briefly double-runs the signer over the same keystore. Same doctrine as hermes and x402-buyer: RWO-backed singletons run Recreate.

Implemented as a post-helmfile-sync kubectl patch (agentruntime.EnforceRemoteSignerRecreate) from both the hermes and openclaw sync paths, mirroring the existing hermes strategy-migration pattern — including the load-bearing explicit rollingUpdate: null that clears the k8s-defaulted block the API would otherwise reject the type flip over. Patching spec.strategy doesn't touch the pod template, so the pin never rolls the signer itself. Helm leaves it alone on later upgrades because the chart never renders the field.

Upstream follow-up: add strategy/updateStrategy support to the remote-signer chart in ObolNetwork/helm-charts, then delete this patch.

Reviewed and deliberately unchanged

• serviceoffer-controller — flagged in the review as "RollingUpdate without leader election", but the controller already has Lease-based leader election (cmd/serviceoffer-controller/main.go:84, on by default), so its surge is safe: the new pod blocks on the Lease until the old leader releases it. No change; the x402.yaml comment already documents this.
• Frontend readiness probe (cosmetic local-UI 502s during rollouts) — left for a UI-side change in obol-stack-front-end/chart.

Tests

• internal/embed/embed_pdb_test.go — table-driven pins for both PDBs (namespace, minAvailable, exact selector — extra labels would silently match no pods) + cloudflared PDB template gating.
• internal/agentruntime/signer_test.go — patch-args shape incl. the explicit rollingUpdate: null.

Live validation (multi-service k3d cluster from the #707 run)

• obol stack up created both PDBs: erpc/erpc and x402/x402-verifier, MIN AVAILABLE 1, ALLOWED DISRUPTIONS 0.
• The stack-up agent re-sync flipped remote-signer from RollingUpdate to {"type":"Recreate"}without rolling the pod (pod age preserved), confirming the patch is non-disruptive.
• helm lint clean; full go test green on touched packages.

https://claude.ai/code/session_01YLuXwN1A4tG3xmEMKvAzeD

[OisinKyne]

Upstream follow-up: add strategy/updateStrategy support to the remote-signer chart in ObolNetwork/helm-charts, then delete this patch.

We control this chart, lets just fix this there rather than a post-install hook? I assume that would shorten install time

[bussyjd]

Closing as superseded by integration/v0.13.0-rc0 @ a29c826 (rolled up in #716 / release v0.13.0-rc0). All commits of this PR are contained in the RC tip (verified by ancestry/patch-id audit).

[bussyjd]

Replica-stance hardening: PDBs for payment/RPC gates + remote-signer Recreate

Follow-up to #707, from the replica-stance review of the remaining services. Targets multi-node clusters, where drains and RWO volume attach are real.

Stacked on #707 (fix/litellm-zero-downtime) — merge that first; this PR's own diff is the last commit.

Changes

PodDisruptionBudgets (minAvailable: 1)

• x402-verifier (base/templates/x402.yaml) — the ForwardAuth gate for every paid route. On multi-node, a kubectl drain would silently evict the only pod and 5xx all /services/* traffic; the PDB makes that require explicit operator intent (delete the pod), same stance as the existing litellm PDB.
• eRPC (base/templates/erpc.yaml) — the RPC front door for agents and the frontend. Lives in the base chart because the ethereum/erpc chart (0.0.4) has no PDB template; its podDisruptionBudget value in our values file is unwired. Selector pinned to the chart's live selectorLabels.
• cloudflared — already ships a PDB (gated on active tunnel + >1 replica); now pinned by a test so it doesn't regress.

Rollouts are unaffected in all cases: at 1 replica the default RollingUpdate rounds to maxUnavailable 0 / maxSurge 1, and both verifier (/readyz gates on config + routes loaded) and eRPC surge gaplessly behind readiness probes.

remote-signer → strategy: Recreate

The obol/remote-signer chart (0.3.3) cannot express spec.strategy, so signer deployments defaulted to RollingUpdate — a singleton over a ReadWriteOnce keystore PVC. On multi-node, a surge pod can land on another node and wedge forever on the volume attach; on any cluster it briefly double-runs the signer over the same keystore. Same doctrine as hermes and x402-buyer: RWO-backed singletons run Recreate.

Implemented as a post-helmfile-sync kubectl patch (agentruntime.EnforceRemoteSignerRecreate) from both the hermes and openclaw sync paths, mirroring the existing hermes strategy-migration pattern — including the load-bearing explicit rollingUpdate: null that clears the k8s-defaulted block the API would otherwise reject the type flip over. Patching spec.strategy doesn't touch the pod template, so the pin never rolls the signer itself. Helm leaves it alone on later upgrades because the chart never renders the field.

Upstream follow-up: add strategy/updateStrategy support to the remote-signer chart in ObolNetwork/helm-charts, then delete this patch.

Reviewed and deliberately unchanged

• serviceoffer-controller — flagged in the review as "RollingUpdate without leader election", but the controller already has Lease-based leader election (cmd/serviceoffer-controller/main.go:84, on by default), so its surge is safe: the new pod blocks on the Lease until the old leader releases it. No change; the x402.yaml comment already documents this.
• Frontend readiness probe (cosmetic local-UI 502s during rollouts) — left for a UI-side change in obol-stack-front-end/chart.

Tests

• internal/embed/embed_pdb_test.go — table-driven pins for both PDBs (namespace, minAvailable, exact selector — extra labels would silently match no pods) + cloudflared PDB template gating.
• internal/agentruntime/signer_test.go — patch-args shape incl. the explicit rollingUpdate: null.

Live validation (multi-service k3d cluster from the #707 run)

• obol stack up created both PDBs: erpc/erpc and x402/x402-verifier, MIN AVAILABLE 1, ALLOWED DISRUPTIONS 0.
• The stack-up agent re-sync flipped remote-signer from RollingUpdate to {"type":"Recreate"}without rolling the pod (pod age preserved), confirming the patch is non-disruptive.
• helm lint clean; full go test green on touched packages.

https://claude.ai/code/session_01YLuXwN1A4tG3xmEMKvAzeD

[OisinKyne]

Upstream follow-up: add strategy/updateStrategy support to the remote-signer chart in ObolNetwork/helm-charts, then delete this patch.

We control this chart, lets just fix this there rather than a post-install hook? I assume that would shorten install time

[bussyjd]

Closing as superseded by integration/v0.13.0-rc0 @ a29c826 (rolled up in #716 / release v0.13.0-rc0). All commits of this PR are contained in the RC tip (verified by ancestry/patch-id audit).

[bussyjd]

Replica-stance hardening: PDBs for payment/RPC gates + remote-signer Recreate

Follow-up to #707, from the replica-stance review of the remaining services. Targets multi-node clusters, where drains and RWO volume attach are real.

Stacked on #707 (fix/litellm-zero-downtime) — merge that first; this PR's own diff is the last commit.

Changes

PodDisruptionBudgets (minAvailable: 1)

• x402-verifier (base/templates/x402.yaml) — the ForwardAuth gate for every paid route. On multi-node, a kubectl drain would silently evict the only pod and 5xx all /services/* traffic; the PDB makes that require explicit operator intent (delete the pod), same stance as the existing litellm PDB.
• eRPC (base/templates/erpc.yaml) — the RPC front door for agents and the frontend. Lives in the base chart because the ethereum/erpc chart (0.0.4) has no PDB template; its podDisruptionBudget value in our values file is unwired. Selector pinned to the chart's live selectorLabels.
• cloudflared — already ships a PDB (gated on active tunnel + >1 replica); now pinned by a test so it doesn't regress.

Rollouts are unaffected in all cases: at 1 replica the default RollingUpdate rounds to maxUnavailable 0 / maxSurge 1, and both verifier (/readyz gates on config + routes loaded) and eRPC surge gaplessly behind readiness probes.

remote-signer → strategy: Recreate

The obol/remote-signer chart (0.3.3) cannot express spec.strategy, so signer deployments defaulted to RollingUpdate — a singleton over a ReadWriteOnce keystore PVC. On multi-node, a surge pod can land on another node and wedge forever on the volume attach; on any cluster it briefly double-runs the signer over the same keystore. Same doctrine as hermes and x402-buyer: RWO-backed singletons run Recreate.

Implemented as a post-helmfile-sync kubectl patch (agentruntime.EnforceRemoteSignerRecreate) from both the hermes and openclaw sync paths, mirroring the existing hermes strategy-migration pattern — including the load-bearing explicit rollingUpdate: null that clears the k8s-defaulted block the API would otherwise reject the type flip over. Patching spec.strategy doesn't touch the pod template, so the pin never rolls the signer itself. Helm leaves it alone on later upgrades because the chart never renders the field.

Upstream follow-up: add strategy/updateStrategy support to the remote-signer chart in ObolNetwork/helm-charts, then delete this patch.

Reviewed and deliberately unchanged

• serviceoffer-controller — flagged in the review as "RollingUpdate without leader election", but the controller already has Lease-based leader election (cmd/serviceoffer-controller/main.go:84, on by default), so its surge is safe: the new pod blocks on the Lease until the old leader releases it. No change; the x402.yaml comment already documents this.
• Frontend readiness probe (cosmetic local-UI 502s during rollouts) — left for a UI-side change in obol-stack-front-end/chart.

Tests

• internal/embed/embed_pdb_test.go — table-driven pins for both PDBs (namespace, minAvailable, exact selector — extra labels would silently match no pods) + cloudflared PDB template gating.
• internal/agentruntime/signer_test.go — patch-args shape incl. the explicit rollingUpdate: null.

Live validation (multi-service k3d cluster from the #707 run)

• obol stack up created both PDBs: erpc/erpc and x402/x402-verifier, MIN AVAILABLE 1, ALLOWED DISRUPTIONS 0.
• The stack-up agent re-sync flipped remote-signer from RollingUpdate to {"type":"Recreate"}without rolling the pod (pod age preserved), confirming the patch is non-disruptive.
• helm lint clean; full go test green on touched packages.

https://claude.ai/code/session_01YLuXwN1A4tG3xmEMKvAzeD

[OisinKyne]

Upstream follow-up: add strategy/updateStrategy support to the remote-signer chart in ObolNetwork/helm-charts, then delete this patch.

We control this chart, lets just fix this there rather than a post-install hook? I assume that would shorten install time

[bussyjd]

Closing as superseded by integration/v0.13.0-rc0 @ a29c826 (rolled up in #716 / release v0.13.0-rc0). All commits of this PR are contained in the RC tip (verified by ancestry/patch-id audit).

[bussyjd]

Replica-stance hardening: PDBs for payment/RPC gates + remote-signer Recreate

Follow-up to #707, from the replica-stance review of the remaining services. Targets multi-node clusters, where drains and RWO volume attach are real.

Stacked on #707 (fix/litellm-zero-downtime) — merge that first; this PR's own diff is the last commit.

Changes

PodDisruptionBudgets (minAvailable: 1)

• x402-verifier (base/templates/x402.yaml) — the ForwardAuth gate for every paid route. On multi-node, a kubectl drain would silently evict the only pod and 5xx all /services/* traffic; the PDB makes that require explicit operator intent (delete the pod), same stance as the existing litellm PDB.
• eRPC (base/templates/erpc.yaml) — the RPC front door for agents and the frontend. Lives in the base chart because the ethereum/erpc chart (0.0.4) has no PDB template; its podDisruptionBudget value in our values file is unwired. Selector pinned to the chart's live selectorLabels.
• cloudflared — already ships a PDB (gated on active tunnel + >1 replica); now pinned by a test so it doesn't regress.

Rollouts are unaffected in all cases: at 1 replica the default RollingUpdate rounds to maxUnavailable 0 / maxSurge 1, and both verifier (/readyz gates on config + routes loaded) and eRPC surge gaplessly behind readiness probes.

remote-signer → strategy: Recreate

The obol/remote-signer chart (0.3.3) cannot express spec.strategy, so signer deployments defaulted to RollingUpdate — a singleton over a ReadWriteOnce keystore PVC. On multi-node, a surge pod can land on another node and wedge forever on the volume attach; on any cluster it briefly double-runs the signer over the same keystore. Same doctrine as hermes and x402-buyer: RWO-backed singletons run Recreate.

Implemented as a post-helmfile-sync kubectl patch (agentruntime.EnforceRemoteSignerRecreate) from both the hermes and openclaw sync paths, mirroring the existing hermes strategy-migration pattern — including the load-bearing explicit rollingUpdate: null that clears the k8s-defaulted block the API would otherwise reject the type flip over. Patching spec.strategy doesn't touch the pod template, so the pin never rolls the signer itself. Helm leaves it alone on later upgrades because the chart never renders the field.

Upstream follow-up: add strategy/updateStrategy support to the remote-signer chart in ObolNetwork/helm-charts, then delete this patch.

Reviewed and deliberately unchanged

• serviceoffer-controller — flagged in the review as "RollingUpdate without leader election", but the controller already has Lease-based leader election (cmd/serviceoffer-controller/main.go:84, on by default), so its surge is safe: the new pod blocks on the Lease until the old leader releases it. No change; the x402.yaml comment already documents this.
• Frontend readiness probe (cosmetic local-UI 502s during rollouts) — left for a UI-side change in obol-stack-front-end/chart.

Tests

• internal/embed/embed_pdb_test.go — table-driven pins for both PDBs (namespace, minAvailable, exact selector — extra labels would silently match no pods) + cloudflared PDB template gating.
• internal/agentruntime/signer_test.go — patch-args shape incl. the explicit rollingUpdate: null.

Live validation (multi-service k3d cluster from the #707 run)

• obol stack up created both PDBs: erpc/erpc and x402/x402-verifier, MIN AVAILABLE 1, ALLOWED DISRUPTIONS 0.
• The stack-up agent re-sync flipped remote-signer from RollingUpdate to {"type":"Recreate"}without rolling the pod (pod age preserved), confirming the patch is non-disruptive.
• helm lint clean; full go test green on touched packages.

https://claude.ai/code/session_01YLuXwN1A4tG3xmEMKvAzeD

[OisinKyne]

Upstream follow-up: add strategy/updateStrategy support to the remote-signer chart in ObolNetwork/helm-charts, then delete this patch.

We control this chart, lets just fix this there rather than a post-install hook? I assume that would shorten install time

[bussyjd]

Closing as superseded by integration/v0.13.0-rc0 @ a29c826 (rolled up in #716 / release v0.13.0-rc0). All commits of this PR are contained in the RC tip (verified by ancestry/patch-id audit).

[bussyjd]

Replica-stance hardening: PDBs for payment/RPC gates + remote-signer Recreate

Follow-up to #707, from the replica-stance review of the remaining services. Targets multi-node clusters, where drains and RWO volume attach are real.

Stacked on #707 (fix/litellm-zero-downtime) — merge that first; this PR's own diff is the last commit.

Changes

PodDisruptionBudgets (minAvailable: 1)

• x402-verifier (base/templates/x402.yaml) — the ForwardAuth gate for every paid route. On multi-node, a kubectl drain would silently evict the only pod and 5xx all /services/* traffic; the PDB makes that require explicit operator intent (delete the pod), same stance as the existing litellm PDB.
• eRPC (base/templates/erpc.yaml) — the RPC front door for agents and the frontend. Lives in the base chart because the ethereum/erpc chart (0.0.4) has no PDB template; its podDisruptionBudget value in our values file is unwired. Selector pinned to the chart's live selectorLabels.
• cloudflared — already ships a PDB (gated on active tunnel + >1 replica); now pinned by a test so it doesn't regress.

Rollouts are unaffected in all cases: at 1 replica the default RollingUpdate rounds to maxUnavailable 0 / maxSurge 1, and both verifier (/readyz gates on config + routes loaded) and eRPC surge gaplessly behind readiness probes.

remote-signer → strategy: Recreate

The obol/remote-signer chart (0.3.3) cannot express spec.strategy, so signer deployments defaulted to RollingUpdate — a singleton over a ReadWriteOnce keystore PVC. On multi-node, a surge pod can land on another node and wedge forever on the volume attach; on any cluster it briefly double-runs the signer over the same keystore. Same doctrine as hermes and x402-buyer: RWO-backed singletons run Recreate.

Implemented as a post-helmfile-sync kubectl patch (agentruntime.EnforceRemoteSignerRecreate) from both the hermes and openclaw sync paths, mirroring the existing hermes strategy-migration pattern — including the load-bearing explicit rollingUpdate: null that clears the k8s-defaulted block the API would otherwise reject the type flip over. Patching spec.strategy doesn't touch the pod template, so the pin never rolls the signer itself. Helm leaves it alone on later upgrades because the chart never renders the field.

Upstream follow-up: add strategy/updateStrategy support to the remote-signer chart in ObolNetwork/helm-charts, then delete this patch.

Reviewed and deliberately unchanged

• serviceoffer-controller — flagged in the review as "RollingUpdate without leader election", but the controller already has Lease-based leader election (cmd/serviceoffer-controller/main.go:84, on by default), so its surge is safe: the new pod blocks on the Lease until the old leader releases it. No change; the x402.yaml comment already documents this.
• Frontend readiness probe (cosmetic local-UI 502s during rollouts) — left for a UI-side change in obol-stack-front-end/chart.

Tests

• internal/embed/embed_pdb_test.go — table-driven pins for both PDBs (namespace, minAvailable, exact selector — extra labels would silently match no pods) + cloudflared PDB template gating.
• internal/agentruntime/signer_test.go — patch-args shape incl. the explicit rollingUpdate: null.

Live validation (multi-service k3d cluster from the #707 run)

• obol stack up created both PDBs: erpc/erpc and x402/x402-verifier, MIN AVAILABLE 1, ALLOWED DISRUPTIONS 0.
• The stack-up agent re-sync flipped remote-signer from RollingUpdate to {"type":"Recreate"}without rolling the pod (pod age preserved), confirming the patch is non-disruptive.
• helm lint clean; full go test green on touched packages.

https://claude.ai/code/session_01YLuXwN1A4tG3xmEMKvAzeD

[OisinKyne]

Upstream follow-up: add strategy/updateStrategy support to the remote-signer chart in ObolNetwork/helm-charts, then delete this patch.

We control this chart, lets just fix this there rather than a post-install hook? I assume that would shorten install time

[bussyjd]

Closing as superseded by integration/v0.13.0-rc0 @ a29c826 (rolled up in #716 / release v0.13.0-rc0). All commits of this PR are contained in the RC tip (verified by ancestry/patch-id audit).