Push gateway-owned desired state to supervisors

[pimlock]

Summary

Replace steady-state supervisor polling for gateway-owned desired state with push delivery. The gateway publishes effective config and policy, provider environment, and inference route snapshots through a transport-neutral control-delivery abstraction, using the existing reverse ConnectSupervisor session as the first transport.

The persistent supervisor session and its SSH/exec relay substrate landed in PR #867. This issue builds on that foundation. It does not introduce the session itself or move every supervisor-triggered operation onto one shared stream.

Resolved design

• Push gateway-owned desired state: effective config and policy, provider environment, and inference routes.
• Keep ReportPolicyStatus and SubmitPolicyAnalysis as unary supervisor-initiated RPCs.
• Keep PushSandboxLogs as a supervisor-initiated client-streaming RPC.
• Keep one initial unary bootstrap fetch for compatibility. Remove steady-state polling after push capability negotiation succeeds.
• Use transport-neutral SupervisorControlDelivery and SupervisorControlResult messages with shared publisher and handler logic.
• Use capability negotiation so mixed-version peers safely retain legacy bootstrap and polling behavior.
• Keep DB-backed desired state as the source of truth, with immediate local notification where practical and bounded reconciliation as the correctness backstop.
• Add only the supervisor-to-gateway acknowledgements needed for pushed desired-state revisions.
• Leave an optional Kubernetes gateway-initiated supervisor control transport as a follow-up that can reuse the same logical messages and processing paths.

Scope

• Define transport-neutral desired-state delivery and result messages for config and policy, provider environment, and inference routes.
• Negotiate push capabilities through the supervisor session handshake.
• Publish current snapshots after session acceptance and newer snapshots when desired state changes.
• Apply snapshots idempotently by revision and acknowledge received, applied, or failed outcomes.
• Separate one-time bootstrap from steady-state polling so negotiated push disables only the polling loop.
• Preserve safe fallback for old gateways, old supervisors, and operational rollback.
• Document ordering, acknowledgement, replay, failure, and reconciliation semantics.

Non-goals

• Moving ReportPolicyStatus, SubmitPolicyAnalysis, or PushSandboxLogs onto ConnectSupervisor.
• Building generic request and response multiplexing over the supervisor session.
• Replacing SSH, exec, or RelayStream behavior.
• Implementing a production gateway-initiated supervisor gRPC server for Kubernetes.
• Removing the initial unary bootstrap fetch.
• Fully solving multi-replica ownership beyond the existing session ownership substrate and bounded DB reconciliation.

Acceptance criteria

• New peers negotiate desired-state push and stop steady-state config, provider-environment, and inference polling.
• Current snapshots are delivered after session acceptance, and newer revisions are coalesced, retried, and acknowledged safely.
• Duplicate, out-of-order, failed, and reconnect-replayed deliveries preserve last-known-good state.
• Old or capability-incompatible peers retain legacy bootstrap and polling behavior.
• Supervisor-triggered status, analysis, and log RPCs remain independent and tested.
• Cross-gateway updates eventually reach the session-owning gateway through bounded reconciliation.
• The architecture documents the desired-state boundary, retained RPCs, compatibility behavior, and fallback state machine.

References

• Session and relay foundation: PR feat(server,sandbox): supervisor-initiated SSH connect and exec over gRPC-multiplexed relay #867
• Detailed implementation plan and rationale: issue comment
• Supervisor-session token expiry: feat(supervisor): enforce supervisor session token expiry with proactive reconnect #1954
• Legacy RPC cleanup after migration: refactor(supervisor): remove legacy supervisor control RPCs after session migration #1955

[pimlock]

🏗️ build-plan

Implementation Plan

Issue type: feat
Complexity: High
Confidence: Medium - the desired-state boundary is now clear, but the change still crosses proto, gateway session ownership, sandbox runtime, sidecar propagation, inference routing, rollout compatibility, and cross-gateway reconciliation.

Summary

Replace steady-state supervisor polling for gateway-owned desired state with push delivery. The gateway will publish config, provider environment, and inference snapshots through a transport-neutral control-delivery abstraction, with the existing reverse ConnectSupervisor session as the first transport and a future gateway-initiated supervisor RPC as an extension point.

Keep supervisor-triggered traffic on its existing RPC shapes. ReportPolicyStatus and SubmitPolicyAnalysis remain unary RPCs, while PushSandboxLogs remains client-streaming. This avoids rebuilding request/response correlation, deadlines, cancellation, retry ambiguity, deduplication, and independent backpressure on the shared session.

Resolved Decisions

• The goal of Push gateway-owned desired state to supervisors #1731 is to remove steady-state polling, not to move every supervisor-to-gateway operation onto one stream.
• Push gateway-owned desired state: effective config, policy, settings, provider environment, and inference routes.
• Keep ReportPolicyStatus, SubmitPolicyAnalysis, and PushSandboxLogs on their existing supervisor-initiated RPCs.
• Add only the supervisor-to-gateway acknowledgements required for pushed desired-state revisions.
• Keep one initial unary bootstrap fetch for compatibility. A one-time startup fetch is not steady-state polling.
• Use transport-neutral SupervisorControlDelivery and SupervisorControlResult messages and shared processing logic. Do not couple the logical protocol to ConnectSupervisor.
• Implement reverse-session delivery in Push gateway-owned desired state to supervisors #1731. Leave Kubernetes gateway-initiated direct RPC delivery as a follow-up that can reuse the same messages, publisher, handler, and acknowledgement semantics.
• Use capability offer/selection so mixed-version gateways and supervisors fall back safely. Missing negotiated support means the supervisor continues its legacy bootstrap and polling behavior.
• Add a gateway rollout mode such as auto | legacy, with auto selecting mutually supported push capabilities and legacy providing an operational rollback.
• Use DB-backed desired state as the source of truth. User writes return after DB commit, not after supervisor application.
• Use immediate local notification where practical plus bounded, jittered owner-side reconciliation as the correctness backstop for missed and cross-gateway updates.
• Keep current accepted-stream authentication semantics in v1. Token-expiry reconnect hardening remains tracked in feat(supervisor): enforce supervisor session token expiry with proactive reconnect #1954.

Message and Processing Model

• SupervisorControlDelivery: carries a delivery ID, sandbox ID, and one desired-state payload for config, provider environment, or inference routes.
• SupervisorControlResult: carries the delivery ID, applied revision, and received, applied, or failed status with an optional error.
• GatewayMessage embeds SupervisorControlDelivery for reverse-session delivery.
• SupervisorMessage embeds SupervisorControlResult for reverse-session acknowledgement.
• A shared supervisor handler owns revision comparison, idempotent application, sidecar propagation, and acknowledgement generation without knowing which transport delivered the message.
• A shared gateway desired-state publisher owns snapshot generation, coalescing, retry policy, and delivery through a transport interface. The reverse-session adapter is implemented now; a future direct-RPC adapter can call the same publisher and handler.

Scope

• proto/openshell.proto: add desired-state push capability negotiation, transport-neutral delivery/result messages, and session envelope variants. Account for any SupervisorHello fields introduced by feat(kubernetes): support HA gateway rebalancing #1868 before assigning field numbers.
• crates/openshell-server/src/supervisor_session.rs: negotiate push support, implement the reverse-session transport adapter, send current and updated snapshots, and process desired-state acknowledgements.
• crates/openshell-server/src/grpc/policy.rs: extract reusable config and provider-environment snapshot builders while leaving policy status, log ingestion, and policy analysis RPC handlers intact.
• crates/openshell-server/src/inference.rs: expose reusable inference bundle resolution for desired-state publishing.
• crates/openshell-supervisor-process/src/supervisor_session.rs: receive desired-state deliveries, dispatch them to shared processing, return acknowledgements, and preserve legacy behavior when push is not negotiated.
• crates/openshell-sandbox/src/lib.rs: consume pushed config and provider snapshots through the existing runtime reconciliation paths, propagate updates through sidecar control, and stop steady-state polling after push is negotiated.
• crates/openshell-supervisor-network/src/inference_routes.rs: consume pushed inference snapshots and stop steady-state polling after push is negotiated.
• crates/openshell-core/src/grpc_client.rs: retain one-time bootstrap and legacy fallback calls; separate bootstrap from steady-state polling so negotiated push can disable only the polling loop.
• architecture/sandbox.md: document the desired-state push boundary, retained supervisor-triggered RPCs, compatibility fallback, acknowledgement semantics, and future direct-transport extension point.

Explicitly Out of Scope

• Moving ReportPolicyStatus, SubmitPolicyAnalysis, or PushSandboxLogs onto ConnectSupervisor.
• Building generic request/response multiplexing over the supervisor session.
• Implementing a production gateway-initiated supervisor gRPC server for Kubernetes.
• Supervisor endpoint discovery, inbound supervisor authentication, server certificate provisioning, or Kubernetes NetworkPolicy changes for a direct transport.
• Removing the initial unary bootstrap fetch.
• Replacing SSH, exec, or RelayStream behavior.
• Fully solving multi-replica ownership beyond using the session ownership substrate and bounded DB reconciliation.

Implementation Steps

1. Reconcile the implementation with the current main branch and PR feat(kubernetes): support HA gateway rebalancing #1868, especially SupervisorHello field allocation and supervisor-session ownership.
2. Add transport-neutral desired-state delivery/result messages and a granular push capability negotiated through SupervisorHello and SessionAccepted.
3. Separate one-time bootstrap from steady-state polling in the supervisor runtime so old peers retain current behavior while negotiated peers can disable polling.
4. Extract reusable config, provider-environment, and inference snapshot builders from the existing RPC handlers.
5. Introduce a gateway desired-state publisher and transport interface, with a fake transport for unit tests and a reverse-session adapter for production.
6. Introduce one shared supervisor desired-state handler used by reverse-session delivery and suitable for a future direct-RPC entry point.
7. After SessionAccepted, send the current desired-state snapshots and process revision-keyed received, applied, and failed acknowledgements.
8. Add update notifications plus bounded, jittered reconciliation so the session-owning gateway eventually sends newer DB-backed revisions, including updates committed through another replica.
9. Stop config/provider/inference steady-state polling only after push capability negotiation succeeds. Resume legacy polling if the negotiated push path becomes unavailable according to the documented fallback policy.
10. Leave status, analysis, and log call sites and handlers on their existing RPCs.

Test Plan

• Unit tests: message conversion, capability negotiation, bootstrap-versus-polling state transitions, shared snapshot builders, revision comparison, idempotent application, acknowledgement handling, coalescing, retry decisions, and fake transport coverage.
• Integration tests: SessionAccepted ordering, connect-time snapshot delivery, skipped delivery for current revisions, duplicate and out-of-order snapshots, failed apply, reconnect replay, fallback when capability or rollout mode is absent, and retained unary/client-streaming report behavior.
• Sidecar tests: config and provider updates delivered to the network sidecar are propagated through the existing sidecar control channel to the process supervisor.
• HA tests: an update committed through a non-owner gateway is eventually observed and pushed by the session owner through reconciliation.
• Compatibility tests: old gateway plus new supervisor uses legacy behavior; new gateway plus old supervisor sends no push messages; new peers use negotiated push. Verify that immutable version-pinned supervisor selection remains the default while mutable overrides do not break protocol fallback.
• E2E tests: run the relevant sandbox E2E paths with desired-state push enabled and assert live config/policy/provider/inference updates without steady-state supervisor polling. Keep a legacy fallback lane where practical.

Follow-up Issues

• feat(supervisor): enforce supervisor session token expiry with proactive reconnect #1954 - enforce supervisor session token expiry with proactive reconnect.
• refactor(supervisor): remove legacy supervisor control RPCs after session migration #1955 - narrow legacy cleanup to RPCs made redundant by desired-state push. Do not remove retained status, analysis, or log RPCs.
• Create a follow-up for an optional Kubernetes gateway-initiated supervisor control transport after the shared message, publisher, and handler abstractions are proven. That follow-up should cover endpoint discovery, supervisor server identity, mTLS, readiness, NetworkPolicy, HA routing, and direct-transport E2E coverage.

Risks and Open Questions

• PR feat(kubernetes): support HA gateway rebalancing #1868 changes supervisor session ownership and SupervisorHello; re-check its live head before implementation and avoid conflicting field numbers or duplicate ownership machinery.
• The fallback transition after a negotiated push stream fails must avoid both an update gap and concurrent polling. Model this as an explicit state machine.
• Provider environment applied semantics must be precise: applied means staged or cached for supervisor use, not injected into processes that are already running.
• Snapshot notification hooks can be incomplete as the set of mutations grows, so bounded reconciliation remains the correctness backstop.
• A future direct transport must not force Kubernetes endpoint discovery, inbound trust, or certificate concerns into the shared desired-state publisher or supervisor handler.
• This change does not touch process identity, /proc, binary execution, or inter-process visibility semantics, so no new SELinux or AppArmor compatibility impact is expected.

Documentation Impact

• Update architecture/sandbox.md with the desired-state push model, retained supervisor-triggered RPCs, transport boundary, compatibility behavior, fallback state machine, and acknowledgement semantics.
• Update docs/reference/gateway-config.mdx if session_control_mode or reconciliation settings are exposed through gateway TOML.
• Update published behavior documentation only where users or operators can observe the rollout mode, update latency, or fallback behavior.

---

Revision 2 - narrow scope to desired-state push, retain supervisor-triggered RPCs, and add a transport-neutral extension point
Revision 1 - initial feature-flagged session-control plan with auth and legacy-RPC cleanup split into follow-up issues