refactor(network): consolidate proxy egress pipeline

[johntmyers]

Summary

Refactor sandbox proxy egress so CONNECT and forward-proxy traffic share explicit destination authorization and relay primitives while preserving the existing user-facing network behavior. This establishes the RFC #2155 foundation without introducing the later DNS or arbitrary TCP features, and keeps the shared path compatible with the middleware framework from #2027.

User-facing behavior

• Existing CONNECT and HTTP proxy flows continue to support the same policy enforcement, metadata access, inference routing, credential injection, upgrades, and denial responses.
• Ambiguous endpoint policies are rejected at policy update time with a clear validation reason instead of becoming partially active.
• Rejected updates fail closed by default. Operators that prefer availability can set policy_validation_failure_mode = "retain_last_valid" under [openshell.gateway] in gateway.toml.
• OCSF events state why validation failed and whether the previous policy remains active.
• This PR does not enable DNS interception or arbitrary TCP egress; those remain follow-up work.

Related Issue

Related design: #2155
Related middleware implementation: #2027

Changes

• Introduce a shared egress pipeline with explicit destination intent, authorization, endpoint decisions, and relay preparation.
• Route CONNECT and non-CONNECT traffic through common HTTP, raw-stream, credential, and policy-generation boundaries.
• Preserve metadata-service handling, inference routing, credential injection, HTTP upgrades, raw relay, and current denial behavior.
• Pin relays to policy generations and close stale HTTP, upgraded, long-lived, and raw streams after policy activation changes.
• Reject overlapping endpoint selectors when connection or request-processing metadata is ambiguous.
• Configure rejected policy generation posture with [openshell.gateway] policy_validation_failure_mode in gateway.toml; default to fail_closed and allow the explicit retain_last_valid availability mode.
• Propagate the gateway TOML posture to sandbox supervisors as part of the effective config revision; individual sandboxes cannot override it.
• Emit OCSF configuration, finding, and denial rationale for validation failures and quarantine state.
• Add compatibility baselines, middleware coverage, ambiguity/recovery coverage, and shared-path E2E tests.
• Update gateway, Helm, architecture, and policy documentation for the new invariants and operator-visible failure posture.

Testing

• mise run pre-commit passes after rebasing onto current main
• Unit and integration tests added/updated
• Targeted config-revision, settings-poll, and proxy-denial tests pass
• Live Docker gateway proxy E2E target: 10 passed on the branch
• mise run helm:lint
• mise run helm:test (64 passed)

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs:https://nvidia-preview-pr-2373.docs.buildwithfern.com/openshell

[johntmyers]

/ok to test 0c67990

[github-actions]

Label test:e2e applied for 0c67990. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[johntmyers]

/ok to test 79d2123

[johntmyers]

/ok to test 1d568e3

[johntmyers]

/ok to test 92efd90

[johntmyers]

/ok to test 344a2eb

[johntmyers]

/ok to test 0f9a6bf

[pimlock]

[agent-review]

Agent infrastructure also needs a companion update: .agents/skills/generate-sandbox-policy/SKILL.md:442 still says all overlapping endpoint coverage causes OPA evaluation errors. This PR permits compatible overlaps and adds rejected-generation quarantine behavior, so the policy-generation, CLI, and cluster-debugging skills should reflect the new semantics.

Focused verification passed: 11 ambiguity tests, 4 relay-generation tests, and 4 validation-failure posture tests.

[johntmyers]

/ok to test 9a4dd3a

[johntmyers]

gator-agent

PR Review Status

Validation: Project-valid maintainer-authored implementation of the open RFC #2155 foundation for a shared proxy-egress pipeline.
Head SHA: 04c16b08142f9de132b86bc01f59baf7c41ea1c3
Base SHA: 596d729e3fbe4f828424cadbe0bf1b3c5b307ada
Merge base SHA: 596d729e3fbe4f828424cadbe0bf1b3c5b307ada
Patch ID: 2405a7f7baf6960ad6784158d7ad68b7b1efefc2
Gator payload: 2
Review mode: initial
Previous reviewed SHA: none

Thanks @pimlock. I checked the endpoint-glob, gateway-preflight, generation-pinning, WebSocket-classification, and agent-skill concerns you raised against the rebased head; those fixes are present. @krishicks, I also checked your JSON-RPC/MCP classification point; the current comparison now requires an exact protocol match before comparing the shared body limit.

Blocking findings:

• GATOR-04c16b08-01: Transient middleware-registry preparation failures incorrectly enter rejected-policy quarantine instead of preserving the last-known-good runtime.
• GATOR-04c16b08-02: Global activation of provider composition can persist before affected effective policies are ambiguity-checked.

Carried findings:

• None

Non-blocking suggestions:

• None

Docs: Fern, architecture, Helm, policy, and agent-skill documentation are updated.

Next state: gator:in-review

[johntmyers]

gator-agent

PR Review Status

Validation: Project-valid maintainer-authored implementation of the open RFC #2155 foundation for a shared proxy-egress pipeline.
Head SHA: 286d02719e3ecb7e0126aa5934c535c72b0567b3
Base SHA: 596d729e3fbe4f828424cadbe0bf1b3c5b307ada
Merge base SHA: 596d729e3fbe4f828424cadbe0bf1b3c5b307ada
Patch ID: 547e500de34791f5040b2f66f9c6bd8326e574e2
Gator payload: 2
Review mode: follow_up
Previous reviewed SHA: 04c16b08142f9de132b86bc01f59baf7c41ea1c3

Thanks @johntmyers. I checked the middleware-outage and provider-composition preflight fixes in the two-commit delta. GATOR-04c16b08-01 and GATOR-04c16b08-02 are resolved by the new classification/preflight code and regression coverage.

Blocking findings:

• GATOR-286d0271-01: Revision-only failure deduplication suppresses a later policy-rejection disposition when an initially unavailable middleware becomes reachable for the same candidate.

Carried findings:

• None

Docs: No new direct UX change in this remediation delta; the already-reviewed Fern and architecture documentation remains applicable.

Next state: gator:in-review

[johntmyers]

gator-agent

PR Review Status

Validation: Project-valid maintainer-authored implementation of the open RFC #2155 foundation for a shared proxy-egress pipeline.
Head SHA: 9f4447f4a9cb2e914bd33619bb1820ce431d4b4f
Base SHA: 596d729e3fbe4f828424cadbe0bf1b3c5b307ada
Merge base SHA: 596d729e3fbe4f828424cadbe0bf1b3c5b307ada
Patch ID: 96ade727c4c42a3c2c050df5ad84bcbbc146222d
Gator payload: 2
Review mode: follow_up
Previous reviewed SHA: 286d02719e3ecb7e0126aa5934c535c72b0567b3

Thanks @johntmyers. I checked the same-candidate failure-transition fix in the latest author delta. The deduplication identity now includes the runtime failure class, so a middleware outage followed by policy rejection is handled as a changed disposition. The focused regression coverage exercises that transition.

Blocking findings:

• No blocking findings remain.

Carried findings:

• GATOR-286d0271-01: Resolved by failure-class-aware deduplication and transition coverage.

Docs: No new direct UX change in this remediation delta; the already-reviewed Fern and architecture documentation remains applicable.

Next state: gator:watch-pipeline

[johntmyers]

gator-agent

Monitoring Complete

Monitoring is complete because this PR has merged.

Head SHA: 9f4447f4a9cb2e914bd33619bb1820ce431d4b4f
Gator payload: 2

Final status: Gator review completed with no blocking findings remaining, required checks passed, and maintainer approval was present before merge.

I removed the active gator:* label because there is nothing left for gator to monitor on this PR.