refactor(compute): register compiled drivers

[drew]

Summary

Introduce an explicit compiled compute-driver registry, modeled after SQLx driver installation, as a behavior-preserving pre-factor. The gateway still ships the same first-party drivers at this boundary; selection and auto-detection now consume generic registrations instead of a closed driver-name switch.

Related Issue

Part of #2417.

Stacked on #2744. Followed by #2822.

Changes

• add ComputeDriverRegistry, ComputeDriverRegistration, and ComputeDriverFactory
• construct in-process compute runtimes through registered factories
• probe installed registrations in deterministic priority order and retain all available results
• compute driver selection once after configuration merging and reuse it for authentication defaults and runtime construction
• preserve configured external endpoint precedence over a registration with the same name
• retain the existing first-party driver set and runtime behavior
• document the compiled-driver registration boundary

This PR intentionally does not add standalone driver binaries or move first-party composition out of openshell-server; those changes are isolated in the following PRs.

Testing

• mise run pre-commit
• cargo test -p openshell-server registry_detection_ --lib

Checklist

• Changes are scoped to accepted issue refactor(compute): define driver shutdown and restart lifecycle contract #2417
• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture documentation is updated
• Public-facing documentation remains unchanged

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs:https://nvidia-preview-pr-2786.docs.buildwithfern.com/openshell

[github-actions]

Label test:e2e applied, but pull-request/2786 is at b8659a8 while the PR head is 0862fc2. A maintainer needs to comment /ok to test 0862fc22b855cf8e296df7355c2fcd1d9aba92d6 to refresh the mirror. Once the mirror catches up, re-run Branch E2E Checks from the Actions tab.

[elezar]

I think this needs to be split before it can be reviewed confidently. The PR currently combines several independently meaningful changes and then repairs their intermediate states in later commits, which makes it difficult to tell whether each boundary is complete and preserves behavior.

Could we also take #2786 out of the active landing stack for now? #2743 and #2744 are independently useful and should be able to progress without waiting for this restructuring. I do not mean that they should land unchanged:

• #2743 still needs its requested lifecycle corrections: bounded shutdown concurrency, acquiring the lifecycle gate and re-reading current intent in both sweeps, and paging through all persisted sandboxes. The e2e_stop_gateway process-wait fix from this PR also belongs naturally with #2743 because it compensates for that PR's graceful-shutdown stop sweep.
• #2744 should then be rebased on the revised #2743. It can provide the external-driver lifecycle-ownership capability that resolves #2743's built-in-only policy, while also incorporating the pending naming and endpoint-override documentation clarifications as appropriate.

Once those PRs have been updated and merged, #2786 can be rebuilt as a smaller stack on top of the resulting base.

Could we restructure it into a stack along these lines?

1. Introduce the compiled-driver registry as a behavior-preserving PREfactor.
2. Add the standalone first-party driver executables and their external-driver E2E paths. Please fold the VM default-image fix, Kubernetes supervisor-image reuse, post-renderer-based Kubernetes isolation, and scoped external-Podman job into this change so it is complete when introduced.
3. Move first-party composition and VM ownership into openshell-gateway and make openshell-server backend-independent. The backend-neutral server comments and the bundled-Z3/CMake corrections should land with this move.
4. Update packaging and build references for the new gateway crate.
5. Finish the compiled-driver boundary simplification as a focused cleanup.

The gateway shutdown wait in e2e_stop_gateway appears independent of this work and tied to the lifecycle sweep introduced by #2743, so I suggest moving that to #2743 or a small follow-up based on #2744 rather than including it in this stack.

I reconstructed this sequence locally on the current #2744 head. Each proposed boundary compiles, the registry-focused test passes, and the final tree matches this PR, so the split appears mechanically feasible.

[drew]

Restructured this work in response to the review. The overall stack is now:

1. refactor(compute): unify gateway restart reconciliation #2743 — gateway shutdown/startup reconciliation, including bounded concurrency, lifecycle gating and intent re-reads, pagination, and the gateway-process wait.
2. refactor(compute): support external driver parity #2744 — public GetCapabilities.gateway_manages_lifecycle parity for in-process and external drivers.
3. refactor(compute): register compiled drivers #2786 — behavior-preserving compiled-driver registry pre-factor only.
4. feat(compute): add standalone first-party drivers #2822 — standalone Docker, Podman, Kubernetes, and VM drivers plus complete public-RPC E2E setup. The VM image fix, Kubernetes image reuse and post-renderer isolation, and scoped Podman coverage are folded into this boundary.
5. refactor(compute): decouple gateway driver composition #2823 — move first-party composition and VM process ownership into openshell-gateway, remove backend construction from openshell-server, and update packaging/build references. The backend-neutral comments and bundled-Z3/CMake fixes land here too.

I consolidated the proposed composition, packaging, and final boundary-cleanup steps into #2823 because the packaging references are direct consumers of the crate move and the cleanup removes the transient server-owned boundary. Keeping them together leaves one complete architectural outcome without an intermediate packaging-only PR.

All three rewritten PRs are drafts. mise run pre-commit passes at every new boundary, the registry selection test passes on #2786, the driver-free gateway build passes on #2822 and #2823, and the final #2823 tree is byte-for-byte identical to the pre-split #2786 head.

[drew]

/ok to test

Changes made.

[elezar]

/ok-to-test 25f9a63

[elezar]

@drew A quick summary of the review changes I folded into the single commit in this PR and restored at 25f9a63:

• Changed auto-detection to evaluate every installed driver probe in deterministic priority order, retain the complete ordered set of available registrations, and select the first one.
• Made the no-driver error generic to the installed registry instead of hard-coding the first-party driver list, which keeps custom gateway binaries understandable.
• Moved driver selection to configuration preparation after config merging and carried that same selection through authentication defaults and runtime construction. This prevents a later probe from selecting a different driver during the same startup.
• Added regression coverage for probe order, multiple available drivers, generic errors, configured custom drivers, and single evaluation during startup.
• Updated the compute-runtime architecture documentation to describe the deterministic probe and single-selection behavior.

This aligns the registry behavior with the extensibility concerns discussed in #2163. The current PR head is 25f9a63.

[elezar]

Reviewed and addressed issues as called out in the comment.

LGTM, but feel free to get a second opinion.

[elezar]

/ok-to-test 25920c4

[drew]

/ok to test 0e2ac3d