VMM: networking mode management by Leechael · Pull Request #756 · Dstack-TEE/dstack

Leechael · 2026-07-07T23:07:17Z

Summary

Adds multi-NIC networking support to the VMM and UI:

supports repeated per-VM network configurations while retaining legacy networking manifest compatibility;
exposes network configuration and capabilities over VMM RPC;
creates one QEMU netdev/device pair per interface with deterministic per-NIC MAC addresses;
validates bridge/custom-mode requests and keeps runtime networking snapshots consistent;
measures the NIC count in RTMR0 so multi-NIC CVMs remain attestable;
adds create/update controls to the VMM UI.

Rebased and relocated onto the unified monorepo layout from PR #770.

Validation

cargo test -p dstack-vmm -p dstack-mr -p dstack-types -p dstack-verifier
npm ci && npm run build in dstack/vmm/ui
reuse lint
prek run --all-files

kvinwang · 2026-07-08T14:33:53Z

A note on the guest_ips addition before we grow it further:

The whole guest_ip machinery only exists to serve VMM's built-in port-forward feature. And the way it learns the guest IP is pretty awkward — it has to talk to an external DHCP server out-of-band (dnsmasq --dhcp-script → dhcp-notify.sh → ReportDhcpLease RPC) just to find out what address the guest got. So the stored value is really "what dnsmasq handed out," not the guest's actual IP: it goes stale (the del/release path is a no-op), it's empty for user/slirp NICs, and the RPC that feeds it is unauthenticated. That's a lot of moving parts for what it delivers.

Context worth flagging: the built-in port forwarding — and guest_ip along with it — was already a candidate for removal; the intent was to deprecate and delete it rather than build on top of it.

Given that, I'd suggest not extending this into a per-MAC guest_ips map / guest-ips.json file plus the new interfaces[].ip surface. That doubles down on a subsystem we meant to drop: it adds a second on-disk file with the same staleness/trust caveats and grows the public API contract around it. If multi-NIC status reporting is genuinely needed, let's settle the fate of port-forward first — and if it stays, the DHCP-report path should get authentication + a release/cleanup path before we persist and expose it more widely.

Happy to help sketch the removal path if we decide to go that way.

Per PR #756 review: the guest_ip machinery exists only to serve VMM's built-in port-forward, a subsystem already slated for deprecation. It learns the guest address out-of-band from an external DHCP server via an unauthenticated ReportDhcpLease RPC, so the stored value is stale-prone and empty for slirp NICs. Extending it into a per-MAC map doubled down on that subsystem: a second on-disk file with the same staleness/trust caveats plus a wider public API surface. Roll back the extension rather than build on top of it: - proto: remove NetworkInterfaceStatus.ip, renumber trailing fields - app.rs: drop guest_ips map from state and DHCP-lease path; port forwarding uses the single guest_ip again - qemu.rs: remove guest_ips field, guest-ips.json persistence (guest_ips_path/guest_ips/set_guest_ip_for_mac), and per-NIC ip population in VmInfo::interfaces - ui: drop the per-interface IP column and its grid track Multi-NIC status reporting can be revisited once the fate of port-forward is settled; if it stays, the DHCP-report path needs authentication and a release/cleanup path before persisting and exposing guest IPs more widely.

Per PR #756 review: the guest_ip machinery exists only to serve VMM's built-in port-forward, a subsystem already slated for deprecation. It learns the guest address out-of-band from an external DHCP server via an unauthenticated ReportDhcpLease RPC, so the stored value is stale-prone and empty for slirp NICs. Extending it into a per-MAC map doubled down on that subsystem: a second on-disk file with the same staleness/trust caveats plus a wider public API surface. Roll back the extension rather than build on top of it: - proto: remove NetworkInterfaceStatus.ip, renumber trailing fields - app.rs: drop guest_ips map from state and DHCP-lease path; port forwarding uses the single guest_ip again - qemu.rs: remove guest_ips field, guest-ips.json persistence (guest_ips_path/guest_ips/set_guest_ip_for_mac), per-NIC ip population in VmInfo::interfaces, and the now-dead VmInfo.guest_ip field that only fed that population - ui: drop the per-interface IP column and its grid track Multi-NIC status reporting can be revisited once the fate of port-forward is settled; if it stays, the DHCP-report path needs authentication and a release/cleanup path before persisting and exposing guest IPs more widely.

kvinwang

LGTM

Per PR #756 review: the guest_ip machinery exists only to serve VMM's built-in port-forward, a subsystem already slated for deprecation. It learns the guest address out-of-band from an external DHCP server via an unauthenticated ReportDhcpLease RPC, so the stored value is stale-prone and empty for slirp NICs. Extending it into a per-MAC map doubled down on that subsystem: a second on-disk file with the same staleness/trust caveats plus a wider public API surface. Roll back the extension rather than build on top of it: - proto: remove NetworkInterfaceStatus.ip, renumber trailing fields - app.rs: drop guest_ips map from state and DHCP-lease path; port forwarding uses the single guest_ip again - qemu.rs: remove guest_ips field, guest-ips.json persistence (guest_ips_path/guest_ips/set_guest_ip_for_mac), per-NIC ip population in VmInfo::interfaces, and the now-dead VmInfo.guest_ip field that only fed that population - ui: drop the per-interface IP column and its grid track Multi-NIC status reporting can be revisited once the fate of port-forward is settled; if it stays, the DHCP-report path needs authentication and a release/cleanup path before persisting and exposing guest IPs more widely.

Single-NIC bridge/user networking could not express mixed topologies (e.g. one bridge NIC for external traffic + one user-mode NIC for isolated management). Extend the model to support multiple NICs per VM. - Manifest field: networking (singular, Optional) -> networks (Vec) with backward-compatible deserialization of the legacy field - Proto: add repeated NetworkingConfig in VmConfiguration/UpdateVmRequest, NetworkInterfaceStatus in VmInfo, and NetworkingCapabilities in GetMetaResponse - QEMU: generate one -netdev/-device pair per entry with indexed net IDs and per-NIC deterministic MACs (SHA256 of vm_id:index) - DHCP: match leases against all NICs, persist per-MAC guest IP map - Port forwarding: select the first bridge NIC with forward_service_enabled - Validation: reject bridge mode when the host interface does not exist, reject custom mode via RPC (manifest-only) - Measurement: networking config excluded from VM measurement to allow topology changes without re-attestation

- Replace hand-written Manifest Deserialize impl with derive + Manifest::from_json() for legacy "networking" field migration - Simplify resolve_networking to clone global config as base then selectively override non-empty per-VM fields - Extract effective_networks() on VmState and VmInfo to eliminate four duplicated runtime_networks fallback blocks - Remove redundant forward_service_enabled pre-assignment in resolve_requested_networks (now inherited from global config clone)

Drop if_name, lowerdev, and pci_bdf — all three were always set to None in the only construction site. Add them back when a backend actually populates them.

Each virtio-net-pci device changes the guest ACPI/DSDT layout and thus RTMR0. Before this change dstack-mr hardcoded exactly one NIC, so a CVM booted with the new multi-network support (repeated `networks`) would produce an RTMR0 the verifier could never reconstruct, and attestation would fail closed. Dropping ACPI from the measurement instead would let an untrusted host silently attach un-attested devices under an unchanged quote, so the NIC count must be measured. Mirror the existing `num_gpus` plumbing: add `num_nics` to `dstack_types::VmConfig` and `dstack_mr::Machine`, emit one `-netdev/-device virtio-net-pci` pair per NIC in the ACPI generator, and thread the resolved interface count from the VMM through to the verifier. Backward compatibility: - `num_nics` defaults to 1 and is omitted from the serialized VmConfig when equal to 1, so existing configs, their measurement cache keys, and RTMR0 values are byte-for-byte unchanged (verified against the previous hardcoded layout). - legacy configs without the field deserialize to 1. Verified: RTMR0(num_nics=1) == RTMR0(previous hardcoded layout); RTMR0(num_nics=2) differs.

Per PR #756 review: the guest_ip machinery exists only to serve VMM's built-in port-forward, a subsystem already slated for deprecation. It learns the guest address out-of-band from an external DHCP server via an unauthenticated ReportDhcpLease RPC, so the stored value is stale-prone and empty for slirp NICs. Extending it into a per-MAC map doubled down on that subsystem: a second on-disk file with the same staleness/trust caveats plus a wider public API surface. Roll back the extension rather than build on top of it: - proto: remove NetworkInterfaceStatus.ip, renumber trailing fields - app.rs: drop guest_ips map from state and DHCP-lease path; port forwarding uses the single guest_ip again - qemu.rs: remove guest_ips field, guest-ips.json persistence (guest_ips_path/guest_ips/set_guest_ip_for_mac), per-NIC ip population in VmInfo::interfaces, and the now-dead VmInfo.guest_ip field that only fed that population - ui: drop the per-interface IP column and its grid track Multi-NIC status reporting can be revisited once the fate of port-forward is settled; if it stays, the DHCP-report path needs authentication and a release/cleanup path before persisting and exposing guest IPs more widely.

Rebasing onto master pulled in the new make_vm_config(.., requirements: Option<&Requirements>) parameter added by the launch-token-requirement work (#763). The multi-NIC bridge/user config-equality test still called the 5-arg form; pass None for the new argument so the crate builds against the rebased signature.

Reject configurations with multiple forwarding-enabled bridges so DHCP lease updates cannot redirect the VM-wide forwarding target between interfaces.\n\nPersist runtime network snapshots atomically before launching QEMU, publish the snapshot to in-memory state before DHCP can arrive, and clear both copies when process startup fails. Add regression coverage for forwarding validation and atomic snapshot replacement.

kvinwang · 2026-07-16T08:12:10Z

Rebased onto current master (7b1b4934, including #787) and force-pushed.

The former qemu.rs changes were reapplied by responsibility rather than restoring the monolith:

network resolution/validation and indexed MAC derivation → app/network.rs
RPC networking presentation and interface status → app/vm_info.rs
manifest compatibility and runtime-network snapshots → app/workdir.rs
repeated -netdev/virtio-net-pci construction → the prepared QemuCommandBuilder path in app/qemu.rs

I also kept bridge/filesystem validation in PreparedQemuLaunch, so command construction remains host-I/O-free, and added coverage for two NICs plus first-user-NIC-only host forwarding. Range-diff confirms original commits 2–9 replay unchanged; the first commit differs only where its monolithic QEMU changes were relocated.

Validation:

cargo test -p dstack-vmm -p dstack-mr -p dstack-types -p dstack-verifier
cargo clippy -p dstack-vmm --all-targets --all-features -- -D warnings
npm ci && npm run build in dstack/vmm/ui
prek run --all-files
reuse lint on a clean archive
./run-tests.sh

Three things that are one thing: a port mapping had no way to say which NIC it uses, the NIC it silently got was often the wrong one, and on a bridge it got nothing at all. `port_map` is implemented as QEMU `hostfwd=` entries, and those need a user-mode netdev: let hostfwd_index = self.prepared.networks.iter() .position(|n| n.mode == NetworkingMode::User); Multi-NIC (#756) made that a choice, and it has been made silently ever since. A bridge NIC for external traffic beside a user-mode NIC for management -- the topology multi-NIC was added for -- puts every published port on the *management* NIC: the traffic reaches the guest, but over slirp, bypassing whatever the bridge NIC's nwfilter was there to enforce and hiding the client's address behind the slirp gateway. A second user-mode NIC can never publish anything, because `position` returns the first. And with no user-mode NIC at all there is no `hostfwd_index`, so every mapping is dropped with no warning while `GetInfo` keeps reporting the ports as though they worked. So say it. `PortMapping.nic_index` names the NIC a mapping enters through, `@<nic>` on the CLI, unset resolving to the first user-mode NIC and failing that the first bridge NIC. Existing VMs keep their behaviour exactly wherever a user-mode NIC exists; where none does, ports now have somewhere to go instead of nowhere. One mapping resolves to exactly one NIC, and that NIC's backend decides the mechanism: `hostfwd=` for user mode, netd for a bridge. Both sides read the same resolution, so no host port can be claimed twice. The VMM cannot forward a bridge NIC's ports itself -- it runs without CAP_NET_ADMIN by design, and the userspace forwarder that used to cover this was removed in e2e607f because proxying on the host hands back the per-packet cost that leaving user mode was meant to escape. netd is privileged and is the only component that sees every VMM instance on a host, so it is also the only one that can arbitrate a host port between them. `ingress` on `prepare_bridge` carries the requirement there. Every field of it is caller-named, which is what `bridge`, `mac` and `queues` already get and the opposite of `filtered`. The line is whether netd can check what it is handed: an nwfilter name cannot be checked for whether it filters anything, while a host port is a closed space a node policy can be stated over. Naming is not deciding. The netd here builds interfaces and does not forward ports. It says so by leaving `ingress` out of its response -- the reading `queues` already has, where absent separates "this netd does not do that" from "nothing was asked for". The VMM warns rather than failing, because a VM deployed before this has been running with its ports dropped and refusing to launch it now would turn a silent misconfiguration into an outage. Also add `remove_all`. Teardown by identity only reaches the NIC indices its caller still has a record of, and that record is written after the interface exists: a VMM killed in between leaves a TAP nothing on disk points at, a lost record reads as "nothing to remove", and a manifest that lost a NIC leaves an index the list no longer reaches. netd derives every name a VM could occupy instead, bounded by what an identity may say. The VMM sweeps before preparing a launch as well as on stop, so a launch is self-healing regardless of what the record says, and an unreachable netd no longer fails a stop. And `workdir` on both prepare operations: untrusted, never read for a decision, there so an operator reading netd's log can get from an opaque TAP name back to the VM that asked for it.

Leechael force-pushed the vmm-networking-rpc-iteration branch 2 times, most recently from 501848b to de1bbb1Compare July 8, 2026 09:46

Leechael marked this pull request as ready for review July 8, 2026 10:09

Leechael force-pushed the vmm-networking-rpc-iteration branch from c447e16 to 76b965aCompare July 9, 2026 15:59

Leechael force-pushed the vmm-networking-rpc-iteration branch from 76b965a to c728dcdCompare July 9, 2026 16:34

Leechael force-pushed the vmm-networking-rpc-iteration branch from 37e88da to 5693786Compare July 9, 2026 23:40

kvinwang approved these changes Jul 10, 2026
View reviewed changes

kvinwang force-pushed the vmm-networking-rpc-iteration branch from 5693786 to 0f812fbCompare July 13, 2026 04:13

Leechaeland others added 10 commits July 16, 2026 00:56

refactor(vmm): remove unused proto fields from NetworkInterfaceStatus
033c80e
Drop if_name, lowerdev, and pci_bdf — all three were always set to None in the only construction site. Add them back when a backend actually populates them.

review(vmm): align networking backend API
66a5fc7

feat(vmm-ui): add VM networking controls
5681141

refactor(vmm): align networking with split qemu modules
1e329bf

kvinwang force-pushed the vmm-networking-rpc-iteration branch from 0f812fb to 1e329bfCompare July 16, 2026 08:11

kvinwang merged commit 7059967 into masterJul 16, 2026
16 checks passed

Leechael deleted the vmm-networking-rpc-iteration branch July 17, 2026 09:18

kvinwang mentioned this pull request Jul 20, 2026
refactor(vmm): remove built-in port forwarding #795
Merged

kvinwang mentioned this pull request Aug 28, 2026
feat(vmm): give port mappings a NIC, and carry them to netd #1154
Open

Navigation Menu

Uh oh!

VMM: networking mode management - #756

VMM: networking mode management#756
kvinwang merged 10 commits into
masterfrom
vmm-networking-rpc-iteration

Leechael commented Jul 7, 2026•
edited by kvinwang
Loading

Uh oh!

kvinwang commented Jul 8, 2026

Uh oh!

kvinwang left a comment

Uh oh!

kvinwang commented Jul 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Leechael commented Jul 7, 2026• edited by kvinwang Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Uh oh!

kvinwang commented Jul 8, 2026

Uh oh!

kvinwang left a comment

Choose a reason for hiding this comment

Uh oh!

kvinwang commented Jul 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Leechael commented Jul 7, 2026•
edited by kvinwang
Loading