feat(netd): carry a VM's host port mappings to whoever configures the host

docs/bridge-networking.md

@@ -199,6 +199,41 @@ The remaining bytes are derived from the VM ID hash. The prefix applies to all n
 - Docker's nftables chains (`DOCKER-FORWARD`) run before libvirt's but do not block virbr0 traffic
 - Use `setup-bridge.sh check --bridge <name>` to diagnose missing rules
 
+### Host port mappings
+
+A VM's `port_map` has always been implemented as QEMU `hostfwd=` entries on a
+user-mode netdev. A bridge NIC has no such netdev, so **the VMM does not forward
+host ports for bridge VMs** — it is deliberately run without `CAP_NET_ADMIN`,
+and a userspace proxy on the host would give back the per-packet cost that
+moving off user mode was meant to escape.
+
+What the VMM does instead is carry the requirement to `netd`, which is
+privileged and is the only component that sees every VMM instance on the host —
+and therefore the only one that can arbitrate a host port between them. Whether
+a given `netd` implements forwarding is its own business; the one in this
+repository does not, and says so through `hello` rather than accepting ports it
+will not forward.
+
+So on a node whose `netd` does not forward:
+
+- the VM starts normally and its bridge networking works
+- its port mappings do not apply, and the VMM says so in its log at every launch
+- `GetInfo` reports an empty `ingress` on the interface, against the non-empty
+ `ports` on the configuration — that difference is the signal
+
+Expose such a VM by reaching its address on the bridge directly, by putting an
+L7 proxy in front of it, or by running a `netd` that forwards. `GetInfo` reports
+the interface's `guest_ip` when `netd` is the authority for addresses on that
+segment; a bridge NIC otherwise takes its address from a DHCP server the VMM
+does not run and cannot ask.
+
+`macvtap` and `custom` NICs can never carry host port mappings — the first
+bypasses the host bridge, and the second hands the whole netdev string to the
+operator. The VMM warns rather than refusing to start, because a VM deployed
+before any of this existed has been running with its ports dropped, and turning
+that into a failed launch on upgrade would make an outage out of a
+misconfiguration that was already there.
+
 ### Mixing networking modes
 
 Bridge and user-mode VMs can coexist. Set the global default in `vmm.toml` and override per-VM as needed:

dstack/vmm/rpc/proto/vmm_rpc.proto

@@ -67,6 +67,26 @@ message NetworkInterfaceStatus {
  // configuration decides it, so it is reported here with the rest of the
  // resolved state rather than on the VM's own NetworkingConfig.
  optional string macvtap_mode = 8;
+ // Address this interface reaches its segment at, when the node's netd is the
+ // one that decides addresses. A bridge NIC gets its address from a DHCP
+ // server the VMM does not run, so absent means "not reported to us", never
+ // "has none".
+ optional string guest_ip = 9;
+ // Host ports actually forwarded to this interface. This is what the node
+ // did; VmConfiguration.ports is what was asked for. They differ whenever the
+ // node's netd allocates a port itself or declines one, and an empty list
+ // against a non-empty request means nothing was forwarded at all.
+ repeated IngressBinding ingress = 10;
+}
+
+// One host port forwarded to a VM, as the node established it.
+message IngressBinding {
+ // "tcp" or "udp".
+ string protocol = 1;
+ // Host address the port is accepted on.
+ string host_address = 2;
+ uint32 host_port = 3;
+ uint32 guest_port = 4;
 }
 
 // Structured log or lifecycle event emitted by the guest or runtime.

dstack/vmm/src/app.rs

@@ -558,6 +558,75 @@ impl App {
  Ok(())
  }
 
+ /// The host ports this VM asks for, if the node can forward any.
+ ///
+ /// `port_map` has always been implemented as QEMU `hostfwd=` entries on a
+ /// user-mode netdev, so a VM that moves to a bridge silently loses every
+ /// one of them: nothing warns, and `GetInfo` keeps reporting the ports as
+ /// though they worked. Handing them to netd is what can make them real,
+ /// and saying so when it cannot is what makes the loss visible.
+ ///
+ /// Returns an empty list rather than failing when the node has no forwarder.
+ /// A VM deployed before this existed has been running with its ports
+ /// dropped, and refusing to start it now would turn a silent
+ /// misconfiguration into an outage on upgrade.
+ async fn ingress_for(
+ &self,
+ vm: &VmConfig,
+ networks: &[Networking],
+ ) -> Vec<netd::IngressRequest> {
+ if vm.manifest.port_map.is_empty() {
+ return Vec::new();
+ }
+ // QEMU's own `hostfwd=` entries ride on a user-mode netdev, so a VM
+ // that still has one needs nothing from netd. This mirrors how the
+ // launch picks the interface to put them on.
+ if networks
+ .iter()
+ .any(|network| network.nic.mode == NetworkingMode::User)
+ {
+ return Vec::new();
+ }
+ let ports = vm.manifest.port_map.len();
+ if !networks
+ .iter()
+ .any(|network| network.nic.mode == NetworkingMode::Bridge)
+ {
+ // macvtap bypasses the host bridge and custom mode owns its own
+ // netdev string, so neither has anywhere for the host to forward to.
+ warn!(
+ vm_id = %vm.manifest.id,
+ ports,
+ "this VM's networking mode cannot carry host port mappings, so \
+ its {ports} port mapping(s) do not apply"
+ );
+ return Vec::new();
+ }
+ let forwards = netd::capabilities(&self.config.netd.socket)
+ .await
+ .map(|capabilities| capabilities.has("ingress"))
+ .unwrap_or(false);
+ if !forwards {
+ warn!(
+ vm_id = %vm.manifest.id,
+ ports,
+ "no netd on this node forwards host ports, so this VM's {ports} \
+ port mapping(s) do not apply to its bridge interface"
+ );
+ return Vec::new();
+ }
+ vm.manifest
+ .port_map
+ .iter()
+ .map(|mapping| netd::IngressRequest {
+ protocol: mapping.protocol.as_str().to_string(),
+ host_address: mapping.address.to_string(),
+ host_port: mapping.from,
+ guest_port: mapping.to,
+ })
+ .collect()
+ }
+
  async fn prepare_filtered_networks(
  &self,
  vm: &VmConfig,
@@ -577,6 +646,7 @@ impl App {
  .work_dir(&vm.manifest.id)
  .map(|dir| dir.path().display().to_string())
  .unwrap_or_default();
+ let ingress = self.ingress_for(vm, networks).await;
  let mut prepared = Vec::new();
  for (nic_index, network) in networks.iter_mut().enumerate() {
  if !needs_netd_interface(network, &self.config.cvm) {
@@ -607,6 +677,15 @@ impl App {
  filtered,
  queues,
  workdir: workdir.clone(),
+ // Only the first NIC carries them, matching where user-mode
+ // networking puts its `hostfwd=` entries. A second NIC that
+ // repeated the list would ask two interfaces to answer on
+ // one host port.
+ ingress: if nic_index == 0 {
+ ingress.clone()
+ } else {
+ Vec::new()
+ },
  }),
  NetworkingMode::Macvtap => NetdRequest::PrepareMacvtap(PrepareMacvtapRequest {
  identity: identity.clone(),
@@ -691,6 +770,18 @@ impl App {
  )
  );
  }
+ // Ports that were asked for and not answered for are the
+ // failure this whole path exists to end. A netd that forwards
+ // says what it built; silence is not consent.
+ let requested = if nic_index == 0 { ingress.len() } else { 0 };
+ if requested > 0 {
+ let established = response
+ .ingress
+ .clone()
+ .context("netd accepted host ports without saying what it forwarded")?;
+ network.ingress = established;
+ }
+ network.guest_ip = response.guest_ip.clone().unwrap_or_default();
  Ok(())
  })();
  if let Err(error) = accepted {

dstack/vmm/src/app/vm_info.rs

@@ -87,6 +87,21 @@ fn interfaces_to_proto(
  // state. The VM's own record cannot carry one.
  macvtap_mode: (networking.nic.mode == NetworkingMode::Macvtap)
  .then(|| networking.macvtap_mode.clone()),
+ // Both are what the node reported at launch, not what the VM
+ // asked for. `port_map` is the request and stays on the
+ // configuration; an empty list here against a non-empty request
+ // is the node saying it forwarded nothing.
+ guest_ip: (!networking.guest_ip.is_empty()).then(|| networking.guest_ip.clone()),
+ ingress: networking
+ .ingress
+ .iter()
+ .map(|binding| pb::IngressBinding {
+ protocol: binding.protocol.clone(),
+ host_address: binding.host_address.clone(),
+ host_port: binding.host_port as u32,
+ guest_port: binding.guest_port as u32,
+ })
+ .collect(),
  }
  })
  .collect()

dstack/vmm/src/config.rs

@@ -10,6 +10,7 @@ use path_absolutize::Absolutize;
 use rocket::figment::Figment;
 use serde::{Deserialize, Serialize};
 
+use crate::netd::IngressBinding;
 use dstack_types::TdxAttestationVariant;
 use lspci::{lspci_filtered, Device};
 use tracing::{info, warn};
@@ -1072,6 +1073,27 @@ pub struct Networking {
  /// the VM runs, and what has to be removed is what was created.
  #[serde(default, skip_serializing_if = "NetdInterface::is_none")]
  pub netd_interface: NetdInterface,
+ /// The address netd says this NIC reaches its segment at.
+ ///
+ /// Runtime state, cleared when the VM stops. A bridge NIC's address comes
+ /// from a DHCP server the VMM does not run, so without netd reporting it the
+ /// VMM can only say which bridge a VM is on, not where it is. Empty
+ /// whenever netd does not assign addresses.
+ ///
+ /// Unlike `device` this is persisted, so a VMM restart does not lose the
+ /// address of a VM that is still running. A stale `/dev/tapN` can name a
+ /// different device after a restart, which is why that one is not; a stale
+ /// address is only ever reported, never acted on.
+ #[serde(default, skip_serializing_if = "String::is_empty")]
+ pub guest_ip: String,
+ /// The host ports netd forwards to this NIC, as netd established them.
+ ///
+ /// Runtime state, persisted for the same reason as `guest_ip`. What was
+ /// asked for lives in the manifest's `port_map`; this is what the node
+ /// actually did, which is not the same thing when netd allocates a port
+ /// from a range or declines one.
+ #[serde(default, skip_serializing_if = "Vec::is_empty")]
+ pub ingress: Vec<IngressBinding>,
 }
 
 /// The host interface netd created for a NIC, if any.