Fix NVIDIA passthrough into Workspace containers

[tigerblue77]

---

• I have read the contributing guideline and understand that I have made the correct modifications

---

Description:

The host's NVIDIA container toolkit injects the driver into this container, so the DinD layer itself has a working GPU — which is exactly what the issue reports: nvidia-smi and vulkaninfo behave there. Nothing carries that injection any further, though:

• root/etc/docker/daemon.json registers an nvidia runtime for the DinD daemon but never makes it the default,
• and no CDI specification describes the devices we were handed.

So the Workspace containers Kasm creates inside the DinD layer come up as plain runc containers, with no driver libraries and no Vulkan ICD.

This adds a GPU step to init-config-kasm, which is a oneshot that completes before svc-docker starts dockerd. It only fires when NVIDIA devices are actually present in the container (/dev/nvidiactl), and then:

1. nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml — describes the device nodes, driver libraries, binaries and Vulkan/EGL/GLX loader configuration that were injected into this container;
2. nvidia-ctk config --in-place --set nvidia-container-runtime.mode=cdi — points the NVIDIA runtime at that specification instead of re-discovering the driver at container creation time;
3. nvidia-ctk runtime configure --runtime=docker --set-as-default — makes the NVIDIA runtime the default one for the DinD daemon.

The specification is removed and regenerated on every start, so a host driver upgrade is picked up rather than cached. Documentation and changelog entries are included.

Benefits of this PR and context:

Closes#83.

Workspace containers get the driver, its libraries and the ICD/vendor configuration injected again, so Vulkan and OpenGL work inside them — whether Kasm asks for the GPU through device_requests, through a "runtime": "nvidia" Docker run config override, or through a Workspace image that already sets NVIDIA_VISIBLE_DEVICES. --device nvidia.com/gpu=all also becomes usable directly, since CDI is enabled by default in dockerd from 28.3.0 onwards and this image pins Docker 29.

Two things about the current state are worth calling out, as they explain why the old advice stopped working:

• Making the NVIDIA runtime the default inside the DinD layer used to be sufficient on its own. Since nvidia-container-toolkit 1.18.0 the runtime defaults to jit-cdi, which builds its specification at container creation time by re-discovering the driver on what it takes to be the host — and that discovery is the step that does not survive being run inside the DinD layer. Doing the generation once, explicitly, at startup, moves that work to a point where it either succeeds or is reported, instead of silently yielding a container without a GPU. 1.18.2 turned the same class of failure into an error rather than an unresolvable device, which points in the same direction.
• A CDI specification is capability-agnostic. The graphics parts of the driver are injected whether or not graphics made it into NVIDIA_DRIVER_CAPABILITIES for that Workspace, which removes the other half of the "Vulkan specifically is missing" symptom.

Hosts without a GPU are unaffected: the whole block is skipped, and /etc/docker/daemon.json and /etc/nvidia-container-runtime/config.toml are left exactly as they ship today.

How Has This Been Tested?

Honest summary: I had no NVIDIA host available, so the GPU path has not been exercised end to end. What has been checked:

• bash -n on the modified init-config-kasm/run, and readme-vars.yml parses.
• Every nvidia-ctk invocation was checked against the toolkit sources for the version this image installs — cdi generate --output, config --in-place --set, and runtime configure --runtime/--set-as-default (which only logs a "restart the daemon" recommendation, so it is safe to call before dockerd is up).
• Ordering: init-config-kasm → init-config-end → … → init-services → svc-docker, so the specification and the daemon configuration are in place before dockerd starts.
• No-GPU behaviour: with /dev/nvidiactl absent the guard is false and nothing is written.

What would need to be confirmed on a GPU host, and I would appreciate a second pair of eyes on:

1. docker exec -it kasm cat /etc/cdi/nvidia.yaml is generated and lists the expected devices;
2. docker exec -it kasm docker info | grep -i runtime shows Default Runtime: nvidia;
3. vulkaninfo --summary and glxinfo -B inside a Workspace session report the NVIDIA driver;
4. a GPU-less host still starts cleanly and its Workspaces are unchanged.

Source / References:

• Issue: [BUG] Nvidia Passthrough no longer works #83
• nvidia-container-toolkit v1.18.0 — "Default to jit-cdi mode in the nvidia runtime": https://github.com/NVIDIA/nvidia-container-toolkit/blob/main/CHANGELOG.md
• nvidia-container-toolkit release notes (1.18.2 error handling for JIT CDI generation): https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/release-notes.html
• CDI support in the NVIDIA Container Toolkit: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/cdi-support.html
• dockerd reference — CDI is enabled by default since Docker Engine 28.3.0: https://docs.docker.com/reference/cli/dockerd/
• Kasm GPU acceleration documentation: https://docs.kasm.com/docs/latest/how-to/gpu/index.html

---

Generated by Claude Code and reviewed by @tigerblue77