Skip to content

driver: request gpu when creating container builder - #3063

Merged
tonistiigi merged 1 commit into
docker:masterfrom
crazy-max:driver-ctn-gpu-request
Mar 13, 2025
Merged

driver: request gpu when creating container builder#3063
tonistiigi merged 1 commit into
docker:masterfrom
crazy-max:driver-ctn-gpu-request

Conversation

@crazy-max

Copy link
Copy Markdown
Member

With CDI support added in BuildKit moby/buildkit#4056, we need to request GPUs when creating a container builder.

This is necessary on WSL for Nvidia GPUs because there is no linux drivers available that we could install in BuildKit container during OnDemand setup. Mounting /usr/lib/wsl works https://github.com/docker/buildx/compare/master...crazy-max:buildx:driver-ctn-mount-wsl-lib?expand=1 but this is not desired and could be error prone if a remote context/builder is used. So instead we can make a device request when creating the container builder.

Can be tested by creating a BuildKit container with ubuntu base:

$ EXPORT_BASE=ubuntu PLATFORMS=linux/amd64,linux/arm64 ./hack/images ubuntu crazymax/buildkit push

You can use crazymax/buildkit:ubuntu for testing on WSL with Nvidia GPU:

$ docker buildx create --name testgpu --bootstrap --driver-opt "image=crazymax/buildkit:ubuntu"
#1 [internal] booting buildkit
#1 pulling image crazymax/buildkit:ubuntu
#1 pulling image crazymax/buildkit:ubuntu 5.3s done
#1 creating container buildx_buildkit_testgpu0
#1 creating container buildx_buildkit_testgpu0 1.9s done
#1 DONE 7.2s
testgpu
$ docker exec -it buildx_buildkit_testgpu0 nvidia-smi
Thu Mar 13 15:41:39 2025 +-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.124.06 Driver Version: 572.70 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3060 Ti On | 00000000:2A:00.0 On | N/A |
| 0% 55C P8 23W / 200W | 2955MiB / 8192MiB | 13% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 23 G /Xwayland N/A |
| 0 N/A N/A 25 G /Xwayland N/A |
| 0 N/A N/A 35 G /Xwayland N/A |
+-----------------------------------------------------------------------------------------+

Comment threaddriver/docker-container/driver.go Outdated
}
useInit := true
hc := &container.HostConfig{
Privileged: true,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess Privileged not really required for this test.

@crazy-max
crazy-maxforce-pushed the driver-ctn-gpu-request branch from 0cebbba to c17ea70CompareMarch 13, 2025 16:22
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
@crazy-max
crazy-maxforce-pushed the driver-ctn-gpu-request branch from c17ea70 to 3375782CompareMarch 13, 2025 16:44
@tonistiigi
tonistiigi merged commit 97f1d47 into docker:masterMar 13, 2025
// hasGPUCapability checks if docker daemon has GPU capability. We need to run
// a dummy container with GPU device to check if the daemon has this capability
// because there is no API to check it yet.
func (d *Driver) hasGPUCapability(ctx context.Context, image string, gpus []container.DeviceRequest) bool {

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Time this func takes:

  • ~15ms on MacOS (no GPU)
  • ~230ms on WSL (with GPU)
  • ~200ms on WSL (no GPU)
  • ~200ms on Windows (with GPU)
  • ~160ms on Windows (no GPU)

@crazy-max
crazy-max deleted the driver-ctn-gpu-request branch March 13, 2025 17:00
@crazy-max

Copy link
Copy Markdown
MemberAuthor

Added follow-up label to have an API for this capability on Moby side

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@crazy-max@tonistiigi