Is there an existing issue for this?
Operating system
Linux (Fedora 43, kernel 7.1.4)
GPU vendor
Nvidia (CUDA)
GPU model
2x RTX 3090 (24 GB each)
GPU VRAM
24 GB per card
Version number
6.14.0-alpha (ghcr.io/invoke-ai/invokeai:main-cuda, image built 2026-07-31T04:12:19Z,
digest sha256:0dab6184f0fa5900d42a75fbcea9d99b86b4b6e78e75454f8619346bc0edaf36)
Browser
n/a - observed server-side
Python dependencies
torch 2.7.1+cu128, driver 610.43.03
What happened
InvokeAI allocates a CUDA context on a GPU at startup, before any generation has been requested,
and holds it for as long as the process runs. On this host that is a flat 256 MiB.
$ docker compose up -d --force-recreate invokeai # no requests sent to the server, ever
+15s invokeai 256 MiB
+45s invokeai 256 MiB
+90s invokeai 256 MiB
+150s invokeai 256 MiB
+240s 256 MiB
+330s 256 MiB
256 MiB is the characteristic size of a bare CUDA context (driver structures, kernel image, allocator
bookkeeping) rather than model weights - consistent with something calling into CUDA during startup.
The server is otherwise idle and correctly releases model weights; this is not the weight-cache
behaviour covered by #6354.
What you expected to happen
A freshly started server that has served no requests holds no VRAM. CUDA initialisation deferred until
the first operation that actually needs a device.
Why this matters
On a single-purpose box 256 MiB is noise. On a shared inference host it is not:
- It is common to run InvokeAI alongside an LLM server (llama.cpp / vLLM / llama-swap) on the same
cards. Those size their KV cache against free VRAM at load time, so a resident 256 MiB directly
reduces achievable context length or forces a smaller quant.
- The cost is paid continuously for a service that is used intermittently. A user who generates a few
images a day pays it 24/7.
- It defeats the usual mitigation. The documented way to reclaim VRAM is to restart the container after
use; that no longer reaches zero, because the fresh process re-allocates immediately.
- On 24 GB cards running 27B-class models the headroom is already measured in hundreds of MiB.
Related config that does not help, since it governs weights rather than the context:
enable_partial_loading: true, model_cache_keep_alive_min, device_working_mem_gb.
How to reproduce
- Start the server with a CUDA GPU available and send it no requests.
nvidia-smi --query-compute-apps=pid,used_memory --format=csv- Observe the
invokeai-web process holding a CUDA allocation within ~10-15 s of startup.
Additional context
This appears to be a regression - the previous main-cuda image did not behave this way. On this
host a freshly restarted server sat at 0 MiB until the first generation, and restarting the container
was an effective way to reclaim VRAM after use. That changed with the image pulled on 2026-07-31.
I cannot supply the last known-good version number: the host auto-updates this tag, so the previous
image had already been replaced by the time the change was noticed. The behaviour difference itself was
unambiguous to someone who watches VRAM on this machine daily. If the diff between the current
main-cuda digest above and its immediate predecessor is small, that should localise it quickly.
If maintainers believe this has always been the case, the report stands as a feature request for lazy
CUDA initialisation instead.
Ruled out: container GPU wiring. This host had recently switched from the legacy
runtime: nvidia + explicit /dev/nvidia* device mounts to CDI (nvidia.com/gpu=all) on the same
afternoon. Reverting that service to the exact previous configuration and recreating it reproduced the
allocation identically (256 MiB within 15 s, flat for 5.5 minutes), so the GPU wiring is not involved.
A guard on device initialisation, or an opt-in lazy_cuda_init style setting, would resolve it for
shared-GPU deployments.
Is there an existing issue for this?
releasing model weights. This is about the CUDA context itself, allocated before any model is loaded.
Operating system
Linux (Fedora 43, kernel 7.1.4)
GPU vendor
Nvidia (CUDA)
GPU model
2x RTX 3090 (24 GB each)
GPU VRAM
24 GB per card
Version number
6.14.0-alpha (
ghcr.io/invoke-ai/invokeai:main-cuda, image built 2026-07-31T04:12:19Z,digest
sha256:0dab6184f0fa5900d42a75fbcea9d99b86b4b6e78e75454f8619346bc0edaf36)Browser
n/a - observed server-side
Python dependencies
torch 2.7.1+cu128, driver 610.43.03
What happened
InvokeAI allocates a CUDA context on a GPU at startup, before any generation has been requested,
and holds it for as long as the process runs. On this host that is a flat 256 MiB.
256 MiB is the characteristic size of a bare CUDA context (driver structures, kernel image, allocator
bookkeeping) rather than model weights - consistent with something calling into CUDA during startup.
The server is otherwise idle and correctly releases model weights; this is not the weight-cache
behaviour covered by #6354.
What you expected to happen
A freshly started server that has served no requests holds no VRAM. CUDA initialisation deferred until
the first operation that actually needs a device.
Why this matters
On a single-purpose box 256 MiB is noise. On a shared inference host it is not:
cards. Those size their KV cache against free VRAM at load time, so a resident 256 MiB directly
reduces achievable context length or forces a smaller quant.
images a day pays it 24/7.
use; that no longer reaches zero, because the fresh process re-allocates immediately.
Related config that does not help, since it governs weights rather than the context:
enable_partial_loading: true,model_cache_keep_alive_min,device_working_mem_gb.How to reproduce
nvidia-smi --query-compute-apps=pid,used_memory --format=csvinvokeai-webprocess holding a CUDA allocation within ~10-15 s of startup.Additional context
This appears to be a regression - the previous
main-cudaimage did not behave this way. On thishost a freshly restarted server sat at 0 MiB until the first generation, and restarting the container
was an effective way to reclaim VRAM after use. That changed with the image pulled on 2026-07-31.
I cannot supply the last known-good version number: the host auto-updates this tag, so the previous
image had already been replaced by the time the change was noticed. The behaviour difference itself was
unambiguous to someone who watches VRAM on this machine daily. If the diff between the current
main-cudadigest above and its immediate predecessor is small, that should localise it quickly.If maintainers believe this has always been the case, the report stands as a feature request for lazy
CUDA initialisation instead.
Ruled out: container GPU wiring. This host had recently switched from the legacy
runtime: nvidia+ explicit/dev/nvidia*device mounts to CDI (nvidia.com/gpu=all) on the sameafternoon. Reverting that service to the exact previous configuration and recreating it reproduced the
allocation identically (256 MiB within 15 s, flat for 5.5 minutes), so the GPU wiring is not involved.
A guard on device initialisation, or an opt-in
lazy_cuda_initstyle setting, would resolve it forshared-GPU deployments.