Unload model from memory/GPU when idle for set period

[jpm-canonical]

This issue was also reported by someone else here: openvinotoolkit/openvino#33665

With the current implementation it is not possible to use the GPU for other workloads while OVMS is running, as it will run out of free vRAM. Other LLM servers approach this by setting a user-defined idle timeout on a model. If no API requests are made to query a specific model for a specific amount of time, that model gets removed from memory. When a new request comes in for the model, it automatically gets loaded into memory again. It would be nice if OVMS had a similar feature.

An explanation of how llama.cpp does it can be found at https://github.com/ggml-org/llama.cpp/tree/master/tools/server#sleeping-on-idle

[proud-nerd]

Would be insanely helpful. I could utilize my GPU way more for different workloads. Right now I need to decide what models I need and make sure they fit into VRAM. Swapping models by hand if necessary is cumbersome and could be easily prevented with this approach. Especially helpful for more VRAM limited, consumer-oriented GPUs.

[exzile]

Picking this up. Here's the approach I'm implementing, scoped to the VRAM-heavy LLM continuous-batching path first. It follows the model llama.cpp established with --sleep-idle-seconds (PR ggml-org/llama.cpp#18228) — unload weights + KV cache on idle, lazy reload on the next request — adapted to the OVMS model-manager lifecycle.

Config (opt-in, no default behavior change)

• New idle_unload_timeout_seconds on the mediapipe/model config entry. 0 = disabled (default).

Idle tracking

• A per-graph "last activity" timestamp, updated only by real inference requests (/v3/chat/completions, /v3/completions, KFS infer). Status/health/readiness/metrics and model-status endpoints are exempt — they neither reset the idle timer nor trigger a reload, so health probes can't keep a model pinned in VRAM. (Same exemption llama.cpp applies to /health and /props.)

Unload (driven by the existing model-manager watcher)

• The watcher thread (already running ~1s) checks each graph; when now - lastActivity > timeout and there are no in-flight requests, it transitions the graph to a new UNLOADED state in PipelineDefinitionStatus and releases the heavy LLM node resource (the GenAiServable / ContinuousBatchingPipeline held via LLM_NODE_RESOURCES), freeing both the compiled model and the KV cache from VRAM while keeping the validated graph definition intact.
• Because OVMS unloads within its single-process model manager (no subprocess router), the teardown frees the VRAM cleanly with no residual per-model process footprint — avoiding the ~600 MiB-per-idle-subprocess limitation noted for llama.cpp's router mode.

Lazy reload on demand

• A request to an UNLOADED graph re-runs initializeNodes() to recreate the pipeline, then serves; the first request after idle pays the reload cost. This composes with --cache_dir on the CB path (--cache_dir not propagated to LLM continuous batching pipeline (regression vs 2025.4 promise) #4230) so the wake-up is a cache import rather than a full recompile.

Concurrency

• Reuses the existing graphqueue/state-machine handling already used for reload/retire: unload only when in-flight count is 0; requests arriving during unload wait for reload via the same mechanism.

Observability

• Log + metric for load/unload transitions, and the loaded/unloaded state surfaced on the model-status endpoint (the analog of llama.cpp's is_sleeping).

I'll keep the existing AVAILABLE/RETIRED semantics untouched and add UNLOADED as a distinct state so nothing changes when the feature is off. PR to follow against main, with tests covering unload-after-idle and reload-on-request.

[exzile]

Opened #4332 implementing this, verified end-to-end on GPU (idle unload frees resources after the timeout; the next request transparently wakes the model and serves).