Pages a guest has freed stay with the machine, because the guest has no way to say it is done with them. They leave only when the host runs short and reclaims them as it would any cold memory, which pages out what the guest would have handed over for nothing.
Nothing in Containerization attaches a memory balloon today. I have this implemented across both backends and would like to check it before opening pull requests, since one of the pieces is a policy decision rather than a mechanism.
Mechanism
Attach a balloon to each virtual machine and expose the size the machine should hold as setTargetMemorySize. Lowering it hands memory back to the host; raising it, up to the size the machine was created with, returns memory to the guest. Backends without a balloon report the operation as unsupported, in the same shape as pause and hotplug. Note that the guest keeps reporting the same total while the balloon holds its pages, and shows them as no longer free, because the balloon driver has to account for them in case it is asked to give them back.
Compact before driving it. Virtualization asks for the guest to be compacted before the balloon is driven, so the pages it gives up sit together well enough to be worth taking. Doing that inside the operation that lowers the target puts the recommended order in the one place that drives the device, rather than leaving each caller to remember it; the agent already carries the setting the guest needs, so it is the interface that grows rather than the protocol between the halves. https://developer.apple.com/documentation/virtualization/vzvirtiotraditionalmemoryballoondevice
Cloud Hypervisor gets a balloon with free page reporting turned on, so the guest hands back the pages it frees without anything having to choose a size. Kata drives the same device the same way, in src/runtime-rs/crates/hypervisor/ch-config/src/convert.rs. It sizes the balloon rather than the machine, so the balloon is asked to hold whatever the machine should not, and both backends answer the same request the same way. The types follow the Cloud Hypervisor OpenAPI spec, which carries size alongside deflate_on_oom and free_page_reporting, and desired_ram alongside desired_balloon on resize.
A pod's containers share one machine and one balloon, so bounding them together is the same operation a single container's machine takes, and the pod had no way to ask for it. The machine gathers the guest's free memory and drives the device, since dialing the agent and holding the balloon both already belong to it.
Policy, and the part I would most like an opinion on
Attaching a balloon leaves the question of what to set it to, and the guest cannot answer it: having freed memory, it has no way to say so. So the reading is taken on the host, from the anonymous memory the guest reports holding, and the rest is asked for back.
Reclaiming too far is the failure worth avoiding, because the guest then fetches back pages it was still using. The refault count is already reported, so a rise in it since the last reading is treated as evidence the last target was too tight, and memory is given back rather than the reading being trusted. The decision is a plain function of one reading, so it is tested without a virtual machine.
On top of that sits a loop that looks at what a machine's containers hold on a cadence and asks the machine to hold that plus headroom, so memory a workload frees goes back proactively instead of waiting for host pressure. It is off by default behind proactiveMemoryReclaim on the pod and container configurations; the cloud-hypervisor backend reports freed pages continuously either way.
Questions:
- Does a reclaim policy belong in this library at all, or should the library expose
setTargetMemorySize and the guest's memory reading and leave the policy to whatever drives it? - If it does belong here, is off-by-default behind a configuration flag the right default?
Testing note
Memory a guest has never touched is not backed on the host, so a balloon that takes only those pages moves nothing at all. A test has to fill a tmpfs and free it, leaving the machine holding pages the guest no longer wants, which is the state the device exists to resolve. Worth stating because a naive test passes while measuring nothing.
Pages a guest has freed stay with the machine, because the guest has no way to say it is done with them. They leave only when the host runs short and reclaims them as it would any cold memory, which pages out what the guest would have handed over for nothing.
Nothing in Containerization attaches a memory balloon today. I have this implemented across both backends and would like to check it before opening pull requests, since one of the pieces is a policy decision rather than a mechanism.
Mechanism
Attach a balloon to each virtual machine and expose the size the machine should hold as
setTargetMemorySize. Lowering it hands memory back to the host; raising it, up to the size the machine was created with, returns memory to the guest. Backends without a balloon report the operation as unsupported, in the same shape as pause and hotplug. Note that the guest keeps reporting the same total while the balloon holds its pages, and shows them as no longer free, because the balloon driver has to account for them in case it is asked to give them back.Compact before driving it. Virtualization asks for the guest to be compacted before the balloon is driven, so the pages it gives up sit together well enough to be worth taking. Doing that inside the operation that lowers the target puts the recommended order in the one place that drives the device, rather than leaving each caller to remember it; the agent already carries the setting the guest needs, so it is the interface that grows rather than the protocol between the halves. https://developer.apple.com/documentation/virtualization/vzvirtiotraditionalmemoryballoondevice
Cloud Hypervisor gets a balloon with free page reporting turned on, so the guest hands back the pages it frees without anything having to choose a size. Kata drives the same device the same way, in
src/runtime-rs/crates/hypervisor/ch-config/src/convert.rs. It sizes the balloon rather than the machine, so the balloon is asked to hold whatever the machine should not, and both backends answer the same request the same way. The types follow the Cloud Hypervisor OpenAPI spec, which carriessizealongsidedeflate_on_oomandfree_page_reporting, anddesired_ramalongsidedesired_balloonon resize.A pod's containers share one machine and one balloon, so bounding them together is the same operation a single container's machine takes, and the pod had no way to ask for it. The machine gathers the guest's free memory and drives the device, since dialing the agent and holding the balloon both already belong to it.
Policy, and the part I would most like an opinion on
Attaching a balloon leaves the question of what to set it to, and the guest cannot answer it: having freed memory, it has no way to say so. So the reading is taken on the host, from the anonymous memory the guest reports holding, and the rest is asked for back.
Reclaiming too far is the failure worth avoiding, because the guest then fetches back pages it was still using. The refault count is already reported, so a rise in it since the last reading is treated as evidence the last target was too tight, and memory is given back rather than the reading being trusted. The decision is a plain function of one reading, so it is tested without a virtual machine.
On top of that sits a loop that looks at what a machine's containers hold on a cadence and asks the machine to hold that plus headroom, so memory a workload frees goes back proactively instead of waiting for host pressure. It is off by default behind
proactiveMemoryReclaimon the pod and container configurations; the cloud-hypervisor backend reports freed pages continuously either way.Questions:
setTargetMemorySizeand the guest's memory reading and leave the policy to whatever drives it?Testing note
Memory a guest has never touched is not backed on the host, so a balloon that takes only those pages moves nothing at all. A test has to fill a tmpfs and free it, leaving the machine holding pages the guest no longer wants, which is the state the device exists to resolve. Worth stating because a naive test passes while measuring nothing.