[Request]: vminit API for block device operations, initially for freeze/thaw.

[jglogan]

Feature or enhancement request details

Adding a guest agent interface call for control operations (typically ioctl()) on block devices can enable some valuable storage features for CZ-based apps:

• Freeze/thaw to ensure clean container commit --live ([Request]: add container export for running containers. container#1400)
• Trim to relinquish freed filesystem blocks to the host ([Request]: Relinquish freed filesystem blocks back to macOS. container#1402).

Let's do the API design in this issue first, before submitting any PR. We don't want individual guest agent calls for each - that would be a mess.

Code of Conduct

• I agree to follow this project's Code of Conduct

[saehejkang]

@jglogan Couple of things

1. Looks like the trim work overlaps with these two issues. Maybe those dupe issues should be closed?

• [Request]: Guest agent API call for returning freed disk blocks to macOS. #542
• [Request]: Storage - Automatically reclaim unused disk space from containers with minimal user interaction #414.

2. Is this issue going to be only focusing on the implementation of freeze/thaw? Or as you mentioned, we want one API that does: freeze, thaw, trim?

3. Any tips on how to get started into understanding the containerization side of things? Here is what I am thinking below:

• Start by looking into Vminitd and understanding how the API works with what exists now.
• Next, I will take a look into how the Ext4 Filesystem works and see how it is being used currently.
• Finally, look into the ContainerizationOS package and understanding how syscall or sysctl works? I feel I need to implement something similar for the ioctl calls (FSTRIM, FSFREEZE).

I want to make sure I gain a solid understanding of things before I even think of coming up with an API design 🚀.

CC: @dcantah in case you had any tips :)

[jglogan]

@saehejkang thanks!

1. I looked for that issue and couldn't find it, so I created a new one. Let's use the earlier [Request]: Storage - Automatically reclaim unused disk space from containers with minimal user interaction #414 and I'll merge any useful informative content from [Request]: Guest agent API call for returning freed disk blocks to macOS. #542 and then close the newer issue.
2. Let's do the general API design for a single call for block device operations here in this issue (and keep bothering @dcantah as he has opinions here). The PR for this issue will cover only implementing the freeze/thaw variant of the API call. [Request]: Storage - Automatically reclaim unused disk space from containers with minimal user interaction #414 will cover trim (which has the potential to be more complicated depending on the mechanism/policy split).
3. Let's look at that step by step.

• Familiarize yourself with the RPC stuff:
  • In the main project, VirtualMachineAgent (the protocol for the client), Vminitd (the implementation over the generated gRPC client), SandboxContext.proto (the protobuf definition, see also the make protos target).
  • In the vminitd subproject, really the main thing you need to look at is Server+GRPC.swift, you'll implement the server side of the call there. Looking at the other code will give you a sense for how to make system calls in Swift.
• You don't need to know anything about ext4fs for this. These ioctls operate at the virtual file system level.
• I don't think you'll need to worry about ContainerizationOS here. This is Linux swift code so you can just call ioctl() (see RunCommand.swift in the vminitd subproject).

The API will definitely need a pathname to the block device, let's call it devicePath for now. I've done the trim function bit in a past life so I can spell out what it will look like in terms of pseudocode; the freeze/thaw case is similar and the device validation code will be shared for whatever operations get performed.

// #define FITRIM _IOWR('X', 121, struct fstrim_range)
FITRIM = UInt32(0xc0185879)

trimBlockDevice(path: FilePath):
 if not path.isAbsolute():
 throw "filesystem trim requires an absolute path"

 finfo = stat()
 rc = lstat(path, finfo)
 if rc != 0:
 throw "filesystem trim stat failed: \(errno)"

 if finfo.mode & MODE_SYMLINK != 0:
 throw"filesystem trim refuses to process a symlink"

 fd = open(r.Path)
 if fd == -1:
 throw "filesystem trim open failed: \(errno)"

 defer fd.close()

 trange = fstrim_range
 trange.start = 0
 trange.len = UInt64.max
 trange.minlen = 0
 rc = ioctl(fd, FITRIM, &trange)
 if rc == -1:
 throw "filesystem trim failed: \(errno)

[saehejkang]

In the vminitd subproject, really the main thing you need to look at is Server+GRPC.swift, you'll implement the server side of the call there. Looking at the other code will give you a sense for how to make system calls in Swift.

@jglogan 😆 My xcode was bugging and the vminitd package was completely missing. I was so confused, as I thought there was some sort of server side call completely missing. I was reviewing your work on #669, when I saw the package! Just as you responded here, I figured out the things you mentioned above (the RPC stuff) and have one clarifying question.

---

I understand that vminitd creates a gRPC server inside the VM listening on a vsock port. So these APIs are created to simply interact with the guest agent (make modifications, etc.). But how does container even use them and is there a good command to follow in order to see how it all comes together?

[jglogan]

Container doesn't know about vminitd.

You need to look for things like this in Containerization: https://github.com/apple/containerization/blob/main/Sources/Containerization/LinuxContainer.swift#L558.

The call site for that create func in container is: https://github.com/apple/container/blob/main/Sources/Services/ContainerSandboxService/Server/SandboxService.swift#L249

[dcantah]

As John said, container is just a consumer of this library, it doesn't care about vminitd. The library has a host-side (macOS) and a guest-side (Linux VM). They communicate over gRPC on vsock port 1024 which you found.

• vminitd (vminitd/Sources/vminitd/) — the guest agent. It boots inside the VM, mounts
  essential filesystems, and starts a gRPC server on yee olde vsock port.
• Vminitd client (Sources/Containerization/Vminitd.swift) — the host-side gRPC client stub that
  wraps all the RPC calls.

The gRPC service is called SandboxContext (defined in Sources/Containerization/SandboxContext/SandboxContext.proto) and exposes a bunch of RPCs (we will add more I promise LOL) for process lifecycle, mounts, networking, sysctl, etc. Essentially everything to setup the system such that it's ready to take workloads (containers).

Following the Container Create + Start Flow

The best command to trace is create → start on LinuxContainer:

1. LinuxContainer.create() (Sources/Containerization/LinuxContainer.swift)

• Creates and boots a VM
• Calls vm.dialAgent() which opens a vsock connection to port 1024 and returns a Vminitd client
• Through that client, sends RPCs to set up the guest:
• standardSetup(): mounts /tmp, /dev/pts, etc.
• mount(): mounts the rootfs and writable overlay
• addressAdd(), up(), routeAddDefault(): configures networking
• configureDNS(), configureHosts(): writes /etc/resolv.conf and /etc/hosts

2. LinuxContainer.start()

• Dials the agent again
• Creates a LinuxProcess which sets up vsock listeners for stdio (stdin/stdout/stderr each get
  their own vsock port)
• Sends CreateProcess RPC: vminitd decodes the OCI spec and creates a ManagedContainer
• Sends StartProcess RPC: vminitd launches the actual process (via vmexec or runc), returns
  the PID.

3. On the vminitd side (vminitd/Sources/vminitd/Server+GRPC.swift)

• createProcess(): decodes the OCI spec, applies alterations (uid/gid resolution,
  PATH/HOME setup), creates a ManagedContainer
• startProcess(): calls ManagedContainer.start() which launches the process

Virtualization.framework

All the virtualization bits right now are backed by interactions to Virtualization.framework. All of this code is mostly shoved in VZVirtualMachineInstance. We made a protocol (VirtualMachineInstance) such that you could kind of plug in any virtualization stack, so long as they support the features we need under the hood (vsock, virtio-block, virtiofs, virtio-net) to do IO. Related to Virtualization.framework, how vminitd even gets in the VM is probably worth mentioning:

vminitd and vmexec binaries are compiled as static Linux binaries (musl libc, cross-compiled via Swift's static Linux SDK; see vminitd/Makefile). They get packaged into an EXT4 disk image that becomes the VM's root filesystem.

The flow:

1. cctl rootfs create (Sources/cctl/RootfsCommand.swift) builds a tar.gz archive containing a minimal rootfs: /sbin/vminitd, /sbin/vmexec, and standard directories (bin, dev, sys, proc, run, tmp, mnt, var).
2. That tar.gz is unpacked into a sparse EXT4 filesystem image via EXT4Unpacker (Sources/Containerization/Image/Unpacker/EXT4Unpacker.swift). The library has its own pure-Swift EXT4 formatter in ContainerizationEXT4.
3. The Makefile orchestrates this with make init (Makefile).

Attaching It to the VM

In Mount.swift, when a mount has runtimeOptions: .virtioblk, it's attached as a VZVirtioBlockDeviceConfiguration backed by a VZDiskImageStorageDeviceAttachment. The init filesystem gets the first block device allocation (/dev/vda) via AttachedFilesystem, which uses a simple character allocator ('a', 'b', 'c'...).

Kernel Command Line

The key piece for asking the guest to run vminitd as its init process is in Kernel.swift, linuxCommandline(initialFilesystem:):

args.append("init=/sbin/vminitd") // run vminitd as PID 1
args.append("ro") // mount root read-only

// for ext4 specifically:
args.append("rootfstype=ext4")
args.append("root=/dev/vda") // the init block device

The default args also include console=hvc0 (so we can get serial console logs) and tsc=reliable.

This gets wired into the boot loader in VZVirtualMachineInstance.swift.

[saehejkang]

@dcantah Thank you for the in-depth explanation! Definitely lots to wrap my head around, but I started with focusing on the connection between the guest-agent/gRPC client and the Virtualization Framework. It is starting to make a lot more sense, but naturally as I dig deeper, I have more questions (that I hope you have answered in the other sections). I am sure as I continue tackling issues in this project, it will all come together nicely 💯 .

[saehejkang]

@jglogan @dcantah

API Design for Freeze/Thaw

The design is based off the assumption that this will be a generic API for filesystem-level operations on block devices. For the first iteration, the API focuses on freeze and thaw only.

Name

Spent a good time mulling on this and trying to come up with a generic name but am open to ideas here 🤔 .

filesystemOperations - This makes sense if we expect to add more operations past freeze/thaw/trim.
manageFilesystem - Similar to the one above and only works with more operations.
freezeFilesystem - This one is good as its scope is just for freeze/thaw (could maybe work with the addition of trim).

Request Body

devicePath: string # Path to the filesystem
duration: int64 # How long the filesystem should remain frozen (ms)

• The duration will be defaulted to a reasonable value.
• For an initial passthrough lets start with a single device path and not worry about multiple paths.
• If no device paths are specified, the command initiates freeze/thaw on all mounts for the running container.

Response Body

success: bool # Whether the filesystem freeze succeeded
frozenAt: int64 # Timestamp when freeze was applied
thawedAt: int64 # Timestamp when thaw was applied
error: string # Error message if freeze/thaw failed (empty if success)

EDIT: The timestamp fields do not add much value and can be omitted

Pseudocode

I did some extensive research into all the ioctl operations and found a good way to intertwine the operations.

1. Perform device validation (from the code John pasted above)
2. Freeze the filesystem
3. Based on the duration, wait, and then thaw the filesystem

Further Enhancements with Trim

If we wanted to perform the trim operation, we could just add an optional flag to the request body. If the flag is set to true, perform trim first and then skip the freeze/thaw operations.

Other Design Decisions

I had an idea of making one of the request fields an array of strings that specifies the operations to be performed. I can see there potentially being some overlap in functionality, but it also feels like it could get quite complicated. I know we don't want multiple APIs for filesystem operations BUT I think splitting up freeze/thaw and trim might make the most sense.

[dcantah]

I think we'll want one RPC, so I agree on that, but protobuf supports a union-like functionality called oneof that we can use to take in different parameters for the various fs ops we may want to do. For example, something like this:

message FitrimParams {
 uint64 start = 1;
 uint64 len = 2;
 uint64 minimum_len = 3;
 }

 message FitrimResult {
 uint64 trimmed_bytes = 1;
 }

 message FilesystemOperationRequest {
 // Path to the mounted filesystem.
 string path = 1;
 oneof operation {
 FitrimParams fitrim = 2;
 google.protobuf.Empty freeze = 3;
 google.protobuf.Empty thaw = 4;
 }
 }

 message FilesystemOperationResponse {
 oneof result {
 FitrimResult fitrim = 1;
 }
 }

 // Perform a filesystem operation on a mounted filesystem.
 rpc FilesystemOperation(FilesystemOperationRequest) returns (FilesystemOperationResponse);

The data that is required for all requests (the path to the fs) lives outside of the oneof, and then the operation specific data lives inside. I didn't see that linux takes in any extra data for freeze or thaw, so left them as empty, but we could just define empty message FiFreezeParams if we want to to be safe so we could add fields later on without it being a breaking change.

[saehejkang]

I think we'll want one RPC, so I agree on that, but protobuf supports a union-like functionality called oneof

I wasn’t aware of oneof and can see how it allows us to accept different parameters for filesystem operations. I think I initially had the right idea of representing multiple operations, but wasn’t sure how that would translate into RPC! I also like how that only allows for a SINGLE operation per request, so it makes things simpler. Looks like I had the right idea about what we could name the request, response, RPC but I was overthinking this entire design 😅 .

I didn't see that linux takes in any extra data for freeze or thaw, so left them as empty, but we could just define empty message FiFreezeParams

Are there any params we want to add now, even though linux does not require anything? My only thought would be duration, so that there does not have to be two separate requests in order to execute a freeze and the immediately after, a thaw.