[Request]: FreeBSD container support

[davidchisnall]

Feature or enhancement request details

The FreeBSD container extensions are nearly standardised and the FreeBSD project is producing base images.

I've been running FreeBSD containers on macOS for about a year with a patched version of Podman that runs a FreeBSD VM to run the containers (I put together some infrastructure for building FreeBSD VM images that did everything that Podman expected), but it would be great to be able to run them with the one-VM-per-container model of this project and have an easy way of working with FreeBSD containers on macOS.

Are the requirements for how the VM image interacts with the host environment documented?

Code of Conduct

• I agree to follow this project's Code of Conduct

[jglogan]

@davidchisnall Good idea! I'm going to move this to the containerization repo as your initial work on this would involve the sorting out how to create a runtime abstraction around a VM running a BSD kernel.

Start with this bit of documentation: https://swiftpackageindex.com/apple/containerization/0.5.0/documentation/containerization/virtualmachineagent.

The implementation of this that runs the VMs for containers today is: https://github.com/apple/containerization/blob/main/Sources/Containerization/Agent/Vminitd.swift.

You'll see gRPC calls in there inside all of the VirtualMachineAgent protocol implementations. Those are going over a vsock into the VZVirtualMachine, and the server side is the Linux guest agent in Sources/vminitd.

Start hacking in containerization with the objective of enhancing the cctl tool so that you can launch a BSD container. There's little point in dealing with the container repo until you can do that. Good luck!

[dcantah]

Mirroring John, I think this would be sweet. Happy to help if you run into anything. This'd be the first time someone would stress the guest agent side of things in a non-linux world, so if any of the rpcs don't get you to the promise land please report what does and doesn't suck. Anything is up for change as it's pre 1.0

[dcantah]

I could either see you editing vminitd to be more agnostic, or honestly my preferred approach would just be a new freebsd guest agent that conforms to the guest agent protocol as I don't think we're ready to take in any drastic changes in vminitd that may be needed.

[davidchisnall]

Thanks. From a quick look at the protocol, I’d expect the FreeBSD version of the daemon to be 90% the same as the Linux one. I had a look at a few of the files in the implementation and there were only a few places that looked like they’d want to be different (and a lot of the differences are things that Swift already papers over).

The initial question was really aimed at an earlier step though. In particular:

• How are host files shared to the guest? (VirtIO-FS?)
• How does the guest get the mount points (explicit mount commands via the daemon? On Podman this is exposed via the qemufwcfg device, so I had to write a driver for that, but if they’re just RPC messages that’s simpler)
• How are commands issued to the guest (gRPC over VirtIO socket? Is it always a specific socket, or is there a dynamic discovery mechanism?)
• Do you expect a container runtime inside the VM, or do you simply treat the entirety of the inside of the VM as the container’s world? If the former, does this use the same protocol as existing container runtimes or something new? (It looks as if the vminitd stuff works with processes, rather than containers?)
• How is the kernel loaded? (Podman just has a disk image, do you do this or use something simpler like the Linuxboot protocol to load an initial image plus RAM disk?)

[dcantah]

• How are host files shared to the guest? (VirtIO-FS?)

Yep!

• How does the guest get the mount points (explicit mount commands via the daemon? On Podman this is exposed via the qemufwcfg device, so I had to write a driver for that, but if they’re just RPC messages that’s simpler)

Explicit mount commands via the framework here, but also for "bind" mounts from the host, we have to obviously tell the VMM that we're exposing this so it can actually be mounted in the guest. https://github.com/apple/containerization/blob/main/Sources/Containerization/VZVirtualMachineInstance.swift#L305-L307 https://github.com/apple/containerization/blob/main/Sources/Containerization/Mount.swift#L135, those two spots should help.

• How are commands issued to the guest (gRPC over VirtIO socket? Is it always a specific socket, or is there a dynamic discovery mechanism?)

Yep! And right now we mostly dial once on a known vsock port that the guest is listening on (1024), and reuse that conn for all communication with the guest (resizing terminal, wait(), kill() etc.)

• Do you expect a container runtime inside the VM, or do you simply treat the entirety of the inside of the VM as the container’s world? If the former, does this use the same protocol as existing container runtimes or something new? (It looks as if the vminitd stuff works with processes, rather than containers?)

We expect just our guest agent and this small runc-like shim binary that actually does the container launches. The entirety of the VM today is the container's playground though. It's a new protocol, although we'd love to either flesh out the shim binary to be a more full fledged OCI runtime, or support just packaging an existing one in the guest. Today the way we describe the container is still the OCI runtime spec, but the APIs on the host (LinuxContainer type) only expose what our shim binary can actually set up (for example it'd be weird to just let the user edit a runtime spec and pass it to our type, as we don't support seccomp and other things so they'd essentially be nops).

• How is the kernel loaded? (Podman just has a disk image, do you do this or use something simpler like the Linuxboot protocol to load an initial image plus RAM disk?)

The latter. Linuxboot and a block device that has the guest agent and shim binary in it that you pass to be the userland of the VM. I'd look at https://github.com/apple/containerization/blob/main/Sources/cctl/RootfsCommand.swift and https://github.com/apple/containerization/blob/main/Sources/cctl/RunCommand.swift for how we both create and use the initial filesystem mount to setup the VM.

[dcantah]

Let us know what your thoughts/plans are after diggin in! It's still obviously an extremely new project so I'm hesitant to take in support for a new platform already, but if the changes are somewhat self contained we'd definitely evaluate. The nice bit is I'd think most of the changes could even be in a separate repo as the bulk of the work likely will be in some new VirtualMachineInstance, Container and guest agent which are all pluggable.

[davidchisnall]

Thanks, that all looks feasible. There's Linuxboot support in FreeBSD already, I think the VirtIO-FS patches are nearly in (FUSE and VirtIO are there already, so it's mostly plumbing), especially if we can start by reusing ocijail as the runtime.

Oh, one more thing:

Do you extract the container images on the host or the guest? If it's the former (I guess it is, since APFS is much more sensible for this kind of thing than any standard Linux FS), do you depend on the page-sharing bits of VirtIO-FS (I think those are still to-do, the externally-owned-page bits in the buffer cache were added for ZFS and should be reusable, but it's not trivial to connect them up correctly)?

Don't expect anything fast from me - it will probably take a few weeks for me to dig through and understand this, but the FreeBSD Foundation is gently pushing on the container bits so I may be able to persuade some other folks to put in a bit more time (I've been trying to gently nudge the pieces into place to enable this kind of use case for the last few years and I think almost everything is there).

[dcantah]

Do you extract the container images on the host or the guest?

On the host. However, I didn't think about the image side of things.. The way we do the extracts and expose the image to a guest is not via virtiofs. We unpack the tar archives directly into a sparse file formatted with ext4 on the fly, and then we clonefile(2) the file when we want to run a container and attach this clone as a block device to the VM. I'm ignorant to FreeBSD's ext4 support, but today that is the only way we support exposing the images to a guest. We don't do unpack directly to files on the host for performance, and because APFS is case insensitive by default so some workloads just won't work (like if you wanted to compile the linux kernel that has quite a few files that only differ in casing).

[davidchisnall]

Do you extract the archives on the host with an ext4 driver or in the VM? If it’s the latter, that’s easy. I don’t think ext4 is a good fit for FreeBSD. There’s an ext2 (maybe 3?) driver and, if I remember correctly, the on-disk format is compatible, but it is not really a native FS and I think extended attributes are probably not working (among other things). Most container systems on FreeBSD use ZFS, but if you’re doing the CoW bits externally then UFS is probably fine. I’m not sure if the macOS UFS driver supports UFS2? If you’re manually creating the filesystem, the on-disk structures for UFS are fairly similar (the FreeBSD ext2 driver shares a lot of code with the UFS driver), they’re both simple inode-based filesystems (and, to create a disk image, you don’t need to touch any of the journalling or soft updates bits). The FreeBSD tools for building disk images work this way, creating a filesystem directly rather than going via the kernel).

[emaste]

ext2fs(4) has some ext3 and ext4 support

It currently implements most of the features required by ext3 and ext4 file systems. Support for Extended Attributes in ext4 is experimental. Journalling and encryption are currently not supported.

but indeed isn't a great fit for FreeBSD containerization

[dcantah]

@davidchisnall We do the extract/formatting on the host, the link I shared is an ext4 formatter implementation in Swift we'd written to do this Archive -> Ext4 block device dance for image pulls. The full flow is:

1. Pull tars
2. Make sparse file of size {configurable, but for the container tool is 512GiB}
3. Write tar data into sparse file with the EXT4 lib we have.

It's obviously non trivial to add new filesystem formatters, but we're open to it (we've been thinking about erofs lately on the linux side).

[jglogan]

The nice bit is I'd think most of the changes could even be in a separate repo as the bulk of the work likely will be in some new VirtualMachineInstance, Container and guest agent which are all pluggable.

It'd be nice in some respects if our Linux vminitd was in its own repo.