Add file handle attachment support

[curdbecker]

Based on @torarnv's existing PR this PR adds also exposes a VZFileHandleNetworkDeviceAttachment via aFileHandleNetworkInterface to allow connection a container to an arbitrary file handle provided by another service.

This attachment is used by tools like nirs/vmnet-helper to bridge the container to an existing host interface. It opens also up the possibility to add a similar service directly into container to circumvent the need for without having the binary signed with the com.apple.vm.networking entitlement, e.g. during container development.

[nirs]

@curdbecker sounds like cool feature! but with VZFileHandleNetworkDeviceAttachment it will not be easy to configure port forwarding and other options that are much easier with the native vmnet support introduced in macOS 26.

[curdbecker]

@nirs thanks a lot :) Yes, the native bridging support is likely the go-to-alternative in any case, but if you maybe want to take a look at the linked PR in container from torarnv and the discussion as well as mine, then unfortunately the entitlement blocks using the integrated bridging support as far as I understand.

I also took a look at your new project and it seems to be missing there as well, right? Without the entitlement one can only choose between SIP and bridging. Then I would always choose SIP in a heartbeat.

The goal here from me is also that I do not want to have classical container features like port forwarding etc., but I would like to have containers where I do have a great deal of control about the underlying VM, e.g. kernel, init ad command line, as well as still benefiting from everything great the container eco system has brought us.

The worse performance (although you seem to have put a great deal of work into optimization - that is something that would be as well also possible here) and the overhead of having a bridge would be okay for me when I need it :)

Btw, I used your vmnet-helper with this little tool when creating a drop-in of Docker Desktop's VM runner to achieve the goals above... That's how I learned of it. Very nice! :)
For me the entire "Docker modding with VM runner replacement" idea immediately became then more of a scientific nature when I learned about containers growing features, since container is so much more suited for this purpose - and it is basically the identical stack - just not hidden and obfuscated.

[curdbecker]

Updated to latest containerization - to work with container version 1.0.0 onwards in the related PR :)

[crosbymichael]

Thanks for the changes. Ping me when done and we can get this PR in

[nirs]

@curdbecker,

This attachment is used by tools like nirs/vmnet-helper to bridge the container to an existing host interface. It opens also up the possibility to add a similar service directly into container to circumvent the need for without having the binary signed with the com.apple.vm.networking entitlement, e.g. during container development.

com.apple.vm.networking entitlement is not required since macOS 26. You can use com.apple.security.virtualization to use vmnet without root. This is not documented but this is the way the container can use vmnet without root.

[curdbecker]

@nirs Yes, vmnet works, but as @torarnv pointed out bridging still apparently still requires the entitlement. He also had to disable SIP in order to make that work. And from what I have seen in general asking for a custom NIC mac address doesn't work as well. Then the API call fails. I think this is likely connected to the bridging restriction. Or am I wrong here? That would surprise me quite a lot 😱

[curdbecker]

@crosbymichael Thanks for the review. Really excited that we can get this in, since especially the DHCP PNP part from tornav is quite interesting. Let me know if I should have missed anything (my ADHD self can't always too certain even after triple checking things...) :)

[torarnv]

@crosbymichael Have you had a look at the original PR in apple/container#1463 and the discussion there with @dcantah and @jglogan?

I don't mind exactly which PR gets merged if it brings bridge-mode to container (and thanks for following up on this @curdbecker!) ☺️ But it's a bit unclear to me what the porcelain side of this PR will look like.

Are we targeting the FileHandleNetworkInterface, or BridgedNetworkInterface (as apple/container#1463 does), or both (as apple/container#1622 does)?

What are the tradeoffs between the two when it comes to:

• Performance
• Configurability (port mapping, etc)
• Setup (additional helpers running)
• Other?

For the com.apple.vm.networking entitlement issue specifically, I don't see that as a blocker for using BridgedNetworkInterface, considering container is an Apple project.

But if we can have both, where BridgedNetworkInterface is available from official releases, with the entitlement, and FileHandleNetworkInterface is available for development, and more esoteric network setups, then I guess win-win? 😄

[curdbecker]

@torarnv Thanks for getting it started. It helped me quite a lot so far in most of my use cases. I actually run docker solely in container now and with bridging, separate docker daemons etc. basically all my issues are solved :)

That's why I also just put the changes on top of your PR and now rebased them. Didn't make sense to open too many several independent PRs here - and it would also have confused me quite a bit locally...
Feel free to also rebase the other PR and I guess we can merge them both, but I made sure that the original attribution is preserved while indicating that I meddled with it 😂
It's quite a cool solution :)

That being said: I actually do not see an "either or" here - both of them make sense from my point of view - even independently. Especially in the context of containerization as a set of tools to build container-based setups.
There are quite a lot of setups beyond bridging where using a simple file handle as a network interface can come in quite a bit handy.

For instance, one of the things I am also using it for is emulating an entire network environment for testing. And it is quite helpful that the environment then is fully configurable inside a single process and also remains completely transparent to the outer OS networking infrastructure. Then things don't get as messy as before with explicit VM networking, container namespaces etc.

What I haven't done yet, but peaked my interested while I was looking into the whole Docker networking mess was also something like ns-3. The way I understand it, file handle support would allow also to plug it directly in there... and then replace the hack-ish solutions that I had before. (Although colleagues warned me that it can also simulate arbitrary networks and can therefore get quite complicated quite soon...)

It would be also interesting to expose file handle networking as an option directly in container. The whole dedicated bridging to get around the entitlement is although another topic anyways... 😂

[torarnv]

@curdbecker Thanks for the detailed summary ☺️ I understand and sympathize with your use case, and I appreciate you preserving the attribution!

I'd be happy if we can have both, so thumbs up from me on this PR 👍🏻

[nirs]

@nirs Yes, vmnet works, but as @torarnv pointed out bridging still apparently still requires the entitlement.

The VZ bridged device attachment requires root or com.apple.networking entitlement, but with VZ file handle device attachment VMNET_BRIDGED_MODE works fine with com.apple.security.virtualization.

Example run in bridged mode:

% build/vmnet-run --operation-mode bridged --shared-interface en10 -- sleep 1 INFO [runner] running on macOS 26.5.1INFO [main] running build/vmnet-helper v0.12.0-5-g29ce071 on macOS 26.5.1INFO [main] running as uid: 501 gid: 20INFO [main] using bulk_forwarding: true{"vmnet_write_max_packets":256,"vmnet_read_max_packets":256,"vmnet_mtu":1500,"vmnet_interface_id":"4FE2BC19-4803-48CA-B4E7-7A01EF6DEF41","vmnet_max_packet_size":1514,"vmnet_mac_address":"d6:f7:f4:ba:ec:95"}INFO [main] started vmnet interfaceINFO [main] started host forwardingINFO [main] started vm forwardingINFO [main] waiting for terminationINFO [main] received signal Terminated: 15INFO [main] stopped vmnet interface

He also had to disable SIP in order to make that work.

I would not disable SIP, this is your safety net.

And from what I have seen in general asking for a custom NIC mac address doesn't work as well.

I had no issues using the mac address returned from vmnet or using my own generated MAC address:
https://github.com/nirs/vmnet-helper/blob/6076be89236965b0751605585062c09c7d622d09/testing/mac.py#L7

Tested with vfkit, krunkit, and qemu.

Then the API call fails. I think this is likely connected to the bridging restriction. Or am I wrong here? That would surprise me quite a lot 😱

Not very clear what you tried and what failed

[torarnv]

The VZ bridged device attachment requires root or com.apple.networking entitlement

He also had to disable SIP in order to make that work.

I would not disable SIP, this is your safety net.

To clarify, a properly signed container binary with com.apple.vm.networking doesn't require SIP disabled, nor root. The premise of landing support for bridged network via BridgedNetworkInterface is that container ships with this entitlement officially. The SIP/AMFI disable case I described earlier was for my own development of the feature itself.

[nirs]

The VZ bridged device attachment requires root or com.apple.networking entitlement

He also had to disable SIP in order to make that work.

I would not disable SIP, this is your safety net.

To clarify, a properly signed container binary with com.apple.networking doesn't require SIP disabled, nor root. The premise of landing support for bridged network via BridgedNetworkInterface is that container ships with this entitlement officially. The SIP/AMFI disable case I described earlier was for my own development of the feature itself.

Interesting - what do you mean by properly singed?

Can this work for vfkit?
crc-org/vfkit#144

[curdbecker]

@nirs Yes, vmnet works, but as @torarnv pointed out bridging still apparently still requires the entitlement.

The VZ bridged device attachment requires root or com.apple.networking entitlement, but with VZ file handle device attachment VMNET_BRIDGED_MODE works fine with com.apple.security.virtualization.

@nirs - sorry, but it seems that we are talking past each other. That fact is really out of the question. This statement was meant in regard to the BridgedNetworkInterface that also ended up in this PR, since it is quite hard to have them separated when building/developing the related PR in container. This one still requires the entitlement as tornav pointed out here :)

And for me running in bridge mode without the OS-level bridging support and a container-integrated version of vmnet-helper now looks like this:

% container run --rm -it --name test --network isolated-test-vlan-44-en13 alpine:3 sh
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether aa:ec:cc:1c:b6:72 brd ff:ff:ff:ff:ff:ff
inet 10.44.44.16/24 brd 10.44.44.255 scope global eth0
valid_lft forever preferred_lft forever

He also had to disable SIP in order to make that work.

I would not disable SIP, this is your safety net.

Yes, of course, sorry - like I said this was not in any way meant a suggestion. In doubt also take a look at the related discussion. Nobody should ever run in production without SIP for basically any reason in my opinion :)

Then the API call fails. I think this is likely connected to the bridging restriction. Or am I wrong here? That would surprise me quite a lot 😱

Not very clear what you tried and what failed

Well, sorry, I have to look it up again, since I am in quite a lot of different topics now, but I basically had accepted it. I think it was vmnet_start_interfacehere. I also took a look at your code for reference again and didn't see anything related to custom MACs, but well, I also do not really need them, so I deferred that to the point where someone would actually interested in merging the code - no need to waste too much effort :)

[curdbecker]

I'd be happy if we can have both, so thumbs up from me on this PR 👍🏻

@tornav Thanks a lot :) Yes, I would also glad to have both - even without the overhead of "manual bridging". Then I would also opt to your implementation directly, since it is certainly way more efficient and straightforward.

It would be awesome if maybe someone here from the amazing Apple folks that created these projects could maybe "lobby" for a way to get around the requirement of an explicit signature for this entitlement from Apple. I would be amazing if we could this also using normal development signing as for most other entitlements.

[torarnv]

To clarify, a properly signed container binary with com.apple.vm.networking doesn't require SIP disabled, nor root. The premise of landing support for bridged network via BridgedNetworkInterface is that container ships with this entitlement officially. The SIP/AMFI disable case I described earlier was for my own development of the feature itself.

Interesting - what do you mean by properly singed?

I mean that the binary is signed with the com.apple.vm.networking entitlement using a signing identity that is approved by Apple to use the com.apple.vm.networking entitlement, so that the resulting binary will run unmodified on vanilla systems.

https://github.com/openai/tart is an example of a product that does that.

Can this work for vfkit? crc-org/vfkit#144

As long as the project (vfkit) is granted the right to use the entitlement by Apple it should work AFAIU.

Which is why having container require the entitlement seems like a small hurdle 😄

[curdbecker]

To clarify, a properly signed container binary with com.apple.vm.networking doesn't require SIP disabled, nor root. The premise of landing support for bridged network via BridgedNetworkInterface is that container ships with this entitlement officially. The SIP/AMFI disable case I described earlier was for my own development of the feature itself.

Interesting - what do you mean by properly singed?

I mean that the binary is signed with the com.apple.vm.networking entitlement using a signing identity that is approved by Apple to use the com.apple.vm.networking entitlement, so that the resulting binary will run unmodified on vanilla systems.

I also just looked up the again the relevant apple documentation, see here. It does not tell you thought that you can't just self-sign this entitlement or use a "normal" developer certificate. Thankfully tornav discovered that before I have, since I would likely have been quite frustrated 😂 😂

[nirs]

Interesting - what do you mean by properly singed?

I mean that the binary is signed with the com.apple.vm.networking entitlement using a signing identity that is approved by Apple to use the com.apple.vm.networking entitlement, so that the resulting binary will run unmodified on vanilla systems.

Apple approval is not an acceptable solution for an open source project.

[torarnv]

Interesting - what do you mean by properly singed?

I mean that the binary is signed with the com.apple.vm.networking entitlement using a signing identity that is approved by Apple to use the com.apple.vm.networking entitlement, so that the resulting binary will run unmodified on vanilla systems.

Apple approval is not an acceptable solution for an open source project.

I'll leave you to make that choice for vfkit and related projects.

For https://github.com/apple/containerization and https://github.com/apple/container I see no reason why the official releases can't be built with a signing identity that is given this entitlement by the very entity that manages the project.

[curdbecker]

Well, of course I forgot to re-execute the tests locally yesterday evening, so the rebase actually was clean, but another non-optional use of the IPv4 address sneaked in there. Just fixed that and all tests pass for me locally now :)

[Rjvs]

@curdbecker just noting the PR still has the changes requested tag, which might be why @crosbymichael hasn't re-reviewed it despite your request... is it worth addressing that last change request some other way, so this PR doesn't just languish?

[curdbecker]

@Rjvs Thanks for the hint... Yeah, I do not know. I already pressed the re-request review button. Now I did it again, but the requested change does not disappear? And I am either ADHD-blind right now or there are really no further changes in the diff anymore and this is really a Github bug? Do you see anything?

I thought actually the heated discussion that ensued here last time was what made the pull request a bit less interesting?

[curdbecker]

When I press the button, then there is at least a 404 that comes back for the unread_timeline request.... Oh, oh 😂

[curdbecker]

@crosbymichael Ok, rebased to the current master again. And adapted to your changes in CHInterface.swift from last month also, so that all tests again pass and we build cleanly also on Linux.

Maybe we can now talk about what's going on here again? It's a bit annoying to keep adding ? repeatedly over months now 😂 :)