Swap to APIServer for all communications by dcantah · Pull Request #628 · apple/container

dcantah · 2025-09-17T18:47:07Z

Today, we have a sort of odd model all things considered. We talk to the APIServer to do the initial container creation, which mostly all of the work is just registering the runtime helper with launchd. After this registration, almost all communication from a client is talking directly to that runtime helper. This has a couple rather annoying issues in that we now need a sort of event/notification channel that the helper will establish with the APIServer for errors/start/exited cases.

If instead of talking to the runtime helper directly, we instead took a detour through the APIServer, the APIServer is clued into exactly what order of operations is occurring. This makes "did starting the container fail? Okay we should clean up" scenarios much simpler, and it also simplifies the clients quite a bit as they don't need this split brained client model, everyone just talks to the APIServer. This change is in pursuit of that. I have reworked our clients, the ContainerService and some of our XPC types to accomplish it.

The biggest "contract" change is in the SandboxService. Today we have on the flag when we register any runtime helper that makes any xpc messages wake up the registered process. This isn't great in scenarios where the process have may crashed, or it exited normally and we're just trying to invoke an RPC on it. Today the helper would spawn again and try and answer our request. It'd be much nicer if we have a connection object that will become invalid if the process that vended it to us is gone. To accomplish this, now the runtime helpers will listen on an anonymous xpc connection and vend endpoints from this via only one handler exposed by the SandboxService (createEndpoint). From that point onwards all communication will be through the endpoint the service vended a client.

Pushed the wrong button

jglogan · 2025-09-19T01:00:24Z

 return container
 }

- private func gracefulStopContainer(_ lc: LinuxContainer, stopOpts: ContainerStopOptions) async throws {


Note about code not modified in this PR:
privatenonisolatedfunc configureProcessConfig()
This should be able to become:
privatestaticfunc configureProcessConfig()
Same for closeHandle().

Another lowkey obsessive nit in the non-modified code above - move getDefaultNameserver out from between the configure...() funcs. All the private methods could probably stand to be reordered sensibly. In a swift file that's pushing 1200 lines it will help new developers.

Mcrich23 · 2025-09-19T04:47:40Z

Hey! So sorry, but in an effort to make plugin development more possible #603 and #635 have lead to the CLI folder being renamed to ContainerCommands and that will impact the merging of your pull request. Just an FYI, so you understand the issue when you are resolving conflicts.

dcantah · 2025-09-19T10:24:02Z

Converting to draft because there's a small cz change that I want in that will allow us to "fix" some behavior in stop()

Today, we have a sort of odd model all things considered. We talk to the APIServer to do the initial container creation, which mostly all of the work is just registering the runtime helper with launchd. After this registration, almost all communication from a client is talking directly to that runtime helper. This has a couple rather annoying issues in that we now need a sort of event/notification channel that the helper will establish with the APIServer for errors/start/exited cases. If instead of talking to the runtime helper directly, we instead took a detour through the APIServer, the APIServer is clued into exactly what order of operations is occurring. This makes "did starting the container fail? Okay we should clean up" scenarios much simpler, and it also simplifies the clients quite a bit as they don't need this split brained client model, everyone just talks to the APIServer. This change is in pursuit of that. I have reworked our clients, the ContainerService and some of our XPC types to accomplish it. The biggest "contract" changes are in the SandboxService. The first is today we have on the flag when we register any runtime helper that makes any xpc messages wake up the registered process. This isn't great in scenarios where the process may have crashed, or it exited normally and we're just trying to invoke an RPC on it. Today the helper would spawn again and try and answer our request. It'd be much nicer if we have a connection object that will become invalid if the process that vended it to us is gone. To accomplish this, now the runtime helpers will listen on an anonymous xpc connection and vend endpoints from this via only one handler exposed by the SandboxService (createEndpoint). From that point onwards all communication will be through the endpoint the service vended a client. The second change is the runtime helper will not exit on its own when the container exits, and the event mechanism has been removed. Now the APIServer simply calls wait() to listen for container exit in the background, and once we get an exit we will explicitly tell the helper to shutdown. The rationale is if shutdown is driven by the APIServer now, we can be certain we received everything we need from the helpers before they power down.

dkovba

👍

dkovba · 2025-09-18T19:04:00Z

+ if let h {
+ request.set(key: key, value: h)
+ }
+ }


This code repeats 4 times. Consider extracting it into helper methods in XPCMessage:
static func stdioKey(for index: Int) throws -> XPCKeys { switch index { case 0: return .stdin case 1: return .stdout case 2: return .stderr default: throw ContainerizationError(.invalidArgument, message: "invalid fd \(index)") } } static func setStdioHandles(on request: XPCMessage, stdio: [FileHandle?]) throws { for (index, handle) in stdio.enumerated() { if let handle { request.set(key: stdioKey(for: index), value: handle) } } }

John had commented the same, I'm going to do that (and some other cleanups) in a followup

adityaramani

Changes look good!

## Motivation and Context #654 forgot to include the shutdown XPC that was added in #628.

dcantah · 2025-09-17T18:47:07Z

Today, we have a sort of odd model all things considered. We talk to the APIServer to do the initial container creation, which mostly all of the work is just registering the runtime helper with launchd. After this registration, almost all communication from a client is talking directly to that runtime helper. This has a couple rather annoying issues in that we now need a sort of event/notification channel that the helper will establish with the APIServer for errors/start/exited cases.

If instead of talking to the runtime helper directly, we instead took a detour through the APIServer, the APIServer is clued into exactly what order of operations is occurring. This makes "did starting the container fail? Okay we should clean up" scenarios much simpler, and it also simplifies the clients quite a bit as they don't need this split brained client model, everyone just talks to the APIServer. This change is in pursuit of that. I have reworked our clients, the ContainerService and some of our XPC types to accomplish it.

The biggest "contract" change is in the SandboxService. Today we have on the flag when we register any runtime helper that makes any xpc messages wake up the registered process. This isn't great in scenarios where the process have may crashed, or it exited normally and we're just trying to invoke an RPC on it. Today the helper would spawn again and try and answer our request. It'd be much nicer if we have a connection object that will become invalid if the process that vended it to us is gone. To accomplish this, now the runtime helpers will listen on an anonymous xpc connection and vend endpoints from this via only one handler exposed by the SandboxService (createEndpoint). From that point onwards all communication will be through the endpoint the service vended a client.

Pushed the wrong button

jglogan · 2025-09-19T01:00:24Z

 return container
 }

- private func gracefulStopContainer(_ lc: LinuxContainer, stopOpts: ContainerStopOptions) async throws {


Note about code not modified in this PR:
privatenonisolatedfunc configureProcessConfig()
This should be able to become:
privatestaticfunc configureProcessConfig()
Same for closeHandle().

Another lowkey obsessive nit in the non-modified code above - move getDefaultNameserver out from between the configure...() funcs. All the private methods could probably stand to be reordered sensibly. In a swift file that's pushing 1200 lines it will help new developers.

Mcrich23 · 2025-09-19T04:47:40Z

Hey! So sorry, but in an effort to make plugin development more possible #603 and #635 have lead to the CLI folder being renamed to ContainerCommands and that will impact the merging of your pull request. Just an FYI, so you understand the issue when you are resolving conflicts.

dcantah · 2025-09-19T10:24:02Z

Converting to draft because there's a small cz change that I want in that will allow us to "fix" some behavior in stop()

Today, we have a sort of odd model all things considered. We talk to the APIServer to do the initial container creation, which mostly all of the work is just registering the runtime helper with launchd. After this registration, almost all communication from a client is talking directly to that runtime helper. This has a couple rather annoying issues in that we now need a sort of event/notification channel that the helper will establish with the APIServer for errors/start/exited cases. If instead of talking to the runtime helper directly, we instead took a detour through the APIServer, the APIServer is clued into exactly what order of operations is occurring. This makes "did starting the container fail? Okay we should clean up" scenarios much simpler, and it also simplifies the clients quite a bit as they don't need this split brained client model, everyone just talks to the APIServer. This change is in pursuit of that. I have reworked our clients, the ContainerService and some of our XPC types to accomplish it. The biggest "contract" changes are in the SandboxService. The first is today we have on the flag when we register any runtime helper that makes any xpc messages wake up the registered process. This isn't great in scenarios where the process may have crashed, or it exited normally and we're just trying to invoke an RPC on it. Today the helper would spawn again and try and answer our request. It'd be much nicer if we have a connection object that will become invalid if the process that vended it to us is gone. To accomplish this, now the runtime helpers will listen on an anonymous xpc connection and vend endpoints from this via only one handler exposed by the SandboxService (createEndpoint). From that point onwards all communication will be through the endpoint the service vended a client. The second change is the runtime helper will not exit on its own when the container exits, and the event mechanism has been removed. Now the APIServer simply calls wait() to listen for container exit in the background, and once we get an exit we will explicitly tell the helper to shutdown. The rationale is if shutdown is driven by the APIServer now, we can be certain we received everything we need from the helpers before they power down.

dkovba

👍

dkovba · 2025-09-18T19:04:00Z

+ if let h {
+ request.set(key: key, value: h)
+ }
+ }


This code repeats 4 times. Consider extracting it into helper methods in XPCMessage:
static func stdioKey(for index: Int) throws -> XPCKeys { switch index { case 0: return .stdin case 1: return .stdout case 2: return .stderr default: throw ContainerizationError(.invalidArgument, message: "invalid fd \(index)") } } static func setStdioHandles(on request: XPCMessage, stdio: [FileHandle?]) throws { for (index, handle) in stdio.enumerated() { if let handle { request.set(key: stdioKey(for: index), value: handle) } } }

John had commented the same, I'm going to do that (and some other cleanups) in a followup

adityaramani

Changes look good!

## Motivation and Context #654 forgot to include the shutdown XPC that was added in #628.

dcantah · 2025-09-17T18:47:07Z

Today, we have a sort of odd model all things considered. We talk to the APIServer to do the initial container creation, which mostly all of the work is just registering the runtime helper with launchd. After this registration, almost all communication from a client is talking directly to that runtime helper. This has a couple rather annoying issues in that we now need a sort of event/notification channel that the helper will establish with the APIServer for errors/start/exited cases.

If instead of talking to the runtime helper directly, we instead took a detour through the APIServer, the APIServer is clued into exactly what order of operations is occurring. This makes "did starting the container fail? Okay we should clean up" scenarios much simpler, and it also simplifies the clients quite a bit as they don't need this split brained client model, everyone just talks to the APIServer. This change is in pursuit of that. I have reworked our clients, the ContainerService and some of our XPC types to accomplish it.

The biggest "contract" change is in the SandboxService. Today we have on the flag when we register any runtime helper that makes any xpc messages wake up the registered process. This isn't great in scenarios where the process have may crashed, or it exited normally and we're just trying to invoke an RPC on it. Today the helper would spawn again and try and answer our request. It'd be much nicer if we have a connection object that will become invalid if the process that vended it to us is gone. To accomplish this, now the runtime helpers will listen on an anonymous xpc connection and vend endpoints from this via only one handler exposed by the SandboxService (createEndpoint). From that point onwards all communication will be through the endpoint the service vended a client.

Pushed the wrong button

jglogan · 2025-09-19T01:00:24Z

 return container
 }

- private func gracefulStopContainer(_ lc: LinuxContainer, stopOpts: ContainerStopOptions) async throws {


Note about code not modified in this PR:
privatenonisolatedfunc configureProcessConfig()
This should be able to become:
privatestaticfunc configureProcessConfig()
Same for closeHandle().

Another lowkey obsessive nit in the non-modified code above - move getDefaultNameserver out from between the configure...() funcs. All the private methods could probably stand to be reordered sensibly. In a swift file that's pushing 1200 lines it will help new developers.

Mcrich23 · 2025-09-19T04:47:40Z

Hey! So sorry, but in an effort to make plugin development more possible #603 and #635 have lead to the CLI folder being renamed to ContainerCommands and that will impact the merging of your pull request. Just an FYI, so you understand the issue when you are resolving conflicts.

dcantah · 2025-09-19T10:24:02Z

Converting to draft because there's a small cz change that I want in that will allow us to "fix" some behavior in stop()

Today, we have a sort of odd model all things considered. We talk to the APIServer to do the initial container creation, which mostly all of the work is just registering the runtime helper with launchd. After this registration, almost all communication from a client is talking directly to that runtime helper. This has a couple rather annoying issues in that we now need a sort of event/notification channel that the helper will establish with the APIServer for errors/start/exited cases. If instead of talking to the runtime helper directly, we instead took a detour through the APIServer, the APIServer is clued into exactly what order of operations is occurring. This makes "did starting the container fail? Okay we should clean up" scenarios much simpler, and it also simplifies the clients quite a bit as they don't need this split brained client model, everyone just talks to the APIServer. This change is in pursuit of that. I have reworked our clients, the ContainerService and some of our XPC types to accomplish it. The biggest "contract" changes are in the SandboxService. The first is today we have on the flag when we register any runtime helper that makes any xpc messages wake up the registered process. This isn't great in scenarios where the process may have crashed, or it exited normally and we're just trying to invoke an RPC on it. Today the helper would spawn again and try and answer our request. It'd be much nicer if we have a connection object that will become invalid if the process that vended it to us is gone. To accomplish this, now the runtime helpers will listen on an anonymous xpc connection and vend endpoints from this via only one handler exposed by the SandboxService (createEndpoint). From that point onwards all communication will be through the endpoint the service vended a client. The second change is the runtime helper will not exit on its own when the container exits, and the event mechanism has been removed. Now the APIServer simply calls wait() to listen for container exit in the background, and once we get an exit we will explicitly tell the helper to shutdown. The rationale is if shutdown is driven by the APIServer now, we can be certain we received everything we need from the helpers before they power down.

dkovba

👍

dkovba · 2025-09-18T19:04:00Z

+ if let h {
+ request.set(key: key, value: h)
+ }
+ }


This code repeats 4 times. Consider extracting it into helper methods in XPCMessage:
static func stdioKey(for index: Int) throws -> XPCKeys { switch index { case 0: return .stdin case 1: return .stdout case 2: return .stderr default: throw ContainerizationError(.invalidArgument, message: "invalid fd \(index)") } } static func setStdioHandles(on request: XPCMessage, stdio: [FileHandle?]) throws { for (index, handle) in stdio.enumerated() { if let handle { request.set(key: stdioKey(for: index), value: handle) } } }

John had commented the same, I'm going to do that (and some other cleanups) in a followup

adityaramani

Changes look good!

## Motivation and Context #654 forgot to include the shutdown XPC that was added in #628.

dcantah · 2025-09-17T18:47:07Z

Today, we have a sort of odd model all things considered. We talk to the APIServer to do the initial container creation, which mostly all of the work is just registering the runtime helper with launchd. After this registration, almost all communication from a client is talking directly to that runtime helper. This has a couple rather annoying issues in that we now need a sort of event/notification channel that the helper will establish with the APIServer for errors/start/exited cases.

If instead of talking to the runtime helper directly, we instead took a detour through the APIServer, the APIServer is clued into exactly what order of operations is occurring. This makes "did starting the container fail? Okay we should clean up" scenarios much simpler, and it also simplifies the clients quite a bit as they don't need this split brained client model, everyone just talks to the APIServer. This change is in pursuit of that. I have reworked our clients, the ContainerService and some of our XPC types to accomplish it.

The biggest "contract" change is in the SandboxService. Today we have on the flag when we register any runtime helper that makes any xpc messages wake up the registered process. This isn't great in scenarios where the process have may crashed, or it exited normally and we're just trying to invoke an RPC on it. Today the helper would spawn again and try and answer our request. It'd be much nicer if we have a connection object that will become invalid if the process that vended it to us is gone. To accomplish this, now the runtime helpers will listen on an anonymous xpc connection and vend endpoints from this via only one handler exposed by the SandboxService (createEndpoint). From that point onwards all communication will be through the endpoint the service vended a client.

Pushed the wrong button

jglogan · 2025-09-19T01:00:24Z

 return container
 }

- private func gracefulStopContainer(_ lc: LinuxContainer, stopOpts: ContainerStopOptions) async throws {


Note about code not modified in this PR:
privatenonisolatedfunc configureProcessConfig()
This should be able to become:
privatestaticfunc configureProcessConfig()
Same for closeHandle().

Another lowkey obsessive nit in the non-modified code above - move getDefaultNameserver out from between the configure...() funcs. All the private methods could probably stand to be reordered sensibly. In a swift file that's pushing 1200 lines it will help new developers.

Mcrich23 · 2025-09-19T04:47:40Z

Hey! So sorry, but in an effort to make plugin development more possible #603 and #635 have lead to the CLI folder being renamed to ContainerCommands and that will impact the merging of your pull request. Just an FYI, so you understand the issue when you are resolving conflicts.

dcantah · 2025-09-19T10:24:02Z

Converting to draft because there's a small cz change that I want in that will allow us to "fix" some behavior in stop()

Today, we have a sort of odd model all things considered. We talk to the APIServer to do the initial container creation, which mostly all of the work is just registering the runtime helper with launchd. After this registration, almost all communication from a client is talking directly to that runtime helper. This has a couple rather annoying issues in that we now need a sort of event/notification channel that the helper will establish with the APIServer for errors/start/exited cases. If instead of talking to the runtime helper directly, we instead took a detour through the APIServer, the APIServer is clued into exactly what order of operations is occurring. This makes "did starting the container fail? Okay we should clean up" scenarios much simpler, and it also simplifies the clients quite a bit as they don't need this split brained client model, everyone just talks to the APIServer. This change is in pursuit of that. I have reworked our clients, the ContainerService and some of our XPC types to accomplish it. The biggest "contract" changes are in the SandboxService. The first is today we have on the flag when we register any runtime helper that makes any xpc messages wake up the registered process. This isn't great in scenarios where the process may have crashed, or it exited normally and we're just trying to invoke an RPC on it. Today the helper would spawn again and try and answer our request. It'd be much nicer if we have a connection object that will become invalid if the process that vended it to us is gone. To accomplish this, now the runtime helpers will listen on an anonymous xpc connection and vend endpoints from this via only one handler exposed by the SandboxService (createEndpoint). From that point onwards all communication will be through the endpoint the service vended a client. The second change is the runtime helper will not exit on its own when the container exits, and the event mechanism has been removed. Now the APIServer simply calls wait() to listen for container exit in the background, and once we get an exit we will explicitly tell the helper to shutdown. The rationale is if shutdown is driven by the APIServer now, we can be certain we received everything we need from the helpers before they power down.

dkovba

👍

dkovba · 2025-09-18T19:04:00Z

+ if let h {
+ request.set(key: key, value: h)
+ }
+ }


This code repeats 4 times. Consider extracting it into helper methods in XPCMessage:
static func stdioKey(for index: Int) throws -> XPCKeys { switch index { case 0: return .stdin case 1: return .stdout case 2: return .stderr default: throw ContainerizationError(.invalidArgument, message: "invalid fd \(index)") } } static func setStdioHandles(on request: XPCMessage, stdio: [FileHandle?]) throws { for (index, handle) in stdio.enumerated() { if let handle { request.set(key: stdioKey(for: index), value: handle) } } }

John had commented the same, I'm going to do that (and some other cleanups) in a followup

adityaramani

Changes look good!

## Motivation and Context #654 forgot to include the shutdown XPC that was added in #628.

dcantah · 2025-09-17T18:47:07Z

Today, we have a sort of odd model all things considered. We talk to the APIServer to do the initial container creation, which mostly all of the work is just registering the runtime helper with launchd. After this registration, almost all communication from a client is talking directly to that runtime helper. This has a couple rather annoying issues in that we now need a sort of event/notification channel that the helper will establish with the APIServer for errors/start/exited cases.

If instead of talking to the runtime helper directly, we instead took a detour through the APIServer, the APIServer is clued into exactly what order of operations is occurring. This makes "did starting the container fail? Okay we should clean up" scenarios much simpler, and it also simplifies the clients quite a bit as they don't need this split brained client model, everyone just talks to the APIServer. This change is in pursuit of that. I have reworked our clients, the ContainerService and some of our XPC types to accomplish it.

The biggest "contract" change is in the SandboxService. Today we have on the flag when we register any runtime helper that makes any xpc messages wake up the registered process. This isn't great in scenarios where the process have may crashed, or it exited normally and we're just trying to invoke an RPC on it. Today the helper would spawn again and try and answer our request. It'd be much nicer if we have a connection object that will become invalid if the process that vended it to us is gone. To accomplish this, now the runtime helpers will listen on an anonymous xpc connection and vend endpoints from this via only one handler exposed by the SandboxService (createEndpoint). From that point onwards all communication will be through the endpoint the service vended a client.

Pushed the wrong button

jglogan · 2025-09-19T01:00:24Z

 return container
 }

- private func gracefulStopContainer(_ lc: LinuxContainer, stopOpts: ContainerStopOptions) async throws {


Note about code not modified in this PR:
privatenonisolatedfunc configureProcessConfig()
This should be able to become:
privatestaticfunc configureProcessConfig()
Same for closeHandle().

Another lowkey obsessive nit in the non-modified code above - move getDefaultNameserver out from between the configure...() funcs. All the private methods could probably stand to be reordered sensibly. In a swift file that's pushing 1200 lines it will help new developers.

Mcrich23 · 2025-09-19T04:47:40Z

Hey! So sorry, but in an effort to make plugin development more possible #603 and #635 have lead to the CLI folder being renamed to ContainerCommands and that will impact the merging of your pull request. Just an FYI, so you understand the issue when you are resolving conflicts.

dcantah · 2025-09-19T10:24:02Z

Converting to draft because there's a small cz change that I want in that will allow us to "fix" some behavior in stop()

Today, we have a sort of odd model all things considered. We talk to the APIServer to do the initial container creation, which mostly all of the work is just registering the runtime helper with launchd. After this registration, almost all communication from a client is talking directly to that runtime helper. This has a couple rather annoying issues in that we now need a sort of event/notification channel that the helper will establish with the APIServer for errors/start/exited cases. If instead of talking to the runtime helper directly, we instead took a detour through the APIServer, the APIServer is clued into exactly what order of operations is occurring. This makes "did starting the container fail? Okay we should clean up" scenarios much simpler, and it also simplifies the clients quite a bit as they don't need this split brained client model, everyone just talks to the APIServer. This change is in pursuit of that. I have reworked our clients, the ContainerService and some of our XPC types to accomplish it. The biggest "contract" changes are in the SandboxService. The first is today we have on the flag when we register any runtime helper that makes any xpc messages wake up the registered process. This isn't great in scenarios where the process may have crashed, or it exited normally and we're just trying to invoke an RPC on it. Today the helper would spawn again and try and answer our request. It'd be much nicer if we have a connection object that will become invalid if the process that vended it to us is gone. To accomplish this, now the runtime helpers will listen on an anonymous xpc connection and vend endpoints from this via only one handler exposed by the SandboxService (createEndpoint). From that point onwards all communication will be through the endpoint the service vended a client. The second change is the runtime helper will not exit on its own when the container exits, and the event mechanism has been removed. Now the APIServer simply calls wait() to listen for container exit in the background, and once we get an exit we will explicitly tell the helper to shutdown. The rationale is if shutdown is driven by the APIServer now, we can be certain we received everything we need from the helpers before they power down.

dkovba

👍

dkovba · 2025-09-18T19:04:00Z

+ if let h {
+ request.set(key: key, value: h)
+ }
+ }


This code repeats 4 times. Consider extracting it into helper methods in XPCMessage:
static func stdioKey(for index: Int) throws -> XPCKeys { switch index { case 0: return .stdin case 1: return .stdout case 2: return .stderr default: throw ContainerizationError(.invalidArgument, message: "invalid fd \(index)") } } static func setStdioHandles(on request: XPCMessage, stdio: [FileHandle?]) throws { for (index, handle) in stdio.enumerated() { if let handle { request.set(key: stdioKey(for: index), value: handle) } } }

John had commented the same, I'm going to do that (and some other cleanups) in a followup

adityaramani

Changes look good!

## Motivation and Context #654 forgot to include the shutdown XPC that was added in #628.

dcantah · 2025-09-17T18:47:07Z

Today, we have a sort of odd model all things considered. We talk to the APIServer to do the initial container creation, which mostly all of the work is just registering the runtime helper with launchd. After this registration, almost all communication from a client is talking directly to that runtime helper. This has a couple rather annoying issues in that we now need a sort of event/notification channel that the helper will establish with the APIServer for errors/start/exited cases.

If instead of talking to the runtime helper directly, we instead took a detour through the APIServer, the APIServer is clued into exactly what order of operations is occurring. This makes "did starting the container fail? Okay we should clean up" scenarios much simpler, and it also simplifies the clients quite a bit as they don't need this split brained client model, everyone just talks to the APIServer. This change is in pursuit of that. I have reworked our clients, the ContainerService and some of our XPC types to accomplish it.

The biggest "contract" change is in the SandboxService. Today we have on the flag when we register any runtime helper that makes any xpc messages wake up the registered process. This isn't great in scenarios where the process have may crashed, or it exited normally and we're just trying to invoke an RPC on it. Today the helper would spawn again and try and answer our request. It'd be much nicer if we have a connection object that will become invalid if the process that vended it to us is gone. To accomplish this, now the runtime helpers will listen on an anonymous xpc connection and vend endpoints from this via only one handler exposed by the SandboxService (createEndpoint). From that point onwards all communication will be through the endpoint the service vended a client.

Pushed the wrong button

jglogan · 2025-09-19T01:00:24Z

 return container
 }

- private func gracefulStopContainer(_ lc: LinuxContainer, stopOpts: ContainerStopOptions) async throws {


Note about code not modified in this PR:
privatenonisolatedfunc configureProcessConfig()
This should be able to become:
privatestaticfunc configureProcessConfig()
Same for closeHandle().

Another lowkey obsessive nit in the non-modified code above - move getDefaultNameserver out from between the configure...() funcs. All the private methods could probably stand to be reordered sensibly. In a swift file that's pushing 1200 lines it will help new developers.

Mcrich23 · 2025-09-19T04:47:40Z

Hey! So sorry, but in an effort to make plugin development more possible #603 and #635 have lead to the CLI folder being renamed to ContainerCommands and that will impact the merging of your pull request. Just an FYI, so you understand the issue when you are resolving conflicts.

dcantah · 2025-09-19T10:24:02Z

Converting to draft because there's a small cz change that I want in that will allow us to "fix" some behavior in stop()

Today, we have a sort of odd model all things considered. We talk to the APIServer to do the initial container creation, which mostly all of the work is just registering the runtime helper with launchd. After this registration, almost all communication from a client is talking directly to that runtime helper. This has a couple rather annoying issues in that we now need a sort of event/notification channel that the helper will establish with the APIServer for errors/start/exited cases. If instead of talking to the runtime helper directly, we instead took a detour through the APIServer, the APIServer is clued into exactly what order of operations is occurring. This makes "did starting the container fail? Okay we should clean up" scenarios much simpler, and it also simplifies the clients quite a bit as they don't need this split brained client model, everyone just talks to the APIServer. This change is in pursuit of that. I have reworked our clients, the ContainerService and some of our XPC types to accomplish it. The biggest "contract" changes are in the SandboxService. The first is today we have on the flag when we register any runtime helper that makes any xpc messages wake up the registered process. This isn't great in scenarios where the process may have crashed, or it exited normally and we're just trying to invoke an RPC on it. Today the helper would spawn again and try and answer our request. It'd be much nicer if we have a connection object that will become invalid if the process that vended it to us is gone. To accomplish this, now the runtime helpers will listen on an anonymous xpc connection and vend endpoints from this via only one handler exposed by the SandboxService (createEndpoint). From that point onwards all communication will be through the endpoint the service vended a client. The second change is the runtime helper will not exit on its own when the container exits, and the event mechanism has been removed. Now the APIServer simply calls wait() to listen for container exit in the background, and once we get an exit we will explicitly tell the helper to shutdown. The rationale is if shutdown is driven by the APIServer now, we can be certain we received everything we need from the helpers before they power down.

dkovba

👍

dkovba · 2025-09-18T19:04:00Z

+ if let h {
+ request.set(key: key, value: h)
+ }
+ }


This code repeats 4 times. Consider extracting it into helper methods in XPCMessage:
static func stdioKey(for index: Int) throws -> XPCKeys { switch index { case 0: return .stdin case 1: return .stdout case 2: return .stderr default: throw ContainerizationError(.invalidArgument, message: "invalid fd \(index)") } } static func setStdioHandles(on request: XPCMessage, stdio: [FileHandle?]) throws { for (index, handle) in stdio.enumerated() { if let handle { request.set(key: stdioKey(for: index), value: handle) } } }

John had commented the same, I'm going to do that (and some other cleanups) in a followup

adityaramani

Changes look good!

## Motivation and Context #654 forgot to include the shutdown XPC that was added in #628.

dcantah · 2025-09-17T18:47:07Z

Today, we have a sort of odd model all things considered. We talk to the APIServer to do the initial container creation, which mostly all of the work is just registering the runtime helper with launchd. After this registration, almost all communication from a client is talking directly to that runtime helper. This has a couple rather annoying issues in that we now need a sort of event/notification channel that the helper will establish with the APIServer for errors/start/exited cases.

If instead of talking to the runtime helper directly, we instead took a detour through the APIServer, the APIServer is clued into exactly what order of operations is occurring. This makes "did starting the container fail? Okay we should clean up" scenarios much simpler, and it also simplifies the clients quite a bit as they don't need this split brained client model, everyone just talks to the APIServer. This change is in pursuit of that. I have reworked our clients, the ContainerService and some of our XPC types to accomplish it.

The biggest "contract" change is in the SandboxService. Today we have on the flag when we register any runtime helper that makes any xpc messages wake up the registered process. This isn't great in scenarios where the process have may crashed, or it exited normally and we're just trying to invoke an RPC on it. Today the helper would spawn again and try and answer our request. It'd be much nicer if we have a connection object that will become invalid if the process that vended it to us is gone. To accomplish this, now the runtime helpers will listen on an anonymous xpc connection and vend endpoints from this via only one handler exposed by the SandboxService (createEndpoint). From that point onwards all communication will be through the endpoint the service vended a client.

Pushed the wrong button

jglogan · 2025-09-19T01:00:24Z

 return container
 }

- private func gracefulStopContainer(_ lc: LinuxContainer, stopOpts: ContainerStopOptions) async throws {


Note about code not modified in this PR:
privatenonisolatedfunc configureProcessConfig()
This should be able to become:
privatestaticfunc configureProcessConfig()
Same for closeHandle().

Another lowkey obsessive nit in the non-modified code above - move getDefaultNameserver out from between the configure...() funcs. All the private methods could probably stand to be reordered sensibly. In a swift file that's pushing 1200 lines it will help new developers.

Mcrich23 · 2025-09-19T04:47:40Z

Hey! So sorry, but in an effort to make plugin development more possible #603 and #635 have lead to the CLI folder being renamed to ContainerCommands and that will impact the merging of your pull request. Just an FYI, so you understand the issue when you are resolving conflicts.

dcantah · 2025-09-19T10:24:02Z

Converting to draft because there's a small cz change that I want in that will allow us to "fix" some behavior in stop()

Today, we have a sort of odd model all things considered. We talk to the APIServer to do the initial container creation, which mostly all of the work is just registering the runtime helper with launchd. After this registration, almost all communication from a client is talking directly to that runtime helper. This has a couple rather annoying issues in that we now need a sort of event/notification channel that the helper will establish with the APIServer for errors/start/exited cases. If instead of talking to the runtime helper directly, we instead took a detour through the APIServer, the APIServer is clued into exactly what order of operations is occurring. This makes "did starting the container fail? Okay we should clean up" scenarios much simpler, and it also simplifies the clients quite a bit as they don't need this split brained client model, everyone just talks to the APIServer. This change is in pursuit of that. I have reworked our clients, the ContainerService and some of our XPC types to accomplish it. The biggest "contract" changes are in the SandboxService. The first is today we have on the flag when we register any runtime helper that makes any xpc messages wake up the registered process. This isn't great in scenarios where the process may have crashed, or it exited normally and we're just trying to invoke an RPC on it. Today the helper would spawn again and try and answer our request. It'd be much nicer if we have a connection object that will become invalid if the process that vended it to us is gone. To accomplish this, now the runtime helpers will listen on an anonymous xpc connection and vend endpoints from this via only one handler exposed by the SandboxService (createEndpoint). From that point onwards all communication will be through the endpoint the service vended a client. The second change is the runtime helper will not exit on its own when the container exits, and the event mechanism has been removed. Now the APIServer simply calls wait() to listen for container exit in the background, and once we get an exit we will explicitly tell the helper to shutdown. The rationale is if shutdown is driven by the APIServer now, we can be certain we received everything we need from the helpers before they power down.

dkovba

👍

dkovba · 2025-09-18T19:04:00Z

+ if let h {
+ request.set(key: key, value: h)
+ }
+ }


This code repeats 4 times. Consider extracting it into helper methods in XPCMessage:
static func stdioKey(for index: Int) throws -> XPCKeys { switch index { case 0: return .stdin case 1: return .stdout case 2: return .stderr default: throw ContainerizationError(.invalidArgument, message: "invalid fd \(index)") } } static func setStdioHandles(on request: XPCMessage, stdio: [FileHandle?]) throws { for (index, handle) in stdio.enumerated() { if let handle { request.set(key: stdioKey(for: index), value: handle) } } }

John had commented the same, I'm going to do that (and some other cleanups) in a followup

adityaramani

Changes look good!

## Motivation and Context #654 forgot to include the shutdown XPC that was added in #628.

dcantah · 2025-09-17T18:47:07Z

Today, we have a sort of odd model all things considered. We talk to the APIServer to do the initial container creation, which mostly all of the work is just registering the runtime helper with launchd. After this registration, almost all communication from a client is talking directly to that runtime helper. This has a couple rather annoying issues in that we now need a sort of event/notification channel that the helper will establish with the APIServer for errors/start/exited cases.

If instead of talking to the runtime helper directly, we instead took a detour through the APIServer, the APIServer is clued into exactly what order of operations is occurring. This makes "did starting the container fail? Okay we should clean up" scenarios much simpler, and it also simplifies the clients quite a bit as they don't need this split brained client model, everyone just talks to the APIServer. This change is in pursuit of that. I have reworked our clients, the ContainerService and some of our XPC types to accomplish it.

The biggest "contract" change is in the SandboxService. Today we have on the flag when we register any runtime helper that makes any xpc messages wake up the registered process. This isn't great in scenarios where the process have may crashed, or it exited normally and we're just trying to invoke an RPC on it. Today the helper would spawn again and try and answer our request. It'd be much nicer if we have a connection object that will become invalid if the process that vended it to us is gone. To accomplish this, now the runtime helpers will listen on an anonymous xpc connection and vend endpoints from this via only one handler exposed by the SandboxService (createEndpoint). From that point onwards all communication will be through the endpoint the service vended a client.

Pushed the wrong button

jglogan · 2025-09-19T01:00:24Z

 return container
 }

- private func gracefulStopContainer(_ lc: LinuxContainer, stopOpts: ContainerStopOptions) async throws {


Note about code not modified in this PR:
privatenonisolatedfunc configureProcessConfig()
This should be able to become:
privatestaticfunc configureProcessConfig()
Same for closeHandle().

Another lowkey obsessive nit in the non-modified code above - move getDefaultNameserver out from between the configure...() funcs. All the private methods could probably stand to be reordered sensibly. In a swift file that's pushing 1200 lines it will help new developers.

Mcrich23 · 2025-09-19T04:47:40Z

Hey! So sorry, but in an effort to make plugin development more possible #603 and #635 have lead to the CLI folder being renamed to ContainerCommands and that will impact the merging of your pull request. Just an FYI, so you understand the issue when you are resolving conflicts.

dcantah · 2025-09-19T10:24:02Z

Converting to draft because there's a small cz change that I want in that will allow us to "fix" some behavior in stop()

Today, we have a sort of odd model all things considered. We talk to the APIServer to do the initial container creation, which mostly all of the work is just registering the runtime helper with launchd. After this registration, almost all communication from a client is talking directly to that runtime helper. This has a couple rather annoying issues in that we now need a sort of event/notification channel that the helper will establish with the APIServer for errors/start/exited cases. If instead of talking to the runtime helper directly, we instead took a detour through the APIServer, the APIServer is clued into exactly what order of operations is occurring. This makes "did starting the container fail? Okay we should clean up" scenarios much simpler, and it also simplifies the clients quite a bit as they don't need this split brained client model, everyone just talks to the APIServer. This change is in pursuit of that. I have reworked our clients, the ContainerService and some of our XPC types to accomplish it. The biggest "contract" changes are in the SandboxService. The first is today we have on the flag when we register any runtime helper that makes any xpc messages wake up the registered process. This isn't great in scenarios where the process may have crashed, or it exited normally and we're just trying to invoke an RPC on it. Today the helper would spawn again and try and answer our request. It'd be much nicer if we have a connection object that will become invalid if the process that vended it to us is gone. To accomplish this, now the runtime helpers will listen on an anonymous xpc connection and vend endpoints from this via only one handler exposed by the SandboxService (createEndpoint). From that point onwards all communication will be through the endpoint the service vended a client. The second change is the runtime helper will not exit on its own when the container exits, and the event mechanism has been removed. Now the APIServer simply calls wait() to listen for container exit in the background, and once we get an exit we will explicitly tell the helper to shutdown. The rationale is if shutdown is driven by the APIServer now, we can be certain we received everything we need from the helpers before they power down.

dkovba

👍

dkovba · 2025-09-18T19:04:00Z

+ if let h {
+ request.set(key: key, value: h)
+ }
+ }


This code repeats 4 times. Consider extracting it into helper methods in XPCMessage:
static func stdioKey(for index: Int) throws -> XPCKeys { switch index { case 0: return .stdin case 1: return .stdout case 2: return .stderr default: throw ContainerizationError(.invalidArgument, message: "invalid fd \(index)") } } static func setStdioHandles(on request: XPCMessage, stdio: [FileHandle?]) throws { for (index, handle) in stdio.enumerated() { if let handle { request.set(key: stdioKey(for: index), value: handle) } } }

John had commented the same, I'm going to do that (and some other cleanups) in a followup

adityaramani

Changes look good!

## Motivation and Context #654 forgot to include the shutdown XPC that was added in #628.

dcantah force-pushed the move-to-api-server-for-everythang branch 3 times, most recently from 4838bcf to e7e4dcbCompare September 18, 2025 03:29

dcantah marked this pull request as ready for review September 18, 2025 03:34

dcantah force-pushed the move-to-api-server-for-everythang branch 3 times, most recently from 1a56c99 to 6a497e6Compare September 18, 2025 07:25

wlan0 reviewed Sep 18, 2025
View reviewed changes

Comment threadSources/ContainerClient/Core/ClientContainer.swift Outdated

wlan0 reviewed Sep 18, 2025
View reviewed changes

Comment threadSources/ContainerClient/Core/ClientContainer.swift Outdated

wlan0 reviewed Sep 18, 2025
View reviewed changes

Comment threadSources/ContainerClient/Core/ClientContainer.swift Outdated

jglogan reviewed Sep 18, 2025
View reviewed changes

Comment threadSources/ContainerClient/ContainerEvents.swift

dcantah force-pushed the move-to-api-server-for-everythang branch from 6a497e6 to 5d5e622Compare September 18, 2025 18:43

jglogan reviewed Sep 18, 2025
View reviewed changes

Comment threadSources/ContainerClient/SandboxClient.swift
Comment threadSources/Services/ContainerNetworkService/NetworkClient.swift

jglogan reviewed Sep 18, 2025
View reviewed changes

Comment threadSources/ContainerClient/Core/ClientProcess.swift

jglogan reviewed Sep 18, 2025
View reviewed changes

Comment threadSources/ContainerClient/Core/ClientProcess.swift Outdated

jglogan reviewed Sep 18, 2025
View reviewed changes

Comment threadSources/ContainerClient/Core/ClientContainer.swift
Comment threadSources/ContainerClient/Core/ClientContainer.swift

jglogan reviewed Sep 18, 2025
View reviewed changes

Comment threadSources/Services/ContainerAPIService/Containers/ContainersService.swift Outdated
Comment threadSources/Services/ContainerAPIService/Containers/ContainersService.swift Outdated

jglogan previously approved these changes Sep 19, 2025
View reviewed changes

Comment threadSources/Services/ContainerAPIService/Containers/ContainersService.swift Outdated
Comment threadSources/Services/ContainerAPIService/Containers/ContainersService.swift
Comment threadSources/Services/ContainerAPIService/Containers/ContainersService.swift Outdated

jglogan requested changes Sep 19, 2025
View reviewed changes

dcantah force-pushed the move-to-api-server-for-everythang branch 2 times, most recently from 560d05a to 07707d5Compare September 19, 2025 10:23

dcantah marked this pull request as draft September 19, 2025 10:23

dcantah force-pushed the move-to-api-server-for-everythang branch from 07707d5 to 66fef47Compare September 19, 2025 19:04

dcantah requested a review from jglogan September 19, 2025 19:26

dcantah marked this pull request as ready for review September 19, 2025 19:26

dcantah requested a review from wlan0 September 19, 2025 19:26

jglogan approved these changes Sep 19, 2025
View reviewed changes

wlan0 approved these changes Sep 20, 2025
View reviewed changes

dcantah merged commit 444064d into apple:mainSep 20, 2025
37 of 38 checks passed

dkovba reviewed Sep 20, 2025
View reviewed changes

adityaramani reviewed Sep 20, 2025
View reviewed changes

jglogan mentioned this pull request Sep 23, 2025
Fix regression - removed shutdown XPC inadvertently. #675
Merged
7 tasks

jglogan added a commit that referenced this pull request Sep 23, 2025
Fix regression - removed shutdown XPC inadvertently. (#675)
e54ee7c
## Motivation and Context #654 forgot to include the shutdown XPC that was added in #628.