Move to TOML configuration for defaults

[noah-thor]

This change migrates away from using UserDefaults, instead providing a TOML configuration mechanism for user configurable settings. All exisitng features of user configurability are maintained. However, users will have to migrate any settings they have configured in the UserDefaults into TOML for these settings to take effect.

Breaking changes:

• container system property get is removed in favor of users directly utilizing container system property list --format toml | jq<>.
• container system property set is removed since the TOML configuration is effectively immutable during the lifetime of the container daemon. Uses can edit the TOML they have in their home directory, however no changes will take effect until the daemon is restarted via container system stop && container system start
• container system property list --format table is removed as generating tabular format is non-trivial and the new TOML format is intended to be human readable

Addresses discussion (#1336)

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update

Motivation and Context

Context within #1336

We're looking to improve the user experience and overall functionality of setting defaults for container to better enable future use cases.

Today container uses macOS's UserDefaults to configure settings needed at runtime. As mentioned in #608, UserDefaults may be idiomatic for macOS, but they apply globally to all sessions and do not handle representing complex, hierarchical data well. Additionally, we currently have two ways of setting these defaults, either directly with macOS's defaults command or through container system property.

#608 proposes moving to use environment variables in place of UserDefaults. However, we do not believe this is sufficient. Environment variables are not in a consistent location, are not sourced from data, and also do not handle representing complex, hierarchical data well.

Testing

• Tested locally
• Added/updated tests
• Added/updated docs

[manojmahapatra]

Suggestion: force unwraps are generally unsafe, I would not use try!

[manojmahapatra]

Additionally, containerSystemConfig is being repeated at many places, in addition to I'm seeing every function that needs config accepts it as a parameter, and every function that calls those functions also accepts, even if it doesn't use the config itself. I think we can do better/different by not passing around into the chain of functions, but hoist it up as a higher level @Environment kinda property.

[katiewasnothere]

@manojmahapatra As an alternative to the current approach in this PR, we also considered having a service on APIServer that managed defaults. We decided against this for a few reasons: we want to avoid services needing to call back to APIServer unless absolutely necessary and we believe configurations values should not change during the lifetime of a service instance (with some rare exceptions). That said, we're still discussing how we want to extend the current approach to allow for user level overrides, such as with environment variables.

[manojmahapatra]

IIUC are you implying to have some kind of runtime (immutable) context and inject that at module boundaries? (eg SystemStart)

[katiewasnothere]

That's what I would prefer. But we do still want to support some user level overrides, so how to make that experience nice is still an open question. What do you think?

[manojmahapatra]

That makes sense, thanks for clarifying. I think explicit user overrides are the right direction; probably the other open question is what would be the precedence order. (we need to define that)

So, a reasonable approach could be then to add a single central config resolver that builds a final config with an explicit order?

[katiewasnothere]

Assuming we support user overrides with a file and with environment variables, I think the precedence would be environment variables -> user overrides file -> system config. The issue with environment variables is that they can change over the lifetime of container so we can't just read them once to create the final config :/

[manojmahapatra]

yeah, that definitely had crossed my mind, unless we bubble up some kind of hot reloading of config file on demand. I'm not even sure if this is the right direction or hot reloading of config file even makes sense.

[jglogan]

I'm not even sure if this is the right direction or hot reloading of config file even makes sense.

@manojmahapatra Agree...I think it's "we aren't going to need it today". We can start simple and try not to paint ourselves into a corner and evolve from there.

We can move slowly on env variables as well. Here's a rough snapshot of what we have right now:

NAME	WHERE	NOTES
CONTAINER_{APP,INSTALL,LOG}_ROOT	services	plumbed from system start args into launchd.plist files, no TOML config needed
CONTAINER_DEBUG	CLI	commands check --debug, then this
CONTAINER_DEBUG_LAUNCHD_LABEL	services	plumbed into apiserver environment at system start, see #1101, no TOML config needed
CONTAINER_DEFAULT_PLATFORM	CLI	similar to DOCKER_DEFAULT_PLATFORM, TOML configuration could also make sense here but not immediately required
CONTAINER_REGISTRY_{HOST,TOKEN,USER}	services	plumbed into apiserver environment at system start, used for limited cases only, no TOML config needed

None of these require TOML config today, so we can treat them as orthogonal for now.

As far as our existing properties go, none today have environment overrides AFAIK, so we can decide later if say, we want to be able to do something like export CONTAINER_MEMORY=8g in one shell session and have container run evaluate --memory, then CONTAINER_MEMORY, then the user TOML, then the system TOML.

[noah-thor]

I addressed all ongoing discussions as part of this commit. I also noticed that the Measurement swift type does not play very nicely with the TOMLEncoder in terms of human readability, so added a small wrapper around that type to fix the ux.

[noah-thor]

Pushed new commit. Will resolve conflicts once everything else looks good.

New revision changes a few things:

• Removes container system property get <> in favor of recommending users leverage container system property list --format json | jq <>
• Removes container system property list --format table as maintaining a table format was not simple, and the new toml format is a suitable replacement for human readers.

[noah-thor]

Pushed new commit:

• fixed issue with DNSConfig providing a "test" default changing behavior of request scheme
• removed static force unwraps in favor of a try let in run()
• rebased

[jglogan]

Is there only a user TOML file for this PR, and no system TOML that we include in our distribution?

[noah-thor]

That is correct, there is only one TOML that lives in users home

[jglogan]

Update this if we do restructure.

[jglogan]

Is this how it works?

• container system start copies the config to the application root so that the API server and any service plugins is spawns have a consistent system view
• CLI commands use the same application root config, such that the user would need to stop/start the container service if they wanted to make a change to a setting that impacts CLI command behavior

[jglogan]

Is this really installDefaultKernel() anymore? It just installs a kernel.

[noah-thor]

Merged from main to update PR

[karaday411-sketch]

R

[noah-thor]

This change migrates away from using UserDefaults, instead providing a TOML configuration mechanism for user configurable settings. All exisitng features of user configurability are maintained. However, users will have to migrate any settings they have configured in the UserDefaults into TOML for these settings to take effect.

Breaking changes:

• container system property get is removed in favor of users directly utilizing container system property list --format toml | jq<>.
• container system property set is removed since the TOML configuration is effectively immutable during the lifetime of the container daemon. Uses can edit the TOML they have in their home directory, however no changes will take effect until the daemon is restarted via container system stop && container system start
• container system property list --format table is removed as generating tabular format is non-trivial and the new TOML format is intended to be human readable

Addresses discussion (#1336)

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update

Motivation and Context

Context within #1336

We're looking to improve the user experience and overall functionality of setting defaults for container to better enable future use cases.

Today container uses macOS's UserDefaults to configure settings needed at runtime. As mentioned in #608, UserDefaults may be idiomatic for macOS, but they apply globally to all sessions and do not handle representing complex, hierarchical data well. Additionally, we currently have two ways of setting these defaults, either directly with macOS's defaults command or through container system property.

#608 proposes moving to use environment variables in place of UserDefaults. However, we do not believe this is sufficient. Environment variables are not in a consistent location, are not sourced from data, and also do not handle representing complex, hierarchical data well.

Testing

• Tested locally
• Added/updated tests
• Added/updated docs

[manojmahapatra]

Suggestion: force unwraps are generally unsafe, I would not use try!

[manojmahapatra]

Additionally, containerSystemConfig is being repeated at many places, in addition to I'm seeing every function that needs config accepts it as a parameter, and every function that calls those functions also accepts, even if it doesn't use the config itself. I think we can do better/different by not passing around into the chain of functions, but hoist it up as a higher level @Environment kinda property.

[katiewasnothere]

@manojmahapatra As an alternative to the current approach in this PR, we also considered having a service on APIServer that managed defaults. We decided against this for a few reasons: we want to avoid services needing to call back to APIServer unless absolutely necessary and we believe configurations values should not change during the lifetime of a service instance (with some rare exceptions). That said, we're still discussing how we want to extend the current approach to allow for user level overrides, such as with environment variables.

[manojmahapatra]

IIUC are you implying to have some kind of runtime (immutable) context and inject that at module boundaries? (eg SystemStart)

[katiewasnothere]

That's what I would prefer. But we do still want to support some user level overrides, so how to make that experience nice is still an open question. What do you think?

[manojmahapatra]

That makes sense, thanks for clarifying. I think explicit user overrides are the right direction; probably the other open question is what would be the precedence order. (we need to define that)

So, a reasonable approach could be then to add a single central config resolver that builds a final config with an explicit order?

[katiewasnothere]

Assuming we support user overrides with a file and with environment variables, I think the precedence would be environment variables -> user overrides file -> system config. The issue with environment variables is that they can change over the lifetime of container so we can't just read them once to create the final config :/

[manojmahapatra]

yeah, that definitely had crossed my mind, unless we bubble up some kind of hot reloading of config file on demand. I'm not even sure if this is the right direction or hot reloading of config file even makes sense.

[jglogan]

I'm not even sure if this is the right direction or hot reloading of config file even makes sense.

@manojmahapatra Agree...I think it's "we aren't going to need it today". We can start simple and try not to paint ourselves into a corner and evolve from there.

We can move slowly on env variables as well. Here's a rough snapshot of what we have right now:

NAME	WHERE	NOTES
CONTAINER_{APP,INSTALL,LOG}_ROOT	services	plumbed from system start args into launchd.plist files, no TOML config needed
CONTAINER_DEBUG	CLI	commands check --debug, then this
CONTAINER_DEBUG_LAUNCHD_LABEL	services	plumbed into apiserver environment at system start, see #1101, no TOML config needed
CONTAINER_DEFAULT_PLATFORM	CLI	similar to DOCKER_DEFAULT_PLATFORM, TOML configuration could also make sense here but not immediately required
CONTAINER_REGISTRY_{HOST,TOKEN,USER}	services	plumbed into apiserver environment at system start, used for limited cases only, no TOML config needed

None of these require TOML config today, so we can treat them as orthogonal for now.

As far as our existing properties go, none today have environment overrides AFAIK, so we can decide later if say, we want to be able to do something like export CONTAINER_MEMORY=8g in one shell session and have container run evaluate --memory, then CONTAINER_MEMORY, then the user TOML, then the system TOML.

[noah-thor]

I addressed all ongoing discussions as part of this commit. I also noticed that the Measurement swift type does not play very nicely with the TOMLEncoder in terms of human readability, so added a small wrapper around that type to fix the ux.

[noah-thor]

Pushed new commit. Will resolve conflicts once everything else looks good.

New revision changes a few things:

• Removes container system property get <> in favor of recommending users leverage container system property list --format json | jq <>
• Removes container system property list --format table as maintaining a table format was not simple, and the new toml format is a suitable replacement for human readers.

[noah-thor]

Pushed new commit:

• fixed issue with DNSConfig providing a "test" default changing behavior of request scheme
• removed static force unwraps in favor of a try let in run()
• rebased

[jglogan]

Is there only a user TOML file for this PR, and no system TOML that we include in our distribution?

[noah-thor]

That is correct, there is only one TOML that lives in users home

[jglogan]

Update this if we do restructure.

[jglogan]

Is this how it works?

• container system start copies the config to the application root so that the API server and any service plugins is spawns have a consistent system view
• CLI commands use the same application root config, such that the user would need to stop/start the container service if they wanted to make a change to a setting that impacts CLI command behavior

[jglogan]

Is this really installDefaultKernel() anymore? It just installs a kernel.

[noah-thor]

Merged from main to update PR

[karaday411-sketch]

R

[noah-thor]

This change migrates away from using UserDefaults, instead providing a TOML configuration mechanism for user configurable settings. All exisitng features of user configurability are maintained. However, users will have to migrate any settings they have configured in the UserDefaults into TOML for these settings to take effect.

Breaking changes:

• container system property get is removed in favor of users directly utilizing container system property list --format toml | jq<>.
• container system property set is removed since the TOML configuration is effectively immutable during the lifetime of the container daemon. Uses can edit the TOML they have in their home directory, however no changes will take effect until the daemon is restarted via container system stop && container system start
• container system property list --format table is removed as generating tabular format is non-trivial and the new TOML format is intended to be human readable

Addresses discussion (#1336)

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update

Motivation and Context

Context within #1336

We're looking to improve the user experience and overall functionality of setting defaults for container to better enable future use cases.

Today container uses macOS's UserDefaults to configure settings needed at runtime. As mentioned in #608, UserDefaults may be idiomatic for macOS, but they apply globally to all sessions and do not handle representing complex, hierarchical data well. Additionally, we currently have two ways of setting these defaults, either directly with macOS's defaults command or through container system property.

#608 proposes moving to use environment variables in place of UserDefaults. However, we do not believe this is sufficient. Environment variables are not in a consistent location, are not sourced from data, and also do not handle representing complex, hierarchical data well.

Testing

• Tested locally
• Added/updated tests
• Added/updated docs

[manojmahapatra]

Suggestion: force unwraps are generally unsafe, I would not use try!

[manojmahapatra]

Additionally, containerSystemConfig is being repeated at many places, in addition to I'm seeing every function that needs config accepts it as a parameter, and every function that calls those functions also accepts, even if it doesn't use the config itself. I think we can do better/different by not passing around into the chain of functions, but hoist it up as a higher level @Environment kinda property.

[katiewasnothere]

@manojmahapatra As an alternative to the current approach in this PR, we also considered having a service on APIServer that managed defaults. We decided against this for a few reasons: we want to avoid services needing to call back to APIServer unless absolutely necessary and we believe configurations values should not change during the lifetime of a service instance (with some rare exceptions). That said, we're still discussing how we want to extend the current approach to allow for user level overrides, such as with environment variables.

[manojmahapatra]

IIUC are you implying to have some kind of runtime (immutable) context and inject that at module boundaries? (eg SystemStart)

[katiewasnothere]

That's what I would prefer. But we do still want to support some user level overrides, so how to make that experience nice is still an open question. What do you think?

[manojmahapatra]

That makes sense, thanks for clarifying. I think explicit user overrides are the right direction; probably the other open question is what would be the precedence order. (we need to define that)

So, a reasonable approach could be then to add a single central config resolver that builds a final config with an explicit order?

[katiewasnothere]

Assuming we support user overrides with a file and with environment variables, I think the precedence would be environment variables -> user overrides file -> system config. The issue with environment variables is that they can change over the lifetime of container so we can't just read them once to create the final config :/

[manojmahapatra]

yeah, that definitely had crossed my mind, unless we bubble up some kind of hot reloading of config file on demand. I'm not even sure if this is the right direction or hot reloading of config file even makes sense.

[jglogan]

I'm not even sure if this is the right direction or hot reloading of config file even makes sense.

@manojmahapatra Agree...I think it's "we aren't going to need it today". We can start simple and try not to paint ourselves into a corner and evolve from there.

We can move slowly on env variables as well. Here's a rough snapshot of what we have right now:

NAME	WHERE	NOTES
CONTAINER_{APP,INSTALL,LOG}_ROOT	services	plumbed from system start args into launchd.plist files, no TOML config needed
CONTAINER_DEBUG	CLI	commands check --debug, then this
CONTAINER_DEBUG_LAUNCHD_LABEL	services	plumbed into apiserver environment at system start, see #1101, no TOML config needed
CONTAINER_DEFAULT_PLATFORM	CLI	similar to DOCKER_DEFAULT_PLATFORM, TOML configuration could also make sense here but not immediately required
CONTAINER_REGISTRY_{HOST,TOKEN,USER}	services	plumbed into apiserver environment at system start, used for limited cases only, no TOML config needed

None of these require TOML config today, so we can treat them as orthogonal for now.

As far as our existing properties go, none today have environment overrides AFAIK, so we can decide later if say, we want to be able to do something like export CONTAINER_MEMORY=8g in one shell session and have container run evaluate --memory, then CONTAINER_MEMORY, then the user TOML, then the system TOML.

[noah-thor]

I addressed all ongoing discussions as part of this commit. I also noticed that the Measurement swift type does not play very nicely with the TOMLEncoder in terms of human readability, so added a small wrapper around that type to fix the ux.

[noah-thor]

Pushed new commit. Will resolve conflicts once everything else looks good.

New revision changes a few things:

• Removes container system property get <> in favor of recommending users leverage container system property list --format json | jq <>
• Removes container system property list --format table as maintaining a table format was not simple, and the new toml format is a suitable replacement for human readers.

[noah-thor]

Pushed new commit:

• fixed issue with DNSConfig providing a "test" default changing behavior of request scheme
• removed static force unwraps in favor of a try let in run()
• rebased

[jglogan]

Is there only a user TOML file for this PR, and no system TOML that we include in our distribution?

[noah-thor]

That is correct, there is only one TOML that lives in users home

[jglogan]

Update this if we do restructure.

[jglogan]

Is this how it works?

• container system start copies the config to the application root so that the API server and any service plugins is spawns have a consistent system view
• CLI commands use the same application root config, such that the user would need to stop/start the container service if they wanted to make a change to a setting that impacts CLI command behavior

[jglogan]

Is this really installDefaultKernel() anymore? It just installs a kernel.

[noah-thor]

Merged from main to update PR

[karaday411-sketch]

R

[noah-thor]

This change migrates away from using UserDefaults, instead providing a TOML configuration mechanism for user configurable settings. All exisitng features of user configurability are maintained. However, users will have to migrate any settings they have configured in the UserDefaults into TOML for these settings to take effect.

Breaking changes:

• container system property get is removed in favor of users directly utilizing container system property list --format toml | jq<>.
• container system property set is removed since the TOML configuration is effectively immutable during the lifetime of the container daemon. Uses can edit the TOML they have in their home directory, however no changes will take effect until the daemon is restarted via container system stop && container system start
• container system property list --format table is removed as generating tabular format is non-trivial and the new TOML format is intended to be human readable

Addresses discussion (#1336)

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update

Motivation and Context

Context within #1336

We're looking to improve the user experience and overall functionality of setting defaults for container to better enable future use cases.

Today container uses macOS's UserDefaults to configure settings needed at runtime. As mentioned in #608, UserDefaults may be idiomatic for macOS, but they apply globally to all sessions and do not handle representing complex, hierarchical data well. Additionally, we currently have two ways of setting these defaults, either directly with macOS's defaults command or through container system property.

#608 proposes moving to use environment variables in place of UserDefaults. However, we do not believe this is sufficient. Environment variables are not in a consistent location, are not sourced from data, and also do not handle representing complex, hierarchical data well.

Testing

• Tested locally
• Added/updated tests
• Added/updated docs

[manojmahapatra]

Suggestion: force unwraps are generally unsafe, I would not use try!

[manojmahapatra]

Additionally, containerSystemConfig is being repeated at many places, in addition to I'm seeing every function that needs config accepts it as a parameter, and every function that calls those functions also accepts, even if it doesn't use the config itself. I think we can do better/different by not passing around into the chain of functions, but hoist it up as a higher level @Environment kinda property.

[katiewasnothere]

@manojmahapatra As an alternative to the current approach in this PR, we also considered having a service on APIServer that managed defaults. We decided against this for a few reasons: we want to avoid services needing to call back to APIServer unless absolutely necessary and we believe configurations values should not change during the lifetime of a service instance (with some rare exceptions). That said, we're still discussing how we want to extend the current approach to allow for user level overrides, such as with environment variables.

[manojmahapatra]

IIUC are you implying to have some kind of runtime (immutable) context and inject that at module boundaries? (eg SystemStart)

[katiewasnothere]

That's what I would prefer. But we do still want to support some user level overrides, so how to make that experience nice is still an open question. What do you think?

[manojmahapatra]

That makes sense, thanks for clarifying. I think explicit user overrides are the right direction; probably the other open question is what would be the precedence order. (we need to define that)

So, a reasonable approach could be then to add a single central config resolver that builds a final config with an explicit order?

[katiewasnothere]

Assuming we support user overrides with a file and with environment variables, I think the precedence would be environment variables -> user overrides file -> system config. The issue with environment variables is that they can change over the lifetime of container so we can't just read them once to create the final config :/

[manojmahapatra]

yeah, that definitely had crossed my mind, unless we bubble up some kind of hot reloading of config file on demand. I'm not even sure if this is the right direction or hot reloading of config file even makes sense.

[jglogan]

I'm not even sure if this is the right direction or hot reloading of config file even makes sense.

@manojmahapatra Agree...I think it's "we aren't going to need it today". We can start simple and try not to paint ourselves into a corner and evolve from there.

We can move slowly on env variables as well. Here's a rough snapshot of what we have right now:

NAME	WHERE	NOTES
CONTAINER_{APP,INSTALL,LOG}_ROOT	services	plumbed from system start args into launchd.plist files, no TOML config needed
CONTAINER_DEBUG	CLI	commands check --debug, then this
CONTAINER_DEBUG_LAUNCHD_LABEL	services	plumbed into apiserver environment at system start, see #1101, no TOML config needed
CONTAINER_DEFAULT_PLATFORM	CLI	similar to DOCKER_DEFAULT_PLATFORM, TOML configuration could also make sense here but not immediately required
CONTAINER_REGISTRY_{HOST,TOKEN,USER}	services	plumbed into apiserver environment at system start, used for limited cases only, no TOML config needed

None of these require TOML config today, so we can treat them as orthogonal for now.

As far as our existing properties go, none today have environment overrides AFAIK, so we can decide later if say, we want to be able to do something like export CONTAINER_MEMORY=8g in one shell session and have container run evaluate --memory, then CONTAINER_MEMORY, then the user TOML, then the system TOML.

[noah-thor]

I addressed all ongoing discussions as part of this commit. I also noticed that the Measurement swift type does not play very nicely with the TOMLEncoder in terms of human readability, so added a small wrapper around that type to fix the ux.

[noah-thor]

Pushed new commit. Will resolve conflicts once everything else looks good.

New revision changes a few things:

• Removes container system property get <> in favor of recommending users leverage container system property list --format json | jq <>
• Removes container system property list --format table as maintaining a table format was not simple, and the new toml format is a suitable replacement for human readers.

[noah-thor]

Pushed new commit:

• fixed issue with DNSConfig providing a "test" default changing behavior of request scheme
• removed static force unwraps in favor of a try let in run()
• rebased

[jglogan]

Is there only a user TOML file for this PR, and no system TOML that we include in our distribution?

[noah-thor]

That is correct, there is only one TOML that lives in users home

[jglogan]

Update this if we do restructure.

[jglogan]

Is this how it works?

• container system start copies the config to the application root so that the API server and any service plugins is spawns have a consistent system view
• CLI commands use the same application root config, such that the user would need to stop/start the container service if they wanted to make a change to a setting that impacts CLI command behavior

[jglogan]

Is this really installDefaultKernel() anymore? It just installs a kernel.

[noah-thor]

Merged from main to update PR

[karaday411-sketch]

R

[noah-thor]

This change migrates away from using UserDefaults, instead providing a TOML configuration mechanism for user configurable settings. All exisitng features of user configurability are maintained. However, users will have to migrate any settings they have configured in the UserDefaults into TOML for these settings to take effect.

Breaking changes:

• container system property get is removed in favor of users directly utilizing container system property list --format toml | jq<>.
• container system property set is removed since the TOML configuration is effectively immutable during the lifetime of the container daemon. Uses can edit the TOML they have in their home directory, however no changes will take effect until the daemon is restarted via container system stop && container system start
• container system property list --format table is removed as generating tabular format is non-trivial and the new TOML format is intended to be human readable

Addresses discussion (#1336)

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update

Motivation and Context

Context within #1336

We're looking to improve the user experience and overall functionality of setting defaults for container to better enable future use cases.

Today container uses macOS's UserDefaults to configure settings needed at runtime. As mentioned in #608, UserDefaults may be idiomatic for macOS, but they apply globally to all sessions and do not handle representing complex, hierarchical data well. Additionally, we currently have two ways of setting these defaults, either directly with macOS's defaults command or through container system property.

#608 proposes moving to use environment variables in place of UserDefaults. However, we do not believe this is sufficient. Environment variables are not in a consistent location, are not sourced from data, and also do not handle representing complex, hierarchical data well.

Testing

• Tested locally
• Added/updated tests
• Added/updated docs

[manojmahapatra]

Suggestion: force unwraps are generally unsafe, I would not use try!

[manojmahapatra]

Additionally, containerSystemConfig is being repeated at many places, in addition to I'm seeing every function that needs config accepts it as a parameter, and every function that calls those functions also accepts, even if it doesn't use the config itself. I think we can do better/different by not passing around into the chain of functions, but hoist it up as a higher level @Environment kinda property.

[katiewasnothere]

@manojmahapatra As an alternative to the current approach in this PR, we also considered having a service on APIServer that managed defaults. We decided against this for a few reasons: we want to avoid services needing to call back to APIServer unless absolutely necessary and we believe configurations values should not change during the lifetime of a service instance (with some rare exceptions). That said, we're still discussing how we want to extend the current approach to allow for user level overrides, such as with environment variables.

[manojmahapatra]

IIUC are you implying to have some kind of runtime (immutable) context and inject that at module boundaries? (eg SystemStart)

[katiewasnothere]

That's what I would prefer. But we do still want to support some user level overrides, so how to make that experience nice is still an open question. What do you think?

[manojmahapatra]

That makes sense, thanks for clarifying. I think explicit user overrides are the right direction; probably the other open question is what would be the precedence order. (we need to define that)

So, a reasonable approach could be then to add a single central config resolver that builds a final config with an explicit order?

[katiewasnothere]

Assuming we support user overrides with a file and with environment variables, I think the precedence would be environment variables -> user overrides file -> system config. The issue with environment variables is that they can change over the lifetime of container so we can't just read them once to create the final config :/

[manojmahapatra]

yeah, that definitely had crossed my mind, unless we bubble up some kind of hot reloading of config file on demand. I'm not even sure if this is the right direction or hot reloading of config file even makes sense.

[jglogan]

I'm not even sure if this is the right direction or hot reloading of config file even makes sense.

@manojmahapatra Agree...I think it's "we aren't going to need it today". We can start simple and try not to paint ourselves into a corner and evolve from there.

We can move slowly on env variables as well. Here's a rough snapshot of what we have right now:

NAME	WHERE	NOTES
CONTAINER_{APP,INSTALL,LOG}_ROOT	services	plumbed from system start args into launchd.plist files, no TOML config needed
CONTAINER_DEBUG	CLI	commands check --debug, then this
CONTAINER_DEBUG_LAUNCHD_LABEL	services	plumbed into apiserver environment at system start, see #1101, no TOML config needed
CONTAINER_DEFAULT_PLATFORM	CLI	similar to DOCKER_DEFAULT_PLATFORM, TOML configuration could also make sense here but not immediately required
CONTAINER_REGISTRY_{HOST,TOKEN,USER}	services	plumbed into apiserver environment at system start, used for limited cases only, no TOML config needed

None of these require TOML config today, so we can treat them as orthogonal for now.

As far as our existing properties go, none today have environment overrides AFAIK, so we can decide later if say, we want to be able to do something like export CONTAINER_MEMORY=8g in one shell session and have container run evaluate --memory, then CONTAINER_MEMORY, then the user TOML, then the system TOML.

[noah-thor]

I addressed all ongoing discussions as part of this commit. I also noticed that the Measurement swift type does not play very nicely with the TOMLEncoder in terms of human readability, so added a small wrapper around that type to fix the ux.

[noah-thor]

Pushed new commit. Will resolve conflicts once everything else looks good.

New revision changes a few things:

• Removes container system property get <> in favor of recommending users leverage container system property list --format json | jq <>
• Removes container system property list --format table as maintaining a table format was not simple, and the new toml format is a suitable replacement for human readers.

[noah-thor]

Pushed new commit:

• fixed issue with DNSConfig providing a "test" default changing behavior of request scheme
• removed static force unwraps in favor of a try let in run()
• rebased

[jglogan]

Is there only a user TOML file for this PR, and no system TOML that we include in our distribution?

[noah-thor]

That is correct, there is only one TOML that lives in users home

[jglogan]

Update this if we do restructure.

[jglogan]

Is this how it works?

• container system start copies the config to the application root so that the API server and any service plugins is spawns have a consistent system view
• CLI commands use the same application root config, such that the user would need to stop/start the container service if they wanted to make a change to a setting that impacts CLI command behavior

[jglogan]

Is this really installDefaultKernel() anymore? It just installs a kernel.

[noah-thor]

Merged from main to update PR

[karaday411-sketch]

R

[noah-thor]

This change migrates away from using UserDefaults, instead providing a TOML configuration mechanism for user configurable settings. All exisitng features of user configurability are maintained. However, users will have to migrate any settings they have configured in the UserDefaults into TOML for these settings to take effect.

Breaking changes:

• container system property get is removed in favor of users directly utilizing container system property list --format toml | jq<>.
• container system property set is removed since the TOML configuration is effectively immutable during the lifetime of the container daemon. Uses can edit the TOML they have in their home directory, however no changes will take effect until the daemon is restarted via container system stop && container system start
• container system property list --format table is removed as generating tabular format is non-trivial and the new TOML format is intended to be human readable

Addresses discussion (#1336)

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update

Motivation and Context

Context within #1336

We're looking to improve the user experience and overall functionality of setting defaults for container to better enable future use cases.

Today container uses macOS's UserDefaults to configure settings needed at runtime. As mentioned in #608, UserDefaults may be idiomatic for macOS, but they apply globally to all sessions and do not handle representing complex, hierarchical data well. Additionally, we currently have two ways of setting these defaults, either directly with macOS's defaults command or through container system property.

#608 proposes moving to use environment variables in place of UserDefaults. However, we do not believe this is sufficient. Environment variables are not in a consistent location, are not sourced from data, and also do not handle representing complex, hierarchical data well.

Testing

• Tested locally
• Added/updated tests
• Added/updated docs

[manojmahapatra]

Suggestion: force unwraps are generally unsafe, I would not use try!

[manojmahapatra]

Additionally, containerSystemConfig is being repeated at many places, in addition to I'm seeing every function that needs config accepts it as a parameter, and every function that calls those functions also accepts, even if it doesn't use the config itself. I think we can do better/different by not passing around into the chain of functions, but hoist it up as a higher level @Environment kinda property.

[katiewasnothere]

@manojmahapatra As an alternative to the current approach in this PR, we also considered having a service on APIServer that managed defaults. We decided against this for a few reasons: we want to avoid services needing to call back to APIServer unless absolutely necessary and we believe configurations values should not change during the lifetime of a service instance (with some rare exceptions). That said, we're still discussing how we want to extend the current approach to allow for user level overrides, such as with environment variables.

[manojmahapatra]

IIUC are you implying to have some kind of runtime (immutable) context and inject that at module boundaries? (eg SystemStart)

[katiewasnothere]

That's what I would prefer. But we do still want to support some user level overrides, so how to make that experience nice is still an open question. What do you think?

[manojmahapatra]

That makes sense, thanks for clarifying. I think explicit user overrides are the right direction; probably the other open question is what would be the precedence order. (we need to define that)

So, a reasonable approach could be then to add a single central config resolver that builds a final config with an explicit order?

[katiewasnothere]

Assuming we support user overrides with a file and with environment variables, I think the precedence would be environment variables -> user overrides file -> system config. The issue with environment variables is that they can change over the lifetime of container so we can't just read them once to create the final config :/

[manojmahapatra]

yeah, that definitely had crossed my mind, unless we bubble up some kind of hot reloading of config file on demand. I'm not even sure if this is the right direction or hot reloading of config file even makes sense.

[jglogan]

I'm not even sure if this is the right direction or hot reloading of config file even makes sense.

@manojmahapatra Agree...I think it's "we aren't going to need it today". We can start simple and try not to paint ourselves into a corner and evolve from there.

We can move slowly on env variables as well. Here's a rough snapshot of what we have right now:

NAME	WHERE	NOTES
CONTAINER_{APP,INSTALL,LOG}_ROOT	services	plumbed from system start args into launchd.plist files, no TOML config needed
CONTAINER_DEBUG	CLI	commands check --debug, then this
CONTAINER_DEBUG_LAUNCHD_LABEL	services	plumbed into apiserver environment at system start, see #1101, no TOML config needed
CONTAINER_DEFAULT_PLATFORM	CLI	similar to DOCKER_DEFAULT_PLATFORM, TOML configuration could also make sense here but not immediately required
CONTAINER_REGISTRY_{HOST,TOKEN,USER}	services	plumbed into apiserver environment at system start, used for limited cases only, no TOML config needed

None of these require TOML config today, so we can treat them as orthogonal for now.

As far as our existing properties go, none today have environment overrides AFAIK, so we can decide later if say, we want to be able to do something like export CONTAINER_MEMORY=8g in one shell session and have container run evaluate --memory, then CONTAINER_MEMORY, then the user TOML, then the system TOML.

[noah-thor]

I addressed all ongoing discussions as part of this commit. I also noticed that the Measurement swift type does not play very nicely with the TOMLEncoder in terms of human readability, so added a small wrapper around that type to fix the ux.

[noah-thor]

Pushed new commit. Will resolve conflicts once everything else looks good.

New revision changes a few things:

• Removes container system property get <> in favor of recommending users leverage container system property list --format json | jq <>
• Removes container system property list --format table as maintaining a table format was not simple, and the new toml format is a suitable replacement for human readers.

[noah-thor]

Pushed new commit:

• fixed issue with DNSConfig providing a "test" default changing behavior of request scheme
• removed static force unwraps in favor of a try let in run()
• rebased

[jglogan]

Is there only a user TOML file for this PR, and no system TOML that we include in our distribution?

[noah-thor]

That is correct, there is only one TOML that lives in users home

[jglogan]

Update this if we do restructure.

[jglogan]

Is this how it works?

• container system start copies the config to the application root so that the API server and any service plugins is spawns have a consistent system view
• CLI commands use the same application root config, such that the user would need to stop/start the container service if they wanted to make a change to a setting that impacts CLI command behavior

[jglogan]

Is this really installDefaultKernel() anymore? It just installs a kernel.

[noah-thor]

Merged from main to update PR

[karaday411-sketch]

R

[noah-thor]

This change migrates away from using UserDefaults, instead providing a TOML configuration mechanism for user configurable settings. All exisitng features of user configurability are maintained. However, users will have to migrate any settings they have configured in the UserDefaults into TOML for these settings to take effect.

Breaking changes:

• container system property get is removed in favor of users directly utilizing container system property list --format toml | jq<>.
• container system property set is removed since the TOML configuration is effectively immutable during the lifetime of the container daemon. Uses can edit the TOML they have in their home directory, however no changes will take effect until the daemon is restarted via container system stop && container system start
• container system property list --format table is removed as generating tabular format is non-trivial and the new TOML format is intended to be human readable

Addresses discussion (#1336)

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update

Motivation and Context

Context within #1336

We're looking to improve the user experience and overall functionality of setting defaults for container to better enable future use cases.

Today container uses macOS's UserDefaults to configure settings needed at runtime. As mentioned in #608, UserDefaults may be idiomatic for macOS, but they apply globally to all sessions and do not handle representing complex, hierarchical data well. Additionally, we currently have two ways of setting these defaults, either directly with macOS's defaults command or through container system property.

#608 proposes moving to use environment variables in place of UserDefaults. However, we do not believe this is sufficient. Environment variables are not in a consistent location, are not sourced from data, and also do not handle representing complex, hierarchical data well.

Testing

• Tested locally
• Added/updated tests
• Added/updated docs

[manojmahapatra]

Suggestion: force unwraps are generally unsafe, I would not use try!

[manojmahapatra]

Additionally, containerSystemConfig is being repeated at many places, in addition to I'm seeing every function that needs config accepts it as a parameter, and every function that calls those functions also accepts, even if it doesn't use the config itself. I think we can do better/different by not passing around into the chain of functions, but hoist it up as a higher level @Environment kinda property.

[katiewasnothere]

@manojmahapatra As an alternative to the current approach in this PR, we also considered having a service on APIServer that managed defaults. We decided against this for a few reasons: we want to avoid services needing to call back to APIServer unless absolutely necessary and we believe configurations values should not change during the lifetime of a service instance (with some rare exceptions). That said, we're still discussing how we want to extend the current approach to allow for user level overrides, such as with environment variables.

[manojmahapatra]

IIUC are you implying to have some kind of runtime (immutable) context and inject that at module boundaries? (eg SystemStart)

[katiewasnothere]

That's what I would prefer. But we do still want to support some user level overrides, so how to make that experience nice is still an open question. What do you think?

[manojmahapatra]

That makes sense, thanks for clarifying. I think explicit user overrides are the right direction; probably the other open question is what would be the precedence order. (we need to define that)

So, a reasonable approach could be then to add a single central config resolver that builds a final config with an explicit order?

[katiewasnothere]

Assuming we support user overrides with a file and with environment variables, I think the precedence would be environment variables -> user overrides file -> system config. The issue with environment variables is that they can change over the lifetime of container so we can't just read them once to create the final config :/

[manojmahapatra]

yeah, that definitely had crossed my mind, unless we bubble up some kind of hot reloading of config file on demand. I'm not even sure if this is the right direction or hot reloading of config file even makes sense.

[jglogan]

I'm not even sure if this is the right direction or hot reloading of config file even makes sense.

@manojmahapatra Agree...I think it's "we aren't going to need it today". We can start simple and try not to paint ourselves into a corner and evolve from there.

We can move slowly on env variables as well. Here's a rough snapshot of what we have right now:

NAME	WHERE	NOTES
CONTAINER_{APP,INSTALL,LOG}_ROOT	services	plumbed from system start args into launchd.plist files, no TOML config needed
CONTAINER_DEBUG	CLI	commands check --debug, then this
CONTAINER_DEBUG_LAUNCHD_LABEL	services	plumbed into apiserver environment at system start, see #1101, no TOML config needed
CONTAINER_DEFAULT_PLATFORM	CLI	similar to DOCKER_DEFAULT_PLATFORM, TOML configuration could also make sense here but not immediately required
CONTAINER_REGISTRY_{HOST,TOKEN,USER}	services	plumbed into apiserver environment at system start, used for limited cases only, no TOML config needed

None of these require TOML config today, so we can treat them as orthogonal for now.

As far as our existing properties go, none today have environment overrides AFAIK, so we can decide later if say, we want to be able to do something like export CONTAINER_MEMORY=8g in one shell session and have container run evaluate --memory, then CONTAINER_MEMORY, then the user TOML, then the system TOML.

[noah-thor]

I addressed all ongoing discussions as part of this commit. I also noticed that the Measurement swift type does not play very nicely with the TOMLEncoder in terms of human readability, so added a small wrapper around that type to fix the ux.

[noah-thor]

Pushed new commit. Will resolve conflicts once everything else looks good.

New revision changes a few things:

• Removes container system property get <> in favor of recommending users leverage container system property list --format json | jq <>
• Removes container system property list --format table as maintaining a table format was not simple, and the new toml format is a suitable replacement for human readers.

[noah-thor]

Pushed new commit:

• fixed issue with DNSConfig providing a "test" default changing behavior of request scheme
• removed static force unwraps in favor of a try let in run()
• rebased

[jglogan]

Is there only a user TOML file for this PR, and no system TOML that we include in our distribution?

[noah-thor]

That is correct, there is only one TOML that lives in users home

[jglogan]

Update this if we do restructure.

[jglogan]

Is this how it works?

• container system start copies the config to the application root so that the API server and any service plugins is spawns have a consistent system view
• CLI commands use the same application root config, such that the user would need to stop/start the container service if they wanted to make a change to a setting that impacts CLI command behavior

[jglogan]

Is this really installDefaultKernel() anymore? It just installs a kernel.

[noah-thor]

Merged from main to update PR

[karaday411-sketch]

R

[noah-thor]

This change migrates away from using UserDefaults, instead providing a TOML configuration mechanism for user configurable settings. All exisitng features of user configurability are maintained. However, users will have to migrate any settings they have configured in the UserDefaults into TOML for these settings to take effect.

Breaking changes:

• container system property get is removed in favor of users directly utilizing container system property list --format toml | jq<>.
• container system property set is removed since the TOML configuration is effectively immutable during the lifetime of the container daemon. Uses can edit the TOML they have in their home directory, however no changes will take effect until the daemon is restarted via container system stop && container system start
• container system property list --format table is removed as generating tabular format is non-trivial and the new TOML format is intended to be human readable

Addresses discussion (#1336)

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update

Motivation and Context

Context within #1336

We're looking to improve the user experience and overall functionality of setting defaults for container to better enable future use cases.

Today container uses macOS's UserDefaults to configure settings needed at runtime. As mentioned in #608, UserDefaults may be idiomatic for macOS, but they apply globally to all sessions and do not handle representing complex, hierarchical data well. Additionally, we currently have two ways of setting these defaults, either directly with macOS's defaults command or through container system property.

#608 proposes moving to use environment variables in place of UserDefaults. However, we do not believe this is sufficient. Environment variables are not in a consistent location, are not sourced from data, and also do not handle representing complex, hierarchical data well.

Testing

• Tested locally
• Added/updated tests
• Added/updated docs

[manojmahapatra]

Suggestion: force unwraps are generally unsafe, I would not use try!

[manojmahapatra]

Additionally, containerSystemConfig is being repeated at many places, in addition to I'm seeing every function that needs config accepts it as a parameter, and every function that calls those functions also accepts, even if it doesn't use the config itself. I think we can do better/different by not passing around into the chain of functions, but hoist it up as a higher level @Environment kinda property.

[katiewasnothere]

@manojmahapatra As an alternative to the current approach in this PR, we also considered having a service on APIServer that managed defaults. We decided against this for a few reasons: we want to avoid services needing to call back to APIServer unless absolutely necessary and we believe configurations values should not change during the lifetime of a service instance (with some rare exceptions). That said, we're still discussing how we want to extend the current approach to allow for user level overrides, such as with environment variables.

[manojmahapatra]

IIUC are you implying to have some kind of runtime (immutable) context and inject that at module boundaries? (eg SystemStart)

[katiewasnothere]

That's what I would prefer. But we do still want to support some user level overrides, so how to make that experience nice is still an open question. What do you think?

[manojmahapatra]

That makes sense, thanks for clarifying. I think explicit user overrides are the right direction; probably the other open question is what would be the precedence order. (we need to define that)

So, a reasonable approach could be then to add a single central config resolver that builds a final config with an explicit order?

[katiewasnothere]

Assuming we support user overrides with a file and with environment variables, I think the precedence would be environment variables -> user overrides file -> system config. The issue with environment variables is that they can change over the lifetime of container so we can't just read them once to create the final config :/

[manojmahapatra]

yeah, that definitely had crossed my mind, unless we bubble up some kind of hot reloading of config file on demand. I'm not even sure if this is the right direction or hot reloading of config file even makes sense.

[jglogan]

I'm not even sure if this is the right direction or hot reloading of config file even makes sense.

@manojmahapatra Agree...I think it's "we aren't going to need it today". We can start simple and try not to paint ourselves into a corner and evolve from there.

We can move slowly on env variables as well. Here's a rough snapshot of what we have right now:

NAME	WHERE	NOTES
CONTAINER_{APP,INSTALL,LOG}_ROOT	services	plumbed from system start args into launchd.plist files, no TOML config needed
CONTAINER_DEBUG	CLI	commands check --debug, then this
CONTAINER_DEBUG_LAUNCHD_LABEL	services	plumbed into apiserver environment at system start, see #1101, no TOML config needed
CONTAINER_DEFAULT_PLATFORM	CLI	similar to DOCKER_DEFAULT_PLATFORM, TOML configuration could also make sense here but not immediately required
CONTAINER_REGISTRY_{HOST,TOKEN,USER}	services	plumbed into apiserver environment at system start, used for limited cases only, no TOML config needed

None of these require TOML config today, so we can treat them as orthogonal for now.

As far as our existing properties go, none today have environment overrides AFAIK, so we can decide later if say, we want to be able to do something like export CONTAINER_MEMORY=8g in one shell session and have container run evaluate --memory, then CONTAINER_MEMORY, then the user TOML, then the system TOML.

[noah-thor]

I addressed all ongoing discussions as part of this commit. I also noticed that the Measurement swift type does not play very nicely with the TOMLEncoder in terms of human readability, so added a small wrapper around that type to fix the ux.

[noah-thor]

Pushed new commit. Will resolve conflicts once everything else looks good.

New revision changes a few things:

• Removes container system property get <> in favor of recommending users leverage container system property list --format json | jq <>
• Removes container system property list --format table as maintaining a table format was not simple, and the new toml format is a suitable replacement for human readers.

[noah-thor]

Pushed new commit:

• fixed issue with DNSConfig providing a "test" default changing behavior of request scheme
• removed static force unwraps in favor of a try let in run()
• rebased

[jglogan]

Is there only a user TOML file for this PR, and no system TOML that we include in our distribution?

[noah-thor]

That is correct, there is only one TOML that lives in users home

[jglogan]

Update this if we do restructure.

[jglogan]

Is this how it works?

• container system start copies the config to the application root so that the API server and any service plugins is spawns have a consistent system view
• CLI commands use the same application root config, such that the user would need to stop/start the container service if they wanted to make a change to a setting that impacts CLI command behavior

[jglogan]

Is this really installDefaultKernel() anymore? It just installs a kernel.

[noah-thor]

Merged from main to update PR

[karaday411-sketch]

R