Shell API and CLI: Add option for staging additional runtime targets by nathanwilliams-ct · Pull Request #2147 · apache/buildstream

nathanwilliams-ct · 2026-07-06T13:31:13Z

This enables users to add additional functionality such as debug tooling in the shell sandbox,
without needing to modify the target element. This is achived through introducing a new option to
shell.

All existing API and UX is maintained, to not break existing scripts. An alternative design was
considered, to have the additional elements as positional arguments similar to the existing element,
but this would need manual parsing to handle the cases where -- is present and not
present, splitting based on a .bst suffix. This UX could be re-visited in future.

Example usage:
bst shell --with base.bst example.bst -- cat example.txt
Where:

example.bst is a simple import element with no dependencies that imports a file called example.txt
base.bst provides a basic alpine sysroot with a standard set of unix tooling (sh, df, cat etc).

Changes:

Introduces test_with_other_targets integration test to the shell test suite.
Adds --with cli option to the shell subcommand and updates it's documentation.
- option can be used multiple times by caller, providing a list of targets.
Extends the shell top level calling interface in Buildstream core to accept a list of
other_targets
- This is where the targets are loaded into elements and checked to make sure they are present
Extends the shell element implementation to accept a list of other targets
- This is where the other elements are staged and integrated into the sandbox

towards: #422

nathanwilliams-ct · 2026-07-10T14:36:34Z

Thanks, whoever triggered the CI... looks like it needs a little work, despite the tests working locally in the tox environment and I missed the linters.

nathanwilliams-ct · 2026-07-16T12:42:57Z

I think this should pass CI now, fixed docs, linters, static type checker, formatter and 3.10 issues

nathanwilliams-ct · 2026-07-16T14:50:02Z

pre-commit hooks would be nice. I keep running the commands locally, but It seems I am doing them in the wrong order.. fixing one thing breaks the others XD

nathanwilliams-ct · 2026-07-16T14:54:15Z

tox
tox -- --integration
tox -e docs
tox -e lint
tox -e mypy
tox -e format

And the tests locally only run on 313 and 314 for me, while CI does also 310 311 312

Is complete list right?

juergbi · 2026-07-17T07:58:04Z

And the tests locally only run on 313 and 314 for me, while CI does also 310 311 312

tox runs the tests on all Python versions from 3.10 to 3.14 by default, but skipping versions where the interpreter is not installed. It's normally fine to test against just one Python version locally, letting CI take care of the others, unless you're actively debugging a version-specific issue.

Is complete list right?

CI also runs buildgrid and buildbarn, tests against master of buildstream-plugins-community, and builds wheels. Unless you're working in these specific areas, it shouldn't be necessary to run those locally.

nathanwilliams-ct · 2026-07-17T08:30:57Z

Ah there's a subtle conflict between format and the docs generation.

nathanwilliams-ct · 2026-07-17T09:15:45Z

Static type checking moved out into it's own PR: #2153

juergbi · 2026-07-17T10:51:22Z

+ with self.timed_activity("Staging other_targets", silent_nested=True), self.__collect_overlaps(sandbox):
+ self.stage_dependency_artifacts(sandbox, other_elements)
+
+ if other_elements:
+ # Stage artifacts from other_elements into the sandbox.
+ for element in other_elements:
+ # Stage deps in the sandbox root
+ with element.timed_activity("Integrating sandbox"), sandbox.batch():
+ for dep in element._dependencies(_Scope.RUN):
+ dep.integrate(sandbox)


The separate staging and integration into the same sandbox could be problematic. The main target element and the other targets may share some dependencies, in which case an element gets staged twice. This may cause confusing overlap warnings or errors (and in some constellations maybe also some real overlap conflicts).
Separate integration means that integration commands of dependencies of the main target can't cover integration with other targets. And integration commands of shared dependencies will be executed twice (or even more with multiple --with). Additional sandbox batch execution for other target integration may also not be the most efficient approach, but performance is not my main concern here.
I'm not saying that this approach is definitely unacceptable, but at the very least it needs to have documented and tested behavior for mentioned aspects such as overlaps and integration commands. Also build shells might not be tested at all right now, if I haven't missed anything.
Regarding overlaps, a possible alternative would be to stage the other elements into a separate sandbox / virtual directory (with the usual overlap processing) and then merge it into the shell where the overlap handling may be different (e.g., the --with tree allowed to always replace files). This was also suggested in https://mail.gnome.org/archives/buildstream-list/2019-February/msg00001.html. Integration commands will still be problematic but maybe some limitations there are acceptable (but should also be clarified). I would definitely at least use a single integration sandbox for all 'other' elements and don't duplicate integration within that part.
Virtual stack element
If it was only for runtime shells, I think the behavior should rather be equivalent to creating a stack element that has the main target and all other targets as dependencies, which would likely not even require any changes in element.py. One caveat is that runtime shells use the environment variables from the target element, so that would break with the (virtual) stack element approach.
However, build shells make things more complicated as there the main target has essentially full control over the sandbox.
Inject other targets as dependencies
One possible alternative that comes to mind is that we may be able to inject the --with elements as dependencies of the main target (runtime dependency for runtime shells and build dependency for build shells). There could be element plugins where this is problematic for build shells but normal build elements should be fine and build shells anyway can't work with all element plugins.
It's possible that I'm missing something why this would be a bad idea, but it might be worth exploring if nobody can think of a clear blocker right away.
One issue I can think of is that it might not work with buildtrees where we get the full sandbox root from CAS and don't stage anything. It may be possible to support an alternative buildtree support (only used with --with) where we first construct a sandbox like for a normal build shell and then only replace the source/build directory with the corresponding directory from the buildtree. If we want to go down this route, this should likely wait for a follow-up PR.

I will look to explore these alternative routes, thanks for the feedback.

$ tox -e venv -- bst --directory tests/integration/project shell manual/import-file.bst [--:--:--][ ][ main:core activity ] START Loading elements [00:00:00][ ][ main:core activity ] SUCCESS Loading elements [--:--:--][ ][ main:core activity ] START Resolving elements [00:00:00][ ][ main:core activity ] SUCCESS Resolving elements [--:--:--][ ][ main:core activity ] START Initializing remote caches [00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches [--:--:--][ ][ main:core activity ] START Query cache [00:00:00][ ][ main:core activity ] SUCCESS Query cache [--:--:--][dc2422ef][ main:manual/import-file.bst ] START Staging dependencies [00:00:00][dc2422ef][ main:manual/import-file.bst ] SUCCESS Staging dependencies [--:--:--][dc2422ef][ main:manual/import-file.bst ] START Integrating sandbox [00:00:00][dc2422ef][ main:manual/import-file.bst ] SUCCESS Integrating sandbox [--:--:--][dc2422ef][ main:manual/import-file.bst ] STATUS Running command sh -i Error launching shell: Staged artifacts do not provide command 'sh' $ tox -e venv -- bst --directory tests/integration/project shell manual/import-file.bst --with base/base-alpine.bst [--:--:--][ ][ main:core activity ] START Loading elements [00:00:00][ ][ main:core activity ] SUCCESS Loading elements [--:--:--][ ][ main:core activity ] START Resolving elements [00:00:00][ ][ main:core activity ] SUCCESS Resolving elements [--:--:--][ ][ main:core activity ] START Initializing remote caches [00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches [--:--:--][ ][ main:core activity ] START Query cache [00:00:00][ ][ main:core activity ] SUCCESS Query cache [--:--:--][e8d846b7][ build:manual/import-file.bst_tempshwrznrg.bst] START test/manual-import-file.bst_tempshwrznrg/e8d846b7-build.20260818-111411.log [--:--:--][e8d846b7][ build:manual/import-file.bst_tempshwrznrg.bst] START Staging sources [00:00:00][e8d846b7][ build:manual/import-file.bst_tempshwrznrg.bst] SUCCESS Staging sources [--:--:--][e8d846b7][ build:manual/import-file.bst_tempshwrznrg.bst] START Caching artifact [00:00:00][e8d846b7][ build:manual/import-file.bst_tempshwrznrg.bst] SUCCESS Caching artifact [00:00:00][e8d846b7][ build:manual/import-file.bst_tempshwrznrg.bst] SUCCESS test/manual-import-file.bst_tempshwrznrg/e8d846b7-build.20260818-111411.log [--:--:--][ ][ main:core activity ] START Loading elements [00:00:00][ ][ main:core activity ] SUCCESS Loading elements [--:--:--][ ][ main:core activity ] START Resolving elements [00:00:00][ ][ main:core activity ] SUCCESS Resolving elements [--:--:--][ ][ main:core activity ] START Initializing remote caches [00:00:00][ ][ main:core activity ] SUCCESS Initializing remote caches [--:--:--][ ][ main:core activity ] START Query cache [00:00:00][ ][ main:core activity ] SUCCESS Query cache [--:--:--][e8d846b7][ main:manual/import-file.bst_tempshwrznrg.bst] START Staging dependencies [00:00:00][e8d846b7][ main:manual/import-file.bst_tempshwrznrg.bst] SUCCESS Staging dependencies [--:--:--][e8d846b7][ main:manual/import-file.bst_tempshwrznrg.bst] START Integrating sandbox [00:00:00][e8d846b7][ main:manual/import-file.bst_tempshwrznrg.bst] SUCCESS Integrating sandbox [--:--:--][e8d846b7][ main:manual/import-file.bst_tempshwrznrg.bst] STATUS Running command sh -i [e8d846b7@manual/import-file.bst_tempshwrznrg.bst:/]$ cat test.txt This is a test [e8d846b7@manual/import-file.bst_tempshwrznrg.bst:/]$ exit
hmm, I went down the route of creating a temporary element.
Due to how the element loading works, it makes it almost impossible to inject additional dependencies in at runtime. Temporary stack element would have made buildtree ones unhelpful. Solving the overlap problem and dealing with overlapping sandboxes and integration commands was too complicated and messy.
The downside of using a temporary element is the shell command needs to attempt to run a build on the temporary element, before it can shell, to cache it's build result.
I'll tidy up my prototype with some tests and push it a bit later on.

Temporary stack element would have made buildtree ones unhelpful.
The downside of using a temporary element is the shell command needs to attempt to run a build on the temporary element, before it can shell, to cache it's build result.
Not being able to use a cached build seems like a major downside to me. I was originally thinking of injecting it in the in-memory representation, not creating a temporary file. But maybe that's not feasible.
A possible mitigation with the temporary file approach could be using different approaches for build and runtime shells. A temporary stack element for runtime shells and what your branch is doing now for build shells. Or have you already considered this as not tenable for some reason?

While injecting deps in the in-memory representation should theoretically be possible, it would probably be too invasive just for this feature.
A possible tweak to the temp file approach could be to add some kind of substitution / path override dict to the Loader. This could be set in Stream.shell_with(). _load_file_no_deps() would then use the path override, instead of constructing the regular fullpath. However, it would still use the regular filename as shortname.
I haven't prototyped this, but the possible advantages are:
No cache key difference for runtime shells, being able to use a previously cached build
Logging wouldn't expose temp filenames
Temp file could be placed in a temporary directory instead of polluting the elements directory with a temporary file
Any thoughts?

Temporary stack element for runtime shell would be possible I did try it out, but because build shells don't work with that approach and with the aim to keep maintenance cost down: I didn't want two separate implementations if we can help it.
I tried a few ways to inject dependencies in-memory, but part the problem I found was a lot of dependency loading and resolving work is done at the early stages immediately after loading the yaml and parsing, where it's basically impossible to inject the dependencies in a sensible way. Doing it at a later stage e.g. in the Stream.shell or the Element.execute_shell method and or trying to add a inject_extra_deps method to the Element class, I couldn't get it to work. Partly due to the complexity of the shell construction, especially where it involves the cached buildtrees being used which means the whole shell construction process is skipped. The lack of type hints to actually understand the control flow also doesn’t help(#2167).
I will have a play with adjusting the tempfile approach to hide it a bit better like you suggest.

It might be nice to have a 'reload' method on Element that can cleanly reload the element if there are changes at runtime, but it would involve some heavy refactoring that I currently don't have the context to do. Element is heavily implemented around the idea of everything is immutable.
We could almost do with a replacing the whole Element class with an explicit state machine to represent elements at different stages of it's lifecycle. There are so many Optional fields and 'state' booleans, it's difficult to untangle and work out which state the element is currently in, and what order to call things in etc.
[*] -->LoadElement: Load from YAML LoadElement-->WeakElement: Calculate weak cache key WeakElement-->StrongElement: Resolve dependencies and calculate strong key StrongElement-->CachedElement: Element has cached artifacts StrongElement-->PreparedElement: Prepare sources and dependencies for a build PreparedElement-->CachedElement: Build the element PreparedElement-->FailedBuildElement: cache buildtree

This enables users to add additional functionality such as debug tooling in the shell sandbox, without needing to modify the target element. This is achived through introducing a new option to shell. All existing API and UX is maintained, to not break existing scripts. An alternative design was considered, to have the additional elements as positional arguments similar to the existing element, but this would need manual parsing to handle the cases where `--` is present and not present, splitting based on a `.bst` suffix. This UX could be re-visited in future. Example usage: bst shell --with base.bst example.bst -- cat example.txt Where: - example.bst is a simple import element with no dependencies that imports a file called example.txt - base.bst provides a basic alpine sysroot with a standard set of unix tooling (sh, df, cat etc). Changes: - Introduces `test_with_other_targets` integration test to the shell test suite. - Adds `--with` cli option to the shell subcommand and updates it's documentation. - option can be used multiple times by caller, providing a list of targets. - Extends the shell top level calling interface in Buildstream core to accept a list of other_targets - This is where the targets are loaded into elements and checked to make sure they are present - Extends the shell element implementation to accept a list of other targets - This is where the other elements are staged and integrated into the sandbox

nathanwilliams-ct · 2026-07-06T13:31:13Z

This enables users to add additional functionality such as debug tooling in the shell sandbox,
without needing to modify the target element. This is achived through introducing a new option to
shell.

All existing API and UX is maintained, to not break existing scripts. An alternative design was
considered, to have the additional elements as positional arguments similar to the existing element,
but this would need manual parsing to handle the cases where -- is present and not
present, splitting based on a .bst suffix. This UX could be re-visited in future.

Example usage:
bst shell --with base.bst example.bst -- cat example.txt
Where:

example.bst is a simple import element with no dependencies that imports a file called example.txt
base.bst provides a basic alpine sysroot with a standard set of unix tooling (sh, df, cat etc).

Changes:

Introduces test_with_other_targets integration test to the shell test suite.
Adds --with cli option to the shell subcommand and updates it's documentation.
- option can be used multiple times by caller, providing a list of targets.
Extends the shell top level calling interface in Buildstream core to accept a list of
other_targets
- This is where the targets are loaded into elements and checked to make sure they are present
Extends the shell element implementation to accept a list of other targets
- This is where the other elements are staged and integrated into the sandbox

towards: #422

nathanwilliams-ct · 2026-07-10T14:36:34Z

Thanks, whoever triggered the CI... looks like it needs a little work, despite the tests working locally in the tox environment and I missed the linters.

nathanwilliams-ct · 2026-07-16T12:42:57Z

I think this should pass CI now, fixed docs, linters, static type checker, formatter and 3.10 issues

nathanwilliams-ct · 2026-07-16T14:50:02Z

pre-commit hooks would be nice. I keep running the commands locally, but It seems I am doing them in the wrong order.. fixing one thing breaks the others XD

nathanwilliams-ct · 2026-07-16T14:54:15Z

tox
tox -- --integration
tox -e docs
tox -e lint
tox -e mypy
tox -e format

And the tests locally only run on 313 and 314 for me, while CI does also 310 311 312

Is complete list right?

juergbi · 2026-07-17T07:58:04Z

And the tests locally only run on 313 and 314 for me, while CI does also 310 311 312

tox runs the tests on all Python versions from 3.10 to 3.14 by default, but skipping versions where the interpreter is not installed. It's normally fine to test against just one Python version locally, letting CI take care of the others, unless you're actively debugging a version-specific issue.

Is complete list right?

CI also runs buildgrid and buildbarn, tests against master of buildstream-plugins-community, and builds wheels. Unless you're working in these specific areas, it shouldn't be necessary to run those locally.

nathanwilliams-ct · 2026-07-17T08:30:57Z

Ah there's a subtle conflict between format and the docs generation.

nathanwilliams-ct · 2026-07-17T09:15:45Z

Static type checking moved out into it's own PR: #2153

juergbi · 2026-07-17T10:51:22Z

+ with self.timed_activity("Staging other_targets", silent_nested=True), self.__collect_overlaps(sandbox):
+ self.stage_dependency_artifacts(sandbox, other_elements)
+
+ if other_elements:
+ # Stage artifacts from other_elements into the sandbox.
+ for element in other_elements:
+ # Stage deps in the sandbox root
+ with element.timed_activity("Integrating sandbox"), sandbox.batch():
+ for dep in element._dependencies(_Scope.RUN):
+ dep.integrate(sandbox)
