cli: add buildtree checkout command by ownik · Pull Request #2129 · apache/buildstream

ownik · 2026-06-01T13:35:42Z

This PR adds a "--use-buildtrees" option to "bst artifacts checkout", similar to the existing option in "bst shell".

The goal is to make it possible to check out build artifacts from the sandbox build tree when it is available.

Would be okay with adding this functionality, or are there reasons not to expose it through "bst artifacts checkout"? Or is there another way to get build artifacts from sandbox now?

ownik · 2026-06-08T09:17:05Z

@BenjaminSchubert @abderrahim @cs-shadow @gtristan @juergbi I've made changes to verify that buildtree exists in the cache, similar to the shell command. Could you please review, please?

abderrahim · 2026-06-08T10:02:02Z

Looks like this is fixing #1822, please link the issue in your PR. Also please add a test for this behaviour.

I'll try to review it today.

abderrahim · 2026-06-08T12:55:20Z

A few things need to be considered in the design of this feature:

Someone recently suggested to implement this as a flag to source checkout rather than artifact checkout. It kinda makes sense that the build tree is related to the source, but it's also stored as part of the artifact. Maybe it makes sense to move it to a separate toplevel command? something like bst buildtree checkout?
This is currently checking out the buildroot rather than what is traditionally considered a "build tree". This means it can work with both build elements and non-build-elements, but it also means we get a whole sandbox with the dependencies, the build tree, and even the artifact (if the build succeeded) which is probably not what people asking for the build tree are looking for.
if we go with a new toplevel command, we might want to expose options to checkout either the build tree or the build root.

ownik · 2026-06-09T12:46:49Z

@abderrahim Thanks for the feedback! I've reworked the implementation into a new bst buildtree checkout command. By default it checks out the buildtree, and a --buildroot option can be used to export the entire buildroot instead.

I've also added an integration test covering the new command and link the issue in PR.

ownik · 2026-06-17T12:33:37Z

@abderrahim Any chance to review this PR?

abderrahim

Thanks for the contributions. See my comments below.

abderrahim · 2026-06-30T08:47:27Z

+ # self.targets contains a list of the loaded target objects
+ # if we specify --deps build, Stream._load() will return a list
+ # of build dependency objects, however, we need to prepare a sandbox
+ # with the target (which has had its appropriate dependencies loaded)
+ element: Element = self.targets[0]


Does this make sense for the build tree? Either we're checking out just the buildtree, and then dependencies don't make sense, or we check out the whole buildroot and it already contains the build dependencies.
Am I missing something?

Sorry for the delayed response, and thanks for pointing this out.
Yes, you're right. I've changed this to element: Element = elements[0], is it ok?

abderrahim · 2026-06-30T08:50:31Z

+ # try to pull them if they are not already cached.
+ #
+ self.query_cache(elements)
+ self._pull_missing_artifacts(elements)


I think we only need to pull the artifact for the element in question. No need for dependencies here.
as an aside, I wonder if we should try to override the pull-buildtrees configuration in this case: if we're pulling to extract a build tree, we should probably try to pull the buildtree even if we usually don't.

abderrahim · 2026-06-30T08:57:41Z

+
+ try:
+ artifact = element._get_artifact()
+ virdir = artifact.get_buildroot() if buildroot else artifact.get_buildtree()


Another thing we need to be aware of (and also give a suggestion about to the user) is that not all elements have a build tree: only build elements do.
So it's not enough to check for the build root, we also need to check for the build tree. Then depending on the availability and the user request, we can suggest an alternative to the user.
buildtree available buildroot available user requested result
Y Y build tree extract build tree
Y Y build root extract build root
N Y build tree error + suggest extract build root
N Y build root extract build root
N N build tree error
N N build root error

Thanks for pointing this out. I've accounted for all of the cases you described and added tests covering them.
Could you please take another look when you have a chance?

abderrahim · 2026-06-30T08:58:48Z

+ project = str(datafiles)
+ element_name = "build-shell/buildtree.bst"
+ checkout = os.path.join(cli.directory, "checkout")
+ tar = os.path.join(cli.directory, "source-checkout.tar")


Suggested change
tar=os.path.join(cli.directory, "source-checkout.tar")
tar=os.path.join(cli.directory, "buildtree.tar")

abderrahim · 2026-06-30T09:03:53Z

+@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox")
+def test_buildtree_checkout_fail(cli, datafiles):
+ project = str(datafiles)
+ element_name = "build-shell/buildtree.bst"


We should also test the buildtree of a script element (using build-shell/script.bst). This is the corner case I mentioned above where we have a build root but not a build tree.

i've added a test case which uses build-shell/script.bst

Add a new `bst buildtree checkout` command for exporting buildtree stored in artifacts. By default, the command checks out the buildtree. The `--buildroot` option can be used to export the entire buildroot sandbox instead. Fixedapache#1822

ownik · 2026-06-01T13:35:42Z

This PR adds a "--use-buildtrees" option to "bst artifacts checkout", similar to the existing option in "bst shell".

The goal is to make it possible to check out build artifacts from the sandbox build tree when it is available.

Would be okay with adding this functionality, or are there reasons not to expose it through "bst artifacts checkout"? Or is there another way to get build artifacts from sandbox now?

ownik · 2026-06-08T09:17:05Z

@BenjaminSchubert @abderrahim @cs-shadow @gtristan @juergbi I've made changes to verify that buildtree exists in the cache, similar to the shell command. Could you please review, please?

abderrahim · 2026-06-08T10:02:02Z

Looks like this is fixing #1822, please link the issue in your PR. Also please add a test for this behaviour.

I'll try to review it today.

abderrahim · 2026-06-08T12:55:20Z

A few things need to be considered in the design of this feature:

Someone recently suggested to implement this as a flag to source checkout rather than artifact checkout. It kinda makes sense that the build tree is related to the source, but it's also stored as part of the artifact. Maybe it makes sense to move it to a separate toplevel command? something like bst buildtree checkout?
This is currently checking out the buildroot rather than what is traditionally considered a "build tree". This means it can work with both build elements and non-build-elements, but it also means we get a whole sandbox with the dependencies, the build tree, and even the artifact (if the build succeeded) which is probably not what people asking for the build tree are looking for.
if we go with a new toplevel command, we might want to expose options to checkout either the build tree or the build root.

ownik · 2026-06-09T12:46:49Z

@abderrahim Thanks for the feedback! I've reworked the implementation into a new bst buildtree checkout command. By default it checks out the buildtree, and a --buildroot option can be used to export the entire buildroot instead.

I've also added an integration test covering the new command and link the issue in PR.

ownik · 2026-06-17T12:33:37Z

@abderrahim Any chance to review this PR?

abderrahim

Thanks for the contributions. See my comments below.

abderrahim · 2026-06-30T08:47:27Z

+ # self.targets contains a list of the loaded target objects
+ # if we specify --deps build, Stream._load() will return a list
+ # of build dependency objects, however, we need to prepare a sandbox
+ # with the target (which has had its appropriate dependencies loaded)
+ element: Element = self.targets[0]


Does this make sense for the build tree? Either we're checking out just the buildtree, and then dependencies don't make sense, or we check out the whole buildroot and it already contains the build dependencies.
Am I missing something?

Sorry for the delayed response, and thanks for pointing this out.
Yes, you're right. I've changed this to element: Element = elements[0], is it ok?

abderrahim · 2026-06-30T08:50:31Z

+ # try to pull them if they are not already cached.
+ #
+ self.query_cache(elements)
+ self._pull_missing_artifacts(elements)


I think we only need to pull the artifact for the element in question. No need for dependencies here.
as an aside, I wonder if we should try to override the pull-buildtrees configuration in this case: if we're pulling to extract a build tree, we should probably try to pull the buildtree even if we usually don't.

abderrahim · 2026-06-30T08:57:41Z

+
+ try:
+ artifact = element._get_artifact()
+ virdir = artifact.get_buildroot() if buildroot else artifact.get_buildtree()


Another thing we need to be aware of (and also give a suggestion about to the user) is that not all elements have a build tree: only build elements do.
So it's not enough to check for the build root, we also need to check for the build tree. Then depending on the availability and the user request, we can suggest an alternative to the user.
buildtree available buildroot available user requested result
Y Y build tree extract build tree
Y Y build root extract build root
N Y build tree error + suggest extract build root
N Y build root extract build root
N N build tree error
N N build root error

Thanks for pointing this out. I've accounted for all of the cases you described and added tests covering them.
Could you please take another look when you have a chance?

abderrahim · 2026-06-30T08:58:48Z

+ project = str(datafiles)
+ element_name = "build-shell/buildtree.bst"
+ checkout = os.path.join(cli.directory, "checkout")
+ tar = os.path.join(cli.directory, "source-checkout.tar")


Suggested change
tar=os.path.join(cli.directory, "source-checkout.tar")
tar=os.path.join(cli.directory, "buildtree.tar")

abderrahim · 2026-06-30T09:03:53Z

+@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox")
+def test_buildtree_checkout_fail(cli, datafiles):
+ project = str(datafiles)
+ element_name = "build-shell/buildtree.bst"


We should also test the buildtree of a script element (using build-shell/script.bst). This is the corner case I mentioned above where we have a build root but not a build tree.

i've added a test case which uses build-shell/script.bst

Add a new `bst buildtree checkout` command for exporting buildtree stored in artifacts. By default, the command checks out the buildtree. The `--buildroot` option can be used to export the entire buildroot sandbox instead. Fixedapache#1822

ownik · 2026-06-01T13:35:42Z

This PR adds a "--use-buildtrees" option to "bst artifacts checkout", similar to the existing option in "bst shell".

The goal is to make it possible to check out build artifacts from the sandbox build tree when it is available.

Would be okay with adding this functionality, or are there reasons not to expose it through "bst artifacts checkout"? Or is there another way to get build artifacts from sandbox now?

ownik · 2026-06-08T09:17:05Z

@BenjaminSchubert @abderrahim @cs-shadow @gtristan @juergbi I've made changes to verify that buildtree exists in the cache, similar to the shell command. Could you please review, please?

abderrahim · 2026-06-08T10:02:02Z

Looks like this is fixing #1822, please link the issue in your PR. Also please add a test for this behaviour.

I'll try to review it today.

abderrahim · 2026-06-08T12:55:20Z

A few things need to be considered in the design of this feature:

Someone recently suggested to implement this as a flag to source checkout rather than artifact checkout. It kinda makes sense that the build tree is related to the source, but it's also stored as part of the artifact. Maybe it makes sense to move it to a separate toplevel command? something like bst buildtree checkout?
This is currently checking out the buildroot rather than what is traditionally considered a "build tree". This means it can work with both build elements and non-build-elements, but it also means we get a whole sandbox with the dependencies, the build tree, and even the artifact (if the build succeeded) which is probably not what people asking for the build tree are looking for.
if we go with a new toplevel command, we might want to expose options to checkout either the build tree or the build root.

ownik · 2026-06-09T12:46:49Z

@abderrahim Thanks for the feedback! I've reworked the implementation into a new bst buildtree checkout command. By default it checks out the buildtree, and a --buildroot option can be used to export the entire buildroot instead.

I've also added an integration test covering the new command and link the issue in PR.

ownik · 2026-06-17T12:33:37Z

@abderrahim Any chance to review this PR?

abderrahim

Thanks for the contributions. See my comments below.

abderrahim · 2026-06-30T08:47:27Z

+ # self.targets contains a list of the loaded target objects
+ # if we specify --deps build, Stream._load() will return a list
+ # of build dependency objects, however, we need to prepare a sandbox
+ # with the target (which has had its appropriate dependencies loaded)
+ element: Element = self.targets[0]


Does this make sense for the build tree? Either we're checking out just the buildtree, and then dependencies don't make sense, or we check out the whole buildroot and it already contains the build dependencies.
Am I missing something?

Sorry for the delayed response, and thanks for pointing this out.
Yes, you're right. I've changed this to element: Element = elements[0], is it ok?

abderrahim · 2026-06-30T08:50:31Z

+ # try to pull them if they are not already cached.
+ #
+ self.query_cache(elements)
+ self._pull_missing_artifacts(elements)


I think we only need to pull the artifact for the element in question. No need for dependencies here.
as an aside, I wonder if we should try to override the pull-buildtrees configuration in this case: if we're pulling to extract a build tree, we should probably try to pull the buildtree even if we usually don't.

abderrahim · 2026-06-30T08:57:41Z

+
+ try:
+ artifact = element._get_artifact()
+ virdir = artifact.get_buildroot() if buildroot else artifact.get_buildtree()


Another thing we need to be aware of (and also give a suggestion about to the user) is that not all elements have a build tree: only build elements do.
So it's not enough to check for the build root, we also need to check for the build tree. Then depending on the availability and the user request, we can suggest an alternative to the user.
buildtree available buildroot available user requested result
Y Y build tree extract build tree
Y Y build root extract build root
N Y build tree error + suggest extract build root
N Y build root extract build root
N N build tree error
N N build root error

Thanks for pointing this out. I've accounted for all of the cases you described and added tests covering them.
Could you please take another look when you have a chance?

abderrahim · 2026-06-30T08:58:48Z

+ project = str(datafiles)
+ element_name = "build-shell/buildtree.bst"
+ checkout = os.path.join(cli.directory, "checkout")
+ tar = os.path.join(cli.directory, "source-checkout.tar")


Suggested change
tar=os.path.join(cli.directory, "source-checkout.tar")
tar=os.path.join(cli.directory, "buildtree.tar")

abderrahim · 2026-06-30T09:03:53Z

+@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox")
+def test_buildtree_checkout_fail(cli, datafiles):
+ project = str(datafiles)
+ element_name = "build-shell/buildtree.bst"


We should also test the buildtree of a script element (using build-shell/script.bst). This is the corner case I mentioned above where we have a build root but not a build tree.

i've added a test case which uses build-shell/script.bst

Add a new `bst buildtree checkout` command for exporting buildtree stored in artifacts. By default, the command checks out the buildtree. The `--buildroot` option can be used to export the entire buildroot sandbox instead. Fixedapache#1822

ownik · 2026-06-01T13:35:42Z

This PR adds a "--use-buildtrees" option to "bst artifacts checkout", similar to the existing option in "bst shell".

The goal is to make it possible to check out build artifacts from the sandbox build tree when it is available.

Would be okay with adding this functionality, or are there reasons not to expose it through "bst artifacts checkout"? Or is there another way to get build artifacts from sandbox now?

ownik · 2026-06-08T09:17:05Z

@BenjaminSchubert @abderrahim @cs-shadow @gtristan @juergbi I've made changes to verify that buildtree exists in the cache, similar to the shell command. Could you please review, please?

abderrahim · 2026-06-08T10:02:02Z

Looks like this is fixing #1822, please link the issue in your PR. Also please add a test for this behaviour.

I'll try to review it today.

abderrahim · 2026-06-08T12:55:20Z

A few things need to be considered in the design of this feature:

Someone recently suggested to implement this as a flag to source checkout rather than artifact checkout. It kinda makes sense that the build tree is related to the source, but it's also stored as part of the artifact. Maybe it makes sense to move it to a separate toplevel command? something like bst buildtree checkout?
This is currently checking out the buildroot rather than what is traditionally considered a "build tree". This means it can work with both build elements and non-build-elements, but it also means we get a whole sandbox with the dependencies, the build tree, and even the artifact (if the build succeeded) which is probably not what people asking for the build tree are looking for.
if we go with a new toplevel command, we might want to expose options to checkout either the build tree or the build root.

ownik · 2026-06-09T12:46:49Z

@abderrahim Thanks for the feedback! I've reworked the implementation into a new bst buildtree checkout command. By default it checks out the buildtree, and a --buildroot option can be used to export the entire buildroot instead.

I've also added an integration test covering the new command and link the issue in PR.

ownik · 2026-06-17T12:33:37Z

@abderrahim Any chance to review this PR?

abderrahim

Thanks for the contributions. See my comments below.

abderrahim · 2026-06-30T08:47:27Z

+ # self.targets contains a list of the loaded target objects
+ # if we specify --deps build, Stream._load() will return a list
+ # of build dependency objects, however, we need to prepare a sandbox
+ # with the target (which has had its appropriate dependencies loaded)
+ element: Element = self.targets[0]


Does this make sense for the build tree? Either we're checking out just the buildtree, and then dependencies don't make sense, or we check out the whole buildroot and it already contains the build dependencies.
Am I missing something?

Sorry for the delayed response, and thanks for pointing this out.
Yes, you're right. I've changed this to element: Element = elements[0], is it ok?

abderrahim · 2026-06-30T08:50:31Z

+ # try to pull them if they are not already cached.
+ #
+ self.query_cache(elements)
+ self._pull_missing_artifacts(elements)


I think we only need to pull the artifact for the element in question. No need for dependencies here.
as an aside, I wonder if we should try to override the pull-buildtrees configuration in this case: if we're pulling to extract a build tree, we should probably try to pull the buildtree even if we usually don't.

abderrahim · 2026-06-30T08:57:41Z

+
+ try:
+ artifact = element._get_artifact()
+ virdir = artifact.get_buildroot() if buildroot else artifact.get_buildtree()


Another thing we need to be aware of (and also give a suggestion about to the user) is that not all elements have a build tree: only build elements do.
So it's not enough to check for the build root, we also need to check for the build tree. Then depending on the availability and the user request, we can suggest an alternative to the user.
buildtree available buildroot available user requested result
Y Y build tree extract build tree
Y Y build root extract build root
N Y build tree error + suggest extract build root
N Y build root extract build root
N N build tree error
N N build root error

Thanks for pointing this out. I've accounted for all of the cases you described and added tests covering them.
Could you please take another look when you have a chance?

abderrahim · 2026-06-30T08:58:48Z

+ project = str(datafiles)
+ element_name = "build-shell/buildtree.bst"
+ checkout = os.path.join(cli.directory, "checkout")
+ tar = os.path.join(cli.directory, "source-checkout.tar")


Suggested change
tar=os.path.join(cli.directory, "source-checkout.tar")
tar=os.path.join(cli.directory, "buildtree.tar")

abderrahim · 2026-06-30T09:03:53Z

+@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox")
+def test_buildtree_checkout_fail(cli, datafiles):
+ project = str(datafiles)
+ element_name = "build-shell/buildtree.bst"


We should also test the buildtree of a script element (using build-shell/script.bst). This is the corner case I mentioned above where we have a build root but not a build tree.

i've added a test case which uses build-shell/script.bst

Add a new `bst buildtree checkout` command for exporting buildtree stored in artifacts. By default, the command checks out the buildtree. The `--buildroot` option can be used to export the entire buildroot sandbox instead. Fixedapache#1822

ownik · 2026-06-01T13:35:42Z

This PR adds a "--use-buildtrees" option to "bst artifacts checkout", similar to the existing option in "bst shell".

The goal is to make it possible to check out build artifacts from the sandbox build tree when it is available.

Would be okay with adding this functionality, or are there reasons not to expose it through "bst artifacts checkout"? Or is there another way to get build artifacts from sandbox now?

ownik · 2026-06-08T09:17:05Z

@BenjaminSchubert @abderrahim @cs-shadow @gtristan @juergbi I've made changes to verify that buildtree exists in the cache, similar to the shell command. Could you please review, please?

abderrahim · 2026-06-08T10:02:02Z

Looks like this is fixing #1822, please link the issue in your PR. Also please add a test for this behaviour.

I'll try to review it today.

abderrahim · 2026-06-08T12:55:20Z

A few things need to be considered in the design of this feature:

Someone recently suggested to implement this as a flag to source checkout rather than artifact checkout. It kinda makes sense that the build tree is related to the source, but it's also stored as part of the artifact. Maybe it makes sense to move it to a separate toplevel command? something like bst buildtree checkout?
This is currently checking out the buildroot rather than what is traditionally considered a "build tree". This means it can work with both build elements and non-build-elements, but it also means we get a whole sandbox with the dependencies, the build tree, and even the artifact (if the build succeeded) which is probably not what people asking for the build tree are looking for.
if we go with a new toplevel command, we might want to expose options to checkout either the build tree or the build root.

ownik · 2026-06-09T12:46:49Z

@abderrahim Thanks for the feedback! I've reworked the implementation into a new bst buildtree checkout command. By default it checks out the buildtree, and a --buildroot option can be used to export the entire buildroot instead.

I've also added an integration test covering the new command and link the issue in PR.

ownik · 2026-06-17T12:33:37Z

@abderrahim Any chance to review this PR?

abderrahim

Thanks for the contributions. See my comments below.

abderrahim · 2026-06-30T08:47:27Z

+ # self.targets contains a list of the loaded target objects
+ # if we specify --deps build, Stream._load() will return a list
+ # of build dependency objects, however, we need to prepare a sandbox
+ # with the target (which has had its appropriate dependencies loaded)
+ element: Element = self.targets[0]


Does this make sense for the build tree? Either we're checking out just the buildtree, and then dependencies don't make sense, or we check out the whole buildroot and it already contains the build dependencies.
Am I missing something?

Sorry for the delayed response, and thanks for pointing this out.
Yes, you're right. I've changed this to element: Element = elements[0], is it ok?

abderrahim · 2026-06-30T08:50:31Z

+ # try to pull them if they are not already cached.
+ #
+ self.query_cache(elements)
+ self._pull_missing_artifacts(elements)


I think we only need to pull the artifact for the element in question. No need for dependencies here.
as an aside, I wonder if we should try to override the pull-buildtrees configuration in this case: if we're pulling to extract a build tree, we should probably try to pull the buildtree even if we usually don't.

abderrahim · 2026-06-30T08:57:41Z

+
+ try:
+ artifact = element._get_artifact()
+ virdir = artifact.get_buildroot() if buildroot else artifact.get_buildtree()


Another thing we need to be aware of (and also give a suggestion about to the user) is that not all elements have a build tree: only build elements do.
So it's not enough to check for the build root, we also need to check for the build tree. Then depending on the availability and the user request, we can suggest an alternative to the user.
buildtree available buildroot available user requested result
Y Y build tree extract build tree
Y Y build root extract build root
N Y build tree error + suggest extract build root
N Y build root extract build root
N N build tree error
N N build root error

Thanks for pointing this out. I've accounted for all of the cases you described and added tests covering them.
Could you please take another look when you have a chance?

abderrahim · 2026-06-30T08:58:48Z

+ project = str(datafiles)
+ element_name = "build-shell/buildtree.bst"
+ checkout = os.path.join(cli.directory, "checkout")
+ tar = os.path.join(cli.directory, "source-checkout.tar")


Suggested change
tar=os.path.join(cli.directory, "source-checkout.tar")
tar=os.path.join(cli.directory, "buildtree.tar")

abderrahim · 2026-06-30T09:03:53Z

+@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox")
+def test_buildtree_checkout_fail(cli, datafiles):
+ project = str(datafiles)
+ element_name = "build-shell/buildtree.bst"


We should also test the buildtree of a script element (using build-shell/script.bst). This is the corner case I mentioned above where we have a build root but not a build tree.

i've added a test case which uses build-shell/script.bst

Add a new `bst buildtree checkout` command for exporting buildtree stored in artifacts. By default, the command checks out the buildtree. The `--buildroot` option can be used to export the entire buildroot sandbox instead. Fixedapache#1822

ownik · 2026-06-01T13:35:42Z

This PR adds a "--use-buildtrees" option to "bst artifacts checkout", similar to the existing option in "bst shell".

The goal is to make it possible to check out build artifacts from the sandbox build tree when it is available.

Would be okay with adding this functionality, or are there reasons not to expose it through "bst artifacts checkout"? Or is there another way to get build artifacts from sandbox now?

ownik · 2026-06-08T09:17:05Z

@BenjaminSchubert @abderrahim @cs-shadow @gtristan @juergbi I've made changes to verify that buildtree exists in the cache, similar to the shell command. Could you please review, please?

abderrahim · 2026-06-08T10:02:02Z

Looks like this is fixing #1822, please link the issue in your PR. Also please add a test for this behaviour.

I'll try to review it today.

abderrahim · 2026-06-08T12:55:20Z

A few things need to be considered in the design of this feature:

Someone recently suggested to implement this as a flag to source checkout rather than artifact checkout. It kinda makes sense that the build tree is related to the source, but it's also stored as part of the artifact. Maybe it makes sense to move it to a separate toplevel command? something like bst buildtree checkout?
This is currently checking out the buildroot rather than what is traditionally considered a "build tree". This means it can work with both build elements and non-build-elements, but it also means we get a whole sandbox with the dependencies, the build tree, and even the artifact (if the build succeeded) which is probably not what people asking for the build tree are looking for.
if we go with a new toplevel command, we might want to expose options to checkout either the build tree or the build root.

ownik · 2026-06-09T12:46:49Z

@abderrahim Thanks for the feedback! I've reworked the implementation into a new bst buildtree checkout command. By default it checks out the buildtree, and a --buildroot option can be used to export the entire buildroot instead.

I've also added an integration test covering the new command and link the issue in PR.

ownik · 2026-06-17T12:33:37Z

@abderrahim Any chance to review this PR?

abderrahim

Thanks for the contributions. See my comments below.

abderrahim · 2026-06-30T08:47:27Z

+ # self.targets contains a list of the loaded target objects
+ # if we specify --deps build, Stream._load() will return a list
+ # of build dependency objects, however, we need to prepare a sandbox
+ # with the target (which has had its appropriate dependencies loaded)
+ element: Element = self.targets[0]


Does this make sense for the build tree? Either we're checking out just the buildtree, and then dependencies don't make sense, or we check out the whole buildroot and it already contains the build dependencies.
Am I missing something?

Sorry for the delayed response, and thanks for pointing this out.
Yes, you're right. I've changed this to element: Element = elements[0], is it ok?

abderrahim · 2026-06-30T08:50:31Z

+ # try to pull them if they are not already cached.
+ #
+ self.query_cache(elements)
+ self._pull_missing_artifacts(elements)


I think we only need to pull the artifact for the element in question. No need for dependencies here.
as an aside, I wonder if we should try to override the pull-buildtrees configuration in this case: if we're pulling to extract a build tree, we should probably try to pull the buildtree even if we usually don't.

abderrahim · 2026-06-30T08:57:41Z

+
+ try:
+ artifact = element._get_artifact()
+ virdir = artifact.get_buildroot() if buildroot else artifact.get_buildtree()


Another thing we need to be aware of (and also give a suggestion about to the user) is that not all elements have a build tree: only build elements do.
So it's not enough to check for the build root, we also need to check for the build tree. Then depending on the availability and the user request, we can suggest an alternative to the user.
buildtree available buildroot available user requested result
Y Y build tree extract build tree
Y Y build root extract build root
N Y build tree error + suggest extract build root
N Y build root extract build root
N N build tree error
N N build root error

Thanks for pointing this out. I've accounted for all of the cases you described and added tests covering them.
Could you please take another look when you have a chance?

abderrahim · 2026-06-30T08:58:48Z

+ project = str(datafiles)
+ element_name = "build-shell/buildtree.bst"
+ checkout = os.path.join(cli.directory, "checkout")
+ tar = os.path.join(cli.directory, "source-checkout.tar")


Suggested change
tar=os.path.join(cli.directory, "source-checkout.tar")
tar=os.path.join(cli.directory, "buildtree.tar")

abderrahim · 2026-06-30T09:03:53Z

+@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox")
+def test_buildtree_checkout_fail(cli, datafiles):
+ project = str(datafiles)
+ element_name = "build-shell/buildtree.bst"


We should also test the buildtree of a script element (using build-shell/script.bst). This is the corner case I mentioned above where we have a build root but not a build tree.

i've added a test case which uses build-shell/script.bst

Add a new `bst buildtree checkout` command for exporting buildtree stored in artifacts. By default, the command checks out the buildtree. The `--buildroot` option can be used to export the entire buildroot sandbox instead. Fixedapache#1822

ownik · 2026-06-01T13:35:42Z

This PR adds a "--use-buildtrees" option to "bst artifacts checkout", similar to the existing option in "bst shell".

The goal is to make it possible to check out build artifacts from the sandbox build tree when it is available.

Would be okay with adding this functionality, or are there reasons not to expose it through "bst artifacts checkout"? Or is there another way to get build artifacts from sandbox now?

ownik · 2026-06-08T09:17:05Z

@BenjaminSchubert @abderrahim @cs-shadow @gtristan @juergbi I've made changes to verify that buildtree exists in the cache, similar to the shell command. Could you please review, please?

abderrahim · 2026-06-08T10:02:02Z

Looks like this is fixing #1822, please link the issue in your PR. Also please add a test for this behaviour.

I'll try to review it today.

abderrahim · 2026-06-08T12:55:20Z

A few things need to be considered in the design of this feature:

Someone recently suggested to implement this as a flag to source checkout rather than artifact checkout. It kinda makes sense that the build tree is related to the source, but it's also stored as part of the artifact. Maybe it makes sense to move it to a separate toplevel command? something like bst buildtree checkout?
This is currently checking out the buildroot rather than what is traditionally considered a "build tree". This means it can work with both build elements and non-build-elements, but it also means we get a whole sandbox with the dependencies, the build tree, and even the artifact (if the build succeeded) which is probably not what people asking for the build tree are looking for.
if we go with a new toplevel command, we might want to expose options to checkout either the build tree or the build root.

ownik · 2026-06-09T12:46:49Z

@abderrahim Thanks for the feedback! I've reworked the implementation into a new bst buildtree checkout command. By default it checks out the buildtree, and a --buildroot option can be used to export the entire buildroot instead.

I've also added an integration test covering the new command and link the issue in PR.

ownik · 2026-06-17T12:33:37Z

@abderrahim Any chance to review this PR?

abderrahim

Thanks for the contributions. See my comments below.

abderrahim · 2026-06-30T08:47:27Z

+ # self.targets contains a list of the loaded target objects
+ # if we specify --deps build, Stream._load() will return a list
+ # of build dependency objects, however, we need to prepare a sandbox
+ # with the target (which has had its appropriate dependencies loaded)
+ element: Element = self.targets[0]


Does this make sense for the build tree? Either we're checking out just the buildtree, and then dependencies don't make sense, or we check out the whole buildroot and it already contains the build dependencies.
Am I missing something?

Sorry for the delayed response, and thanks for pointing this out.
Yes, you're right. I've changed this to element: Element = elements[0], is it ok?

abderrahim · 2026-06-30T08:50:31Z

+ # try to pull them if they are not already cached.
+ #
+ self.query_cache(elements)
+ self._pull_missing_artifacts(elements)


I think we only need to pull the artifact for the element in question. No need for dependencies here.
as an aside, I wonder if we should try to override the pull-buildtrees configuration in this case: if we're pulling to extract a build tree, we should probably try to pull the buildtree even if we usually don't.

abderrahim · 2026-06-30T08:57:41Z

+
+ try:
+ artifact = element._get_artifact()
+ virdir = artifact.get_buildroot() if buildroot else artifact.get_buildtree()


Another thing we need to be aware of (and also give a suggestion about to the user) is that not all elements have a build tree: only build elements do.
So it's not enough to check for the build root, we also need to check for the build tree. Then depending on the availability and the user request, we can suggest an alternative to the user.
buildtree available buildroot available user requested result
Y Y build tree extract build tree
Y Y build root extract build root
N Y build tree error + suggest extract build root
N Y build root extract build root
N N build tree error
N N build root error

Thanks for pointing this out. I've accounted for all of the cases you described and added tests covering them.
Could you please take another look when you have a chance?

abderrahim · 2026-06-30T08:58:48Z

+ project = str(datafiles)
+ element_name = "build-shell/buildtree.bst"
+ checkout = os.path.join(cli.directory, "checkout")
+ tar = os.path.join(cli.directory, "source-checkout.tar")


Suggested change
tar=os.path.join(cli.directory, "source-checkout.tar")
tar=os.path.join(cli.directory, "buildtree.tar")

abderrahim · 2026-06-30T09:03:53Z

+@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox")
+def test_buildtree_checkout_fail(cli, datafiles):
+ project = str(datafiles)
+ element_name = "build-shell/buildtree.bst"


We should also test the buildtree of a script element (using build-shell/script.bst). This is the corner case I mentioned above where we have a build root but not a build tree.

i've added a test case which uses build-shell/script.bst

Add a new `bst buildtree checkout` command for exporting buildtree stored in artifacts. By default, the command checks out the buildtree. The `--buildroot` option can be used to export the entire buildroot sandbox instead. Fixedapache#1822

ownik · 2026-06-01T13:35:42Z

This PR adds a "--use-buildtrees" option to "bst artifacts checkout", similar to the existing option in "bst shell".

The goal is to make it possible to check out build artifacts from the sandbox build tree when it is available.

Would be okay with adding this functionality, or are there reasons not to expose it through "bst artifacts checkout"? Or is there another way to get build artifacts from sandbox now?

ownik · 2026-06-08T09:17:05Z

@BenjaminSchubert @abderrahim @cs-shadow @gtristan @juergbi I've made changes to verify that buildtree exists in the cache, similar to the shell command. Could you please review, please?

abderrahim · 2026-06-08T10:02:02Z

Looks like this is fixing #1822, please link the issue in your PR. Also please add a test for this behaviour.

I'll try to review it today.

abderrahim · 2026-06-08T12:55:20Z

A few things need to be considered in the design of this feature:

Someone recently suggested to implement this as a flag to source checkout rather than artifact checkout. It kinda makes sense that the build tree is related to the source, but it's also stored as part of the artifact. Maybe it makes sense to move it to a separate toplevel command? something like bst buildtree checkout?
This is currently checking out the buildroot rather than what is traditionally considered a "build tree". This means it can work with both build elements and non-build-elements, but it also means we get a whole sandbox with the dependencies, the build tree, and even the artifact (if the build succeeded) which is probably not what people asking for the build tree are looking for.
if we go with a new toplevel command, we might want to expose options to checkout either the build tree or the build root.

ownik · 2026-06-09T12:46:49Z

@abderrahim Thanks for the feedback! I've reworked the implementation into a new bst buildtree checkout command. By default it checks out the buildtree, and a --buildroot option can be used to export the entire buildroot instead.

I've also added an integration test covering the new command and link the issue in PR.

ownik · 2026-06-17T12:33:37Z

@abderrahim Any chance to review this PR?

abderrahim

Thanks for the contributions. See my comments below.

abderrahim · 2026-06-30T08:47:27Z

+ # self.targets contains a list of the loaded target objects
+ # if we specify --deps build, Stream._load() will return a list
+ # of build dependency objects, however, we need to prepare a sandbox
+ # with the target (which has had its appropriate dependencies loaded)
+ element: Element = self.targets[0]


Does this make sense for the build tree? Either we're checking out just the buildtree, and then dependencies don't make sense, or we check out the whole buildroot and it already contains the build dependencies.
Am I missing something?

Sorry for the delayed response, and thanks for pointing this out.
Yes, you're right. I've changed this to element: Element = elements[0], is it ok?

abderrahim · 2026-06-30T08:50:31Z

+ # try to pull them if they are not already cached.
+ #
+ self.query_cache(elements)
+ self._pull_missing_artifacts(elements)


I think we only need to pull the artifact for the element in question. No need for dependencies here.
as an aside, I wonder if we should try to override the pull-buildtrees configuration in this case: if we're pulling to extract a build tree, we should probably try to pull the buildtree even if we usually don't.

abderrahim · 2026-06-30T08:57:41Z

+
+ try:
+ artifact = element._get_artifact()
+ virdir = artifact.get_buildroot() if buildroot else artifact.get_buildtree()


Another thing we need to be aware of (and also give a suggestion about to the user) is that not all elements have a build tree: only build elements do.
So it's not enough to check for the build root, we also need to check for the build tree. Then depending on the availability and the user request, we can suggest an alternative to the user.
buildtree available buildroot available user requested result
Y Y build tree extract build tree
Y Y build root extract build root
N Y build tree error + suggest extract build root
N Y build root extract build root
N N build tree error
N N build root error

Thanks for pointing this out. I've accounted for all of the cases you described and added tests covering them.
Could you please take another look when you have a chance?

abderrahim · 2026-06-30T08:58:48Z

+ project = str(datafiles)
+ element_name = "build-shell/buildtree.bst"
+ checkout = os.path.join(cli.directory, "checkout")
+ tar = os.path.join(cli.directory, "source-checkout.tar")


Suggested change
tar=os.path.join(cli.directory, "source-checkout.tar")
tar=os.path.join(cli.directory, "buildtree.tar")

abderrahim · 2026-06-30T09:03:53Z

+@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox")
+def test_buildtree_checkout_fail(cli, datafiles):
+ project = str(datafiles)
+ element_name = "build-shell/buildtree.bst"


We should also test the buildtree of a script element (using build-shell/script.bst). This is the corner case I mentioned above where we have a build root but not a build tree.

i've added a test case which uses build-shell/script.bst

Add a new `bst buildtree checkout` command for exporting buildtree stored in artifacts. By default, the command checks out the buildtree. The `--buildroot` option can be used to export the entire buildroot sandbox instead. Fixedapache#1822

ownik requested review from BenjaminSchubert, abderrahim, cs-shadow, gtristan and juergbi as code owners June 1, 2026 13:35

ownik force-pushed the master branch 3 times, most recently from 948c3f3 to cb60d5bCompare June 5, 2026 11:56

ownik force-pushed the master branch 2 times, most recently from 79fb228 to fa9662aCompare June 9, 2026 12:35

ownik changed the title ~~cli: add --use-buildtrees option for artifacts checkout~~cli: add buildtree checkout commandJun 9, 2026

abderrahim reviewed Jun 30, 2026
View reviewed changes

ownik force-pushed the master branch from fa9662a to f2fb37cCompare August 10, 2026 07:10

cli: add buildtree checkout command
a634866
Add a new `bst buildtree checkout` command for exporting buildtree stored in artifacts. By default, the command checks out the buildtree. The `--buildroot` option can be used to export the entire buildroot sandbox instead. Fixedapache#1822

ownik force-pushed the master branch from f2fb37c to a634866Compare August 10, 2026 07:15

ownik requested review from BenjaminSchubert, abderrahim, cs-shadow, gtristan and juergbi as code owners June 1, 2026 13:35

ownik force-pushed the master branch 3 times, most recently from 948c3f3 to cb60d5bCompare June 5, 2026 11:56

ownik force-pushed the master branch 2 times, most recently from 79fb228 to fa9662aCompare June 9, 2026 12:35

ownik changed the title ~~cli: add --use-buildtrees option for artifacts checkout~~cli: add buildtree checkout commandJun 9, 2026

abderrahim reviewed Jun 30, 2026
View reviewed changes

ownik force-pushed the master branch from fa9662a to f2fb37cCompare August 10, 2026 07:10

cli: add buildtree checkout command
a634866
Add a new `bst buildtree checkout` command for exporting buildtree stored in artifacts. By default, the command checks out the buildtree. The `--buildroot` option can be used to export the entire buildroot sandbox instead. Fixedapache#1822

ownik force-pushed the master branch from f2fb37c to a634866Compare August 10, 2026 07:15

ownik requested review from BenjaminSchubert, abderrahim, cs-shadow, gtristan and juergbi as code owners June 1, 2026 13:35

ownik force-pushed the master branch 3 times, most recently from 948c3f3 to cb60d5bCompare June 5, 2026 11:56

ownik force-pushed the master branch 2 times, most recently from 79fb228 to fa9662aCompare June 9, 2026 12:35

ownik changed the title ~~cli: add --use-buildtrees option for artifacts checkout~~cli: add buildtree checkout commandJun 9, 2026

abderrahim reviewed Jun 30, 2026
View reviewed changes

ownik force-pushed the master branch from fa9662a to f2fb37cCompare August 10, 2026 07:10

cli: add buildtree checkout command
a634866
Add a new `bst buildtree checkout` command for exporting buildtree stored in artifacts. By default, the command checks out the buildtree. The `--buildroot` option can be used to export the entire buildroot sandbox instead. Fixedapache#1822

ownik force-pushed the master branch from f2fb37c to a634866Compare August 10, 2026 07:15

ownik requested review from BenjaminSchubert, abderrahim, cs-shadow, gtristan and juergbi as code owners June 1, 2026 13:35

ownik force-pushed the master branch 3 times, most recently from 948c3f3 to cb60d5bCompare June 5, 2026 11:56

ownik force-pushed the master branch 2 times, most recently from 79fb228 to fa9662aCompare June 9, 2026 12:35

ownik changed the title ~~cli: add --use-buildtrees option for artifacts checkout~~cli: add buildtree checkout commandJun 9, 2026

abderrahim reviewed Jun 30, 2026
View reviewed changes

ownik force-pushed the master branch from fa9662a to f2fb37cCompare August 10, 2026 07:10

cli: add buildtree checkout command
a634866
Add a new `bst buildtree checkout` command for exporting buildtree stored in artifacts. By default, the command checks out the buildtree. The `--buildroot` option can be used to export the entire buildroot sandbox instead. Fixedapache#1822

ownik force-pushed the master branch from f2fb37c to a634866Compare August 10, 2026 07:15

ownik requested review from BenjaminSchubert, abderrahim, cs-shadow, gtristan and juergbi as code owners June 1, 2026 13:35

ownik force-pushed the master branch 3 times, most recently from 948c3f3 to cb60d5bCompare June 5, 2026 11:56

ownik force-pushed the master branch 2 times, most recently from 79fb228 to fa9662aCompare June 9, 2026 12:35

ownik changed the title ~~cli: add --use-buildtrees option for artifacts checkout~~cli: add buildtree checkout commandJun 9, 2026

abderrahim reviewed Jun 30, 2026
View reviewed changes

ownik force-pushed the master branch from fa9662a to f2fb37cCompare August 10, 2026 07:10

cli: add buildtree checkout command
a634866
Add a new `bst buildtree checkout` command for exporting buildtree stored in artifacts. By default, the command checks out the buildtree. The `--buildroot` option can be used to export the entire buildroot sandbox instead. Fixedapache#1822

ownik force-pushed the master branch from f2fb37c to a634866Compare August 10, 2026 07:15

ownik requested review from BenjaminSchubert, abderrahim, cs-shadow, gtristan and juergbi as code owners June 1, 2026 13:35

ownik force-pushed the master branch 3 times, most recently from 948c3f3 to cb60d5bCompare June 5, 2026 11:56

ownik force-pushed the master branch 2 times, most recently from 79fb228 to fa9662aCompare June 9, 2026 12:35

ownik changed the title ~~cli: add --use-buildtrees option for artifacts checkout~~cli: add buildtree checkout commandJun 9, 2026

abderrahim reviewed Jun 30, 2026
View reviewed changes

ownik force-pushed the master branch from fa9662a to f2fb37cCompare August 10, 2026 07:10

cli: add buildtree checkout command
a634866
Add a new `bst buildtree checkout` command for exporting buildtree stored in artifacts. By default, the command checks out the buildtree. The `--buildroot` option can be used to export the entire buildroot sandbox instead. Fixedapache#1822

ownik force-pushed the master branch from f2fb37c to a634866Compare August 10, 2026 07:15

ownik requested review from BenjaminSchubert, abderrahim, cs-shadow, gtristan and juergbi as code owners June 1, 2026 13:35

ownik force-pushed the master branch 3 times, most recently from 948c3f3 to cb60d5bCompare June 5, 2026 11:56

ownik force-pushed the master branch 2 times, most recently from 79fb228 to fa9662aCompare June 9, 2026 12:35

ownik changed the title ~~cli: add --use-buildtrees option for artifacts checkout~~cli: add buildtree checkout commandJun 9, 2026

abderrahim reviewed Jun 30, 2026
View reviewed changes

ownik force-pushed the master branch from fa9662a to f2fb37cCompare August 10, 2026 07:10

cli: add buildtree checkout command
a634866
Add a new `bst buildtree checkout` command for exporting buildtree stored in artifacts. By default, the command checks out the buildtree. The `--buildroot` option can be used to export the entire buildroot sandbox instead. Fixedapache#1822

ownik force-pushed the master branch from f2fb37c to a634866Compare August 10, 2026 07:15

ownik requested review from BenjaminSchubert, abderrahim, cs-shadow, gtristan and juergbi as code owners June 1, 2026 13:35

ownik force-pushed the master branch 3 times, most recently from 948c3f3 to cb60d5bCompare June 5, 2026 11:56

ownik force-pushed the master branch 2 times, most recently from 79fb228 to fa9662aCompare June 9, 2026 12:35

ownik changed the title ~~cli: add --use-buildtrees option for artifacts checkout~~cli: add buildtree checkout commandJun 9, 2026

abderrahim reviewed Jun 30, 2026
View reviewed changes

ownik force-pushed the master branch from fa9662a to f2fb37cCompare August 10, 2026 07:10

cli: add buildtree checkout command
a634866
Add a new `bst buildtree checkout` command for exporting buildtree stored in artifacts. By default, the command checks out the buildtree. The `--buildroot` option can be used to export the entire buildroot sandbox instead. Fixedapache#1822

ownik force-pushed the master branch from f2fb37c to a634866Compare August 10, 2026 07:15

buildtree available	buildroot available	user requested	result
Y	Y	build tree	extract build tree
Y	Y	build root	extract build root
N	Y	build tree	error + suggest extract build root
N	Y	build root	extract build root
N	N	build tree	error
N	N	build root	error

	tar=os.path.join(cli.directory, "source-checkout.tar")
	tar=os.path.join(cli.directory, "buildtree.tar")

Uh oh!

Conversation

ownik commented Jun 1, 2026

Uh oh!

ownik commented Jun 8, 2026

Uh oh!

abderrahim commented Jun 8, 2026

Uh oh!

abderrahim commented Jun 8, 2026

Uh oh!

ownik commented Jun 9, 2026

Uh oh!

ownik commented Jun 17, 2026

Uh oh!

abderrahim left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ownik commented Jun 1, 2026

Uh oh!

ownik commented Jun 8, 2026

Uh oh!

abderrahim commented Jun 8, 2026

Uh oh!

abderrahim commented Jun 8, 2026

Uh oh!

ownik commented Jun 9, 2026

Uh oh!

ownik commented Jun 17, 2026

Uh oh!

abderrahim left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!