Add linux/arm64 platforms to image builds

[timower]

This should allow the toolchain to be easily used on apple silicon macs (or any other arm64 device).

Closes#28

[Eeems]

My understanding is that if you use buildx, DOCKER_BUILDKIT=1 is no longer needed

[timower]

I'll try something like https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners, otherwise I guess this isn't feasible

[Eeems]

To avoid timeouts due to how long it takes, probably. It might also make sense to split out the images into their own jobs as well. I've done that for the images I have generating nightly: https://github.com/Eeems/images/blob/main/.github/workflows/build.yml

[trixmoe]

My optimism in regards to M1 runners was short-sighted: GitHub (and simply Apple) does not support (nested) virtualization on those runners, which results in the VM not being able to boot up. There's thus no way to really improve the earlier amd64 Linux runner solution('s run time).

I found out about this through the issue I opened on the colima repo, and the announcement PR on actions/runner-images, which pointed at the GitHub docs:

(...) nested-virtualization is not supported by arm64 runners

Tl;dr of this whole ordeal

Stick with amd64 Linux runners (for the foreseeable future).
GH doesn't seem to be currently working on Linux arm64 runners (actions/runner-images#5631).

[jandubois]

GitHub (and simply Apple) does not support (nested) virtualization

Note that while GitHub claims it to be a limitation of Apple's Virtualization Framework, it really is a limitation of the M1 chip architecture; nested virtualization is supported by Apple's Virtualization Framework on M2 and M3 chips, which are based on a newer ARM spec.

[trixmoe]

To avoid timeouts due to how long it takes, probably. It might also make sense to split out the images into their own jobs as well. I've done that for the images I have generating nightly: https://github.com/Eeems/images/blob/main/.github/workflows/build.yml

I've looked at the dependencies between the images, and I've realized that one could have 3 different stages, with each image having it's own separate job. Using different jobs also allows it to go over the 6 hour limit.

Stage 1 would be the initial one, Stage 2 starts once toolchain is done, Stage 3 starts once base is done. Each of the blocks are separate jobs.

One would have to look into the exporting of images, to transfer them between jobs, for builds that are not published.
One could also separate amd64 and arm64 builds, but I'm unsure if this is doable when publishing to the image repository.

[Eeems]

I would recommend moving dotnet6 to stage 3 as well, make that dynamic, and have toolchain and base be static steps.

[trixmoe]

I looked more into multi-platform Docker images, as I had some time.

Separating the building of the images per platform is possible (exporting/importing single-platform images is no issue), but combining the images at the end is slightly hacky, as Docker does not support direct publishing of multi-platform "images" (manifest), but rather forces the individual images to be published already before creating the multi-platform manifest. (docker/cli#3350)

To get around this, one can publish the single-platform images onto a local registry ran through GHA services. Once they are published, create the manifest, and push it locally. Finally, it's possible to copy the multi-platform "image" to the public registry. (Similarly to rapidsai/gha-tools#51)

[timower]

This should allow the toolchain to be easily used on apple silicon macs (or any other arm64 device).

Closes#28

[Eeems]

My understanding is that if you use buildx, DOCKER_BUILDKIT=1 is no longer needed

[timower]

I'll try something like https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners, otherwise I guess this isn't feasible

[Eeems]

To avoid timeouts due to how long it takes, probably. It might also make sense to split out the images into their own jobs as well. I've done that for the images I have generating nightly: https://github.com/Eeems/images/blob/main/.github/workflows/build.yml

[trixmoe]

My optimism in regards to M1 runners was short-sighted: GitHub (and simply Apple) does not support (nested) virtualization on those runners, which results in the VM not being able to boot up. There's thus no way to really improve the earlier amd64 Linux runner solution('s run time).

I found out about this through the issue I opened on the colima repo, and the announcement PR on actions/runner-images, which pointed at the GitHub docs:

(...) nested-virtualization is not supported by arm64 runners

Tl;dr of this whole ordeal

Stick with amd64 Linux runners (for the foreseeable future).
GH doesn't seem to be currently working on Linux arm64 runners (actions/runner-images#5631).

[jandubois]

GitHub (and simply Apple) does not support (nested) virtualization

Note that while GitHub claims it to be a limitation of Apple's Virtualization Framework, it really is a limitation of the M1 chip architecture; nested virtualization is supported by Apple's Virtualization Framework on M2 and M3 chips, which are based on a newer ARM spec.

[trixmoe]

To avoid timeouts due to how long it takes, probably. It might also make sense to split out the images into their own jobs as well. I've done that for the images I have generating nightly: https://github.com/Eeems/images/blob/main/.github/workflows/build.yml

I've looked at the dependencies between the images, and I've realized that one could have 3 different stages, with each image having it's own separate job. Using different jobs also allows it to go over the 6 hour limit.

Stage 1 would be the initial one, Stage 2 starts once toolchain is done, Stage 3 starts once base is done. Each of the blocks are separate jobs.

One would have to look into the exporting of images, to transfer them between jobs, for builds that are not published.
One could also separate amd64 and arm64 builds, but I'm unsure if this is doable when publishing to the image repository.

[Eeems]

I would recommend moving dotnet6 to stage 3 as well, make that dynamic, and have toolchain and base be static steps.

[trixmoe]

I looked more into multi-platform Docker images, as I had some time.

Separating the building of the images per platform is possible (exporting/importing single-platform images is no issue), but combining the images at the end is slightly hacky, as Docker does not support direct publishing of multi-platform "images" (manifest), but rather forces the individual images to be published already before creating the multi-platform manifest. (docker/cli#3350)

To get around this, one can publish the single-platform images onto a local registry ran through GHA services. Once they are published, create the manifest, and push it locally. Finally, it's possible to copy the multi-platform "image" to the public registry. (Similarly to rapidsai/gha-tools#51)

[timower]

This should allow the toolchain to be easily used on apple silicon macs (or any other arm64 device).

Closes#28

[Eeems]

My understanding is that if you use buildx, DOCKER_BUILDKIT=1 is no longer needed

[timower]

I'll try something like https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners, otherwise I guess this isn't feasible

[Eeems]

To avoid timeouts due to how long it takes, probably. It might also make sense to split out the images into their own jobs as well. I've done that for the images I have generating nightly: https://github.com/Eeems/images/blob/main/.github/workflows/build.yml

[trixmoe]

My optimism in regards to M1 runners was short-sighted: GitHub (and simply Apple) does not support (nested) virtualization on those runners, which results in the VM not being able to boot up. There's thus no way to really improve the earlier amd64 Linux runner solution('s run time).

I found out about this through the issue I opened on the colima repo, and the announcement PR on actions/runner-images, which pointed at the GitHub docs:

(...) nested-virtualization is not supported by arm64 runners

Tl;dr of this whole ordeal

Stick with amd64 Linux runners (for the foreseeable future).
GH doesn't seem to be currently working on Linux arm64 runners (actions/runner-images#5631).

[jandubois]

GitHub (and simply Apple) does not support (nested) virtualization

Note that while GitHub claims it to be a limitation of Apple's Virtualization Framework, it really is a limitation of the M1 chip architecture; nested virtualization is supported by Apple's Virtualization Framework on M2 and M3 chips, which are based on a newer ARM spec.

[trixmoe]

To avoid timeouts due to how long it takes, probably. It might also make sense to split out the images into their own jobs as well. I've done that for the images I have generating nightly: https://github.com/Eeems/images/blob/main/.github/workflows/build.yml

I've looked at the dependencies between the images, and I've realized that one could have 3 different stages, with each image having it's own separate job. Using different jobs also allows it to go over the 6 hour limit.

Stage 1 would be the initial one, Stage 2 starts once toolchain is done, Stage 3 starts once base is done. Each of the blocks are separate jobs.

One would have to look into the exporting of images, to transfer them between jobs, for builds that are not published.
One could also separate amd64 and arm64 builds, but I'm unsure if this is doable when publishing to the image repository.

[Eeems]

I would recommend moving dotnet6 to stage 3 as well, make that dynamic, and have toolchain and base be static steps.

[trixmoe]

I looked more into multi-platform Docker images, as I had some time.

Separating the building of the images per platform is possible (exporting/importing single-platform images is no issue), but combining the images at the end is slightly hacky, as Docker does not support direct publishing of multi-platform "images" (manifest), but rather forces the individual images to be published already before creating the multi-platform manifest. (docker/cli#3350)

To get around this, one can publish the single-platform images onto a local registry ran through GHA services. Once they are published, create the manifest, and push it locally. Finally, it's possible to copy the multi-platform "image" to the public registry. (Similarly to rapidsai/gha-tools#51)

[timower]

This should allow the toolchain to be easily used on apple silicon macs (or any other arm64 device).

Closes#28

[Eeems]

My understanding is that if you use buildx, DOCKER_BUILDKIT=1 is no longer needed

[timower]

I'll try something like https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners, otherwise I guess this isn't feasible

[Eeems]

To avoid timeouts due to how long it takes, probably. It might also make sense to split out the images into their own jobs as well. I've done that for the images I have generating nightly: https://github.com/Eeems/images/blob/main/.github/workflows/build.yml

[trixmoe]

My optimism in regards to M1 runners was short-sighted: GitHub (and simply Apple) does not support (nested) virtualization on those runners, which results in the VM not being able to boot up. There's thus no way to really improve the earlier amd64 Linux runner solution('s run time).

I found out about this through the issue I opened on the colima repo, and the announcement PR on actions/runner-images, which pointed at the GitHub docs:

(...) nested-virtualization is not supported by arm64 runners

Tl;dr of this whole ordeal

Stick with amd64 Linux runners (for the foreseeable future).
GH doesn't seem to be currently working on Linux arm64 runners (actions/runner-images#5631).

[jandubois]

GitHub (and simply Apple) does not support (nested) virtualization

Note that while GitHub claims it to be a limitation of Apple's Virtualization Framework, it really is a limitation of the M1 chip architecture; nested virtualization is supported by Apple's Virtualization Framework on M2 and M3 chips, which are based on a newer ARM spec.

[trixmoe]

To avoid timeouts due to how long it takes, probably. It might also make sense to split out the images into their own jobs as well. I've done that for the images I have generating nightly: https://github.com/Eeems/images/blob/main/.github/workflows/build.yml

I've looked at the dependencies between the images, and I've realized that one could have 3 different stages, with each image having it's own separate job. Using different jobs also allows it to go over the 6 hour limit.

Stage 1 would be the initial one, Stage 2 starts once toolchain is done, Stage 3 starts once base is done. Each of the blocks are separate jobs.

One would have to look into the exporting of images, to transfer them between jobs, for builds that are not published.
One could also separate amd64 and arm64 builds, but I'm unsure if this is doable when publishing to the image repository.

[Eeems]

I would recommend moving dotnet6 to stage 3 as well, make that dynamic, and have toolchain and base be static steps.

[trixmoe]

I looked more into multi-platform Docker images, as I had some time.

Separating the building of the images per platform is possible (exporting/importing single-platform images is no issue), but combining the images at the end is slightly hacky, as Docker does not support direct publishing of multi-platform "images" (manifest), but rather forces the individual images to be published already before creating the multi-platform manifest. (docker/cli#3350)

To get around this, one can publish the single-platform images onto a local registry ran through GHA services. Once they are published, create the manifest, and push it locally. Finally, it's possible to copy the multi-platform "image" to the public registry. (Similarly to rapidsai/gha-tools#51)

[timower]

This should allow the toolchain to be easily used on apple silicon macs (or any other arm64 device).

Closes#28

[Eeems]

My understanding is that if you use buildx, DOCKER_BUILDKIT=1 is no longer needed

[timower]

I'll try something like https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners, otherwise I guess this isn't feasible

[Eeems]

To avoid timeouts due to how long it takes, probably. It might also make sense to split out the images into their own jobs as well. I've done that for the images I have generating nightly: https://github.com/Eeems/images/blob/main/.github/workflows/build.yml

[trixmoe]

My optimism in regards to M1 runners was short-sighted: GitHub (and simply Apple) does not support (nested) virtualization on those runners, which results in the VM not being able to boot up. There's thus no way to really improve the earlier amd64 Linux runner solution('s run time).

I found out about this through the issue I opened on the colima repo, and the announcement PR on actions/runner-images, which pointed at the GitHub docs:

(...) nested-virtualization is not supported by arm64 runners

Tl;dr of this whole ordeal

Stick with amd64 Linux runners (for the foreseeable future).
GH doesn't seem to be currently working on Linux arm64 runners (actions/runner-images#5631).

[jandubois]

GitHub (and simply Apple) does not support (nested) virtualization

Note that while GitHub claims it to be a limitation of Apple's Virtualization Framework, it really is a limitation of the M1 chip architecture; nested virtualization is supported by Apple's Virtualization Framework on M2 and M3 chips, which are based on a newer ARM spec.

[trixmoe]

To avoid timeouts due to how long it takes, probably. It might also make sense to split out the images into their own jobs as well. I've done that for the images I have generating nightly: https://github.com/Eeems/images/blob/main/.github/workflows/build.yml

I've looked at the dependencies between the images, and I've realized that one could have 3 different stages, with each image having it's own separate job. Using different jobs also allows it to go over the 6 hour limit.

Stage 1 would be the initial one, Stage 2 starts once toolchain is done, Stage 3 starts once base is done. Each of the blocks are separate jobs.

One would have to look into the exporting of images, to transfer them between jobs, for builds that are not published.
One could also separate amd64 and arm64 builds, but I'm unsure if this is doable when publishing to the image repository.

[Eeems]

I would recommend moving dotnet6 to stage 3 as well, make that dynamic, and have toolchain and base be static steps.

[trixmoe]

I looked more into multi-platform Docker images, as I had some time.

Separating the building of the images per platform is possible (exporting/importing single-platform images is no issue), but combining the images at the end is slightly hacky, as Docker does not support direct publishing of multi-platform "images" (manifest), but rather forces the individual images to be published already before creating the multi-platform manifest. (docker/cli#3350)

To get around this, one can publish the single-platform images onto a local registry ran through GHA services. Once they are published, create the manifest, and push it locally. Finally, it's possible to copy the multi-platform "image" to the public registry. (Similarly to rapidsai/gha-tools#51)

[timower]

This should allow the toolchain to be easily used on apple silicon macs (or any other arm64 device).

Closes#28

[Eeems]

My understanding is that if you use buildx, DOCKER_BUILDKIT=1 is no longer needed

[timower]

I'll try something like https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners, otherwise I guess this isn't feasible

[Eeems]

To avoid timeouts due to how long it takes, probably. It might also make sense to split out the images into their own jobs as well. I've done that for the images I have generating nightly: https://github.com/Eeems/images/blob/main/.github/workflows/build.yml

[trixmoe]

My optimism in regards to M1 runners was short-sighted: GitHub (and simply Apple) does not support (nested) virtualization on those runners, which results in the VM not being able to boot up. There's thus no way to really improve the earlier amd64 Linux runner solution('s run time).

I found out about this through the issue I opened on the colima repo, and the announcement PR on actions/runner-images, which pointed at the GitHub docs:

(...) nested-virtualization is not supported by arm64 runners

Tl;dr of this whole ordeal

Stick with amd64 Linux runners (for the foreseeable future).
GH doesn't seem to be currently working on Linux arm64 runners (actions/runner-images#5631).

[jandubois]

GitHub (and simply Apple) does not support (nested) virtualization

Note that while GitHub claims it to be a limitation of Apple's Virtualization Framework, it really is a limitation of the M1 chip architecture; nested virtualization is supported by Apple's Virtualization Framework on M2 and M3 chips, which are based on a newer ARM spec.

[trixmoe]

To avoid timeouts due to how long it takes, probably. It might also make sense to split out the images into their own jobs as well. I've done that for the images I have generating nightly: https://github.com/Eeems/images/blob/main/.github/workflows/build.yml

I've looked at the dependencies between the images, and I've realized that one could have 3 different stages, with each image having it's own separate job. Using different jobs also allows it to go over the 6 hour limit.

Stage 1 would be the initial one, Stage 2 starts once toolchain is done, Stage 3 starts once base is done. Each of the blocks are separate jobs.

One would have to look into the exporting of images, to transfer them between jobs, for builds that are not published.
One could also separate amd64 and arm64 builds, but I'm unsure if this is doable when publishing to the image repository.

[Eeems]

I would recommend moving dotnet6 to stage 3 as well, make that dynamic, and have toolchain and base be static steps.

[trixmoe]

I looked more into multi-platform Docker images, as I had some time.

Separating the building of the images per platform is possible (exporting/importing single-platform images is no issue), but combining the images at the end is slightly hacky, as Docker does not support direct publishing of multi-platform "images" (manifest), but rather forces the individual images to be published already before creating the multi-platform manifest. (docker/cli#3350)

To get around this, one can publish the single-platform images onto a local registry ran through GHA services. Once they are published, create the manifest, and push it locally. Finally, it's possible to copy the multi-platform "image" to the public registry. (Similarly to rapidsai/gha-tools#51)

[timower]

This should allow the toolchain to be easily used on apple silicon macs (or any other arm64 device).

Closes#28

[Eeems]

My understanding is that if you use buildx, DOCKER_BUILDKIT=1 is no longer needed

[timower]

I'll try something like https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners, otherwise I guess this isn't feasible

[Eeems]

To avoid timeouts due to how long it takes, probably. It might also make sense to split out the images into their own jobs as well. I've done that for the images I have generating nightly: https://github.com/Eeems/images/blob/main/.github/workflows/build.yml

[trixmoe]

My optimism in regards to M1 runners was short-sighted: GitHub (and simply Apple) does not support (nested) virtualization on those runners, which results in the VM not being able to boot up. There's thus no way to really improve the earlier amd64 Linux runner solution('s run time).

I found out about this through the issue I opened on the colima repo, and the announcement PR on actions/runner-images, which pointed at the GitHub docs:

(...) nested-virtualization is not supported by arm64 runners

Tl;dr of this whole ordeal

Stick with amd64 Linux runners (for the foreseeable future).
GH doesn't seem to be currently working on Linux arm64 runners (actions/runner-images#5631).

[jandubois]

GitHub (and simply Apple) does not support (nested) virtualization

Note that while GitHub claims it to be a limitation of Apple's Virtualization Framework, it really is a limitation of the M1 chip architecture; nested virtualization is supported by Apple's Virtualization Framework on M2 and M3 chips, which are based on a newer ARM spec.

[trixmoe]

To avoid timeouts due to how long it takes, probably. It might also make sense to split out the images into their own jobs as well. I've done that for the images I have generating nightly: https://github.com/Eeems/images/blob/main/.github/workflows/build.yml

I've looked at the dependencies between the images, and I've realized that one could have 3 different stages, with each image having it's own separate job. Using different jobs also allows it to go over the 6 hour limit.

Stage 1 would be the initial one, Stage 2 starts once toolchain is done, Stage 3 starts once base is done. Each of the blocks are separate jobs.

One would have to look into the exporting of images, to transfer them between jobs, for builds that are not published.
One could also separate amd64 and arm64 builds, but I'm unsure if this is doable when publishing to the image repository.

[Eeems]

I would recommend moving dotnet6 to stage 3 as well, make that dynamic, and have toolchain and base be static steps.

[trixmoe]

I looked more into multi-platform Docker images, as I had some time.

Separating the building of the images per platform is possible (exporting/importing single-platform images is no issue), but combining the images at the end is slightly hacky, as Docker does not support direct publishing of multi-platform "images" (manifest), but rather forces the individual images to be published already before creating the multi-platform manifest. (docker/cli#3350)

To get around this, one can publish the single-platform images onto a local registry ran through GHA services. Once they are published, create the manifest, and push it locally. Finally, it's possible to copy the multi-platform "image" to the public registry. (Similarly to rapidsai/gha-tools#51)

[timower]

This should allow the toolchain to be easily used on apple silicon macs (or any other arm64 device).

Closes#28

[Eeems]

My understanding is that if you use buildx, DOCKER_BUILDKIT=1 is no longer needed

[timower]

I'll try something like https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners, otherwise I guess this isn't feasible

[Eeems]

To avoid timeouts due to how long it takes, probably. It might also make sense to split out the images into their own jobs as well. I've done that for the images I have generating nightly: https://github.com/Eeems/images/blob/main/.github/workflows/build.yml

[trixmoe]

My optimism in regards to M1 runners was short-sighted: GitHub (and simply Apple) does not support (nested) virtualization on those runners, which results in the VM not being able to boot up. There's thus no way to really improve the earlier amd64 Linux runner solution('s run time).

I found out about this through the issue I opened on the colima repo, and the announcement PR on actions/runner-images, which pointed at the GitHub docs:

(...) nested-virtualization is not supported by arm64 runners

Tl;dr of this whole ordeal

Stick with amd64 Linux runners (for the foreseeable future).
GH doesn't seem to be currently working on Linux arm64 runners (actions/runner-images#5631).

[jandubois]

GitHub (and simply Apple) does not support (nested) virtualization

Note that while GitHub claims it to be a limitation of Apple's Virtualization Framework, it really is a limitation of the M1 chip architecture; nested virtualization is supported by Apple's Virtualization Framework on M2 and M3 chips, which are based on a newer ARM spec.

[trixmoe]

To avoid timeouts due to how long it takes, probably. It might also make sense to split out the images into their own jobs as well. I've done that for the images I have generating nightly: https://github.com/Eeems/images/blob/main/.github/workflows/build.yml

I've looked at the dependencies between the images, and I've realized that one could have 3 different stages, with each image having it's own separate job. Using different jobs also allows it to go over the 6 hour limit.

Stage 1 would be the initial one, Stage 2 starts once toolchain is done, Stage 3 starts once base is done. Each of the blocks are separate jobs.

One would have to look into the exporting of images, to transfer them between jobs, for builds that are not published.
One could also separate amd64 and arm64 builds, but I'm unsure if this is doable when publishing to the image repository.

[Eeems]

I would recommend moving dotnet6 to stage 3 as well, make that dynamic, and have toolchain and base be static steps.

[trixmoe]

I looked more into multi-platform Docker images, as I had some time.

Separating the building of the images per platform is possible (exporting/importing single-platform images is no issue), but combining the images at the end is slightly hacky, as Docker does not support direct publishing of multi-platform "images" (manifest), but rather forces the individual images to be published already before creating the multi-platform manifest. (docker/cli#3350)

To get around this, one can publish the single-platform images onto a local registry ran through GHA services. Once they are published, create the manifest, and push it locally. Finally, it's possible to copy the multi-platform "image" to the public registry. (Similarly to rapidsai/gha-tools#51)