Self-contained docker build with ARM64 publishing

[keynmol]

Fixes#2132

At the moment, the scala-cli docker image is built using a combination of mill tasks, scripts, and dockerfiles that require externally built binary. What's more, it's only built for x86.

With ARM64 runners on GHA being GA, I propose a simplification - a self-contained multi-stage dockerfile, with a custom Github Workflow that merges the images into a single manifest, meaning that docker will pull the correct image no matter the target platform.

Manifest merging and pushing is a complicated step, and Docker have been amending their docs with the example, which is what this workflow is based on.

I've been using this setup in multiple apps, e.g. Mimalyzer.

---

There are some aspects of the build I don't really understand – static, mostly static images, customisation for Linux x86, etc.

This proposal attempts to do the simplest possible thing, so that building a docker image is just docker build . -t VirtusLab/scala-cli.

Currently, it only publishes to ghcr.io to test out the workflows without disturbing the main image on docker hub. But additional publishing steps are easily added, of course

[Gedochao]

@tgodzik@zielinsky can you take a look as well? I think it'd be good to have another pair of eyes on this (or two)

[Gedochao]

There are some aspects of the build I don't really understand – static, mostly static images, customisation for Linux x86, etc.

Static and mostly static images are in the understanding of GraalVM, as per this doc: https://www.graalvm.org/21.3/reference-manual/native-image/StaticImages/index.html
The way they're built is based on the mill-native-image plugin: https://github.com/alexarchambault/mill-native-image

(...) customisation for Linux x86 (...)

What particular customisation for Linux x86 did you mean?

(...) etc.

Shoot away, I'll try to answer, or at least direct you in the direction of an answer.

What's there in this area was initially coded by @alexarchambault (who's also the author of mill-native-image). While I've tinkered with this here and there, most of it has been lying untouched for a long, long time, and I'm not all that familiar with it either.

[tgodzik]

I don't know much about this, I only built basic docker images. Overall look ok as long as it work 😅

[zielinsky]

A few comments from me 😅

[keynmol]

@Gedochao

What particular customisation for Linux x86 did you mean?

This line threw me off

https://github.com/keynmol/scala-cli/blob/93079619648e0b475f874ee2676387b488d7a637/project/settings/package.mill.scala#L406

[Gedochao]

Just to make sure.

We currently have the following images, which are updated with each Scala CLI release:

• https://hub.docker.com/r/virtuslab/scala-cli-slim
• https://hub.docker.com/r/virtuslab/scala-cli

@keynmol your intention is to create a third one, yes?
smth like virtuslab/scala-cli-platform-native?

[Gedochao]

Also, how can we safely test this?

[keynmol]

Here's my plan:

1. Merge this workflow as is, make sure it publishes a working ghcr.io/virtuslab/scala-cli image (similar to existing DockerHub virtuslab/scala-cli)
2. Modify the workflow in a separate PR to build the slim image in the same way (similar to existing DockerHub virtuslab/scala-cli-slim)
3. Add DockerHub publishing steps to both images
4. Remove old docker publishing logic

Up to step 3, the existing docker images on DockerHub are safe and won't be touched, as we only publish to ghcr.io, using it as our testing area.

Steps 1-3 can be done in separate PRs as they won't affect existing dockerhub publishing steps.

In the end, we should have virtuslab/scala-cli and virtuslab/scala-cli-slim that have a manifest merging two platforms, so users automatically get the right image (see screenshot)

[Gedochao]

Okay... sounds reasonable.
Let's get the CI green and do it one step at a time.

[Gedochao]

The CI is green.
@keynmol the workflow is still on Ubuntu 22.04, while the other workflows already rely on 24.04 (I left comments earlier).
Otherwise, we seem to be good to merge here (might want to rebase on main while you're at it)

[keynmol]

I have responded to the Ubuntu version comment: #3962 (comment)

Actually scratch that, in this instance it won't have any effect, I will update

[Gedochao]

LGTM
@keynmol ready to merge?

[Gedochao]

@zielinsky I will wait for your okay before clicking merge.

[keynmol]

I think so, no doubt it will require more iterations when the work flow actually runs :)

[zielinsky]

One question from me before approval.

[zielinsky]

LGTM for now, let's test it!

[keynmol]

Workflow succeeded, and the image is published correctly, containing two platform-specific images: https://github.com/VirtusLab/scala-cli/actions/runs/20428482272/job/58694886877#step:7:1

But I think the visibility of packages needs to be changed in repository and package settings

[keynmol]

Fixes#2132

At the moment, the scala-cli docker image is built using a combination of mill tasks, scripts, and dockerfiles that require externally built binary. What's more, it's only built for x86.

With ARM64 runners on GHA being GA, I propose a simplification - a self-contained multi-stage dockerfile, with a custom Github Workflow that merges the images into a single manifest, meaning that docker will pull the correct image no matter the target platform.

Manifest merging and pushing is a complicated step, and Docker have been amending their docs with the example, which is what this workflow is based on.

I've been using this setup in multiple apps, e.g. Mimalyzer.

---

There are some aspects of the build I don't really understand – static, mostly static images, customisation for Linux x86, etc.

This proposal attempts to do the simplest possible thing, so that building a docker image is just docker build . -t VirtusLab/scala-cli.

Currently, it only publishes to ghcr.io to test out the workflows without disturbing the main image on docker hub. But additional publishing steps are easily added, of course

[Gedochao]

@tgodzik@zielinsky can you take a look as well? I think it'd be good to have another pair of eyes on this (or two)

[Gedochao]

There are some aspects of the build I don't really understand – static, mostly static images, customisation for Linux x86, etc.

Static and mostly static images are in the understanding of GraalVM, as per this doc: https://www.graalvm.org/21.3/reference-manual/native-image/StaticImages/index.html
The way they're built is based on the mill-native-image plugin: https://github.com/alexarchambault/mill-native-image

(...) customisation for Linux x86 (...)

What particular customisation for Linux x86 did you mean?

(...) etc.

Shoot away, I'll try to answer, or at least direct you in the direction of an answer.

What's there in this area was initially coded by @alexarchambault (who's also the author of mill-native-image). While I've tinkered with this here and there, most of it has been lying untouched for a long, long time, and I'm not all that familiar with it either.

[tgodzik]

I don't know much about this, I only built basic docker images. Overall look ok as long as it work 😅

[zielinsky]

A few comments from me 😅

[keynmol]

@Gedochao

What particular customisation for Linux x86 did you mean?

This line threw me off

https://github.com/keynmol/scala-cli/blob/93079619648e0b475f874ee2676387b488d7a637/project/settings/package.mill.scala#L406

[Gedochao]

Just to make sure.

We currently have the following images, which are updated with each Scala CLI release:

• https://hub.docker.com/r/virtuslab/scala-cli-slim
• https://hub.docker.com/r/virtuslab/scala-cli

@keynmol your intention is to create a third one, yes?
smth like virtuslab/scala-cli-platform-native?

[Gedochao]

Also, how can we safely test this?

[keynmol]

Here's my plan:

1. Merge this workflow as is, make sure it publishes a working ghcr.io/virtuslab/scala-cli image (similar to existing DockerHub virtuslab/scala-cli)
2. Modify the workflow in a separate PR to build the slim image in the same way (similar to existing DockerHub virtuslab/scala-cli-slim)
3. Add DockerHub publishing steps to both images
4. Remove old docker publishing logic

Up to step 3, the existing docker images on DockerHub are safe and won't be touched, as we only publish to ghcr.io, using it as our testing area.

Steps 1-3 can be done in separate PRs as they won't affect existing dockerhub publishing steps.

In the end, we should have virtuslab/scala-cli and virtuslab/scala-cli-slim that have a manifest merging two platforms, so users automatically get the right image (see screenshot)

[Gedochao]

Okay... sounds reasonable.
Let's get the CI green and do it one step at a time.

[Gedochao]

The CI is green.
@keynmol the workflow is still on Ubuntu 22.04, while the other workflows already rely on 24.04 (I left comments earlier).
Otherwise, we seem to be good to merge here (might want to rebase on main while you're at it)

[keynmol]

I have responded to the Ubuntu version comment: #3962 (comment)

Actually scratch that, in this instance it won't have any effect, I will update

[Gedochao]

LGTM
@keynmol ready to merge?

[Gedochao]

@zielinsky I will wait for your okay before clicking merge.

[keynmol]

I think so, no doubt it will require more iterations when the work flow actually runs :)

[zielinsky]

One question from me before approval.

[zielinsky]

LGTM for now, let's test it!

[keynmol]

Workflow succeeded, and the image is published correctly, containing two platform-specific images: https://github.com/VirtusLab/scala-cli/actions/runs/20428482272/job/58694886877#step:7:1

But I think the visibility of packages needs to be changed in repository and package settings

[keynmol]

Fixes#2132

At the moment, the scala-cli docker image is built using a combination of mill tasks, scripts, and dockerfiles that require externally built binary. What's more, it's only built for x86.

With ARM64 runners on GHA being GA, I propose a simplification - a self-contained multi-stage dockerfile, with a custom Github Workflow that merges the images into a single manifest, meaning that docker will pull the correct image no matter the target platform.

Manifest merging and pushing is a complicated step, and Docker have been amending their docs with the example, which is what this workflow is based on.

I've been using this setup in multiple apps, e.g. Mimalyzer.

---

There are some aspects of the build I don't really understand – static, mostly static images, customisation for Linux x86, etc.

This proposal attempts to do the simplest possible thing, so that building a docker image is just docker build . -t VirtusLab/scala-cli.

Currently, it only publishes to ghcr.io to test out the workflows without disturbing the main image on docker hub. But additional publishing steps are easily added, of course

[Gedochao]

@tgodzik@zielinsky can you take a look as well? I think it'd be good to have another pair of eyes on this (or two)

[Gedochao]

There are some aspects of the build I don't really understand – static, mostly static images, customisation for Linux x86, etc.

Static and mostly static images are in the understanding of GraalVM, as per this doc: https://www.graalvm.org/21.3/reference-manual/native-image/StaticImages/index.html
The way they're built is based on the mill-native-image plugin: https://github.com/alexarchambault/mill-native-image

(...) customisation for Linux x86 (...)

What particular customisation for Linux x86 did you mean?

(...) etc.

Shoot away, I'll try to answer, or at least direct you in the direction of an answer.

What's there in this area was initially coded by @alexarchambault (who's also the author of mill-native-image). While I've tinkered with this here and there, most of it has been lying untouched for a long, long time, and I'm not all that familiar with it either.

[tgodzik]

I don't know much about this, I only built basic docker images. Overall look ok as long as it work 😅

[zielinsky]

A few comments from me 😅

[keynmol]

@Gedochao

What particular customisation for Linux x86 did you mean?

This line threw me off

https://github.com/keynmol/scala-cli/blob/93079619648e0b475f874ee2676387b488d7a637/project/settings/package.mill.scala#L406

[Gedochao]

Just to make sure.

We currently have the following images, which are updated with each Scala CLI release:

• https://hub.docker.com/r/virtuslab/scala-cli-slim
• https://hub.docker.com/r/virtuslab/scala-cli

@keynmol your intention is to create a third one, yes?
smth like virtuslab/scala-cli-platform-native?

[Gedochao]

Also, how can we safely test this?

[keynmol]

Here's my plan:

1. Merge this workflow as is, make sure it publishes a working ghcr.io/virtuslab/scala-cli image (similar to existing DockerHub virtuslab/scala-cli)
2. Modify the workflow in a separate PR to build the slim image in the same way (similar to existing DockerHub virtuslab/scala-cli-slim)
3. Add DockerHub publishing steps to both images
4. Remove old docker publishing logic

Up to step 3, the existing docker images on DockerHub are safe and won't be touched, as we only publish to ghcr.io, using it as our testing area.

Steps 1-3 can be done in separate PRs as they won't affect existing dockerhub publishing steps.

In the end, we should have virtuslab/scala-cli and virtuslab/scala-cli-slim that have a manifest merging two platforms, so users automatically get the right image (see screenshot)

[Gedochao]

Okay... sounds reasonable.
Let's get the CI green and do it one step at a time.

[Gedochao]

The CI is green.
@keynmol the workflow is still on Ubuntu 22.04, while the other workflows already rely on 24.04 (I left comments earlier).
Otherwise, we seem to be good to merge here (might want to rebase on main while you're at it)

[keynmol]

I have responded to the Ubuntu version comment: #3962 (comment)

Actually scratch that, in this instance it won't have any effect, I will update

[Gedochao]

LGTM
@keynmol ready to merge?

[Gedochao]

@zielinsky I will wait for your okay before clicking merge.

[keynmol]

I think so, no doubt it will require more iterations when the work flow actually runs :)

[zielinsky]

One question from me before approval.

[zielinsky]

LGTM for now, let's test it!

[keynmol]

Workflow succeeded, and the image is published correctly, containing two platform-specific images: https://github.com/VirtusLab/scala-cli/actions/runs/20428482272/job/58694886877#step:7:1

But I think the visibility of packages needs to be changed in repository and package settings

[keynmol]

Fixes#2132

At the moment, the scala-cli docker image is built using a combination of mill tasks, scripts, and dockerfiles that require externally built binary. What's more, it's only built for x86.

With ARM64 runners on GHA being GA, I propose a simplification - a self-contained multi-stage dockerfile, with a custom Github Workflow that merges the images into a single manifest, meaning that docker will pull the correct image no matter the target platform.

Manifest merging and pushing is a complicated step, and Docker have been amending their docs with the example, which is what this workflow is based on.

I've been using this setup in multiple apps, e.g. Mimalyzer.

---

There are some aspects of the build I don't really understand – static, mostly static images, customisation for Linux x86, etc.

This proposal attempts to do the simplest possible thing, so that building a docker image is just docker build . -t VirtusLab/scala-cli.

Currently, it only publishes to ghcr.io to test out the workflows without disturbing the main image on docker hub. But additional publishing steps are easily added, of course

[Gedochao]

@tgodzik@zielinsky can you take a look as well? I think it'd be good to have another pair of eyes on this (or two)

[Gedochao]

There are some aspects of the build I don't really understand – static, mostly static images, customisation for Linux x86, etc.

Static and mostly static images are in the understanding of GraalVM, as per this doc: https://www.graalvm.org/21.3/reference-manual/native-image/StaticImages/index.html
The way they're built is based on the mill-native-image plugin: https://github.com/alexarchambault/mill-native-image

(...) customisation for Linux x86 (...)

What particular customisation for Linux x86 did you mean?

(...) etc.

Shoot away, I'll try to answer, or at least direct you in the direction of an answer.

What's there in this area was initially coded by @alexarchambault (who's also the author of mill-native-image). While I've tinkered with this here and there, most of it has been lying untouched for a long, long time, and I'm not all that familiar with it either.

[tgodzik]

I don't know much about this, I only built basic docker images. Overall look ok as long as it work 😅

[zielinsky]

A few comments from me 😅

[keynmol]

@Gedochao

What particular customisation for Linux x86 did you mean?

This line threw me off

https://github.com/keynmol/scala-cli/blob/93079619648e0b475f874ee2676387b488d7a637/project/settings/package.mill.scala#L406

[Gedochao]

Just to make sure.

We currently have the following images, which are updated with each Scala CLI release:

• https://hub.docker.com/r/virtuslab/scala-cli-slim
• https://hub.docker.com/r/virtuslab/scala-cli

@keynmol your intention is to create a third one, yes?
smth like virtuslab/scala-cli-platform-native?

[Gedochao]

Also, how can we safely test this?

[keynmol]

Here's my plan:

1. Merge this workflow as is, make sure it publishes a working ghcr.io/virtuslab/scala-cli image (similar to existing DockerHub virtuslab/scala-cli)
2. Modify the workflow in a separate PR to build the slim image in the same way (similar to existing DockerHub virtuslab/scala-cli-slim)
3. Add DockerHub publishing steps to both images
4. Remove old docker publishing logic

Up to step 3, the existing docker images on DockerHub are safe and won't be touched, as we only publish to ghcr.io, using it as our testing area.

Steps 1-3 can be done in separate PRs as they won't affect existing dockerhub publishing steps.

In the end, we should have virtuslab/scala-cli and virtuslab/scala-cli-slim that have a manifest merging two platforms, so users automatically get the right image (see screenshot)

[Gedochao]

Okay... sounds reasonable.
Let's get the CI green and do it one step at a time.

[Gedochao]

The CI is green.
@keynmol the workflow is still on Ubuntu 22.04, while the other workflows already rely on 24.04 (I left comments earlier).
Otherwise, we seem to be good to merge here (might want to rebase on main while you're at it)

[keynmol]

I have responded to the Ubuntu version comment: #3962 (comment)

Actually scratch that, in this instance it won't have any effect, I will update

[Gedochao]

LGTM
@keynmol ready to merge?

[Gedochao]

@zielinsky I will wait for your okay before clicking merge.

[keynmol]

I think so, no doubt it will require more iterations when the work flow actually runs :)

[zielinsky]

One question from me before approval.

[zielinsky]

LGTM for now, let's test it!

[keynmol]

Workflow succeeded, and the image is published correctly, containing two platform-specific images: https://github.com/VirtusLab/scala-cli/actions/runs/20428482272/job/58694886877#step:7:1

But I think the visibility of packages needs to be changed in repository and package settings

[keynmol]

Fixes#2132

At the moment, the scala-cli docker image is built using a combination of mill tasks, scripts, and dockerfiles that require externally built binary. What's more, it's only built for x86.

With ARM64 runners on GHA being GA, I propose a simplification - a self-contained multi-stage dockerfile, with a custom Github Workflow that merges the images into a single manifest, meaning that docker will pull the correct image no matter the target platform.

Manifest merging and pushing is a complicated step, and Docker have been amending their docs with the example, which is what this workflow is based on.

I've been using this setup in multiple apps, e.g. Mimalyzer.

---

There are some aspects of the build I don't really understand – static, mostly static images, customisation for Linux x86, etc.

This proposal attempts to do the simplest possible thing, so that building a docker image is just docker build . -t VirtusLab/scala-cli.

Currently, it only publishes to ghcr.io to test out the workflows without disturbing the main image on docker hub. But additional publishing steps are easily added, of course

[Gedochao]

@tgodzik@zielinsky can you take a look as well? I think it'd be good to have another pair of eyes on this (or two)

[Gedochao]

There are some aspects of the build I don't really understand – static, mostly static images, customisation for Linux x86, etc.

Static and mostly static images are in the understanding of GraalVM, as per this doc: https://www.graalvm.org/21.3/reference-manual/native-image/StaticImages/index.html
The way they're built is based on the mill-native-image plugin: https://github.com/alexarchambault/mill-native-image

(...) customisation for Linux x86 (...)

What particular customisation for Linux x86 did you mean?

(...) etc.

Shoot away, I'll try to answer, or at least direct you in the direction of an answer.

What's there in this area was initially coded by @alexarchambault (who's also the author of mill-native-image). While I've tinkered with this here and there, most of it has been lying untouched for a long, long time, and I'm not all that familiar with it either.

[tgodzik]

I don't know much about this, I only built basic docker images. Overall look ok as long as it work 😅

[zielinsky]

A few comments from me 😅

[keynmol]

@Gedochao

What particular customisation for Linux x86 did you mean?

This line threw me off

https://github.com/keynmol/scala-cli/blob/93079619648e0b475f874ee2676387b488d7a637/project/settings/package.mill.scala#L406

[Gedochao]

Just to make sure.

We currently have the following images, which are updated with each Scala CLI release:

• https://hub.docker.com/r/virtuslab/scala-cli-slim
• https://hub.docker.com/r/virtuslab/scala-cli

@keynmol your intention is to create a third one, yes?
smth like virtuslab/scala-cli-platform-native?

[Gedochao]

Also, how can we safely test this?

[keynmol]

Here's my plan:

1. Merge this workflow as is, make sure it publishes a working ghcr.io/virtuslab/scala-cli image (similar to existing DockerHub virtuslab/scala-cli)
2. Modify the workflow in a separate PR to build the slim image in the same way (similar to existing DockerHub virtuslab/scala-cli-slim)
3. Add DockerHub publishing steps to both images
4. Remove old docker publishing logic

Up to step 3, the existing docker images on DockerHub are safe and won't be touched, as we only publish to ghcr.io, using it as our testing area.

Steps 1-3 can be done in separate PRs as they won't affect existing dockerhub publishing steps.

In the end, we should have virtuslab/scala-cli and virtuslab/scala-cli-slim that have a manifest merging two platforms, so users automatically get the right image (see screenshot)

[Gedochao]

Okay... sounds reasonable.
Let's get the CI green and do it one step at a time.

[Gedochao]

The CI is green.
@keynmol the workflow is still on Ubuntu 22.04, while the other workflows already rely on 24.04 (I left comments earlier).
Otherwise, we seem to be good to merge here (might want to rebase on main while you're at it)

[keynmol]

I have responded to the Ubuntu version comment: #3962 (comment)

Actually scratch that, in this instance it won't have any effect, I will update

[Gedochao]

LGTM
@keynmol ready to merge?

[Gedochao]

@zielinsky I will wait for your okay before clicking merge.

[keynmol]

I think so, no doubt it will require more iterations when the work flow actually runs :)

[zielinsky]

One question from me before approval.

[zielinsky]

LGTM for now, let's test it!

[keynmol]

Workflow succeeded, and the image is published correctly, containing two platform-specific images: https://github.com/VirtusLab/scala-cli/actions/runs/20428482272/job/58694886877#step:7:1

But I think the visibility of packages needs to be changed in repository and package settings

[keynmol]

Fixes#2132

At the moment, the scala-cli docker image is built using a combination of mill tasks, scripts, and dockerfiles that require externally built binary. What's more, it's only built for x86.

With ARM64 runners on GHA being GA, I propose a simplification - a self-contained multi-stage dockerfile, with a custom Github Workflow that merges the images into a single manifest, meaning that docker will pull the correct image no matter the target platform.

Manifest merging and pushing is a complicated step, and Docker have been amending their docs with the example, which is what this workflow is based on.

I've been using this setup in multiple apps, e.g. Mimalyzer.

---

There are some aspects of the build I don't really understand – static, mostly static images, customisation for Linux x86, etc.

This proposal attempts to do the simplest possible thing, so that building a docker image is just docker build . -t VirtusLab/scala-cli.

Currently, it only publishes to ghcr.io to test out the workflows without disturbing the main image on docker hub. But additional publishing steps are easily added, of course

[Gedochao]

@tgodzik@zielinsky can you take a look as well? I think it'd be good to have another pair of eyes on this (or two)

[Gedochao]

There are some aspects of the build I don't really understand – static, mostly static images, customisation for Linux x86, etc.

Static and mostly static images are in the understanding of GraalVM, as per this doc: https://www.graalvm.org/21.3/reference-manual/native-image/StaticImages/index.html
The way they're built is based on the mill-native-image plugin: https://github.com/alexarchambault/mill-native-image

(...) customisation for Linux x86 (...)

What particular customisation for Linux x86 did you mean?

(...) etc.

Shoot away, I'll try to answer, or at least direct you in the direction of an answer.

What's there in this area was initially coded by @alexarchambault (who's also the author of mill-native-image). While I've tinkered with this here and there, most of it has been lying untouched for a long, long time, and I'm not all that familiar with it either.

[tgodzik]

I don't know much about this, I only built basic docker images. Overall look ok as long as it work 😅

[zielinsky]

A few comments from me 😅

[keynmol]

@Gedochao

What particular customisation for Linux x86 did you mean?

This line threw me off

https://github.com/keynmol/scala-cli/blob/93079619648e0b475f874ee2676387b488d7a637/project/settings/package.mill.scala#L406

[Gedochao]

Just to make sure.

We currently have the following images, which are updated with each Scala CLI release:

• https://hub.docker.com/r/virtuslab/scala-cli-slim
• https://hub.docker.com/r/virtuslab/scala-cli

@keynmol your intention is to create a third one, yes?
smth like virtuslab/scala-cli-platform-native?

[Gedochao]

Also, how can we safely test this?

[keynmol]

Here's my plan:

1. Merge this workflow as is, make sure it publishes a working ghcr.io/virtuslab/scala-cli image (similar to existing DockerHub virtuslab/scala-cli)
2. Modify the workflow in a separate PR to build the slim image in the same way (similar to existing DockerHub virtuslab/scala-cli-slim)
3. Add DockerHub publishing steps to both images
4. Remove old docker publishing logic

Up to step 3, the existing docker images on DockerHub are safe and won't be touched, as we only publish to ghcr.io, using it as our testing area.

Steps 1-3 can be done in separate PRs as they won't affect existing dockerhub publishing steps.

In the end, we should have virtuslab/scala-cli and virtuslab/scala-cli-slim that have a manifest merging two platforms, so users automatically get the right image (see screenshot)

[Gedochao]

Okay... sounds reasonable.
Let's get the CI green and do it one step at a time.

[Gedochao]

The CI is green.
@keynmol the workflow is still on Ubuntu 22.04, while the other workflows already rely on 24.04 (I left comments earlier).
Otherwise, we seem to be good to merge here (might want to rebase on main while you're at it)

[keynmol]

I have responded to the Ubuntu version comment: #3962 (comment)

Actually scratch that, in this instance it won't have any effect, I will update

[Gedochao]

LGTM
@keynmol ready to merge?

[Gedochao]

@zielinsky I will wait for your okay before clicking merge.

[keynmol]

I think so, no doubt it will require more iterations when the work flow actually runs :)

[zielinsky]

One question from me before approval.

[zielinsky]

LGTM for now, let's test it!

[keynmol]

Workflow succeeded, and the image is published correctly, containing two platform-specific images: https://github.com/VirtusLab/scala-cli/actions/runs/20428482272/job/58694886877#step:7:1

But I think the visibility of packages needs to be changed in repository and package settings

[keynmol]

Fixes#2132

At the moment, the scala-cli docker image is built using a combination of mill tasks, scripts, and dockerfiles that require externally built binary. What's more, it's only built for x86.

With ARM64 runners on GHA being GA, I propose a simplification - a self-contained multi-stage dockerfile, with a custom Github Workflow that merges the images into a single manifest, meaning that docker will pull the correct image no matter the target platform.

Manifest merging and pushing is a complicated step, and Docker have been amending their docs with the example, which is what this workflow is based on.

I've been using this setup in multiple apps, e.g. Mimalyzer.

---

There are some aspects of the build I don't really understand – static, mostly static images, customisation for Linux x86, etc.

This proposal attempts to do the simplest possible thing, so that building a docker image is just docker build . -t VirtusLab/scala-cli.

Currently, it only publishes to ghcr.io to test out the workflows without disturbing the main image on docker hub. But additional publishing steps are easily added, of course

[Gedochao]

@tgodzik@zielinsky can you take a look as well? I think it'd be good to have another pair of eyes on this (or two)

[Gedochao]

There are some aspects of the build I don't really understand – static, mostly static images, customisation for Linux x86, etc.

Static and mostly static images are in the understanding of GraalVM, as per this doc: https://www.graalvm.org/21.3/reference-manual/native-image/StaticImages/index.html
The way they're built is based on the mill-native-image plugin: https://github.com/alexarchambault/mill-native-image

(...) customisation for Linux x86 (...)

What particular customisation for Linux x86 did you mean?

(...) etc.

Shoot away, I'll try to answer, or at least direct you in the direction of an answer.

What's there in this area was initially coded by @alexarchambault (who's also the author of mill-native-image). While I've tinkered with this here and there, most of it has been lying untouched for a long, long time, and I'm not all that familiar with it either.

[tgodzik]

I don't know much about this, I only built basic docker images. Overall look ok as long as it work 😅

[zielinsky]

A few comments from me 😅

[keynmol]

@Gedochao

What particular customisation for Linux x86 did you mean?

This line threw me off

https://github.com/keynmol/scala-cli/blob/93079619648e0b475f874ee2676387b488d7a637/project/settings/package.mill.scala#L406

[Gedochao]

Just to make sure.

We currently have the following images, which are updated with each Scala CLI release:

• https://hub.docker.com/r/virtuslab/scala-cli-slim
• https://hub.docker.com/r/virtuslab/scala-cli

@keynmol your intention is to create a third one, yes?
smth like virtuslab/scala-cli-platform-native?

[Gedochao]

Also, how can we safely test this?

[keynmol]

Here's my plan:

1. Merge this workflow as is, make sure it publishes a working ghcr.io/virtuslab/scala-cli image (similar to existing DockerHub virtuslab/scala-cli)
2. Modify the workflow in a separate PR to build the slim image in the same way (similar to existing DockerHub virtuslab/scala-cli-slim)
3. Add DockerHub publishing steps to both images
4. Remove old docker publishing logic

Up to step 3, the existing docker images on DockerHub are safe and won't be touched, as we only publish to ghcr.io, using it as our testing area.

Steps 1-3 can be done in separate PRs as they won't affect existing dockerhub publishing steps.

In the end, we should have virtuslab/scala-cli and virtuslab/scala-cli-slim that have a manifest merging two platforms, so users automatically get the right image (see screenshot)

[Gedochao]

Okay... sounds reasonable.
Let's get the CI green and do it one step at a time.

[Gedochao]

The CI is green.
@keynmol the workflow is still on Ubuntu 22.04, while the other workflows already rely on 24.04 (I left comments earlier).
Otherwise, we seem to be good to merge here (might want to rebase on main while you're at it)

[keynmol]

I have responded to the Ubuntu version comment: #3962 (comment)

Actually scratch that, in this instance it won't have any effect, I will update

[Gedochao]

LGTM
@keynmol ready to merge?

[Gedochao]

@zielinsky I will wait for your okay before clicking merge.

[keynmol]

I think so, no doubt it will require more iterations when the work flow actually runs :)

[zielinsky]

One question from me before approval.

[zielinsky]

LGTM for now, let's test it!

[keynmol]

Workflow succeeded, and the image is published correctly, containing two platform-specific images: https://github.com/VirtusLab/scala-cli/actions/runs/20428482272/job/58694886877#step:7:1

But I think the visibility of packages needs to be changed in repository and package settings

[keynmol]

Fixes#2132

At the moment, the scala-cli docker image is built using a combination of mill tasks, scripts, and dockerfiles that require externally built binary. What's more, it's only built for x86.

With ARM64 runners on GHA being GA, I propose a simplification - a self-contained multi-stage dockerfile, with a custom Github Workflow that merges the images into a single manifest, meaning that docker will pull the correct image no matter the target platform.

Manifest merging and pushing is a complicated step, and Docker have been amending their docs with the example, which is what this workflow is based on.

I've been using this setup in multiple apps, e.g. Mimalyzer.

---

There are some aspects of the build I don't really understand – static, mostly static images, customisation for Linux x86, etc.

This proposal attempts to do the simplest possible thing, so that building a docker image is just docker build . -t VirtusLab/scala-cli.

Currently, it only publishes to ghcr.io to test out the workflows without disturbing the main image on docker hub. But additional publishing steps are easily added, of course

[Gedochao]

@tgodzik@zielinsky can you take a look as well? I think it'd be good to have another pair of eyes on this (or two)

[Gedochao]

There are some aspects of the build I don't really understand – static, mostly static images, customisation for Linux x86, etc.

Static and mostly static images are in the understanding of GraalVM, as per this doc: https://www.graalvm.org/21.3/reference-manual/native-image/StaticImages/index.html
The way they're built is based on the mill-native-image plugin: https://github.com/alexarchambault/mill-native-image

(...) customisation for Linux x86 (...)

What particular customisation for Linux x86 did you mean?

(...) etc.

Shoot away, I'll try to answer, or at least direct you in the direction of an answer.

What's there in this area was initially coded by @alexarchambault (who's also the author of mill-native-image). While I've tinkered with this here and there, most of it has been lying untouched for a long, long time, and I'm not all that familiar with it either.

[tgodzik]

I don't know much about this, I only built basic docker images. Overall look ok as long as it work 😅

[zielinsky]

A few comments from me 😅

[keynmol]

@Gedochao

What particular customisation for Linux x86 did you mean?

This line threw me off

https://github.com/keynmol/scala-cli/blob/93079619648e0b475f874ee2676387b488d7a637/project/settings/package.mill.scala#L406

[Gedochao]

Just to make sure.

We currently have the following images, which are updated with each Scala CLI release:

• https://hub.docker.com/r/virtuslab/scala-cli-slim
• https://hub.docker.com/r/virtuslab/scala-cli

@keynmol your intention is to create a third one, yes?
smth like virtuslab/scala-cli-platform-native?

[Gedochao]

Also, how can we safely test this?

[keynmol]

Here's my plan:

1. Merge this workflow as is, make sure it publishes a working ghcr.io/virtuslab/scala-cli image (similar to existing DockerHub virtuslab/scala-cli)
2. Modify the workflow in a separate PR to build the slim image in the same way (similar to existing DockerHub virtuslab/scala-cli-slim)
3. Add DockerHub publishing steps to both images
4. Remove old docker publishing logic

Up to step 3, the existing docker images on DockerHub are safe and won't be touched, as we only publish to ghcr.io, using it as our testing area.

Steps 1-3 can be done in separate PRs as they won't affect existing dockerhub publishing steps.

In the end, we should have virtuslab/scala-cli and virtuslab/scala-cli-slim that have a manifest merging two platforms, so users automatically get the right image (see screenshot)

[Gedochao]

Okay... sounds reasonable.
Let's get the CI green and do it one step at a time.

[Gedochao]

The CI is green.
@keynmol the workflow is still on Ubuntu 22.04, while the other workflows already rely on 24.04 (I left comments earlier).
Otherwise, we seem to be good to merge here (might want to rebase on main while you're at it)

[keynmol]

I have responded to the Ubuntu version comment: #3962 (comment)

Actually scratch that, in this instance it won't have any effect, I will update

[Gedochao]

LGTM
@keynmol ready to merge?

[Gedochao]

@zielinsky I will wait for your okay before clicking merge.

[keynmol]

I think so, no doubt it will require more iterations when the work flow actually runs :)

[zielinsky]

One question from me before approval.

[zielinsky]

LGTM for now, let's test it!

[keynmol]

Workflow succeeded, and the image is published correctly, containing two platform-specific images: https://github.com/VirtusLab/scala-cli/actions/runs/20428482272/job/58694886877#step:7:1

But I think the visibility of packages needs to be changed in repository and package settings