With containerd, I pulled images debian, ubuntu, nginx all latest, using --platform=linux/arm64/v8 and multiple strange observations.

[chetanshivaji]

Description

docker pull --platform=linux/arm64/v8 nginx:latest && docker pull --platform=linux/arm64/v8 debian:latest && docker pull --platform=linux/arm64/v8 ubuntu:latest
------------------------------------------------------------------------------------------------------------------
[root@localhost tryNested]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nginx latest d2eb56950b84 6 days ago 278MB
debian latest 27586f460943 12 days ago 383MB
ubuntu latest b359f1067efa 54 years ago 28.9MB
------------------------------------------------------------------------------------------------------------------
[root@localhost tryNested]# docker images --tree
WARNING: This is an experimental feature. The output may change and shouldn't be depended on.

IMAGE ID DISK USAGE CONTENT SIZE USED
nginx:latest d2eb56950b84 278MB 69.6MB 
├**─ linux/arm64/v8 96c43ba31637 278MB 69.6MB** 
├─ linux/amd64 396c6e925f28 0B 0B 
├─ linux/arm/v5 583c1fe635d5 0B 0B 
├─ linux/arm/v7 0c07ab67bdea 0B 0B 
├─ linux/386 1608b0773886 0B 0B 
├─ linux/mips64le 767d520bfd3e 0B 0B 
├─ linux/ppc64le 5d944fb59cfb 0B 0B 
└─ linux/s390x 708aae785214 0B 0B 

debian:latest 27586f460943 383MB 99.1MB ✔ 
**├─ linux/arm64/v8 4466c6813e1d 203MB 49.6MB 
├─ linux/amd64 e225d70fafe8 180MB 49.6MB ✔** 
├─ linux/arm/v5 baabbe93525e 0B 0B 
├─ linux/arm/v7 926de0d2617f 0B 0B 
├─ linux/386 f0d9b867cc9d 0B 0B 
├─ linux/mips64le a198b97962c6 0B 0B 
├─ linux/ppc64le 47585ee2d5a4 0B 0B 
└─ linux/s390x 5245f9d6107f 0B 0B 

**ubuntu:latest b359f1067efa 28.9MB 0B** 

------------------------------------------------------------------------------------------------------------------

[root@localhost tryNested]# docker inspect d2eb56950b84|grep -ani arch
54: "Architecture": "arm64",
[root@localhost tryNested]# docker inspect 27586f460943|grep -ani arch
38: "Architecture": "amd64",
[root@localhost tryNested]# docker inspect b359f1067efa|grep -ani arch
41: "Architecture": "arm64",
------------------------------------------------------------------------------------------------------------------

Questions
1. why ubuntu shows nothing in --tree command.
2. why arch of debian shows amd64?
4. why for debian both arch got pulled i.e. arm64 and amd64 but for nginx and ubuntu only linux/arm64/v8 got pulled?

Reproduce

.

Expected behavior

Questions
**1. why ubuntu shows nothing in --tree command.
Should show it downloaded.

2. why arch of debian shows amd64?
   It should show arm64 because we pulled arm64.

3. why for debian both arch got pulled i.e. arm64 and amd64 but for nginx and ubuntu only linux/arm64/v8 got pulled?**
   it should output in same way for all images, why it's different for different images.?

docker version

[root@localhost tryNested]# docker version
Client:
 Version: 27.3.1
 API version: 1.47
 Go version: go1.22.7
 Git commit: ce12230
 Built: Fri Sep 20 11:39:44 2024
 OS/Arch: linux/amd64
 Context: default

Server: Docker Engine - Community
 Engine:
 Version: 27.3.1
 API version: 1.47 (minimum version 1.24)
 Go version: go1.22.7
 Git commit: 41ca978
 Built: Fri Sep 20 11:41:02 2024
 OS/Arch: linux/amd64
 Experimental: false
 containerd:
 Version: v1.7.22
 GitCommit: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc:
 Version: 1.1.14
 GitCommit: v1.1.14-0-g2c9f560
 docker-init:
 Version: 0.19.0
 GitCommit: de40ad0
[root@localhost tryNested]# 

docker info

[root@localhost tryNested]# docker info
Client:
 Version: 27.3.1
 Context: default
 Debug Mode: false
 Plugins:
 buildx: Docker Buildx (Docker Inc.)
 Version: v0.14.1
 Path: /usr/libexec/docker/cli-plugins/docker-buildx
 compose: Docker Compose (Docker Inc.)
 Version: v2.27.1
 Path: /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 5
 Running: 0
 Paused: 0
 Stopped: 5
 Images: 3
 Server Version: 27.3.1
 Storage Driver: overlayfs
 driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc version: v1.1.14-0-g2c9f560
 init version: de40ad0
 Security Options:
 seccomp
 Profile: builtin
 Kernel Version: 5.4.17-2136.323.8.2.el7uek.x86_64
 Operating System: Oracle Linux Server 7.9
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.544GiB
 Name: localhost.localdomain
 ID: 6fdc46df-4874-45fd-94bd-7b4a1295ffc0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTPS Proxy: http://10.115.27.54:3128/
 No Proxy: localhost,127.0.0.1,localaddress,.localdomain.com
 Username: mchetan.more@gmail.com
 Experimental: false
 Insecure Registries:
 127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

[root@localhost tryNested]# 

Additional Info

I see same behavior on docker desktop

[chetanshivaji]

@thaJeztah do you see this same on your setup.?

[chetanshivaji]

Question 4. why we see 0B at multiple places

#docker pull debian:latest

#docker images --tree
WARNING: This is an experimental feature. The output may change and shouldn't be depended on.

IMAGE ID DISK USAGE CONTENT SIZE USED
debian:latest 27586f460943 232MB 99.1MB Γ£ö
Γö£ΓöÇ linux/arm64/v8 4466c6813e1d 49.6MB 49.6MB
Γö£ΓöÇ linux/amd64 e225d70fafe8 182MB 49.6MB Γ£ö
Γö£ΓöÇ linux/arm/v5 baabbe93525e **0B 0B**
Γö£ΓöÇ linux/arm/v7 926de0d2617f **0B 0B**
Γö£ΓöÇ linux/386 f0d9b867cc9d **0B 0B**
Γö£ΓöÇ linux/mips64le a198b97962c6 0B 0B
Γö£ΓöÇ linux/ppc64le 47585ee2d5a4 0B 0B
ΓööΓöÇ linux/s390x 5245f9d6107f 0B 0B

[thaJeztah]

I tried reproducing this issue, but didn't get the same problem;

docker pull --platform=linux/arm64/v8 nginx:latest && docker pull --platform=linux/arm64/v8 debian:latest && docker pull --platform=linux/arm64/v8 ubuntu:latest

latest: Pulling from library/nginx
14c9d9d19932: Download complete
c62fa0108604: Download complete
783efa720071: Download complete
66b98279bfb9: Download complete
e9346e20ffd2: Download complete
e0917afa5d8b: Download complete
e905ecdf6023: Download complete
Digest: sha256:d2eb56950b84efe34f966a2b92efb1a1a2ea53e7e93b94cdf45a27cf3cd47fc0
Status: Downloaded newer image for nginx:latest
docker.io/library/nginx:latest

latest: Pulling from library/debian
6d11c181ebb3: Download complete
Digest: sha256:27586f4609433f2f49a9157405b473c62c3cb28a581c413393975b4e8496d0ab
Status: Downloaded newer image for debian:latest
docker.io/library/debian:latest

latest: Pulling from library/ubuntu
25a614108e8d: Download complete
Digest: sha256:b359f1067efa76f37863778f7b6d0e8d911e3ee8efa807ad01fbf5dc1ef9006b
Status: Downloaded newer image for ubuntu:latest
docker.io/library/ubuntu:latest

docker image ls --tree
WARNING: This is an experimental feature. The output may change and shouldn't be depended on.

IMAGE ID DISK USAGE CONTENT SIZE USED
nginx:latest d2eb56950b84 281MB 69.6MB
├─ linux/arm64/v8 96c43ba31637 281MB 69.6MB
├─ linux/amd64 396c6e925f28 0B 0B
├─ linux/arm/v5 583c1fe635d5 0B 0B
├─ linux/arm/v7 0c07ab67bdea 0B 0B
├─ linux/386 1608b0773886 0B 0B
├─ linux/mips64le 767d520bfd3e 0B 0B
├─ linux/ppc64le 5d944fb59cfb 0B 0B
└─ linux/s390x 708aae785214 0B 0B

debian:latest 27586f460943 205MB 49.6MB
├─ linux/arm64/v8 4466c6813e1d 205MB 49.6MB
├─ linux/amd64 e225d70fafe8 0B 0B
├─ linux/arm/v5 baabbe93525e 0B 0B
├─ linux/arm/v7 926de0d2617f 0B 0B
├─ linux/386 f0d9b867cc9d 0B 0B
├─ linux/mips64le a198b97962c6 0B 0B
├─ linux/ppc64le 47585ee2d5a4 0B 0B
└─ linux/s390x 5245f9d6107f 0B 0B

ubuntu:latest b359f1067efa 139MB 28.9MB
├─ linux/arm64/v8 1d392dca9b70 139MB 28.9MB
├─ linux/amd64 74f92a6b3589 0B 0B
├─ linux/arm/v7 4192b2c3b431 0B 0B
├─ linux/ppc64le 294858195461 0B 0B
├─ linux/riscv64 08d0f41b17da 0B 0B
└─ linux/s390x 48459f9f4d0b 0B 0B

1. why ubuntu shows nothing in --tree command.

Not sure about that one; did an error show up during the pull? Is there information in the daemon logs that may be relevant?

I do notice some things in your setup;

• It looks like you're running on Oracle Linux; we don't have official packages for that Linux distribution; while it's possible packages built for another distro "work", it's definitely not tested.
• I see you have a proxy configured; this could be irrelevant, but does add an extra moving part where things could go wrong.

Was this the first time the ubuntu:latest image was pulled, or was the image already present before you pulled? (possibly for a different architecture)? Trying to narrow-down what could've happened (i.e., pull that appends a platform to an image that's already in the store vs a fully new multi-platform image being pulled).

2. why arch of debian shows amd64?

That is currently a limitation; various commands were designed before multi-platform images existed, and only handle a single architecture. Currently that means that (e.g.) inspect defaults to showing information for a single architecture; if multiple platforms are pulled, then it picks the "best matching" platform, which in your case is the daemon's native platform (linux/amd64).

Work is in progress to improve this;

• c8d: Add --platform flag to history, save and load #5331
• c8d/inspect: Add Manifests field moby/moby#48264

3. why for debian both arch got pulled i.e. arm64 and amd64 but for nginx and ubuntu only linux/arm64/v8 got pulled?

In both cases, the platform that was specified for the pull (linux/arm64/v8) was pulled. For the debian:latest image, you likely already had the linuxx/amd64 variant pulled before you did this pull. In that case, if the image index (the "multi-platform" image, which is the top-level "wrapper" around the platform variants) was still up-to-date, the existing variant is kept, and the newly pulled variant is added in the local image store.

Multi-platform images are effectively a "bundle" of multiple variants of an image. When you pull the image, the registry resolves the name of the image to the wrapper, after which the Docker Engine pulls the platform variant(s) of the image (by default it pulls the daemon's "native" platform/architecture). By passing the --platform flag, you can override that default, and ask it to pull an image-variant that may not match the daemon's native architecture.

Question 4. why we see 0B at multiple places

This is to indicate that the given variant of the image is not present in the daemon's image store.

For example, in the output below, the debian:latest image on Docker Hub provides 8 platform variants, but only the linux/arm64/v8 was pulled (so only that variant is present in the daemon's image store). The other variants were not pulled, so show as 0B in the output.

IMAGE ID DISK USAGE CONTENT SIZE USED
debian:latest 27586f460943 205MB 49.6MB
├─ linux/arm64/v8 4466c6813e1d 205MB 49.6MB
├─ linux/amd64 e225d70fafe8 0B 0B
├─ linux/arm/v5 baabbe93525e 0B 0B
├─ linux/arm/v7 926de0d2617f 0B 0B
├─ linux/386 f0d9b867cc9d 0B 0B
├─ linux/mips64le a198b97962c6 0B 0B
├─ linux/ppc64le 47585ee2d5a4 0B 0B
└─ linux/s390x 5245f9d6107f 0B 0B

After pulling another variant (I pulled linux/s390x), that variant will now also show up with the size consumed in the daemon's image store;

IMAGE ID DISK USAGE CONTENT SIZE USED
debian:latest 27586f460943 383MB 97.5MB
├─ linux/s390x 5245f9d6107f 178MB 47.9MB
├─ linux/arm64/v8 4466c6813e1d 205MB 49.6MB
├─ linux/amd64 e225d70fafe8 0B 0B
├─ linux/arm/v5 baabbe93525e 0B 0B
├─ linux/arm/v7 926de0d2617f 0B 0B
├─ linux/386 f0d9b867cc9d 0B 0B
├─ linux/mips64le a198b97962c6 0B 0B
└─ linux/ppc64le 47585ee2d5a4 0B 0B

The output currently shows size that's consumed locally, as that's been the information that's been relevant for most users (how much space does this image occupy on my machine?).

We may be able to show the size of variants in the registry (i.e., "how much data would be downloaded if I pull this variant"), which may be relevant for some use-cases. There's some limitations to that because the registry only holds information about the compressed size of images ("content size"), but not the size used after extracting ("disk usage").

These are things that are still being discussed though; our goal is to provide sensible defaults (information most users are looking for), which may not always be all information. Some additional information may be added as an optional / customisable option, but also through other commands. For example, we're looking at providing a docker image info command as alternative to docker image inspect, which can provide more in-depth information about an image, but in a human-readable (non-JSON) output.

☝️ feedback on the above is welcome; we're still designing some of these bits (which is also why the "WARNING: This is an experimental feature. The output may change and shouldn't be depended on." is shown).

[chetanshivaji]

@thaJeztah is there a way to pick up arch specific image using imageId and not --platform flag while doing docker save?
eg. for this image
debian:latest 27586f460943 383MB 99.1MB ✔
├─ linux/arm64/v8 4466c6813e1d 203MB 49.6MB
├─ linux/amd64 e225d70fafe8 180MB 49.6MB ✔

Is there a way to docker save using image id as below,
docker save 4466c6813e1d -o debianArm.tar
docker save e225d70fafe8 -o debianAmd64.tar

I know I can do,
docker save -o 27586f460943 -o debian.tar

[chetanshivaji]

@thaJeztah Thanks for the details above, I agree to your points, and I tried steps I mentioned in fresh Ubuntu and seems no issue.
As I have been experimenting with building multiplatform images, and pulling them from docker hub, can't remember sequence of pulling deleting of images and archs. So no track record of it.

I understand --platform feature is on they way to be released in 28.0.0
but it is not possible to save arch specific image with default containerd overlayfs coming as default enabled on Docker Desktop.
It is like overlayfs is coming default, but saving arch specific image code is yet to be released.

No other solution than building image one by one platform specific and do docker save. or wait for 28.0.0 moby.

[thaJeztah]

Is there a way to docker save using image id as below,

I need to check up on that; I agree that is something that ideally would be supported. It's possible that some additional logic is needed on the daemon side to handle resolving variants based on that). Currently, resolving images by their ID is only done for the top-level image. If no match is found, it falls back to assuming it's an image reference;

docker image save -o foo.tar 5245f9d6107f
Error response from daemon: No such image: 5245f9d6107f:latest

I did a quick check as well if it would do better with the full digest, but doesn't look like it either;

docker image save -o foo.tar 5245f9d6107f040e740721f86a1d74ca28e12dd9a073df7c5ba0490fdf394e4b
Error response from daemon: No such image: sha256:5245f9d6107f040e740721f86a1d74ca28e12dd9a073df7c5ba0490fdf394e4b

docker image save -o foo.tar sha256:5245f9d6107f040e740721f86a1d74ca28e12dd9a073df7c5ba0490fdf394e4b
Error response from daemon: No such image: sha256:5245f9d6107f040e740721f86a1d74ca28e12dd9a073df7c5ba0490fdf394e4b

docker image save -o foo.tar docker.io/library/debian@sha256:5245f9d6107f040e740721f86a1d74ca28e12dd9a073df7c5ba0490fdf394e4b
Error response from daemon: No such image: debian@sha256:5245f9d6107f040e740721f86a1d74ca28e12dd9a073df7c5ba0490fdf394e4b

docker image save -o foo.tar sha256:5245f9d6107f040e740721f86a1d74ca28e12dd9a073df7c5ba0490fdf394e4b
Error response from daemon: No such image: sha256:5245f9d6107f040e740721f86a1d74ca28e12dd9a073df7c5ba0490fdf394e4b

I should probably check if there's a tracking ticket for that already, and if not, create one.

[thaJeztah]

No other solution than building image one by one platform specific and do docker save. or wait for 28.0.0 moby.

I'm curious if there's a specific reason you need to save each variant separately; it should still be possible to save the image as a whole (multiple architectures), and load those into another engine (pushing to a registry is probably the more conventional way though).

It's worth noting that saving an individual variant of an image "dismantles" the multi-platform image, which may also discard attestations / SBOMS (if present). We currently don't show those in the tree view, but some discussion on that is happening, and a PoC pull request was opened;

• display presence of attestations in docker images --tree #5451