Bug: rootless docker-in-docker in Gitlab CI can't find containers

[Mjb141]

Describe the bug

Trying to start containers in rootless docker-in-docker running on Gitlab self-hosted runners results in docker.errors.NotFound: 404 Client Error for https://docker:2376/v1.44/containers/fa2ce8ad268341b04006af38e93bd4ce9873df0de7f1686ffc3eba63558a5c9f/json: Not Found ("No such container: fa2ce8ad268341b04006af38e93bd4ce9873df0de7f1686ffc3eba63558a5c9f")

To Reproduce

• Gitlab self-hosted runner
• Docker autoscaler executor
• Rootless docker-in-docker
• cp -r /certs/client/* /root/.docker because the Docker dependency doesn't respect DOCKER_TLS_CERTDIR and tcp://docker:2376 fails without certs in this dir (got this from here: https://github.com/testcontainers/testcontainers-python/blob/main/core/tests/test_docker_in_docker.py#L221)
• Run pytest that uses testcontainers

State:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0083e6a1e9f6 47494f49e760 "dockerd-entrypoint.…" 6 minutes ago Up 6 minutes 2375-2376/tcp <long_container_name_ending_in>-build
60dba14e1956 4f1ab7591b83 "dockerd-entrypoint.…" 6 minutes ago Up 6 minutes 2375-2376/tcp <long_container_name_ending_in>__dind-rootless-0

Local:

test_everything.py::test_get_all_customers PASSED [ 50%]
test_everything.py::test_get_customer_by_email PASSED [100%]

Remote:

`docker.errors.NotFound: 404 Client Error for https://docker:2376/v1.44/containers/fa2ce8ad268341b04006af38e93bd4ce9873df0de7f1686ffc3eba63558a5c9f/json: Not Found ("No such container: fa2ce8ad268341b04006af38e93bd4ce9873df0de7f1686ffc3eba63558a5c9f")`

Runtime environment

Provide a summary of your runtime environment. Which operating system, python version, and docker version are you using? What is the version of testcontainers-python you are using? You can run the following commands to get the relevant information.

# Get the operating system information (on a unix os).
$ uname -a
# Get the python version.
$ python --version
# Get the docker version and other docker information.
$ docker info
# Get all python packages.
$ pip freeze

amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Nov 5 17:36:57 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

python 3.11.11 in build container, requires_python = ">=3.11,<3.12" in pyproject.toml

Docker info:

Worker instance:

Client:
 Version: 25.0.5
 Context: default
 Debug Mode: false
 Plugins:
 buildx: Docker Buildx (Docker Inc.)
 Version: v0.0.0+unknown
 Path: /usr/libexec/docker/cli-plugins/docker-buildx

Server:
 Containers: 4
 Running: 3
 Paused: 0
 Stopped: 1
 Images: 5
 Server Version: 25.0.6
 Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Using metacopy: false
 Native Overlay Diff: true
 userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 57f17b0a6295a39009d861b89e3b3b87b005ca27
 runc version: 2c9f5602f0ba3d9da1c2596322dfc4e156844890
 init version: de40ad0
 Security Options:
 seccomp
 Profile: builtin
 cgroupns
 Kernel Version: 6.1.115-126.197.amzn2023.x86_64
 Operating System: Amazon Linux 2023.6.20241121
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 30.64GiB
 Name: ip-10-121-36-254.eu-central-1.compute.internal
 ID: 6ec60948-bbef-4c17-96de-7bee4fab9940
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
 127.0.0.0/8
 Live Restore Enabled: false

Build container:

Client:
 Version: 25.0.5
 Context: default
 Debug Mode: false
 Plugins:
 buildx: Docker Buildx (Docker Inc.)
 Version: v0.14.0
 Path: /usr/local/libexec/docker/cli-plugins/docker-buildx
 compose: Docker Compose (Docker Inc.)
 Version: v2.27.0
 Path: /usr/local/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
 Images: 1
 Server Version: 25.0.5
 Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Using metacopy: false
 Native Overlay Diff: false
 userxattr: true
 Logging Driver: json-file
 Cgroup Driver: none
 Cgroup Version: 2
 Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7c3aca7a610df76212171d200ca3811ff6096eb8
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
 seccomp
 Profile: builtin
 rootless
 cgroupns
 Kernel Version: 6.1.115-126.197.amzn2023.x86_64
 Operating System: Alpine Linux v3.19
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 30.64GiB
 Name: 60dba14e1956
 ID: 184c448b-2a74-4591-8333-b60346a901bf
 Docker Root Dir: /home/rootless/.local/share/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
 127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: Running in rootless-mode without cgroups. Systemd is required to enable cgroups in rootless-mode.

Dependencies for job:

"psycopg2-binary>=2.9.10",
"pytest>=8.3.4",
"testcontainers[postgres]>=4.8.2",

@alexanderankin Perhaps one for you given #517?

[Mjb141]

I seem to have found a workaround in #537 (comment), TESTCONTAINERS_RYUK_DISABLED: true in my gitlab job works. I'll use this for now, disabling Ryuk isn't a concern on ephemeral workers.

[CarliJoy]

Are you using rootless docker on the worker?
If yes this was fixed with #779

If yes this issues is a duplicate of #537

Once a new version is released (see #829) please retry.