[BUG] since 2.31.0 neworks are removed on up without reasons and fail since containers still exist

[EugenMayer]

Description

The following is broken since 2.31 (including 2.32.4), works with 2.29.7 and before

assume you have 2 docker-compose files (docker-compose.yml and docker-compose-second.yml) in the same folder (thus same COMPOSE_PROJECT), after starting the first one, and then the second one with docker compose -f docker-compose-second.yml up -d, dc fails with this error

Error response from daemon: error while removing network: network docker_default id 6660eebfb4fe91f6cdc67d1570c3cde2dbf9a2f620e77ce1b8b023123b20377f has active endpoints

since it tries to remove the network docker_default which the first one has created and using (legitimately).

I would expect, as it was before since ages, that the services within the docker-compose-second.yml just start within the same network.

Steps To Reproduce

It seems like to reproduce this, the network needs an ipam config - without it, it cannot be reproduced

put those files in one older:

docker-compose.yml

services:
 service1:
 image: busybox

networks:
 default:
 ipam:
 config:
 - subnet: 10.249.240.128/25
 ip_range: 10.249.240.128/25

docker-compose-second.yml

services:
 service2:
 image: busybox

in the folder run

docker compose up -d
docker compose -f docker-compose-second.yml up-d

You should now see

Error response from daemon: error while removing network: network dockerfoo_default id 4b55fc5521daa1341e0bda488d77eae7898a683d4af7afd882677b16814f3218 has active endpoints

Running this with docker-compose < 2.31.0 works as expected

Compose Version

2.34.4

Docker Environment

Client: Docker Engine - Community
 Version: 27.4.1
 Context: default
 Debug Mode: false
 Plugins:
 buildx: Docker Buildx (Docker Inc.)
 Version: v0.20.0
 Path: /usr/libexec/docker/cli-plugins/docker-buildx
 compose: Docker Compose (Docker Inc.)
 Version: v2.32.4
 Path: /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 28
 Running: 28
 Paused: 0
 Stopped: 0
 Images: 40
 Server Version: 27.4.1
 Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Using metacopy: false
 Native Overlay Diff: true
 userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc version: v1.2.4-0-g6c52b3f
 init version: de40ad0
 Security Options:
 apparmor
 seccomp
 Profile: builtin
 cgroupns
 Kernel Version: 6.1.0-30-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 8.519GiB
 Name: instance1
 ID: 5d254a6d-5b7f-4416-a449-2a2aae0477eb
 Docker Root Dir: /mnt/data/docker/data
 Debug Mode: false
 Experimental: false
 Insecure Registries:
 127.0.0.0/8
 Live Restore Enabled: false
 Default Address Pools:
 Base: 10.249.249.0/25, Size: 29

Anything else?

I assume, the second one, since it is not used with an overload like

docker compose -f docker-compose.yml -f docker-compose-second.yml

Does have a different 'network configuration' for the network that should be created, since the ipam definition does not exist within the docker-compose-second.yml file. In the past though, it just used the already existing network, which has been created (after running docker compose up -d thus based on docker-compose.yml.

I assume it is related to Detect network config changes and recreate if needed by @ndeloof #12267 introduced in 2.31.0 - i'am not sure this was an expected side effect though

[ndeloof]

Since 2.31, Docker Compose checks network config and actual network to match, and recreate if necessary. As you run the second compose file in isolation, which involves a "default" implicit network - without any IPAM config - compose detect this network exists but doesn't match the desired configuration (no IPAM) so tries to recreate.

If you want to run a compose application that is split into two files, you should repeat the first one when invoking the second command, so that compose merge both model and get a complete definition of the expected state:

$ docker compose up -d 
# I assume you have some reason to run only this subset at first

$ docker compose -f docker-compose.yml -f docker-compose-second.yml up-d
# Give compose a full understanding of the expected state.

[EugenMayer]

@ndeloof I understand the idea - but is it remotely clear how huge this impact is?

I mean for volumes, we have something that Is 'external' means, keep what there is, just reference it. There is something similar in network https://docs.docker.com/compose/how-tos/networking/#use-a-pre-existing-network

Is it respected, that if I set external: true. e.g

services:
 service2:
 image: busybox

networks:
 default:
 external: true

is that implemented?

[EugenMayer]

I just tested that, even though external works, it is fairly broken.

this will no work,

services:
 service2:
 image: busybox

networks:
 default:
 external: true

it will fail with network default declared as external, but could not be found

The really bad news is, that even

services:
 service2:
 image: busybox

networks:
 default:
 name: default
 external: true

wont work, since name needs to be FQDN (why so ever...) and thus only

services:
 service2:
 image: busybox

networks:
 default:
 name: dockerfoo_default
 external: true

will work, with dockerfoo being my COMPOSER_PROJECT value aka my folder name.

So while I can fix this with external, the name part is by no means as portable as the old variant was. And as far as I know there is now way to expand COMPOSER_PROJECT in the yaml file dynamically right?

[ndeloof]

While you maybe can find a workaround using it, external isn't designed for this purpose.
The main issue here is that you assume Compose can run with partial project model. This isn't true, and not desired. Not being able to detect actual network matches the desired state has been a long running bug. You should reconsider the way you run a compose application.