docker-ce package makes existing KVM guests unusable

[TheCasualObserver]

• This is a bug report
• This is a feature request
• I searched existing issues for the keyword 'KVM' and 'QEMU' before opening this one

Expected behavior

The act of installing the package 'docker-ce' should not interfere with existing KVM-guests and make them unable to access the network on Debian Buster.

Actual behavior

Installing the package 'docker-ce' causes, that KVM guests can no longer access the network.
In some cases a KVM guest can immediately after the installation no longer access the networt. After rebooting any KVM-guest, this KVM guest has no longer access to the network.

After deinstallation of the package 'docker-ce' and rebooting the system, all KVM guests can access the networt again.

Steps to reproduce the behavior

Prerequisite: Debian Buster 64 bit host with KVM-QEMU debian guests

1. Follow the chapter "Install using the repository" on https://docs.docker.com/install/linux/docker-ce/debian/ and install the packages 'docker-ce-cli' and 'containerd.io' (after this step the KVM guests work without problems.)
2. install 'docker-ce' (after this step KVM guests can no longer access the network)

Output of docker version:

:~# docker version
Client: Docker Engine - Community
 Version: 19.03.6
 API version: 1.40
 Go version: go1.12.16
 Git commit: 369ce74a3c
 Built: Thu Feb 13 01:27:58 2020
 OS/Arch: linux/amd64
 Experimental: false

Server: Docker Engine - Community
 Engine:
 Version: 19.03.6
 API version: 1.40 (minimum version 1.12)
 Go version: go1.12.16
 Git commit: 369ce74a3c
 Built: Thu Feb 13 01:26:32 2020
 OS/Arch: linux/amd64
 Experimental: false
 containerd:
 Version: 1.2.13
 GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
 Version: 1.0.0-rc10
 GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
 Version: 0.18.0
 GitCommit: fec3683

Output of docker info:

:~# docker info
Client:
 Debug Mode: false

Server:
 Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
 Images: 0
 Server Version: 19.03.6
 Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
 apparmor
 seccomp
 Profile: default
 Kernel Version: 4.19.0-8-amd64
 Operating System: Debian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 31.13GiB
 Name: REDACTED
 ID: FKUI:EV5T:GBFV:LJYK:UNRV:GDKL:I5DO:3BAF:KDRI:L3XY:NWVN:EBLN
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
 127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.)

The host is a physical 64-bit system.

:~# kvm --version
QEMU emulator version 3.1.0 (Debian 1:3.1+dfsg-8+deb10u4)
Copyright (c) 2003-2018 Fabrice Bellard and the QEMU Project developers

Thank you for having a look at this issue.

[cpuguy83]

It sounds likely that the docker bridge network is conflicting with your kvm net? Docker tries to pick an unused address space. I'm not sure I've seen it pick one that was in use before only error out because it can't find one.

What you might want to do is either:

1. Specify a custom subnet for the bridge (daemon option --bip=<bridge ip>/<subnet>)
2. Specify a custom bridge (daemon option --bridge=myBridge)
3. Disable the bridge (daemon option --bridge=none)... you may also want to disable iptables management if you are going to do this --iptables=false

[TheCasualObserver]

Thank you for taking the time to compile these hints. However they do not help in my situation, because the problem happens during the installation of the 'docker-ce' package - long before I have an opportunity to configure the docker daemon options.

Today I found a workaround which allows me to install 'docker-ce' without breaking my KVM guests.
I changed /etc/sysctl.conf and added net.ipv4.ip_forward=1.

[cpuguy83]

Ah yes... libnetwork messes with that...
It used to enable it explicitly... then it disabled it explicitly... I don't know why it even touches it.

[cpuguy83]

@arkodg WDYT? Can libnetwork just not change net.ipv4.ip_forward?

[arkodg]

@cpuguy83 I don't understand how libnetwork get called when you install docker-ce
In 19.03.6 when dockerd is spawned this https://github.com/moby/libnetwork/blob/74191015cdfdfae1c1774c07e1c46e590d4adc13/drivers/bridge/setup_ip_forwarding.go#L24 gets called, and it looks like it does the following

1. if Ipv4 forwarding is disabled, it enables it
2. if --iptables=false it bails out
3. else it attempts to add a iptables default drop policy for the forward chain
4. if that fails it disables Ipv4 forwarding

Step 4 might not be ideal, we might want to reset it the original value that was read

Is this what is happening ?

@TheCasualObserver
do you see this error (https://github.com/moby/libnetwork/blob/5eb06eb42dec3cb0586ea7ce39d7829580e712f2/iptables/iptables.go#L520) in your dockerd logs

[cpuguy83]

Because, as I recall, dockerd is started on install.

I think actually in the past we used to set the forward policy to accept, and now we set it to drop.

[thaJeztah]

I think that was changed, related to moby/moby#14041 (in moby/libnetwork#1526)

[arkodg]

@cpuguy83 that change you are referring to is only on master (moby/libnetwork#2450), not in 19.03 :)

[wyphan]

Sorry for bumping up an old issue, but I can confirm that this also happens with Ubuntu 20.04.1 LTS (focal) amd64 host. And the workaround described above still works.

[nazar-pc]

I have the same issue after starting Docker at system boot in Alpine Linux (docker package from their official repos), but not when starting after.

Interestingly, default policy for FORWARD changes in that case 🤔

When booting without Docker installed, iptables rules look like this:

Details

Chain INPUT (policy ACCEPT)
target prot opt source destination 
LIBVIRT_INP all -- anywhere anywhere 

Chain FORWARD (policy ACCEPT)
target prot opt source destination 
LIBVIRT_FWX all -- anywhere anywhere 
LIBVIRT_FWI all -- anywhere anywhere 
LIBVIRT_FWO all -- anywhere anywhere 

Chain OUTPUT (policy ACCEPT)
target prot opt source destination 
LIBVIRT_OUT all -- anywhere anywhere 

Chain LIBVIRT_FWI (1 references)
target prot opt source destination 
ACCEPT all -- anywhere 192.168.122.0/24 ctstate RELATED,ESTABLISHED
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable

Chain LIBVIRT_FWO (1 references)
target prot opt source destination 
ACCEPT all -- 192.168.122.0/24 anywhere 
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable

Chain LIBVIRT_FWX (1 references)
target prot opt source destination 
ACCEPT all -- anywhere anywhere 

Chain LIBVIRT_INP (1 references)
target prot opt source destination 
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootps
ACCEPT tcp -- anywhere anywhere tcp dpt:67

Chain LIBVIRT_OUT (1 references)
target prot opt source destination 
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootpc
ACCEPT tcp -- anywhere anywhere tcp dpt:68

If I start Docker afterwards, it is updated to the following:

Details

Chain INPUT (policy ACCEPT)
target prot opt source destination 
LIBVIRT_INP all -- anywhere anywhere 

Chain FORWARD (policy ACCEPT)
target prot opt source destination 
DOCKER-USER all -- anywhere anywhere 
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere 
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere 
ACCEPT all -- anywhere anywhere 
ACCEPT all -- anywhere anywhere 
LIBVIRT_FWX all -- anywhere anywhere 
LIBVIRT_FWI all -- anywhere anywhere 
LIBVIRT_FWO all -- anywhere anywhere 

Chain OUTPUT (policy ACCEPT)
target prot opt source destination 
LIBVIRT_OUT all -- anywhere anywhere 

Chain DOCKER (1 references)
target prot opt source destination 

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination 
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere 
RETURN all -- anywhere anywhere 

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination 
DROP all -- anywhere anywhere 
RETURN all -- anywhere anywhere 

Chain DOCKER-USER (1 references)
target prot opt source destination 
RETURN all -- anywhere anywhere 

Chain LIBVIRT_FWI (1 references)
target prot opt source destination 
ACCEPT all -- anywhere 192.168.122.0/24 ctstate RELATED,ESTABLISHED
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable

Chain LIBVIRT_FWO (1 references)
target prot opt source destination 
ACCEPT all -- 192.168.122.0/24 anywhere 
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable

Chain LIBVIRT_FWX (1 references)
target prot opt source destination 
ACCEPT all -- anywhere anywhere 

Chain LIBVIRT_INP (1 references)
target prot opt source destination 
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootps
ACCEPT tcp -- anywhere anywhere tcp dpt:67

Chain LIBVIRT_OUT (1 references)
target prot opt source destination 
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootpc
ACCEPT tcp -- anywhere anywhere tcp dpt:68

However, if Docker was enabled from the start (and before libvirt because it is alphabetically first on Alpine Linux), then I get this and libvirt is broken:

Details

Chain INPUT (policy ACCEPT)
target prot opt source destination 
LIBVIRT_INP all -- anywhere anywhere 

Chain FORWARD (policy DROP)
target prot opt source destination 
LIBVIRT_FWX all -- anywhere anywhere 
LIBVIRT_FWI all -- anywhere anywhere 
LIBVIRT_FWO all -- anywhere anywhere 
DOCKER-USER all -- anywhere anywhere 
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere 
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere 
ACCEPT all -- anywhere anywhere 
ACCEPT all -- anywhere anywhere 

Chain OUTPUT (policy ACCEPT)
target prot opt source destination 
LIBVIRT_OUT all -- anywhere anywhere 

Chain DOCKER (1 references)
target prot opt source destination 

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination 
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere 
RETURN all -- anywhere anywhere 

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination 
DROP all -- anywhere anywhere 
RETURN all -- anywhere anywhere 

Chain DOCKER-USER (1 references)
target prot opt source destination 
RETURN all -- anywhere anywhere 

Chain LIBVIRT_FWI (1 references)
target prot opt source destination 
ACCEPT all -- anywhere 192.168.122.0/24 ctstate RELATED,ESTABLISHED
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable

Chain LIBVIRT_FWO (1 references)
target prot opt source destination 
ACCEPT all -- 192.168.122.0/24 anywhere 
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable

Chain LIBVIRT_FWX (1 references)
target prot opt source destination 
ACCEPT all -- anywhere anywhere 

Chain LIBVIRT_INP (1 references)
target prot opt source destination 
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootps
ACCEPT tcp -- anywhere anywhere tcp dpt:67

Chain LIBVIRT_OUT (1 references)
target prot opt source destination 
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootpc
ACCEPT tcp -- anywhere anywhere tcp dpt:68

I'm wondering why this happens and whether it works properly with default policy ACCEPT as well? I do not know much about networking, but would be surprised if it did.