Proposal: DNS service discovery in Docker to replace /etc/hosts

[sanimej]

Service discovery in Docker currently relies on updating the container's /etc/hosts file. This resulted in issues like corrupted /etc/hosts file in some cases #17190. Its also not a scalable approach.

This proposal outlines the design for a DNS based service discovery in Docker with the following goals..

• replace the /etc/hosts baed approach as seamlessly as possible
• handle the name resolution in docker(libnetwork) without...
  1. duplicating the host/endpoint/IP info elsewhere
  2. requiring changes to network drivers.

Going forward we will support a pluggable DNS model (similar to IPAM). But its not covered in this proposal.

Docker daemon will act as a embedded DNS server for all the containers in a host. From the container point of view nameserver will be available at the loopback address. When the container is launched the loopback address will be the only nameserver populated in /etc/resolv.conf file. DNS queries from the container will be handled by the Docker process.

Short name (ie: only container name) resolutions will be done in the scope of the networks to which the container is connected to. For a qualified name container_name.network_name resolution will be done only if the container is attached to that network.

If --dns or the dns optoins were passed in the docker run command, DNS queries will be forwarded to those external servers if the name can't be resolved by the embedded DNS server.

The embedded DNS server will be enabled for all user created networks. /etc/hosts file will not be updated for containers on those networks. Only exception is the
containers in the default bridge network (docker0 bridge) with the link option which will continue to use /etc/hosts file. This is to provide backward compatibility for apps relying on that behavior.

The embedded DNS server in Docker is to resolve only the service names offered by Docker containers (local or across hosts). It can't replace any other local or non-local DNS servers used to resolve other services/host names in the cluster.

[mavenugo]

ping @phemmer

[dave-tucker]

👍
Only one question, what if a container has a name of host.example.com, does this hit the embedded DNS server first or is it forwarded to the external servers?

[phemmer]

From the container point of view nameserver will be available at the loopback address

I strongly advise against this. By implementing this behavior, you are preventing users from running DNS servers within docker.

Short name (ie: only container name) resolutions will be done in the scope of the networks to which the container is connected to.

I think this should be avoided as well. If you want the container to search multiple networks, adjust the resolv.conf to search those networks.

If --dns or the dns optoins were passed in the docker run command, DNS queries will be forwarded to those external servers if the name can't be resolved by the embedded DNS server.

I think it might be better to instead forward all and only queries which are not part of a docker "network_name" domain.

 

The 2nd & 3rd items are actually related. In a nutshell, both boil down to the idea that docker provided DNS records should live within their own domain. Meaning the full DNS record for a container would be something like e15fb5654664.docker.local or e15fb5654664.user-network. This way we're sticking with standard DNS design that a single domain does not have multiple authoritative servers with different results.

[aidanhs]

From the container point of view nameserver will be available at the loopback address.

As @phemmer says, this isn't ok. The same thing was said on the docker-dev list - https://groups.google.com/d/msg/docker-dev/WXkMiPJqh7I/onY0k7yLAwAJ

[sanimej]

@phemmer @aidanhs Yes, @thockin brought up the same issue. Though the embedded dns server will bind only to loopback:53 it will break cases where a DNS server is doing *:53 bind in the container. We will look at the possibilities to avoid this issue.

[mariusGundersen]

Couple of (maybe stupid) questions I have:

• what would things TTL of the DNS responses be? There might be performance issue if it's too low, or is that not something to worry about?
• will the DNS service use the hostnameor the container name? Using container names means that it has to be globally unique, not just network unique, while hostname means it can't change.
• how long until this is plugin-able? I, and it sounds like many others, have some specific scenarios that we want to support, and having non-replacable batteries makes it worse than it is today.

[sanimej]

@mariusGundersen

what would things TTL of the DNS responses be? There might be performance issue if it's too low, or is that not something to worry about?

For the Address records it can be 1 hour. We can even go lower because this is only for the records in the namespace of docker networks; Hence its going to be a local lookup.

will the DNS service use the hostnameor the container name? Using container names means that it has to be globally unique, not just network unique, while hostname means it can't change.

Today the service discovery is based on the container name passed via --name option. The same behavior will continue for DNS service discovery as well.

how long until this is plugin-able? I, and it sounds like many others, have some specific scenarios that we want to support, and having non-replacable batteries makes it worse than it is today.

The intent is to seamlessly replace the current /etc/hosts based service discovery. So the user experience will not change. Pluggable DNS will be added after 1.10 release.

[sanimej]

@phemmer

If --dns or the dns optoins were passed in the docker run command, DNS queries will be forwarded to those external servers if the name can't be resolved by the embedded DNS server.

I think it might be better to instead forward all and only queries which are not part of a docker "network_name" domain.

Yes, that is what I implied because the docker DNS server can only resolve the names in the domain of docker networks. I will reword it to be explicit.

[dave-tucker]

@sanimej

Today the service discovery is based on the container name passed via --name option. The same behavior will continue for DNS service discovery as well.

That would be a mistake. We should have never confused container name and hostname. We should fix that behaviour in 1.10:

• --name is supposed to be a friendly name that can be used instead of the container ID with no significance inside the container
• --hostname the containers network name

I know why we made the choice we did (--link blurring the lines by using the --name in the link) but it's very wrong. I already met a few early adopters that were hitting collisions at DockerCon.

[mavenugo]

@dave-tucker @sanimej lets have all the discussions around --name, --link, --hostname or alias in #737. This proposal is to migrate the SD from /etc/hosts to DNS based solution. The actual content that goes inside should be part of #737 discussion.

[fermayo]

+1 to replace updating /etc/hosts with an embedded DNS server.

Does the engine have an IP on all networks that it creates? Couldn't we just use that IP as the nameserver set on the /etc/resolv.conf?
Regarding the scoping of the resolution: if each network is a DNS domain, can't we just have all container's network names as search domains so short hostname resolution works out of the box? Or at least provide a default domain for the containers so they don't need to know any network names.