Extend compose files by allowing multiple files

[dnephin]

Resolves#1987

Add support for specifying multiple -f arguments to docker-compose. Multiple files are merged onto each other to allow users to customize the composition for different environments.

The default overrides file is docker-compose.override.yml which is applied over the default docker-compose.yml if it exists.

[dnephin]

Ok, I've added some basic docs to the reference, cc @moxiegirl for review

I think we'll probably want to add an example which uses this new default override file, but I hope we can work on that as a separate branch.

[moxiegirl]

Do you want to log.info if you find an override file? You might be doing this somewhere and I missed it.

[dnephin]

We don't currently log anything about which file is picked if we hit the defaults. A log.debug() might be good for debugging.

[moxiegirl]

Works for me. Troubleshooting is the exact use case I was thinking of. :-D

[dnephin]

@aanand I think I've addressed your comments

Docs are updated as well, and added debug logging for filenames

[mnowster]

I think this level of terseness effects the readability a bit too much and could do with expanding out, being a bit more verbose. However, I think that's something we can address in a separate PR and not block this one for now.

[dnephin]

I find it hard to be verbose (at least in code!) and always aim for greater density. Let me know how I can make this more readable.

[mnowster]

Overall I think this is good and great to get in before code freeze and some concerns around increasing level of verbosity is non blocking, which can be addressed outside of this PR. 🌴

LGTM.

[Vanuan]

Somehow I thought that docker-compose.override.yml would be applied even if I specified an additional file through -f. Isn't that the case?

[dnephin]

It is not the case.

The default is -f docker-compose.yml -f docker-compose.override.yml . If you specify any -f the default is not used.

[Vanuan]

The default is -f docker-compose.yml -f docker-compose.override.yml

If it were defaults, it would fail with "docker-compose.override.yml is not found"
Is there a flag to ignore the "file doesn't exist" error?

[dnephin]

I left out that bit, but yes, the override is considered optional. There is no flag to define an optional config

[greggwon]

I have a complex C++ application that needs to build with a specific set of tools and libraries. I want to perform the build inside of a container, with everything installed into system directories /usr/lib, /lib, /usr/local/bin, /usr/local/lib etc. To do the build, I mount an "external: false" volume on /usr/local/src. In the Dockerfile, I just install linux with the build tools, and then cd to /usr/local/src, and do "git clone" to download the app which has submodule dependencies for the right versions of things for each branch/tag version I have. The build works. But, I then need to have an saved container without the /usr/local/src mount (and some secrets for GitHub access) that I can then use to run the application on the target hardware/environment. How can I startup the container again, without the volume mounted nor the secrets in view?

[glours]

hey @greggwon
You should use multi-stage builds for this, a stage to build your source and a production one which only contains your final binary.
https://docs.docker.com/build/building/multi-stage/

For the secret and git credentials you can take a look to https://docs.docker.com/engine/reference/builder/#run---mounttypesecret and https://docs.docker.com/engine/reference/builder/#run---mounttypessh

[greggwon]

The problem is that I need the whole build environment because of the libraries that I am building and installing. But, I don't need the source tree I build from. I don't see how this removes the source tree from the resulting container.