add port range support to docker compose

[yuval-k]

fixes#1102
tests passed

[aanand]

Thanks for contributing!

• Could you add some integration tests?
• Could any of this logic be moved into docker-py?

[dnephin]

Can anyone confirm this needs to be handled by the client?

It seems like it should be handled by dockerd. I can't find any reference to it in the remote api docs.

[yuval-k]

When I looked at the docker API, the data structure that I saw there only contained one port, that's why I chose to implement it on the client.
see ExposedPorts at https://docs.docker.com/reference/api/docker_remote_api_v1.17/

[yuval-k]

Also looking at the docker client source code, you can also see that the ports are used individually.
see runconfig/parse.go line 229-238 (the line with ParsePortRange)

[yuval-k]

As for integration tests - I can add those later tonight, but let's first decide if the general idea is acceptable. i.e. it should be done here, vs the docker server and docker-py

[aanand]

Looks like Docker does indeed do it client-side, so that's where we'll need to do it too. Moving it to the server is a discussion to be had on docker/docker.

Paging @shin-: does processing of port ranges belong in docker-py, in your opinion?

[shin-]

Hmm, possibly. But probably as a function in the tools module rather than inside the API methods.

[aanand]

@shin- Thanks, I agree - I think we can move build_port_bindings and its dependencies (to_port_range, split_port, add_port_mapping) all over to docker.utils, perhaps as a submodule (docker.utils.ports).

@uvgroovy Could you submit a PR to docker-py?

[yuval-k]

@aanand So basically move build_port_bindings and deps to docker.utils.ports and then import it in compose/service.py?

I can do that, just one question -
doing that will temporary break the code, unless requirments-dev is updated.
so basically, i think we need to:

1. add build_port_bindings to docker-py
2. release new docker-py
3. remove build_port_bindings from compose/service.py, import it from docker-py and update requirements-dev.txt to the new docker-py version

sounds good? any comments\feedback?

[aanand]

I think it should go:

1. Submit PR to docker-py
2. Update this PR with the code changes and set the docker-py requirement in requirements.txt to point at your repo and branch with a git URL (see the pyinstaller requirement in requirements-dev.txt for an example)
3. When the docker-py PR is merged, update the git URL to point at the docker-py merge commit
4. When the next docker-py is released, update requirements.txt to the stable version
5. Merge this PR

[yuval-k]

sounds good.

[dougborg]

Hey @uvgroovy, just wanted to check in with you to see if this was still moving forward. Do you need any help with testing, etc.? Let me know. I'd be glad to help any way I can.

[yuval-k]

Hi, Sorry for the delay, jewish holidays are slowing me down...
I got the changes to my branch in the docker-py repository, and now just trying to verify that it works with compose, and then i'll submit a PR for docker-py

[yuval-k]

I updated the version of requests to version 2.5.2 to match the one required from docker-py;

but now, I encounter this error:
ImportError: No module named 'requests.packages.chardet.sys'
see:
http://stackoverflow.com/questions/28775276/pyinstaller-error-importerror-no-module-named-requests-packages-chardet-sys

@dougborg \ @aanand would love for some advice :)

[dougborg]

I tried to reproduce this error on my local development environment with your branch, but was unable to. Can you provide any more details as to what your setup looks like and how you are invoking your tests?

I am running:

OSX Yosemite 10.10.2
docker-machine version 0.2.0 (48bea37)
docker-compose 1.1.0 (ee64e0d)
docker 1.6-rc4

[yuval-k]

So basically, When I tried to build compose with the docker-py repo directly. this is my requirements.txt:

PyYAML==3.10
git+https://github.com/uvgroovy/docker-py.git@8028c988ce363345988bc23444f08e27394a7fb5#egg=docker-py
dockerpty==0.3.2
docopt==0.6.1
requests==2.2.1
six==1.7.3
texttable==0.8.2
websocket-client==0.11.0

it complained that requests 2.2.1 is too old:

./script/test
...
Installed /usr/local/lib/python2.7/dist-packages/docker_compose-1.1.0-py2.7.egg
Processing dependencies for docker-compose==1.1.0
error: Installed distribution requests 2.2.1 conflicts with requirement requests>=2.5.2
...

so I changed the requirements.txt for compose in my branch to contain requests 2.5.2 and got the error i mentioned.

 ImportError: No module named 'requests.packages.chardet.sys'

[awsmsrc]

hey guys whats happening with this?
Anything I can do to help?

[aanand]

Same issue as #1075 and #1269 - we're waiting for an upstream fix in requests (https://github.com/kennethreitz/requests/pull/2533) to make it into a release.

[yuval-k]

@aanand , perhaps can you assist with merging the docker-py PR?
docker/docker-py#544

[yuval-k]

requests 2.6 was released with a fix to this issue. updating both docker-py and compose to 2.6.2 seems to be working (tried it in my own laptop and the tests pass) - but here jenkins is specifically looking for requests>=2.2.1,<2.6
and as a result uses
Best match: requests 2.5.3

I couldn't find a reference to requests "<2.6" in the repo it self.

Any idea why that happens\how to fix this?

[aanand]

Sorry I missed this. Needs rebasing on master, then it should pick up the new requests version range.

[yuval-k]

i re-did my commit on the master from upstream compose. all should be well now

[aanm]

How will this behave if we scale containers that are greater than the number of ports available?

[yuval-k]

@aanm not sure what you are asking. this PR just allows you to specify multiple ports in the configuration. it is as if you wrote them individually (i.e. the code 'translates' the port range to multiple single ports for the rest of the program).

[aanm]

@uvgroovy Currently, compose sends a user error if you try to scale a container that has ports defined. For example if you have port: 8888 and you want to scale that service to 2, it won't work because the compose.yml only has one port defined. What I was asking is, if the same occurs if you have port: 8888-8889 and you try to scale that service to 3.

[aanand]

@aanm The same will happen, yes.

It would be nice to be able to specify a set of ports where each one is assigned to a single container in the service, but it's not what the Docker CLI's port range syntax means, so we'd need to come up with an alternate way of specifying it.

[aanm]

@aanand I think it would also be nice to NOT throw an error on the case I was talking about, either with a range or a single port specification. I think it would be better to just throw a warning. I'm saying this because compose + swarm fail if we try to scale up with a single port defined.

[dnephin]

Replacing the error with a warning seems appropriate now that swarm makes that case possible.

[aanm]

@dnephin look at #1466

[yuval-k]

@aanand re-did the commit to the updated compose master.

[aanand]

Thanks!

LGTM

[aanand]

Strike that - still needs docs.

[yuval-k]

@aanand sorry for the delay (i missed your last comment) - I added documentation to yml.md - please let me know if that works.

Also, while it seems that tests failed, it actually looks like an IT issue:
"unable to access 'https://github.com/docker/compose.git/': server certificate verification failed"
any way to re-run the tests?

[aanand]

ping @moxiegirl

My take: this seems to suggest on first reading that you can specify a range of container ports but a single host port, which I don't think is correct. Perhaps:

---

Expose ports. You can specify just a container port (which will be mapped to a random host port), or both (HOST:CONTAINER).

You can optionally pass in ranges of port numbers. If you're specifying host ports as well as container ports, the host and container port ranges must be the same size.

[yangxuesong]

the lastest release 1.4.0rc2 still not support port range.
when will it be available?

[mnowster]

Merged in #1827