Don't hard-code ports

[manics]

Hard-coded mapped ports are probably the main blocker to running multiple copies of devspace on the same host. This removes the mapping, leaving docker to dynamically map the ports.

To run multiple copies of devspace you must use the docker-compose -p, --project-name NAME flag to distinguish between multiple copies. e.g.

$ docker-compose -p altspace up
...
# Get the published jenkins web port:
$ docker-compose -p altspace port nginxjenkins 443
0.0.0.0:32805

which means altspace should be accessible on https://docker-host:32805/

See #73

[sbesson]

Tested this PR on a Docker host with a new devspace following the manual instructions in the README.
The deployment steps works as expected with all the container starting. The only undocumented step was the required modification of the various Dockerfile to use the correct USER_ID.

For the port mapping, the following services were tested successfully. For each test, the PORT was obtained by executing docker-compose -f docker-compose.yml -p $PROJECT port $SERVICE $PRIVATE_PORT:

Service	Private port	Test	Result
nginxjenkins	443	https://$DOCKERHOST:$PORT	Access to Jenkins UI
nginx	80	http://$DOCKERHOST:$PORT/web	Login via OMERO.web
omero	4064	bin/omero login root@$DOCKERHOST:$PORT	Login via OMERO.cli
omero	4064	Add $DOCKERHOST $PORT as server	Login via OMERO.insight *

• requires an encrypted connection to the server (lock icon)

The only connections which failed were:

• nginx/443 (OMERO.web / HTTPS) but this is not related to this PR
• unencrypted connection to the server

[joshmoore]

Happy to get this in (as 0.6.0?) and then look into either non-shared filesystems and/or services in a follow-up.

[jburel]

Deploying a devspace in openstack with the changes (i.e. nginxjenkins) in this PR and the current version of https://github.com/openmicroscopy/ansible-role-devspace is not working
For the time being we will have to use openstack since we cannot all use Docker host e.g. I cannot use minke

Discovered while working on the prep, blocker for now
cc @sbesson@joshmoore

[jburel]

I could add extra instruction to retrieve the dynamic port (not tested yet)
e.g. docker-compose port nginxjenkins 443
unless there is a better solution.

[jburel]

This will require more change in the role since docker-py 1.9.0 is installed and we need docker-py>=1.10.6,<2.0 in order to run the command.

[manics]

Can you run it in docker without openstack?

[joshmoore]

@jburel : can you perhaps just back up to an older version?

[manics]

@jburel@sbesson reckons this PR works on a shared Docker host. Can you outline how this doesn't fit your requirements, then we can see if there are any workarounds?

[jburel]

As mentioned previously:

• We will have to use openstack for the training since the Docker host option is not possible due to permissions. For example, I cannot deploy a devpace on minke.
• The playbook runs to completion without error but it is not possible to access the jenkins web UI as specified in the readme. Accessing the devspace via ssh then running the comment described in this PR does not work so it is difficult/impossible to have clear instruction on how to access for example https://10.0.51.135:port
• I could add my commits on top of 0.5.2 but that will defeat the purpose of a 0.6.0 for the training as discussed. Minimally the devspace version in the role needs to be bumped to 0.5.2 (currently it is at 0.4.1). This is another problem, it will lead to encoding issue w/o Bump devslave-c7 Docker image version to 0.3.1 #71
• I am going to see if I can bump the docker-py version and take it from there. If it is a case of adding a manual step to the instruction that will be fine. At the moment there are too many moving parts than I try to put together.

[sbesson]

With regard to the current production version used in the Ansible role, I think ome/ansible-role-devspace#4 bumped it to 0.5.2. I realized that we have not set up the Travis deployment so the tag was not pushed to Galaxy. I manually reimported the role for now so that https://galaxy.ansible.com/openmicroscopy/devspace/ should now be at 0.1.2.

Totally agreed on trying to limit the moving parts given the short deadline. With the latest Galaxy role, how far are we from being able to use the latest production role openmicroscopy.devspace:0.1.2 to deploy devspace 0.5.2 on Openstack for the scope of the training? If possible, we could certainly us this as a base of discussion for all the limitations of the current design (user, shared Docker host, documentation) and agree on the priorities of 0.6.0.

[jburel]

• 0.5.2 will work
• So far using key other than the snoopy key does not for some reasons I have not yet identified. Correction: I found the problem
• The instructions in the readme have been updated
• A training playbook has been prepared to simplify the set-up
  I update https://docs.google.com/document/d/1aaQIqumfrLvRfcts4YcXgNRsvNz-o-Q7Z7XucBcEYtQ/edit as I go

I will have to revert some changes now that few things we wanted to do currently do not work

[jburel]

Now that I finally solved the problem with the key. I will check if I can upgrade docker-py allowing us to use determine the port as described in this PR.

[jburel]

I made the modification to the role it order to be able to run
docker-compose port nginxjenkins 443

The output is:

WARNING: The JENKINS_PASSWORD variable is not set. Defaulting to a blank string.
WARNING: The USER_ID variable is not set. Defaulting to a blank string.
0.0.0.0:32840

This will not be useful in the context of devspace in openstack since https://DEVSPACE_IP:32840
does not give access to the Jenkins UI

I will have to either add my commits (installation update and removal of snoopy key usage) on top of 0.5.2 or remove this PR for now and review post training. The first option is not ideal since people attending the training will have to work of my branch and not the new tag.

@sbesson changes on top of 0.5.2 are valid and work (#72 and #77). #77 is a useful PR in the context of the training

I have not tested but I reckon I will have similar issue with connection via web/insight etc.

[manics]

This will not be useful in the context of devspace in openstack since https://DEVSPACE_IP:32840
does not give access to the Jenkins UI

What ports are allowed by the security groups on the instance?

[jburel]

(devspace) ls30569:ansible jmarie$ openstack security group list
+--------------------------------------+--------------------------+--------------------------------------------------------+----------------------------------+
| ID | Name | Description | Project |
+--------------------------------------+--------------------------+--------------------------------------------------------+----------------------------------+
| 06578e77-24a0-4565-b09d-69d53b3358f3 | default | Default security group | 5db80a7920424cef8eb6f0eba86cd078 |
| 52d56fca-28e3-40e7-bb57-80bed3b91032 | all | Allow all input/output | 5db80a7920424cef8eb6f0eba86cd078 |
| c562cbc7-4cbf-40d1-94f8-f2d7e2147cbd | docker-devspace-external | External access to Docker servers (managed by Ansible) | 5db80a7920424cef8eb6f0eba86cd078 |
| cad43cdf-6905-41e5-ac49-54179e4c99ad | docker-external | External access to Docker servers (managed by Ansible) | 5db80a7920424cef8eb6f0eba86cd078 |
| fd01cbe6-fe60-45f3-ac4f-93d164b59d67 | ssh | External SSH from anywhere | 5db80a7920424cef8eb6f0eba86cd078 |
+--------------------------------------+--------------------------+--------------------------------------------------------+----------------------------------+
(devspace) ls30569:ansible jmarie$ openstack security group rule list docker-devspace-external
+--------------------------------------+-------------+-----------+-------------+-----------------------+
| ID | IP Protocol | IP Range | Port Range | Remote Security Group |
+--------------------------------------+-------------+-----------+-------------+-----------------------+
| 065f8075-1866-47c4-801d-a66d912bafc0 | tcp | 0.0.0.0/0 | 14063:14063 | None |
| 088dad15-30f9-4742-852d-04b5e643497f | tcp | 0.0.0.0/0 | 80:80 | None |
| 37c4a580-6535-4cc9-9164-f7f32befab59 | None | None | | None |
| 482b5994-4311-41d6-9181-d30d605c58a1 | tcp | 0.0.0.0/0 | 8443:8443 | None |
| 53be806a-ff9f-4725-9dae-14b21b46968b | tcp | 0.0.0.0/0 | 4243:4243 | None |
| 65e05c42-c462-4844-a214-0fa533a1840f | tcp | 0.0.0.0/0 | 8080:8080 | None |
| 80f78484-5c13-4f3f-82c4-2592a1477b94 | tcp | 0.0.0.0/0 | 4064:4064 | None |
| 88fab42f-7b6e-492b-9463-c38692534f05 | tcp | 0.0.0.0/0 | 4444:4444 | None |
| bf15cd25-02ec-4184-a77c-7d8265ef4f2b | tcp | 0.0.0.0/0 | 14064:14064 | None |
| c02e523c-41e6-4d41-a9b6-9e38014f0750 | tcp | 0.0.0.0/0 | 443:443 | None |
| f6faaa20-c6ef-45c2-89d7-1e52c70c8e60 | tcp | 0.0.0.0/0 | 4063:4063 | None |
| fbdc7f7e-a6f8-4871-b523-885e0f0b8aed | tcp | 0.0.0.0/0 | 22:22 | None |
| fc2792d9-72f6-4af4-bdc4-44d750c65c49 | None | None | | None |
+--------------------------------------+-------------+-----------+-------------+-----------------------+
(devspace) ls30569:ansible jmarie$ openstack security group rule list docker-external
+--------------------------------------+-------------+-----------+------------+-----------------------+
| ID | IP Protocol | IP Range | Port Range | Remote Security Group |
+--------------------------------------+-------------+-----------+------------+-----------------------+
| 352207ad-c331-46b4-a316-7f5760621f94 | tcp | 0.0.0.0/0 | 8008:8008 | None |
| 5c0270e8-1819-4241-adda-68a2be127244 | tcp | 0.0.0.0/0 | 443:443 | None |
| 7b01e1f2-ae00-4191-83fe-33cfd853de6c | tcp | 0.0.0.0/0 | 80:80 | None |
| a608322a-d932-4f97-ad3c-941285996ff3 | tcp | 0.0.0.0/0 | 4063:4063 | None |
| a8739fe8-a029-4509-aeb8-6b6c0b30b797 | None | None | | None |
| a879d846-91be-4c02-bb0b-c6e72f167be2 | tcp | 0.0.0.0/0 | 22:22 | None |
| c6744b97-04fa-4e12-8c36-da443ca1dd34 | tcp | 0.0.0.0/0 | 4064:4064 | None |
| cd264f20-43e3-4636-871e-6def98e88121 | None | None | | None |
| e6d1a2f6-f803-4848-a547-7d5d5ef0ca62 | tcp | 0.0.0.0/0 | 8888:8888 | None |
+--------------------------------------+-------------+-----------+------------+-----------------------+
(devspace) ls30569:ansible jmarie$ openstack security group rule list ssh
+--------------------------------------+-------------+-----------+------------+-----------------------+
| ID | IP Protocol | IP Range | Port Range | Remote Security Group |
+--------------------------------------+-------------+-----------+------------+-----------------------+
| 689ebc81-de15-422f-ab2d-7cc5d949ef8f | tcp | 0.0.0.0/0 | 22:22 | None |
| a03f07a1-a04f-4653-a106-3718848bf4b3 | None | None | | None |
| ea1faf04-2762-4148-a196-626fa9896f6d | None | None | | None |
+--------------------------------------+-------------+-----------+------------+-----------------------+

[manics]

You can either add the dynamically assigned port to one of the rules, or apply the all security group to the instance.

[jburel]

I modified the playbook used to create the instance and apply the all security group.
After checking the dynamically assigned port, I was then able to access the jenkins UI
I have not tried the rest
I hope to have everything in place for early next week

[jburel]

It looks that there is a problem to access server via java
I can create a session via python but not java
this prevents any import for example

[joshmoore]

This is likely the use of createClient to create an unsecure connection.

[jburel]

@joshmoore you are correct I can connect now if click the secure option in insight.
Is there an option via cli to do so for an import? I don't think there is.
I will add that to the README