Fix broken swarm commands with Kubernetes defined as orchestrator by silvin-lubecki · Pull Request #1137 · docker/cli

silvin-lubecki · 2018-06-21T16:58:13Z

- What I did

All swarm commands (node, service, etc...) were erroring out if the orchestrator was defined as Kubernetes. I moved the orchestrator flag, the environment variable and the config file value to more locally values regarding stack command.

- How I did it

Renamed DOCKER_ORCHESTRATOR to DOCKER_STACK_ORCHESTRATOR
Renamed config file option "orchestrator" to "stackOrchestrator"
"--orchestrator" flag is no more global but local to stack command and subcommands
Cleaned all global orchestrator code
Replicated Hidden flags in help and Supported flags from root command to stack command

- How to verify it

$ DOCKER_STACK_ORCHESTRATOR=kubernetes docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
1u26o5yll4mdqxfr1o86xphb5 * linuxkit-025000000001 Ready Active Leader 18.06.0-ce-dev

- Description for the changelog

Fixed broken swarm commands with Kubernetes defined as orchestrator
Renamed DOCKER_ORCHESTRATOR to DOCKER_STACK_ORCHESTRATOR
Renamed config file option "orchestrator" to "stackOrchestrator"
"--orchestrator" flag is no more global but local to stack command and subcommands

- A picture of a cute animal (not mandatory but encouraged)

silvin-lubecki · 2018-06-21T16:59:13Z

Please review with attention stack/cmd.go, and if you can make some manual tests too, it would be great 😄

mat007

A few comments!

mat007 · 2018-06-21T17:31:48Z

+ return
+ }
+ if _, ok := f.Annotations["kubernetes"]; ok && !orchestrator.HasKubernetes() {
+ errs = append(errs, fmt.Sprintf("\"--%s\" is only supported on a Docker cli with kubernetes features enabled", f.Name))


nit: back quotes would remove the need to escape the double quotes:
errs = append(errs, fmt.Sprintf(`"--%s" is only supported on a Docker cli with kubernetes features enabled`, f.Name))

Yep, I just moved the code but you're right I'll fix it.

mat007 · 2018-06-21T17:35:29Z

 )

-func newPsCommand(dockerCli command.Cli) *cobra.Command {
+func newPsCommand(dockerCli command.Cli, common *commonOptions) *cobra.Command {


I don't think common needs to be a pointer here as it's always non-nil?

Well, it's because newPsCommand is called during command instantiation, but the common.orchestrator value will be updated in the stack/cmd.go:PersistentPreRunE. That's why it is passed as a pointer and not as a copied value.

mat007 · 2018-06-21T17:36:21Z

 )

-func newListCommand(dockerCli command.Cli) *cobra.Command {
+func newListCommand(dockerCli command.Cli, common *commonOptions) *cobra.Command {


Same here, it doesn't seem common can ever be nil.

mat007 · 2018-06-21T17:37:50Z


+type commonOptions struct {
+	orchestrator command.Orchestrator
+}


Why do we need this type? Can't we use command.Orchestrator directly?

I need to update the value while resolving the orchestrator (conf file -> env -> flag), but we need to wait for the PersistentPreRun to be able to read the orchestrator flag value.

mat007 · 2018-06-21T17:39:15Z

 )

+var (
+	orchestrator = commonOptions{orchestrator: command.OrchestratorSwarm}


orchestrator is actually a commonOptions?

Yes I know, I was lazy... It was a first attempt with orchestrator, didn't want to rename 😅

mat007 · 2018-06-21T17:40:09Z

 	return buildTime
 }

+// nolint: gocyclo


Yeah, don't know why...

mat007 · 2018-06-21T17:42:21Z

- kubeVersion := getKubernetesVersion(dockerCli, opts.kubeConfig)
+ var kubeVersion *kubernetesVersion
+ if orchestrator.HasKubernetes() {
+ kubeVersion = getKubernetesVersion(opts.kubeConfig)


nit: or we could pass orchestrator to getKubernetesVersion instead of kubeConfig

Why instead of? I think getKubernetesVersion needs the kubeConfig to instantiate the kube client.

Sorry got mixed up, I meant instead of the dockerCli which was passed before.

mat007 · 2018-06-21T17:53:38Z


 // WrapCli wraps command.Cli with kubernetes specifics
-func WrapCli(dockerCli command.Cli, opts Options) (*KubeCli, error) {
+func WrapCli(dockerCli command.Cli, opts Options, orchestrator command.Orchestrator) (*KubeCli, error) {


Can't orchestrator be put inside Options somehow?

I'll try that!

silvin-lubecki · 2018-06-21T18:27:52Z

The metalinter is not happy, nor the unit testing... I'll fix this!

silvin-lubecki · 2018-06-21T19:31:19Z

PTAL

antonybichon17 · 2018-06-21T20:25:57Z

I see all those testOrchestrator being removed but nothing replacing them... or am I missing something?

silvin-lubecki · 2018-06-21T20:41:03Z

Tests were moved here

thaJeztah

left some comments for discussion; trying to give it a spin, but nuked my k8s, lol

thaJeztah · 2018-06-21T17:14:23Z

@@ -17,10 +17,25 @@ const (
 	OrchestratorAll = Orchestrator("all")


Looks like these constants can be un-exported? (and the GoDoc removed)

Nevermind; I see they're used in a test; can be fixed, but no urgency

thaJeztah · 2018-06-21T17:17:50Z

+	return command.GetStackOrchestrator(orchestratorFlag, config.StackOrchestrator)
+}
+
+func hideFlag(cmd *cobra.Command, orchestrator command.Orchestrator) {


hideOrchestrationFlags perhaps

thaJeztah · 2018-06-21T17:36:08Z

+ Built: Wed May 30 22:21:05 2018
+ OS/Arch: linux/amd64
+ Experimental: true
+Stack Orchestrator: swarm


Looking at this now, a bit concerned how this will scale if support for more options is added 😞

Talked with @thaJeztah @dhiltgen and it seems more desirable to just get rid of the Orchestrator line for now and revisit.

thaJeztah · 2018-06-21T20:40:33Z

 	orchestratorUnset = Orchestrator("unset")

-	defaultOrchestrator = OrchestratorSwarm
-envVarDockerOrchestrator = "DOCKER_ORCHESTRATOR"


We may have to think about printing a warning if someone has this variable set.
I realize so far it's been experimental, so we're allowed to change things, but it's definitely possible people had this option set, and now it's being ignored.
Of course, to further complicate stuff; some day it may find its way back, once k8s integration on the cli is complete 😅

Just added the warning!

thaJeztah · 2018-06-21T20:42:55Z

- newPsCommand(dockerCli),
- newRemoveCommand(dockerCli),
- newServicesCommand(dockerCli),
+ newDeployCommand(dockerCli, &opts),


It's a pity the change has such a deep impact. Don't see another solution at this time though 😕

thaJeztah · 2018-06-21T20:46:28Z

 OS/Arch:	{{.Os}}/{{.Arch}}
 Experimental:	{{.Experimental}}
- Orchestrator:	{{.Orchestrator}}
+Stack Orchestrator:	{{.StackOrchestrator}}


Still on the fence wether we should always print this, or only if any orchestrator is enabled

Looks like we may want to; here's against a host that doesn't have swarm (or kube) enabled;
Client: Version: 18.06.0-dev API version: 1.37 (downgraded from 1.38) Go version: go1.10.3 Git commit: ba061224 Built: Thu Jun 21 20:48:25 2018 OS/Arch: linux/amd64 Experimental: false Stack Orchestrator: swarm Server: Engine: Version: 18.05.0-ce API version: 1.37 (minimum version 1.12) Go version: go1.10.1 Git commit: f150324 Built: Wed May 9 22:20:16 2018 OS/Arch: linux/amd64 Experimental: true
Output for all may be a bit confusing as well, but don't have better suggestions for that at this point
Stack Orchestrator: all

thaJeztah · 2018-06-21T22:31:05Z

Giving this a spin;

Using "all" orchestrator:

$ docker stack deploy --orchestrator=swarm -c docker-compose.yml foobar
Creating network foobar_default
Creating service foobar_web
$ docker stack ls
NAME SERVICES ORCHESTRATOR NAMESPACE
foobar 1 Swarm

$ docker stack deploy -c docker-compose.yml foobar no orchestrator specified: use either "kubernetes" or "swarm"
$ docker stack deploy --orchestrator=kubernetes -c docker-compose.yml foobar
Waiting for the stack to be stable and running...
web: Ready [pod status: 1/1 ready, 0/1 pending, 0/1 failed]
Stack foobar is stable and running
$ docker stack ls
NAME SERVICES ORCHESTRATOR NAMESPACE
foobar 1 Kubernetes default
foobar 1 Swarm something 1 Swarm

This is something we can improve (i.e.; if it's non-ambiguous, show the result);

$ docker stack ps something
no orchestrator specified: use either "kubernetes" or "swarm"

$ docker stack ps --orchestrator=kubernetes foobar
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
1fc7d698-75a foobar_web-789778cdbf-8tshh nginx:alpine docker-for-desktop Running Running 3 minutes ago *:0->80/tcp
$ docker stack ps --orchestrator=swarm foobar
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
otpw084mgv9f foobar_web.1 nginx:alpine linuxkit-025000000001 Running Running 6 minutes ago

(actually curious how we deal with overlaps in published ports)

$ docker stack services foobar --orchestrator=kubernetes
ID NAME MODE REPLICAS IMAGE PORTS
1fd9a4d4-75a foobar_web replicated 1/1 nginx:alpine *:8080->80/tcp
$ docker stack services foobar --orchestrator=swarm
ID NAME MODE REPLICAS IMAGE PORTS
9x0iam28mlsl foobar_web replicated 1/1 nginx:alpine *:8080->80/tcp

* Renaming DOCKER_ORCHESTRATOR to DOCKER_STACK_ORCHESTRATOR * Renaming config file option "orchestrator" to "stackOrchestrator" * "--orchestrator" flag is no more global but local to stack command and subcommands * Cleaning all global orchestrator code * Replicating Hidden flags in help and Supported flags from root command to stack command Signed-off-by: Silvin Lubecki <silvin.lubecki@docker.com>

thaJeztah · 2018-06-22T01:10:48Z

I squashed @silvin-lubecki's commits, and added an extra commit that (at least for now) removes the "Orchestration" information from docker info output until we have a better design for that

ping @dhiltgen @andrewhsu @vdemeester PTAL

thaJeztah

LGTM

andrewhsu

LGTM

dhiltgen

lgtm

dhiltgen · 2018-06-22T04:09:41Z

 	BuildTime string `json:",omitempty"`
 	Experimental bool
-Orchestrator  string `json:",omitempty"`
+StackOrchestrator string `json:",omitempty"`


@thaJeztah it looks like this got left in (presumably to keep the changes to a minimum) - seems OK, but we should clean this up as part of a follow up pass.

Good one; was actually doubting whether or not we wanted to keep it; perhaps better to remove it yes, as it will show up in docker version --format '{{json .}}'
I'll grab my laptop and update

The output of this information can be confusing, so removing until we have a better design for this. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

The `-k` shorthand was alreaady removed from other commands, so best to be consistent. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

thaJeztah · 2018-06-22T06:41:19Z

@dhiltgen @silvin-lubecki @vdemeester I pushed some additional updates;

Removed the StackOrchestrator from the version struct (as discussed above Fix broken swarm commands with Kubernetes defined as orchestrator #1137 (comment))
Updated the documentation, and fixed some minor issues with the USAGE command
- note that the command-reference documentation on docs.docker.com is generated from a YAML file, so changes the markdown for individual commands is only to make GitHub match reality
Removed the -k shorthand on docker version; this shorthand was already removed (or not added) to any of the other commands. this is a new change

Remaining changes

documentation likely needs additional work (already being discussed on slack)
Bash/Fish/zsh completion scripts do not have all the options

thaJeztah · 2018-06-22T06:46:36Z

+	var opts commonOptions
 	cmd := &cobra.Command{
- Use: "stack",
+ Use: "stack [OPTIONS]",


this output isn't actually used for commands that have subcommands, but good to have in case that situation changes in future

vdemeester

SGTM 💦

vdemeester · 2018-06-22T07:44:38Z


 {{- if not .HasSubCommands}}	{{.UseLine}}{{end}}
-{{- if .HasSubCommands}}	{{ .CommandPath}} COMMAND{{end}}
+{{- if .HasSubCommands}}	{{ .CommandPath}}{{- if .HasAvailableFlags}} [OPTIONS]{{end}} COMMAND{{end}}


nit: Isn't this change unrelated ? 😝

The docker stack command itself now has options, but there was no [OPTIONS] printed 😞

vdemeester · 2018-06-22T07:54:56Z

- newPsCommand(dockerCli),
- newRemoveCommand(dockerCli),
- newServicesCommand(dockerCli),
+ newDeployCommand(dockerCli, &opts),


silvin-lubecki · 2018-06-22T14:27:50Z

PTAL

vdemeester

SGTM 🐸

andrewhsu · 2018-06-22T18:43:54Z

@silvin-lubecki talked with @dhiltgen and @thaJeztah ... would prefer to have this PR without 8fb2e7d. the DOCKER_ORCHESTRATOR env var was in experimental mode so we don't have to treat that as a supported interface that needs migration.

if still strongly desired for inclusion, i'd suggest putting into a new PR.

thaJeztah · 2018-06-22T18:57:03Z

 	}
 	defaultHelpFunc := cmd.HelpFunc()
 	cmd.SetHelpFunc(func(cmd *cobra.Command, args []string) {
- config := cliconfig.LoadDefaultConfigFile(dockerCli.Err()) // dockerCli is not yet initialized, but we only need config file here


Modified the last commit, and removed the warning, but kept these changes

Discussing with @dhiltgen - dropping this commit for now, and keeping it for #1139

thaJeztah · 2018-06-22T19:00:33Z

Moved the warning to #1139

dhiltgen

lgtm

silvin-lubecki requested a review from vdemeester as a code owner June 21, 2018 16:58

GordonTheTurtle added the status/0-triage label Jun 21, 2018

silvin-lubecki requested review from dhiltgen, mat007 and thaJeztah June 21, 2018 16:58

silvin-lubecki force-pushed the stack-orchestrator branch from 0e358b6 to ba06122Compare June 21, 2018 17:09

vdemeester added status/2-code-review and removed status/0-triage labels Jun 21, 2018

mat007 reviewed Jun 21, 2018
View reviewed changes

silvin-lubecki force-pushed the stack-orchestrator branch from 8c67022 to a935af8Compare June 21, 2018 19:22

thaJeztah requested changes Jun 21, 2018
View reviewed changes

thaJeztah force-pushed the stack-orchestrator branch 3 times, most recently from 312408d to 87c39daCompare June 22, 2018 00:53

thaJeztah approved these changes Jun 22, 2018
View reviewed changes

andrewhsu approved these changes Jun 22, 2018
View reviewed changes

dhiltgen approved these changes Jun 22, 2018
View reviewed changes

thaJeztah added 3 commits June 21, 2018 22:02

Remove "Stack Orchestrator" from "docker version"
a63252b
The output of this information can be confusing, so removing until we have a better design for this. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Update command usage and documentation
0f07b9f
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Remove shorthand -k option from docker version
f0a8598
The `-k` shorthand was alreaady removed from other commands, so best to be consistent. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

thaJeztah force-pushed the stack-orchestrator branch from 87c39da to f0a8598Compare June 22, 2018 06:35

thaJeztah reviewed Jun 22, 2018
View reviewed changes

thaJeztah added impact/changelog impact/documentation area/stack docs/revisit area/kubernetes labels Jun 22, 2018

vdemeester approved these changes Jun 22, 2018
View reviewed changes

thaJeztah force-pushed the stack-orchestrator branch from 8fb2e7d to c2c317cCompare June 22, 2018 18:56

thaJeztah reviewed Jun 22, 2018
View reviewed changes

thaJeztah mentioned this pull request Jun 22, 2018
Warn if DOCKER_ORCHESTRATOR is still used but not DOCKER_STACK_ORCHESTRATOR #1139
Merged

thaJeztah force-pushed the stack-orchestrator branch from c2c317c to f0a8598Compare June 22, 2018 20:12

dhiltgen approved these changes Jun 22, 2018
View reviewed changes

andrewhsu merged commit 151990d into docker:masterJun 22, 2018

GordonTheTurtle added this to the 18.06.0 milestone Jun 22, 2018

albers mentioned this pull request Aug 1, 2018
Add bash completion for experimental CLI commands (manifest) #1252
Merged

thaJeztah mentioned this pull request Nov 29, 2018
[18.09 backport] Add bash completion for experimental CLI commands (manifest) #1542
Merged

silvin-lubecki deleted the stack-orchestrator branch January 23, 2019 10:17

		@@ -17,10 +17,25 @@ const (
		OrchestratorAll = Orchestrator("all")

Uh oh!

Conversation

silvin-lubecki commented Jun 21, 2018

Uh oh!

silvin-lubecki commented Jun 21, 2018

Uh oh!

mat007 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

silvin-lubecki commented Jun 21, 2018

Uh oh!

silvin-lubecki commented Jun 21, 2018

Uh oh!

antonybichon17 commented Jun 21, 2018

Uh oh!

silvin-lubecki commented Jun 21, 2018

Uh oh!

thaJeztah left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thaJeztahJun 21, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thaJeztah commented Jun 21, 2018

Uh oh!

thaJeztahJun 21, 2018•
edited
Loading