[19.03 backport] integration-cli: fix swarm tests flakiness by thaJeztah · Pull Request #345 · docker-archive/engine

thaJeztah · 2019-09-01T10:46:03Z

based on top of / depends on #353 rebased

Partial backport of integration-cli: fix swarm tests flakiness moby/moby#39531 integration-cli: fix swarm tests flakiness
- skipped second commit (SwarmKit bump) as that requires backport to the SwarmKit bump_v19.03 branch
- addresses Flaky test: TestAPISwarmServicesCreate moby/moby#36903 Flaky test: TestAPISwarmServicesCreate
- addresses Flaky test: TestAPISwarmRaftQuorum moby/moby#34988 Flaky test: TestAPISwarmRaftQuorum
- addresses Flaky test: TestServicePlugin, TestCreateServiceSecretFileMode moby/moby#37132 Flaky test: TestServicePlugin, TestCreateServiceSecretFileMode
- addresses Flaky test: TestAPISwarmLeaderElection moby/moby#32673 Flaky test: TestAPISwarmLeaderElection
- addresses Flaky test: DockerSwarmSuite.TestSwarmRotateUnlockKey moby/moby#28240 Flaky test: DockerSwarmSuite.TestSwarmRotateUnlockKey
- addresses Flaky test: DockerSwarmSuite.TestSwarmClusterRotateUnlockKey moby/moby#38885 Flaky test: DockerSwarmSuite.TestSwarmClusterRotateUnlockKey
Fix TestSwarmClusterRotateUnlockKey moby/moby#39616 Fix TestSwarmClusterRotateUnlockKey
- fixesFlaky test: DockerSwarmSuite.TestSwarmClusterRotateUnlockKey moby/moby#38885 Flaky test: DockerSwarmSuite.TestSwarmClusterRotateUnlockKey
- addresses flaky DockerSwarmSuite tests moby/moby#39499 flaky DockerSwarmSuite tests
- addresses [epic] flaky tests moby/moby#37306 EPIC: flaky tests
- addresses flaky test: DockerSwarmSuite.TearDownTest on PowerPC moby/moby#33041 flaky test: DockerSwarmSuite.TearDownTest on PowerPC

# (partial backport of) https://github.com/moby/moby/pull/39531 integration-cli: fix swarm tests flakiness
git cherry-pick -s -S -x 3df1095bbdc331d4effa5452d8aafd5aaead5789
# git cherry-pick -s -S -x 096a7afd37d688332b961994116a101d9f3ffab9 # SKIPPED, requires swarmkit backport to bump_v19.03 branch
git cherry-pick -s -S -x 52e0dfef9090fa3c6003115a2c82238b189ebe42
# https://github.com/docker/swarmkit/pull/2808 Fix flaky tests
git cherry-pick -s -S -x b79adac339173bf8bb6de6d0a061a97973c4b62b

thaJeztah · 2019-09-01T10:51:17Z

@tonistiigi @dperny PTAL

@dperny perhaps you can have a look at backporting the changes in swarmkit for 19.03, so that we can update the vendor in a follow up? I updated the PR description of moby#39531 (comment) to show what changes were in that bump; probably only the last two of that are really related to this, but the gometalinter and flaky test changes look like they won't hurt to have in the "bump" branches

full diff: https://github.com/docker/swarmkit/compare/fb584e7b501ec4683b5c3e62476d76b8a7e7d9f6...7dded76ec532741c1ad9736cd2bb6d6661f0a386
includes:
- docker/swarmkit#2808 Fix flaky tests
- docker/swarmkit#2866 Swap gometalinter for golangci-lint
- docker/swarmkit#2869 Increase max recv gRPC message size to initialize connection broker
- docker/swarmkit#2870 Fix update out of sequence

thaJeztah · 2019-09-06T21:59:16Z

opened moby/swarmkit#2889 to backport those fixes in swarmkit

thaJeztah · 2019-09-06T23:07:24Z

Janky failed on a flaky https://ci.docker.com/public/blue/rest/organizations/jenkins/pipelines/engine/branches/PR-345/runs/3/nodes/149/log/?start=0

tracked through moby#39426
and on master is currently skipped (see moby#39453)

[2019-09-06T21:43:12.379Z] --- FAIL: TestServiceRemoveKeepsIngressNetwork (31.44s)
[2019-09-06T21:43:12.379Z] service_test.go:232: Creating a new daemon at: /go/src/github.com/docker/docker/bundles/test-integration/1/TestServiceRemoveKeepsIngressNetwork
[2019-09-06T21:43:12.379Z] daemon.go:321: [df2ee9961c8e2] waiting for daemon to start
[2019-09-06T21:43:12.379Z] daemon.go:339: [df2ee9961c8e2] daemon started
[2019-09-06T21:43:12.379Z] service_test.go:237: timeout hit after 30s: ingress not ready: 0 peers and 0 containers
[2019-09-06T21:43:12.379Z] daemon.go:449: [df2ee9961c8e2] Stopping daemon
[2019-09-06T21:43:12.379Z] daemon.go:288: [df2ee9961c8e2] exiting daemon
[2019-09-06T21:43:12.379Z] daemon.go:436: [df2ee9961c8e2] Daemon stopped
[2019-09-06T21:43:12.379Z] === RUN TestServiceWithDataPathPortInit
[2019-09-06T21:43:12.635Z] PASS: docker_cli_inspect_test.go:309: DockerSuite.TestInspectSizeFlagContainer	0.908s
[2019-09-06T21:43:13.995Z] PASS: docker_cli_inspect_test.go:56: DockerSuite.TestInspectStatus	1.095s
[2019-09-06T21:43:15.878Z] PASS: docker_cli_inspect_test.go:353: DockerSuite.TestInspectStopWhenNotFound	1.529s
[2019-09-06T21:43:16.801Z] PASS: docker_cli_inspect_test.go:320: DockerSuite.TestInspectTemplateError	0.798s
[2019-09-06T21:43:18.163Z] PASS: docker_cli_inspect_test.go:263: DockerSuite.TestInspectTimesAsRFC3339Nano	1.092s
[2019-09-06T21:43:18.418Z] --- PASS: TestServiceWithDataPathPortInit (6.01s)
[2019-09-06T21:43:18.418Z] service_test.go:337: Creating a new daemon at: /go/src/github.com/docker/docker/bundles/test-integration/1/TestServiceWithDataPathPortInit
[2019-09-06T21:43:18.418Z] daemon.go:321: [d0c42b6ea3887] waiting for daemon to start
[2019-09-06T21:43:18.418Z] daemon.go:339: [d0c42b6ea3887] daemon started
[2019-09-06T21:43:18.418Z] daemon.go:449: [d0c42b6ea3887] Stopping daemon
[2019-09-06T21:43:18.418Z] daemon.go:288: [d0c42b6ea3887] exiting daemon
[2019-09-06T21:43:18.418Z] daemon.go:436: [d0c42b6ea3887] Daemon stopped
[2019-09-06T21:43:18.418Z] service_test.go:365: Creating a new daemon at: /go/src/github.com/docker/docker/bundles/test-integration/1/TestServiceWithDataPathPortInit
[2019-09-06T21:43:18.418Z] daemon.go:321: [d4ec035907cb1] waiting for daemon to start
[2019-09-06T21:43:18.418Z] daemon.go:339: [d4ec035907cb1] daemon started
[2019-09-06T21:43:18.418Z] daemon.go:449: [d4ec035907cb1] Stopping daemon
[2019-09-06T21:43:18.418Z] daemon.go:288: [d4ec035907cb1] exiting daemon
[2019-09-06T21:43:18.418Z] daemon.go:436: [d4ec035907cb1] Daemon stopped

thaJeztah · 2019-09-06T23:12:21Z

opened #350 for that remaining flaky test

…3 branch) full diff: moby/swarmkit@4fb9e96...bbe3418 changes included: - moby/swarmkit#2889 [19.03 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets Which relates to - moby#39531 integration-cli: fix swarm tests flakiness - docker-archive#345 [19.03 backport] integration-cli: fix swarm tests flakiness And includes backports of - moby/swarmkit#2808 Fix flaky tests - moby/swarmkit#2866 Swap gometalinter for golangci-lint - moby/swarmkit#2869 Increase max recv gRPC message size to initialize connection broker - related / similar to moby#38103 / docker-archive#102 cluster: set bigger grpc limit for array requests - related / similar to moby#39306 Increase max recv gRPC message size for nodes and secrets - fixesmoby/swarmkit#2733 Error generated when messages size is too big - moby/swarmkit#2870 Fix update out of sequence Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

thaJeztah · 2019-09-12T08:09:57Z

opened #353 for the swarmkit bump, and rebased this PR on top of that

thaJeztah · 2019-09-12T16:06:07Z

this is all green, but I'll rebase to get rid of the commit from #353

thaJeztah · 2019-09-12T16:07:30Z

rebased

…3 branch) full diff: moby/swarmkit@4fb9e96...bbe3418 changes included: - moby/swarmkit#2889 [19.03 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets Which relates to - moby/moby#39531 integration-cli: fix swarm tests flakiness - docker-archive/engine#345 [19.03 backport] integration-cli: fix swarm tests flakiness And includes backports of - moby/swarmkit#2808 Fix flaky tests - moby/swarmkit#2866 Swap gometalinter for golangci-lint - moby/swarmkit#2869 Increase max recv gRPC message size to initialize connection broker - related / similar to moby/moby#38103 / docker-archive/engine#102 cluster: set bigger grpc limit for array requests - related / similar to moby/moby#39306 Increase max recv gRPC message size for nodes and secrets - fixesmoby/swarmkit#2733 Error generated when messages size is too big - moby/swarmkit#2870 Fix update out of sequence Signed-off-by: Sebastiaan van Stijn <github@gone.nl> Upstream-commit: f7dbee3eeaa1dd218116f85b8f60361acbd5b214 Component: engine

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> (cherry picked from commit 3df1095) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> (cherry picked from commit 52e0dfe) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

TestSwarmClusterRotateUnlockKey had been identified as a flaky test. It turns out that the test code was wrong: where we should have been checking the string output of a command, we were instead checking the value of the error. This means that the error case we were expecting was not being matched, and the test was failing when it should have just retried. Signed-off-by: Drew Erny <drew.erny@docker.com> (cherry picked from commit b79adac) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

thaJeztah · 2019-09-16T14:00:52Z

all green now; I'm bringing this one in

thaJeztah added this to the 19.03.3 milestone Sep 1, 2019

thaJeztah force-pushed the 19.03_backport_swarm_flaky branch from 9d6c8c3 to 4961b0aCompare September 6, 2019 21:08

thaJeztah mentioned this pull request Sep 6, 2019
[19.03 backport] Fix update out of sequence and increase max recv gRPC message size for nodes and secrets moby/swarmkit#2889
Merged

thaJeztah assigned tonistiigi Sep 6, 2019

tiborvass approved these changes Sep 9, 2019
View reviewed changes

thaJeztah mentioned this pull request Sep 10, 2019
[19.03 backport] Update containerd to v1.2.9 #349
Merged

thaJeztah mentioned this pull request Sep 12, 2019
[19.03 backport] bump swarmkit to bbe341867eae1615faf8a702ec05bfe986e73e06 (bump_v19.03 branch) #353
Merged

thaJeztah force-pushed the 19.03_backport_swarm_flaky branch from 4961b0a to 1694454Compare September 12, 2019 08:09

thaJeztah force-pushed the 19.03_backport_swarm_flaky branch from 1694454 to 1011e37Compare September 12, 2019 16:07

tonistiigiand others added 3 commits September 16, 2019 13:45

integration-cli: allow temporary errors on leader switch
c502db4
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> (cherry picked from commit 3df1095) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

integration-cli: allow temporary no-leader error
cf05755
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> (cherry picked from commit 52e0dfe) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

thaJeztah force-pushed the 19.03_backport_swarm_flaky branch from 1011e37 to c93da8dCompare September 16, 2019 11:45

thaJeztah merged commit b262d40 into docker-archive:19.03Sep 16, 2019

thaJeztah deleted the 19.03_backport_swarm_flaky branch September 16, 2019 14:01

Navigation Menu

Uh oh!

[19.03 backport] integration-cli: fix swarm tests flakiness - #345

[19.03 backport] integration-cli: fix swarm tests flakiness#345
thaJeztah merged 3 commits into
docker-archive:19.03from
thaJeztah:19.03_backport_swarm_flaky

thaJeztah commented Sep 1, 2019•
edited
Loading

Uh oh!

thaJeztah commented Sep 1, 2019

Uh oh!

thaJeztah commented Sep 6, 2019

Uh oh!

thaJeztah commented Sep 6, 2019

Uh oh!

thaJeztah commented Sep 6, 2019

Uh oh!

thaJeztah commented Sep 12, 2019

Uh oh!

thaJeztah commented Sep 12, 2019

Uh oh!

thaJeztah commented Sep 12, 2019

Uh oh!

thaJeztah commented Sep 16, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

thaJeztah commented Sep 1, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

based on top of / depends on #353 rebased

Uh oh!

thaJeztah commented Sep 1, 2019

Uh oh!

thaJeztah commented Sep 6, 2019

Uh oh!

thaJeztah commented Sep 6, 2019

Uh oh!

thaJeztah commented Sep 6, 2019

Uh oh!

thaJeztah commented Sep 12, 2019

Uh oh!

thaJeztah commented Sep 12, 2019

Uh oh!

thaJeztah commented Sep 12, 2019

Uh oh!

thaJeztah commented Sep 16, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

thaJeztah commented Sep 1, 2019•
edited
Loading