Fix flaky TestServiceWithDefaultAddressPoolInit - #39671

Merged
tiborvass merged 2 commits into
moby:masterfrom
arkodg:fix-flaky-addr-pool-init-test
Sep 9, 2019
Merged

Fix flaky TestServiceWithDefaultAddressPoolInit#39671
tiborvass merged 2 commits into
moby:masterfrom
arkodg:fix-flaky-addr-pool-init-test

Conversation

@arkodg

@arkodgarkodg commented Aug 5, 2019•
edited by thaJeztah
Loading

Copy link
Copy Markdown
Contributor

fixes#38514 Flaky test: TestServiceWithDefaultAddressPoolInit

This commit replaces serviceRunningCount with
swarm.RunningTasksCount to accurately check if the
service is running with the accurate number of instances
or not. serviceRunningCount was only checking the ServiceList
and was not checking if the tasks were running or not

This adds a safe barrier to execute docker network inspect
commands for overlay networks which get created
asynchronously via swarmkit

Signed-off-by: Arko Dasgupta arko.dasgupta@docker.com

@arkodg
arkodgforce-pushed the fix-flaky-addr-pool-init-test branch from 7146630 to c186d46CompareAugust 5, 2019 21:38
@arkodgarkodg changed the title Fix flaky TestServiceWithDefaultAddressPoolInit[WIP] Fix flaky TestServiceWithDefaultAddressPoolInitAug 5, 2019
@arkodg
arkodgforce-pushed the fix-flaky-addr-pool-init-test branch 4 times, most recently from 608d40e to 5d50ffaCompareAugust 5, 2019 22:30
@arkodg
arkodgforce-pushed the fix-flaky-addr-pool-init-test branch 4 times, most recently from 34b90f4 to fafea21CompareAugust 6, 2019 19:03
@arkodg

arkodg commented Aug 6, 2019

Copy link
Copy Markdown
ContributorAuthor

@thaJeztah any idea why the test case is failing ?
I removed all the SwarmLeave calls but I still see Calling POST /v1.41/swarm/leave?force=1" which is causing the service to not start

@arkodg
arkodgforce-pushed the fix-flaky-addr-pool-init-test branch from fafea21 to 53b8069CompareAugust 6, 2019 20:07
@arkodg

arkodg commented Aug 6, 2019

Copy link
Copy Markdown
ContributorAuthor

and changing the order of the operation breaks the next test TestServiceWithDataPathPortInit

@thaJeztah

thaJeztah commented Aug 6, 2019

Copy link
Copy Markdown
Member

is there a manager that failed to stop perhaps?

error is not nil: Error response from daemon: manager stopped: failed to listen on remote API address: listen tcp 0.0.0.0:2477: bind: address already in use: initializing swarm

@arkodg
arkodgforce-pushed the fix-flaky-addr-pool-init-test branch from 53b8069 to c763936CompareAugust 6, 2019 21:01
@kolyshkin

kolyshkin commented Aug 6, 2019

Copy link
Copy Markdown
Contributor

bind: address already in use

Means someone is already listening on this port. Not sure what it means in this exact context.

@arkodg
arkodgforce-pushed the fix-flaky-addr-pool-init-test branch 9 times, most recently from c76e7bb to 9114ccaCompareAugust 7, 2019 00:19
thaJeztah
thaJeztah reviewed Aug 13, 2019
View reviewed changes
Comment threadintegration/network/service_test.go Outdated
thaJeztah
thaJeztah requested changes Aug 13, 2019
View reviewed changes

@thaJeztahthaJeztah left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall looks good; could you have a look at my comments, and move the function back to its old position (which makes it easier to double-check I didn't miss anything)

Comment threadintegration/network/service_test.go Outdated
Comment threadintegration/network/service_test.go Outdated
@arkodg
arkodgforce-pushed the fix-flaky-addr-pool-init-test branch from 2549161 to 952d643CompareAugust 13, 2019 18:55
@arkodg

arkodg commented Aug 13, 2019

Copy link
Copy Markdown
ContributorAuthor

Derek add label: rebuild/windowsRS1

@arkodg

arkodg commented Aug 14, 2019

Copy link
Copy Markdown
ContributorAuthor

Derek add label: rebuild/windowsRS1

Arko Dasgupta
Fix flaky TestServiceWithDefaultAddressPoolInit
f3a3ea0
1.This commit replaces serviceRunningCount with
swarm.RunningTasksCount to accurately check if the
service is running with the accurate number of instances
or not. serviceRunningCount was only checking the ServiceList
and was not checking if the tasks were running or not
This adds a safe barrier to execute docker network inspect
commands for overlay networks which get created
asynchronously via Swarm
2. Make sure client connections are closed
3. Make sure every service and network name is unique
4. Make sure services and networks are cleaned up
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
@arkodg
arkodgforce-pushed the fix-flaky-addr-pool-init-test branch from 952d643 to f3a3ea0CompareAugust 14, 2019 15:03
@thaJeztahthaJeztah changed the title [WIP] Fix flaky TestServiceWithDefaultAddressPoolInitFix flaky TestServiceWithDefaultAddressPoolInitAug 14, 2019
thaJeztah
thaJeztah requested changes Aug 15, 2019
View reviewed changes
Comment threadintegration/network/service_test.go Outdated
Comment threadintegration/network/service_test.go
Comment threadintegration/network/service_test.go
Comment threadintegration/network/service_test.go Outdated
Comment threadintegration/network/service_test.go Outdated
Comment threadintegration/network/service_test.go Outdated
Arko Dasgupta
Move defer method to the top right after New is called
a65dee3
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
thaJeztah
thaJeztah approved these changes Aug 16, 2019
View reviewed changes

@thaJeztahthaJeztah left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@arkodg

arkodg commented Aug 16, 2019

Copy link
Copy Markdown
ContributorAuthor

PTAL @kolyshkin

@thaJeztah

thaJeztah commented Aug 16, 2019

Copy link
Copy Markdown
Member

@tiborvass PTAL

@thaJeztah

thaJeztah commented Aug 27, 2019

Copy link
Copy Markdown
Member

ping @tiborvass@kolyshkin PTAL 🤗

kolyshkin
kolyshkin reviewed Aug 27, 2019
View reviewed changes
Comment threadintegration/network/service_test.go
poll.WaitOn(t, swarm.NoTasks(ctx, c), swarm.ServicePoll)
err = c.NetworkRemove(ctx, overlayID)
assert.NilError(t, err)
c.Close()

@kolyshkinkolyshkinAug 27, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see one potential problem with all the assert statements here. Once any assert fails, the text execution is cancelled, meaning the above cleanup code (remove/close/etc) won't be run.

Perhaps we need to change those assert. calls to check. ones. The difference is check. won't abort test execution immediately (but still mark the test as failed).

If it's not possible to use check. everywhere, maybe we need to do cleanup in a defer.

@arkodgarkodgAug 27, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but AFAIK the RootDir will be cleaned up so it won't affect another PR, as for the current PR, this test failed so the reason for the assert failure should be addressed before moving to the next test

@tiborvass
tiborvass merged commit 000b563 into moby:masterSep 9, 2019
@arkodg
arkodg deleted the fix-flaky-addr-pool-init-test branch September 9, 2019 22:01
This was referenced Sep 21, 2019
Merged
Merged
@thaJeztahthaJeztah mentioned this pull request Oct 8, 2019
Merged
@thaJeztahthaJeztah mentioned this pull request Oct 23, 2019
Closed
@thaJeztahthaJeztah added this to the 20.03.0 milestone Apr 2, 2020
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kolyshkinkolyshkinkolyshkin left review comments
@tiborvasstiborvasstiborvass approved these changes
@thaJeztahthaJeztahthaJeztah approved these changes

Assignees

@MHBauerMHBauer

Labels

area/testingprocess/cherry-pickedstatus/2-code-review

Projects

None yet

Milestone

20.10.0

Development

Successfully merging this pull request may close these issues.

Flaky test: TestServiceWithDefaultAddressPoolInit

6 participants

@arkodg@thaJeztah@kolyshkin@tiborvass@GordonTheTurtle@MHBauer
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Fix flaky TestServiceWithDefaultAddressPoolInit - #39671
Merged
tiborvass merged 2 commits into
moby:masterfrom 
arkodg:fix-flaky-addr-pool-init-test
Sep 9, 2019
Merged
Fix flaky TestServiceWithDefaultAddressPoolInit#39671
tiborvass merged 2 commits into
moby:masterfrom 
arkodg:fix-flaky-addr-pool-init-test
Conversation
@arkodg
@arkodgarkodg
commented
Aug 5, 2019•

edited by thaJeztah

Loading

Copy link


Copy Markdown

Contributor
fixes#38514 Flaky test: TestServiceWithDefaultAddressPoolInit
This commit replaces serviceRunningCount with

swarm.RunningTasksCount to accurately check if the

service is running with the accurate number of instances

or not. serviceRunningCount was only checking the ServiceList

and was not checking if the tasks were running or not
This adds a safe barrier to execute docker network inspect

commands for overlay networks which get created

asynchronously via swarmkit
Signed-off-by: Arko Dasgupta arko.dasgupta@docker.com
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
7146630 to
c186d46CompareAugust 5, 2019 21:38
@arkodgarkodg
changed the title
Fix flaky TestServiceWithDefaultAddressPoolInit[WIP] Fix flaky TestServiceWithDefaultAddressPoolInitAug 5, 2019
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
4 times, most recently
from
608d40e to
5d50ffaCompareAugust 5, 2019 22:30
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
4 times, most recently
from
34b90f4 to
fafea21CompareAugust 6, 2019 19:03
@arkodg
arkodg
commented
Aug 6, 2019

Copy link


Copy Markdown

ContributorAuthor
@thaJeztah any idea why the test case is failing ?

I removed all the SwarmLeave calls but I still see Calling POST /v1.41/swarm/leave?force=1" which is causing the service to not start
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
fafea21 to
53b8069CompareAugust 6, 2019 20:07
@arkodg
arkodg
commented
Aug 6, 2019

Copy link


Copy Markdown

ContributorAuthor
and changing the order of the operation breaks the next test TestServiceWithDataPathPortInit
@thaJeztah
thaJeztah
commented
Aug 6, 2019

Copy link


Copy Markdown

Member
is there a manager that failed to stop perhaps?
error is not nil: Error response from daemon: manager stopped: failed to listen on remote API address: listen tcp 0.0.0.0:2477: bind: address already in use: initializing swarm

@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
53b8069 to
c763936CompareAugust 6, 2019 21:01
@kolyshkin
kolyshkin
commented
Aug 6, 2019

Copy link


Copy Markdown

Contributor
bind: address already in use
Means someone is already listening on this port. Not sure what it means in this exact context.
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
9 times, most recently
from
c76e7bb to
9114ccaCompareAugust 7, 2019 00:19
thaJeztah
thaJeztah
reviewed
Aug 13, 2019
View reviewed changes
Comment threadintegration/network/service_test.go
Outdated

thaJeztah
thaJeztah
requested changes
Aug 13, 2019
View reviewed changes
@thaJeztahthaJeztah
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall looks good; could you have a look at my comments, and move the function back to its old position (which makes it easier to double-check I didn't miss anything)
Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Outdated

@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
2549161 to
952d643CompareAugust 13, 2019 18:55
@arkodg
arkodg
commented
Aug 13, 2019

Copy link


Copy Markdown

ContributorAuthor
Derek add label: rebuild/windowsRS1
@arkodg
arkodg
commented
Aug 14, 2019

Copy link


Copy Markdown

ContributorAuthor
Derek add label: rebuild/windowsRS1
Arko Dasgupta
Fix flaky TestServiceWithDefaultAddressPoolInit
f3a3ea0
1.This commit replaces serviceRunningCount with
swarm.RunningTasksCount to accurately check if the
service is running with the accurate number of instances
or not. serviceRunningCount was only checking the ServiceList
and was not checking if the tasks were running or not
This adds a safe barrier to execute docker network inspect
commands for overlay networks which get created
asynchronously via Swarm
2. Make sure client connections are closed
3. Make sure every service and network name is unique
4. Make sure services and networks are cleaned up
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
952d643 to
f3a3ea0CompareAugust 14, 2019 15:03
@thaJeztahthaJeztah
changed the title
[WIP] Fix flaky TestServiceWithDefaultAddressPoolInitFix flaky TestServiceWithDefaultAddressPoolInitAug 14, 2019
thaJeztah
thaJeztah
requested changes
Aug 15, 2019
View reviewed changes
Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Comment threadintegration/network/service_test.go
Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Outdated

Arko Dasgupta
Move defer method to the top right after New is called
a65dee3
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
thaJeztah
thaJeztah
approved these changes
Aug 16, 2019
View reviewed changes
@thaJeztahthaJeztah
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!
@arkodg
arkodg
commented
Aug 16, 2019

Copy link


Copy Markdown

ContributorAuthor
PTAL @kolyshkin
@thaJeztah
thaJeztah
commented
Aug 16, 2019

Copy link


Copy Markdown

Member
@tiborvass PTAL
@thaJeztah
thaJeztah
commented
Aug 27, 2019

Copy link


Copy Markdown

Member
ping @tiborvass@kolyshkin PTAL 🤗
kolyshkin
kolyshkin
reviewed
Aug 27, 2019
View reviewed changes
Comment threadintegration/network/service_test.go
	poll.WaitOn(t, swarm.NoTasks(ctx, c), swarm.ServicePoll)
	err = c.NetworkRemove(ctx, overlayID)
	assert.NilError(t, err)
	c.Close()
@kolyshkinkolyshkinAug 27, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see one potential problem with all the assert statements here. Once any assert fails, the text execution is cancelled, meaning the above cleanup code (remove/close/etc) won't be run.
Perhaps we need to change those assert. calls to check. ones. The difference is check. won't abort test execution immediately (but still mark the test as failed).
If it's not possible to use check. everywhere, maybe we need to do cleanup in a defer.
@arkodgarkodgAug 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but AFAIK the RootDir will be cleaned up so it won't affect another PR, as for the current PR, this test failed so the reason for the assert failure should be addressed before moving to the next test
@tiborvass
tiborvass
merged commit 000b563
into
moby:masterSep 9, 2019
@arkodg
arkodg
deleted the

fix-flaky-addr-pool-init-test

branch
September 9, 2019 22:01

This was referenced Sep 21, 2019
 Merged

 Merged

@thaJeztahthaJeztah
mentioned this pull request
Oct 8, 2019
 Merged

@thaJeztahthaJeztah
mentioned this pull request
Oct 23, 2019
 Closed

@thaJeztahthaJeztah
added this to the 20.03.0 milestone
Apr 2, 2020
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kolyshkinkolyshkinkolyshkin left review comments
@tiborvasstiborvasstiborvass approved these changes
@thaJeztahthaJeztahthaJeztah approved these changes

Assignees

@MHBauerMHBauer

Labels

area/testingprocess/cherry-pickedstatus/2-code-review

Projects


None yet


Milestone

20.10.0

Development

Successfully merging this pull request may close these issues.

Flaky test: TestServiceWithDefaultAddressPoolInit


6 participants

@arkodg@thaJeztah@kolyshkin@tiborvass@GordonTheTurtle@MHBauer

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Fix flaky TestServiceWithDefaultAddressPoolInit - #39671
Merged
tiborvass merged 2 commits into
moby:masterfrom 
arkodg:fix-flaky-addr-pool-init-test
Sep 9, 2019
Merged
Fix flaky TestServiceWithDefaultAddressPoolInit#39671
tiborvass merged 2 commits into
moby:masterfrom 
arkodg:fix-flaky-addr-pool-init-test
Conversation
@arkodg
@arkodgarkodg
commented
Aug 5, 2019•

edited by thaJeztah

Loading

Copy link


Copy Markdown

Contributor
fixes#38514 Flaky test: TestServiceWithDefaultAddressPoolInit
This commit replaces serviceRunningCount with

swarm.RunningTasksCount to accurately check if the

service is running with the accurate number of instances

or not. serviceRunningCount was only checking the ServiceList

and was not checking if the tasks were running or not
This adds a safe barrier to execute docker network inspect

commands for overlay networks which get created

asynchronously via swarmkit
Signed-off-by: Arko Dasgupta arko.dasgupta@docker.com
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
7146630 to
c186d46CompareAugust 5, 2019 21:38
@arkodgarkodg
changed the title
Fix flaky TestServiceWithDefaultAddressPoolInit[WIP] Fix flaky TestServiceWithDefaultAddressPoolInitAug 5, 2019
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
4 times, most recently
from
608d40e to
5d50ffaCompareAugust 5, 2019 22:30
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
4 times, most recently
from
34b90f4 to
fafea21CompareAugust 6, 2019 19:03
@arkodg
arkodg
commented
Aug 6, 2019

Copy link


Copy Markdown

ContributorAuthor
@thaJeztah any idea why the test case is failing ?

I removed all the SwarmLeave calls but I still see Calling POST /v1.41/swarm/leave?force=1" which is causing the service to not start
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
fafea21 to
53b8069CompareAugust 6, 2019 20:07
@arkodg
arkodg
commented
Aug 6, 2019

Copy link


Copy Markdown

ContributorAuthor
and changing the order of the operation breaks the next test TestServiceWithDataPathPortInit
@thaJeztah
thaJeztah
commented
Aug 6, 2019

Copy link


Copy Markdown

Member
is there a manager that failed to stop perhaps?
error is not nil: Error response from daemon: manager stopped: failed to listen on remote API address: listen tcp 0.0.0.0:2477: bind: address already in use: initializing swarm

@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
53b8069 to
c763936CompareAugust 6, 2019 21:01
@kolyshkin
kolyshkin
commented
Aug 6, 2019

Copy link


Copy Markdown

Contributor
bind: address already in use
Means someone is already listening on this port. Not sure what it means in this exact context.
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
9 times, most recently
from
c76e7bb to
9114ccaCompareAugust 7, 2019 00:19
thaJeztah
thaJeztah
reviewed
Aug 13, 2019
View reviewed changes
Comment threadintegration/network/service_test.go
Outdated

thaJeztah
thaJeztah
requested changes
Aug 13, 2019
View reviewed changes
@thaJeztahthaJeztah
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall looks good; could you have a look at my comments, and move the function back to its old position (which makes it easier to double-check I didn't miss anything)
Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Outdated

@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
2549161 to
952d643CompareAugust 13, 2019 18:55
@arkodg
arkodg
commented
Aug 13, 2019

Copy link


Copy Markdown

ContributorAuthor
Derek add label: rebuild/windowsRS1
@arkodg
arkodg
commented
Aug 14, 2019

Copy link


Copy Markdown

ContributorAuthor
Derek add label: rebuild/windowsRS1
Arko Dasgupta
Fix flaky TestServiceWithDefaultAddressPoolInit
f3a3ea0
1.This commit replaces serviceRunningCount with
swarm.RunningTasksCount to accurately check if the
service is running with the accurate number of instances
or not. serviceRunningCount was only checking the ServiceList
and was not checking if the tasks were running or not
This adds a safe barrier to execute docker network inspect
commands for overlay networks which get created
asynchronously via Swarm
2. Make sure client connections are closed
3. Make sure every service and network name is unique
4. Make sure services and networks are cleaned up
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
952d643 to
f3a3ea0CompareAugust 14, 2019 15:03
@thaJeztahthaJeztah
changed the title
[WIP] Fix flaky TestServiceWithDefaultAddressPoolInitFix flaky TestServiceWithDefaultAddressPoolInitAug 14, 2019
thaJeztah
thaJeztah
requested changes
Aug 15, 2019
View reviewed changes
Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Comment threadintegration/network/service_test.go
Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Outdated

Arko Dasgupta
Move defer method to the top right after New is called
a65dee3
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
thaJeztah
thaJeztah
approved these changes
Aug 16, 2019
View reviewed changes
@thaJeztahthaJeztah
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!
@arkodg
arkodg
commented
Aug 16, 2019

Copy link


Copy Markdown

ContributorAuthor
PTAL @kolyshkin
@thaJeztah
thaJeztah
commented
Aug 16, 2019

Copy link


Copy Markdown

Member
@tiborvass PTAL
@thaJeztah
thaJeztah
commented
Aug 27, 2019

Copy link


Copy Markdown

Member
ping @tiborvass@kolyshkin PTAL 🤗
kolyshkin
kolyshkin
reviewed
Aug 27, 2019
View reviewed changes
Comment threadintegration/network/service_test.go
	poll.WaitOn(t, swarm.NoTasks(ctx, c), swarm.ServicePoll)
	err = c.NetworkRemove(ctx, overlayID)
	assert.NilError(t, err)
	c.Close()
@kolyshkinkolyshkinAug 27, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see one potential problem with all the assert statements here. Once any assert fails, the text execution is cancelled, meaning the above cleanup code (remove/close/etc) won't be run.
Perhaps we need to change those assert. calls to check. ones. The difference is check. won't abort test execution immediately (but still mark the test as failed).
If it's not possible to use check. everywhere, maybe we need to do cleanup in a defer.
@arkodgarkodgAug 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but AFAIK the RootDir will be cleaned up so it won't affect another PR, as for the current PR, this test failed so the reason for the assert failure should be addressed before moving to the next test
@tiborvass
tiborvass
merged commit 000b563
into
moby:masterSep 9, 2019
@arkodg
arkodg
deleted the

fix-flaky-addr-pool-init-test

branch
September 9, 2019 22:01

This was referenced Sep 21, 2019
 Merged

 Merged

@thaJeztahthaJeztah
mentioned this pull request
Oct 8, 2019
 Merged

@thaJeztahthaJeztah
mentioned this pull request
Oct 23, 2019
 Closed

@thaJeztahthaJeztah
added this to the 20.03.0 milestone
Apr 2, 2020
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kolyshkinkolyshkinkolyshkin left review comments
@tiborvasstiborvasstiborvass approved these changes
@thaJeztahthaJeztahthaJeztah approved these changes

Assignees

@MHBauerMHBauer

Labels

area/testingprocess/cherry-pickedstatus/2-code-review

Projects


None yet


Milestone

20.10.0

Development

Successfully merging this pull request may close these issues.

Flaky test: TestServiceWithDefaultAddressPoolInit


6 participants

@arkodg@thaJeztah@kolyshkin@tiborvass@GordonTheTurtle@MHBauer

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Fix flaky TestServiceWithDefaultAddressPoolInit - #39671
Merged
tiborvass merged 2 commits into
moby:masterfrom 
arkodg:fix-flaky-addr-pool-init-test
Sep 9, 2019
Merged
Fix flaky TestServiceWithDefaultAddressPoolInit#39671
tiborvass merged 2 commits into
moby:masterfrom 
arkodg:fix-flaky-addr-pool-init-test
Conversation
@arkodg
@arkodgarkodg
commented
Aug 5, 2019•

edited by thaJeztah

Loading

Copy link


Copy Markdown

Contributor
fixes#38514 Flaky test: TestServiceWithDefaultAddressPoolInit
This commit replaces serviceRunningCount with

swarm.RunningTasksCount to accurately check if the

service is running with the accurate number of instances

or not. serviceRunningCount was only checking the ServiceList

and was not checking if the tasks were running or not
This adds a safe barrier to execute docker network inspect

commands for overlay networks which get created

asynchronously via swarmkit
Signed-off-by: Arko Dasgupta arko.dasgupta@docker.com
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
7146630 to
c186d46CompareAugust 5, 2019 21:38
@arkodgarkodg
changed the title
Fix flaky TestServiceWithDefaultAddressPoolInit[WIP] Fix flaky TestServiceWithDefaultAddressPoolInitAug 5, 2019
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
4 times, most recently
from
608d40e to
5d50ffaCompareAugust 5, 2019 22:30
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
4 times, most recently
from
34b90f4 to
fafea21CompareAugust 6, 2019 19:03
@arkodg
arkodg
commented
Aug 6, 2019

Copy link


Copy Markdown

ContributorAuthor
@thaJeztah any idea why the test case is failing ?

I removed all the SwarmLeave calls but I still see Calling POST /v1.41/swarm/leave?force=1" which is causing the service to not start
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
fafea21 to
53b8069CompareAugust 6, 2019 20:07
@arkodg
arkodg
commented
Aug 6, 2019

Copy link


Copy Markdown

ContributorAuthor
and changing the order of the operation breaks the next test TestServiceWithDataPathPortInit
@thaJeztah
thaJeztah
commented
Aug 6, 2019

Copy link


Copy Markdown

Member
is there a manager that failed to stop perhaps?
error is not nil: Error response from daemon: manager stopped: failed to listen on remote API address: listen tcp 0.0.0.0:2477: bind: address already in use: initializing swarm

@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
53b8069 to
c763936CompareAugust 6, 2019 21:01
@kolyshkin
kolyshkin
commented
Aug 6, 2019

Copy link


Copy Markdown

Contributor
bind: address already in use
Means someone is already listening on this port. Not sure what it means in this exact context.
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
9 times, most recently
from
c76e7bb to
9114ccaCompareAugust 7, 2019 00:19
thaJeztah
thaJeztah
reviewed
Aug 13, 2019
View reviewed changes
Comment threadintegration/network/service_test.go
Outdated

thaJeztah
thaJeztah
requested changes
Aug 13, 2019
View reviewed changes
@thaJeztahthaJeztah
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall looks good; could you have a look at my comments, and move the function back to its old position (which makes it easier to double-check I didn't miss anything)
Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Outdated

@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
2549161 to
952d643CompareAugust 13, 2019 18:55
@arkodg
arkodg
commented
Aug 13, 2019

Copy link


Copy Markdown

ContributorAuthor
Derek add label: rebuild/windowsRS1
@arkodg
arkodg
commented
Aug 14, 2019

Copy link


Copy Markdown

ContributorAuthor
Derek add label: rebuild/windowsRS1
Arko Dasgupta
Fix flaky TestServiceWithDefaultAddressPoolInit
f3a3ea0
1.This commit replaces serviceRunningCount with
swarm.RunningTasksCount to accurately check if the
service is running with the accurate number of instances
or not. serviceRunningCount was only checking the ServiceList
and was not checking if the tasks were running or not
This adds a safe barrier to execute docker network inspect
commands for overlay networks which get created
asynchronously via Swarm
2. Make sure client connections are closed
3. Make sure every service and network name is unique
4. Make sure services and networks are cleaned up
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
952d643 to
f3a3ea0CompareAugust 14, 2019 15:03
@thaJeztahthaJeztah
changed the title
[WIP] Fix flaky TestServiceWithDefaultAddressPoolInitFix flaky TestServiceWithDefaultAddressPoolInitAug 14, 2019
thaJeztah
thaJeztah
requested changes
Aug 15, 2019
View reviewed changes
Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Comment threadintegration/network/service_test.go
Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Outdated

Arko Dasgupta
Move defer method to the top right after New is called
a65dee3
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
thaJeztah
thaJeztah
approved these changes
Aug 16, 2019
View reviewed changes
@thaJeztahthaJeztah
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!
@arkodg
arkodg
commented
Aug 16, 2019

Copy link


Copy Markdown

ContributorAuthor
PTAL @kolyshkin
@thaJeztah
thaJeztah
commented
Aug 16, 2019

Copy link


Copy Markdown

Member
@tiborvass PTAL
@thaJeztah
thaJeztah
commented
Aug 27, 2019

Copy link


Copy Markdown

Member
ping @tiborvass@kolyshkin PTAL 🤗
kolyshkin
kolyshkin
reviewed
Aug 27, 2019
View reviewed changes
Comment threadintegration/network/service_test.go
	poll.WaitOn(t, swarm.NoTasks(ctx, c), swarm.ServicePoll)
	err = c.NetworkRemove(ctx, overlayID)
	assert.NilError(t, err)
	c.Close()
@kolyshkinkolyshkinAug 27, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see one potential problem with all the assert statements here. Once any assert fails, the text execution is cancelled, meaning the above cleanup code (remove/close/etc) won't be run.
Perhaps we need to change those assert. calls to check. ones. The difference is check. won't abort test execution immediately (but still mark the test as failed).
If it's not possible to use check. everywhere, maybe we need to do cleanup in a defer.
@arkodgarkodgAug 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but AFAIK the RootDir will be cleaned up so it won't affect another PR, as for the current PR, this test failed so the reason for the assert failure should be addressed before moving to the next test
@tiborvass
tiborvass
merged commit 000b563
into
moby:masterSep 9, 2019
@arkodg
arkodg
deleted the

fix-flaky-addr-pool-init-test

branch
September 9, 2019 22:01

This was referenced Sep 21, 2019
 Merged

 Merged

@thaJeztahthaJeztah
mentioned this pull request
Oct 8, 2019
 Merged

@thaJeztahthaJeztah
mentioned this pull request
Oct 23, 2019
 Closed

@thaJeztahthaJeztah
added this to the 20.03.0 milestone
Apr 2, 2020
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kolyshkinkolyshkinkolyshkin left review comments
@tiborvasstiborvasstiborvass approved these changes
@thaJeztahthaJeztahthaJeztah approved these changes

Assignees

@MHBauerMHBauer

Labels

area/testingprocess/cherry-pickedstatus/2-code-review

Projects


None yet


Milestone

20.10.0

Development

Successfully merging this pull request may close these issues.

Flaky test: TestServiceWithDefaultAddressPoolInit


6 participants

@arkodg@thaJeztah@kolyshkin@tiborvass@GordonTheTurtle@MHBauer

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Fix flaky TestServiceWithDefaultAddressPoolInit - #39671
Merged
tiborvass merged 2 commits into
moby:masterfrom 
arkodg:fix-flaky-addr-pool-init-test
Sep 9, 2019
Merged
Fix flaky TestServiceWithDefaultAddressPoolInit#39671
tiborvass merged 2 commits into
moby:masterfrom 
arkodg:fix-flaky-addr-pool-init-test
Conversation
@arkodg
@arkodgarkodg
commented
Aug 5, 2019•

edited by thaJeztah

Loading

Copy link


Copy Markdown

Contributor
fixes#38514 Flaky test: TestServiceWithDefaultAddressPoolInit
This commit replaces serviceRunningCount with

swarm.RunningTasksCount to accurately check if the

service is running with the accurate number of instances

or not. serviceRunningCount was only checking the ServiceList

and was not checking if the tasks were running or not
This adds a safe barrier to execute docker network inspect

commands for overlay networks which get created

asynchronously via swarmkit
Signed-off-by: Arko Dasgupta arko.dasgupta@docker.com
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
7146630 to
c186d46CompareAugust 5, 2019 21:38
@arkodgarkodg
changed the title
Fix flaky TestServiceWithDefaultAddressPoolInit[WIP] Fix flaky TestServiceWithDefaultAddressPoolInitAug 5, 2019
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
4 times, most recently
from
608d40e to
5d50ffaCompareAugust 5, 2019 22:30
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
4 times, most recently
from
34b90f4 to
fafea21CompareAugust 6, 2019 19:03
@arkodg
arkodg
commented
Aug 6, 2019

Copy link


Copy Markdown

ContributorAuthor
@thaJeztah any idea why the test case is failing ?

I removed all the SwarmLeave calls but I still see Calling POST /v1.41/swarm/leave?force=1" which is causing the service to not start
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
fafea21 to
53b8069CompareAugust 6, 2019 20:07
@arkodg
arkodg
commented
Aug 6, 2019

Copy link


Copy Markdown

ContributorAuthor
and changing the order of the operation breaks the next test TestServiceWithDataPathPortInit
@thaJeztah
thaJeztah
commented
Aug 6, 2019

Copy link


Copy Markdown

Member
is there a manager that failed to stop perhaps?
error is not nil: Error response from daemon: manager stopped: failed to listen on remote API address: listen tcp 0.0.0.0:2477: bind: address already in use: initializing swarm

@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
53b8069 to
c763936CompareAugust 6, 2019 21:01
@kolyshkin
kolyshkin
commented
Aug 6, 2019

Copy link


Copy Markdown

Contributor
bind: address already in use
Means someone is already listening on this port. Not sure what it means in this exact context.
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
9 times, most recently
from
c76e7bb to
9114ccaCompareAugust 7, 2019 00:19
thaJeztah
thaJeztah
reviewed
Aug 13, 2019
View reviewed changes
Comment threadintegration/network/service_test.go
Outdated

thaJeztah
thaJeztah
requested changes
Aug 13, 2019
View reviewed changes
@thaJeztahthaJeztah
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall looks good; could you have a look at my comments, and move the function back to its old position (which makes it easier to double-check I didn't miss anything)
Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Outdated

@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
2549161 to
952d643CompareAugust 13, 2019 18:55
@arkodg
arkodg
commented
Aug 13, 2019

Copy link


Copy Markdown

ContributorAuthor
Derek add label: rebuild/windowsRS1
@arkodg
arkodg
commented
Aug 14, 2019

Copy link


Copy Markdown

ContributorAuthor
Derek add label: rebuild/windowsRS1
Arko Dasgupta
Fix flaky TestServiceWithDefaultAddressPoolInit
f3a3ea0
1.This commit replaces serviceRunningCount with
swarm.RunningTasksCount to accurately check if the
service is running with the accurate number of instances
or not. serviceRunningCount was only checking the ServiceList
and was not checking if the tasks were running or not
This adds a safe barrier to execute docker network inspect
commands for overlay networks which get created
asynchronously via Swarm
2. Make sure client connections are closed
3. Make sure every service and network name is unique
4. Make sure services and networks are cleaned up
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
952d643 to
f3a3ea0CompareAugust 14, 2019 15:03
@thaJeztahthaJeztah
changed the title
[WIP] Fix flaky TestServiceWithDefaultAddressPoolInitFix flaky TestServiceWithDefaultAddressPoolInitAug 14, 2019
thaJeztah
thaJeztah
requested changes
Aug 15, 2019
View reviewed changes
Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Comment threadintegration/network/service_test.go
Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Outdated

Arko Dasgupta
Move defer method to the top right after New is called
a65dee3
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
thaJeztah
thaJeztah
approved these changes
Aug 16, 2019
View reviewed changes
@thaJeztahthaJeztah
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!
@arkodg
arkodg
commented
Aug 16, 2019

Copy link


Copy Markdown

ContributorAuthor
PTAL @kolyshkin
@thaJeztah
thaJeztah
commented
Aug 16, 2019

Copy link


Copy Markdown

Member
@tiborvass PTAL
@thaJeztah
thaJeztah
commented
Aug 27, 2019

Copy link


Copy Markdown

Member
ping @tiborvass@kolyshkin PTAL 🤗
kolyshkin
kolyshkin
reviewed
Aug 27, 2019
View reviewed changes
Comment threadintegration/network/service_test.go
	poll.WaitOn(t, swarm.NoTasks(ctx, c), swarm.ServicePoll)
	err = c.NetworkRemove(ctx, overlayID)
	assert.NilError(t, err)
	c.Close()
@kolyshkinkolyshkinAug 27, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see one potential problem with all the assert statements here. Once any assert fails, the text execution is cancelled, meaning the above cleanup code (remove/close/etc) won't be run.
Perhaps we need to change those assert. calls to check. ones. The difference is check. won't abort test execution immediately (but still mark the test as failed).
If it's not possible to use check. everywhere, maybe we need to do cleanup in a defer.
@arkodgarkodgAug 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but AFAIK the RootDir will be cleaned up so it won't affect another PR, as for the current PR, this test failed so the reason for the assert failure should be addressed before moving to the next test
@tiborvass
tiborvass
merged commit 000b563
into
moby:masterSep 9, 2019
@arkodg
arkodg
deleted the

fix-flaky-addr-pool-init-test

branch
September 9, 2019 22:01

This was referenced Sep 21, 2019
 Merged

 Merged

@thaJeztahthaJeztah
mentioned this pull request
Oct 8, 2019
 Merged

@thaJeztahthaJeztah
mentioned this pull request
Oct 23, 2019
 Closed

@thaJeztahthaJeztah
added this to the 20.03.0 milestone
Apr 2, 2020
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kolyshkinkolyshkinkolyshkin left review comments
@tiborvasstiborvasstiborvass approved these changes
@thaJeztahthaJeztahthaJeztah approved these changes

Assignees

@MHBauerMHBauer

Labels

area/testingprocess/cherry-pickedstatus/2-code-review

Projects


None yet


Milestone

20.10.0

Development

Successfully merging this pull request may close these issues.

Flaky test: TestServiceWithDefaultAddressPoolInit


6 participants

@arkodg@thaJeztah@kolyshkin@tiborvass@GordonTheTurtle@MHBauer

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Fix flaky TestServiceWithDefaultAddressPoolInit - #39671
Merged
tiborvass merged 2 commits into
moby:masterfrom 
arkodg:fix-flaky-addr-pool-init-test
Sep 9, 2019
Merged
Fix flaky TestServiceWithDefaultAddressPoolInit#39671
tiborvass merged 2 commits into
moby:masterfrom 
arkodg:fix-flaky-addr-pool-init-test
Conversation
@arkodg
@arkodgarkodg
commented
Aug 5, 2019•

edited by thaJeztah

Loading

Copy link


Copy Markdown

Contributor
fixes#38514 Flaky test: TestServiceWithDefaultAddressPoolInit
This commit replaces serviceRunningCount with

swarm.RunningTasksCount to accurately check if the

service is running with the accurate number of instances

or not. serviceRunningCount was only checking the ServiceList

and was not checking if the tasks were running or not
This adds a safe barrier to execute docker network inspect

commands for overlay networks which get created

asynchronously via swarmkit
Signed-off-by: Arko Dasgupta arko.dasgupta@docker.com
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
7146630 to
c186d46CompareAugust 5, 2019 21:38
@arkodgarkodg
changed the title
Fix flaky TestServiceWithDefaultAddressPoolInit[WIP] Fix flaky TestServiceWithDefaultAddressPoolInitAug 5, 2019
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
4 times, most recently
from
608d40e to
5d50ffaCompareAugust 5, 2019 22:30
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
4 times, most recently
from
34b90f4 to
fafea21CompareAugust 6, 2019 19:03
@arkodg
arkodg
commented
Aug 6, 2019

Copy link


Copy Markdown

ContributorAuthor
@thaJeztah any idea why the test case is failing ?

I removed all the SwarmLeave calls but I still see Calling POST /v1.41/swarm/leave?force=1" which is causing the service to not start
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
fafea21 to
53b8069CompareAugust 6, 2019 20:07
@arkodg
arkodg
commented
Aug 6, 2019

Copy link


Copy Markdown

ContributorAuthor
and changing the order of the operation breaks the next test TestServiceWithDataPathPortInit
@thaJeztah
thaJeztah
commented
Aug 6, 2019

Copy link


Copy Markdown

Member
is there a manager that failed to stop perhaps?
error is not nil: Error response from daemon: manager stopped: failed to listen on remote API address: listen tcp 0.0.0.0:2477: bind: address already in use: initializing swarm

@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
53b8069 to
c763936CompareAugust 6, 2019 21:01
@kolyshkin
kolyshkin
commented
Aug 6, 2019

Copy link


Copy Markdown

Contributor
bind: address already in use
Means someone is already listening on this port. Not sure what it means in this exact context.
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
9 times, most recently
from
c76e7bb to
9114ccaCompareAugust 7, 2019 00:19
thaJeztah
thaJeztah
reviewed
Aug 13, 2019
View reviewed changes
Comment threadintegration/network/service_test.go
Outdated

thaJeztah
thaJeztah
requested changes
Aug 13, 2019
View reviewed changes
@thaJeztahthaJeztah
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall looks good; could you have a look at my comments, and move the function back to its old position (which makes it easier to double-check I didn't miss anything)
Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Outdated

@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
2549161 to
952d643CompareAugust 13, 2019 18:55
@arkodg
arkodg
commented
Aug 13, 2019

Copy link


Copy Markdown

ContributorAuthor
Derek add label: rebuild/windowsRS1
@arkodg
arkodg
commented
Aug 14, 2019

Copy link


Copy Markdown

ContributorAuthor
Derek add label: rebuild/windowsRS1
Arko Dasgupta
Fix flaky TestServiceWithDefaultAddressPoolInit
f3a3ea0
1.This commit replaces serviceRunningCount with
swarm.RunningTasksCount to accurately check if the
service is running with the accurate number of instances
or not. serviceRunningCount was only checking the ServiceList
and was not checking if the tasks were running or not
This adds a safe barrier to execute docker network inspect
commands for overlay networks which get created
asynchronously via Swarm
2. Make sure client connections are closed
3. Make sure every service and network name is unique
4. Make sure services and networks are cleaned up
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
952d643 to
f3a3ea0CompareAugust 14, 2019 15:03
@thaJeztahthaJeztah
changed the title
[WIP] Fix flaky TestServiceWithDefaultAddressPoolInitFix flaky TestServiceWithDefaultAddressPoolInitAug 14, 2019
thaJeztah
thaJeztah
requested changes
Aug 15, 2019
View reviewed changes
Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Comment threadintegration/network/service_test.go
Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Outdated

Arko Dasgupta
Move defer method to the top right after New is called
a65dee3
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
thaJeztah
thaJeztah
approved these changes
Aug 16, 2019
View reviewed changes
@thaJeztahthaJeztah
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!
@arkodg
arkodg
commented
Aug 16, 2019

Copy link


Copy Markdown

ContributorAuthor
PTAL @kolyshkin
@thaJeztah
thaJeztah
commented
Aug 16, 2019

Copy link


Copy Markdown

Member
@tiborvass PTAL
@thaJeztah
thaJeztah
commented
Aug 27, 2019

Copy link


Copy Markdown

Member
ping @tiborvass@kolyshkin PTAL 🤗
kolyshkin
kolyshkin
reviewed
Aug 27, 2019
View reviewed changes
Comment threadintegration/network/service_test.go
	poll.WaitOn(t, swarm.NoTasks(ctx, c), swarm.ServicePoll)
	err = c.NetworkRemove(ctx, overlayID)
	assert.NilError(t, err)
	c.Close()
@kolyshkinkolyshkinAug 27, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see one potential problem with all the assert statements here. Once any assert fails, the text execution is cancelled, meaning the above cleanup code (remove/close/etc) won't be run.
Perhaps we need to change those assert. calls to check. ones. The difference is check. won't abort test execution immediately (but still mark the test as failed).
If it's not possible to use check. everywhere, maybe we need to do cleanup in a defer.
@arkodgarkodgAug 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but AFAIK the RootDir will be cleaned up so it won't affect another PR, as for the current PR, this test failed so the reason for the assert failure should be addressed before moving to the next test
@tiborvass
tiborvass
merged commit 000b563
into
moby:masterSep 9, 2019
@arkodg
arkodg
deleted the

fix-flaky-addr-pool-init-test

branch
September 9, 2019 22:01

This was referenced Sep 21, 2019
 Merged

 Merged

@thaJeztahthaJeztah
mentioned this pull request
Oct 8, 2019
 Merged

@thaJeztahthaJeztah
mentioned this pull request
Oct 23, 2019
 Closed

@thaJeztahthaJeztah
added this to the 20.03.0 milestone
Apr 2, 2020
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kolyshkinkolyshkinkolyshkin left review comments
@tiborvasstiborvasstiborvass approved these changes
@thaJeztahthaJeztahthaJeztah approved these changes

Assignees

@MHBauerMHBauer

Labels

area/testingprocess/cherry-pickedstatus/2-code-review

Projects


None yet


Milestone

20.10.0

Development

Successfully merging this pull request may close these issues.

Flaky test: TestServiceWithDefaultAddressPoolInit


6 participants

@arkodg@thaJeztah@kolyshkin@tiborvass@GordonTheTurtle@MHBauer

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Fix flaky TestServiceWithDefaultAddressPoolInit - #39671
Merged
tiborvass merged 2 commits into
moby:masterfrom 
arkodg:fix-flaky-addr-pool-init-test
Sep 9, 2019
Merged
Fix flaky TestServiceWithDefaultAddressPoolInit#39671
tiborvass merged 2 commits into
moby:masterfrom 
arkodg:fix-flaky-addr-pool-init-test
Conversation
@arkodg
@arkodgarkodg
commented
Aug 5, 2019•

edited by thaJeztah

Loading

Copy link


Copy Markdown

Contributor
fixes#38514 Flaky test: TestServiceWithDefaultAddressPoolInit
This commit replaces serviceRunningCount with

swarm.RunningTasksCount to accurately check if the

service is running with the accurate number of instances

or not. serviceRunningCount was only checking the ServiceList

and was not checking if the tasks were running or not
This adds a safe barrier to execute docker network inspect

commands for overlay networks which get created

asynchronously via swarmkit
Signed-off-by: Arko Dasgupta arko.dasgupta@docker.com
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
7146630 to
c186d46CompareAugust 5, 2019 21:38
@arkodgarkodg
changed the title
Fix flaky TestServiceWithDefaultAddressPoolInit[WIP] Fix flaky TestServiceWithDefaultAddressPoolInitAug 5, 2019
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
4 times, most recently
from
608d40e to
5d50ffaCompareAugust 5, 2019 22:30
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
4 times, most recently
from
34b90f4 to
fafea21CompareAugust 6, 2019 19:03
@arkodg
arkodg
commented
Aug 6, 2019

Copy link


Copy Markdown

ContributorAuthor
@thaJeztah any idea why the test case is failing ?

I removed all the SwarmLeave calls but I still see Calling POST /v1.41/swarm/leave?force=1" which is causing the service to not start
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
fafea21 to
53b8069CompareAugust 6, 2019 20:07
@arkodg
arkodg
commented
Aug 6, 2019

Copy link


Copy Markdown

ContributorAuthor
and changing the order of the operation breaks the next test TestServiceWithDataPathPortInit
@thaJeztah
thaJeztah
commented
Aug 6, 2019

Copy link


Copy Markdown

Member
is there a manager that failed to stop perhaps?
error is not nil: Error response from daemon: manager stopped: failed to listen on remote API address: listen tcp 0.0.0.0:2477: bind: address already in use: initializing swarm

@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
53b8069 to
c763936CompareAugust 6, 2019 21:01
@kolyshkin
kolyshkin
commented
Aug 6, 2019

Copy link


Copy Markdown

Contributor
bind: address already in use
Means someone is already listening on this port. Not sure what it means in this exact context.
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
9 times, most recently
from
c76e7bb to
9114ccaCompareAugust 7, 2019 00:19
thaJeztah
thaJeztah
reviewed
Aug 13, 2019
View reviewed changes
Comment threadintegration/network/service_test.go
Outdated

thaJeztah
thaJeztah
requested changes
Aug 13, 2019
View reviewed changes
@thaJeztahthaJeztah
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall looks good; could you have a look at my comments, and move the function back to its old position (which makes it easier to double-check I didn't miss anything)
Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Outdated

@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
2549161 to
952d643CompareAugust 13, 2019 18:55
@arkodg
arkodg
commented
Aug 13, 2019

Copy link


Copy Markdown

ContributorAuthor
Derek add label: rebuild/windowsRS1
@arkodg
arkodg
commented
Aug 14, 2019

Copy link


Copy Markdown

ContributorAuthor
Derek add label: rebuild/windowsRS1
Arko Dasgupta
Fix flaky TestServiceWithDefaultAddressPoolInit
f3a3ea0
1.This commit replaces serviceRunningCount with
swarm.RunningTasksCount to accurately check if the
service is running with the accurate number of instances
or not. serviceRunningCount was only checking the ServiceList
and was not checking if the tasks were running or not
This adds a safe barrier to execute docker network inspect
commands for overlay networks which get created
asynchronously via Swarm
2. Make sure client connections are closed
3. Make sure every service and network name is unique
4. Make sure services and networks are cleaned up
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
952d643 to
f3a3ea0CompareAugust 14, 2019 15:03
@thaJeztahthaJeztah
changed the title
[WIP] Fix flaky TestServiceWithDefaultAddressPoolInitFix flaky TestServiceWithDefaultAddressPoolInitAug 14, 2019
thaJeztah
thaJeztah
requested changes
Aug 15, 2019
View reviewed changes
Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Comment threadintegration/network/service_test.go
Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Outdated

Arko Dasgupta
Move defer method to the top right after New is called
a65dee3
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
thaJeztah
thaJeztah
approved these changes
Aug 16, 2019
View reviewed changes
@thaJeztahthaJeztah
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!
@arkodg
arkodg
commented
Aug 16, 2019

Copy link


Copy Markdown

ContributorAuthor
PTAL @kolyshkin
@thaJeztah
thaJeztah
commented
Aug 16, 2019

Copy link


Copy Markdown

Member
@tiborvass PTAL
@thaJeztah
thaJeztah
commented
Aug 27, 2019

Copy link


Copy Markdown

Member
ping @tiborvass@kolyshkin PTAL 🤗
kolyshkin
kolyshkin
reviewed
Aug 27, 2019
View reviewed changes
Comment threadintegration/network/service_test.go
	poll.WaitOn(t, swarm.NoTasks(ctx, c), swarm.ServicePoll)
	err = c.NetworkRemove(ctx, overlayID)
	assert.NilError(t, err)
	c.Close()
@kolyshkinkolyshkinAug 27, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see one potential problem with all the assert statements here. Once any assert fails, the text execution is cancelled, meaning the above cleanup code (remove/close/etc) won't be run.
Perhaps we need to change those assert. calls to check. ones. The difference is check. won't abort test execution immediately (but still mark the test as failed).
If it's not possible to use check. everywhere, maybe we need to do cleanup in a defer.
@arkodgarkodgAug 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but AFAIK the RootDir will be cleaned up so it won't affect another PR, as for the current PR, this test failed so the reason for the assert failure should be addressed before moving to the next test
@tiborvass
tiborvass
merged commit 000b563
into
moby:masterSep 9, 2019
@arkodg
arkodg
deleted the

fix-flaky-addr-pool-init-test

branch
September 9, 2019 22:01

This was referenced Sep 21, 2019
 Merged

 Merged

@thaJeztahthaJeztah
mentioned this pull request
Oct 8, 2019
 Merged

@thaJeztahthaJeztah
mentioned this pull request
Oct 23, 2019
 Closed

@thaJeztahthaJeztah
added this to the 20.03.0 milestone
Apr 2, 2020
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kolyshkinkolyshkinkolyshkin left review comments
@tiborvasstiborvasstiborvass approved these changes
@thaJeztahthaJeztahthaJeztah approved these changes

Assignees

@MHBauerMHBauer

Labels

area/testingprocess/cherry-pickedstatus/2-code-review

Projects


None yet


Milestone

20.10.0

Development

Successfully merging this pull request may close these issues.

Flaky test: TestServiceWithDefaultAddressPoolInit


6 participants

@arkodg@thaJeztah@kolyshkin@tiborvass@GordonTheTurtle@MHBauer

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Fix flaky TestServiceWithDefaultAddressPoolInit - #39671
Merged
tiborvass merged 2 commits into
moby:masterfrom 
arkodg:fix-flaky-addr-pool-init-test
Sep 9, 2019
Merged
Fix flaky TestServiceWithDefaultAddressPoolInit#39671
tiborvass merged 2 commits into
moby:masterfrom 
arkodg:fix-flaky-addr-pool-init-test
Conversation
@arkodg
@arkodgarkodg
commented
Aug 5, 2019•

edited by thaJeztah

Loading

Copy link


Copy Markdown

Contributor
fixes#38514 Flaky test: TestServiceWithDefaultAddressPoolInit
This commit replaces serviceRunningCount with

swarm.RunningTasksCount to accurately check if the

service is running with the accurate number of instances

or not. serviceRunningCount was only checking the ServiceList

and was not checking if the tasks were running or not
This adds a safe barrier to execute docker network inspect

commands for overlay networks which get created

asynchronously via swarmkit
Signed-off-by: Arko Dasgupta arko.dasgupta@docker.com
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
7146630 to
c186d46CompareAugust 5, 2019 21:38
@arkodgarkodg
changed the title
Fix flaky TestServiceWithDefaultAddressPoolInit[WIP] Fix flaky TestServiceWithDefaultAddressPoolInitAug 5, 2019
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
4 times, most recently
from
608d40e to
5d50ffaCompareAugust 5, 2019 22:30
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
4 times, most recently
from
34b90f4 to
fafea21CompareAugust 6, 2019 19:03
@arkodg
arkodg
commented
Aug 6, 2019

Copy link


Copy Markdown

ContributorAuthor
@thaJeztah any idea why the test case is failing ?

I removed all the SwarmLeave calls but I still see Calling POST /v1.41/swarm/leave?force=1" which is causing the service to not start
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
fafea21 to
53b8069CompareAugust 6, 2019 20:07
@arkodg
arkodg
commented
Aug 6, 2019

Copy link


Copy Markdown

ContributorAuthor
and changing the order of the operation breaks the next test TestServiceWithDataPathPortInit
@thaJeztah
thaJeztah
commented
Aug 6, 2019

Copy link


Copy Markdown

Member
is there a manager that failed to stop perhaps?
error is not nil: Error response from daemon: manager stopped: failed to listen on remote API address: listen tcp 0.0.0.0:2477: bind: address already in use: initializing swarm

@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
53b8069 to
c763936CompareAugust 6, 2019 21:01
@kolyshkin
kolyshkin
commented
Aug 6, 2019

Copy link


Copy Markdown

Contributor
bind: address already in use
Means someone is already listening on this port. Not sure what it means in this exact context.
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
9 times, most recently
from
c76e7bb to
9114ccaCompareAugust 7, 2019 00:19
thaJeztah
thaJeztah
reviewed
Aug 13, 2019
View reviewed changes
Comment threadintegration/network/service_test.go
Outdated

thaJeztah
thaJeztah
requested changes
Aug 13, 2019
View reviewed changes
@thaJeztahthaJeztah
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall looks good; could you have a look at my comments, and move the function back to its old position (which makes it easier to double-check I didn't miss anything)
Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Outdated

@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
2549161 to
952d643CompareAugust 13, 2019 18:55
@arkodg
arkodg
commented
Aug 13, 2019

Copy link


Copy Markdown

ContributorAuthor
Derek add label: rebuild/windowsRS1
@arkodg
arkodg
commented
Aug 14, 2019

Copy link


Copy Markdown

ContributorAuthor
Derek add label: rebuild/windowsRS1
Arko Dasgupta
Fix flaky TestServiceWithDefaultAddressPoolInit
f3a3ea0
1.This commit replaces serviceRunningCount with
swarm.RunningTasksCount to accurately check if the
service is running with the accurate number of instances
or not. serviceRunningCount was only checking the ServiceList
and was not checking if the tasks were running or not
This adds a safe barrier to execute docker network inspect
commands for overlay networks which get created
asynchronously via Swarm
2. Make sure client connections are closed
3. Make sure every service and network name is unique
4. Make sure services and networks are cleaned up
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
@arkodg
arkodgforce-pushed
the
fix-flaky-addr-pool-init-test
branch
from
952d643 to
f3a3ea0CompareAugust 14, 2019 15:03
@thaJeztahthaJeztah
changed the title
[WIP] Fix flaky TestServiceWithDefaultAddressPoolInitFix flaky TestServiceWithDefaultAddressPoolInitAug 14, 2019
thaJeztah
thaJeztah
requested changes
Aug 15, 2019
View reviewed changes
Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Comment threadintegration/network/service_test.go
Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Outdated

Comment threadintegration/network/service_test.go
Outdated

Arko Dasgupta
Move defer method to the top right after New is called
a65dee3
Signed-off-by: Arko Dasgupta <arko.dasgupta@docker.com>
thaJeztah
thaJeztah
approved these changes
Aug 16, 2019
View reviewed changes
@thaJeztahthaJeztah
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!
@arkodg
arkodg
commented
Aug 16, 2019

Copy link


Copy Markdown

ContributorAuthor
PTAL @kolyshkin
@thaJeztah
thaJeztah
commented
Aug 16, 2019

Copy link


Copy Markdown

Member
@tiborvass PTAL
@thaJeztah
thaJeztah
commented
Aug 27, 2019

Copy link


Copy Markdown

Member
ping @tiborvass@kolyshkin PTAL 🤗
kolyshkin
kolyshkin
reviewed
Aug 27, 2019
View reviewed changes
Comment threadintegration/network/service_test.go
	poll.WaitOn(t, swarm.NoTasks(ctx, c), swarm.ServicePoll)
	err = c.NetworkRemove(ctx, overlayID)
	assert.NilError(t, err)
	c.Close()
@kolyshkinkolyshkinAug 27, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see one potential problem with all the assert statements here. Once any assert fails, the text execution is cancelled, meaning the above cleanup code (remove/close/etc) won't be run.
Perhaps we need to change those assert. calls to check. ones. The difference is check. won't abort test execution immediately (but still mark the test as failed).
If it's not possible to use check. everywhere, maybe we need to do cleanup in a defer.
@arkodgarkodgAug 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but AFAIK the RootDir will be cleaned up so it won't affect another PR, as for the current PR, this test failed so the reason for the assert failure should be addressed before moving to the next test
@tiborvass
tiborvass
merged commit 000b563
into
moby:masterSep 9, 2019
@arkodg
arkodg
deleted the

fix-flaky-addr-pool-init-test

branch
September 9, 2019 22:01

This was referenced Sep 21, 2019
 Merged

 Merged

@thaJeztahthaJeztah
mentioned this pull request
Oct 8, 2019
 Merged

@thaJeztahthaJeztah
mentioned this pull request
Oct 23, 2019
 Closed

@thaJeztahthaJeztah
added this to the 20.03.0 milestone
Apr 2, 2020
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@kolyshkinkolyshkinkolyshkin left review comments
@tiborvasstiborvasstiborvass approved these changes
@thaJeztahthaJeztahthaJeztah approved these changes

Assignees

@MHBauerMHBauer

Labels

area/testingprocess/cherry-pickedstatus/2-code-review

Projects


None yet


Milestone

20.10.0

Development

Successfully merging this pull request may close these issues.

Flaky test: TestServiceWithDefaultAddressPoolInit


6 participants

@arkodg@thaJeztah@kolyshkin@tiborvass@GordonTheTurtle@MHBauer