Uh oh!
There was an error while loading. Please reload this page.
Netlink timeouts - #1976
Conversation
fcrisciani
commented
Oct 10, 2017
@mavenugo can you give a pass to the changes, the risk is that the new vendoring of the netlink is pretty heavy in changes. We should try to reproduce the issue and then apply this patch |
Codecov Report
@@ Coverage Diff @@## master #1976 +/- ##
=========================================
Coverage ? 38.94% =========================================
Files ? 137 Lines ? 27390 Branches ? 0 =========================================
Hits ? 10666 Misses ? 15426 Partials ? 1298
Continue to review full report at Codecov.
|
bb38615 to
20c86dbCompareandrewhsu
commented
Nov 8, 2017
How do I make sure this PR does the fix? cc @antonybichon17 |
fcrisciani
commented
Nov 9, 2017
@andrewhsu the deadlock condition triggered on socket closed is tested in the netlink library project. This commit simply uses the same timeout |
antonybichon17
commented
Nov 9, 2017
You are using the timeout in a different part of the code, triggered by different actions. |
fcrisciani
commented
Nov 9, 2017
@antonybichon17 for the validation IMHO we should try to reproduce the customer deadlock that would be the best way to have the validation. |
| if nlFd == -1 { | ||
| // The netlink socket got closed, simply exit to not leak this goroutine | ||
| return | ||
| } |
There was a problem hiding this comment.
better to add a continue here to avoid spam logs
a8dd1c0 to
e78b4ffComparefcrisciani
commented
Nov 17, 2017
@antonybichon17 test added |
thaJeztah
commented
Nov 20, 2017
| msgs, err := nlSock.Receive() | ||
| if err != nil { | ||
| // When the receive timeout expires the receive will return EAGAIN | ||
| if err == syscall.EAGAIN { |
There was a problem hiding this comment.
minor nit
if err != nil{
if err == syscall.EAGAIN{
....
}
....
}
| "github.com/vishvananda/netns" | ||
| ) | ||
| var netlinkSocketsTimeout = 3 * time.Second |
There was a problem hiding this comment.
For my understanding is this a standard timeout ?
There was a problem hiding this comment.
nope is decided from us, the tradeoff is between waking up too ofter or being block too much time. Don't have an opinion on how much is the "right" time here
| if err := sock.SetReceiveTimeout(&tv); err != nil { | ||
| return nil, err | ||
| } | ||
| if err := sock.SetReceiveTimeout(&tv); err != nil { |
There was a problem hiding this comment.
good catch it should be ReceiveTimeout and SendTimeout
There was a problem hiding this comment.
Shouldn't netlinkSendSocketTimeout be netlinkRecvSocketsTimeout here ?
| ) | ||
| if err := s.Send(req); err != nil { | ||
| err := s.Send(req) |
There was a problem hiding this comment.
nit :
if err := s.Send(req);err != nil {
.....
}
| done: | ||
| for { | ||
| msgs, err := s.Receive() | ||
| if err == syscall.EAGAIN { |
There was a problem hiding this comment.
same as above.
if err != nil {
if err == syscall.EAGAIN {
....
}
....
}
19f93aa to
fc7323dCompare| const ( | ||
| netlinkRecvSocketsTimeout = 3 * time.Second | ||
| netlinkSendSocketTimeout = 30 * time.Second |
There was a problem hiding this comment.
this is set to ah higher value to avoid timeout error in case the system is busy. I noticed that the error coming from the write failure to IPVS is not percolated up, so if the addLBBackend backend failed in the configuration, the error is not handled at all. I will follow up on this on a different PR though because the change will be potentially non trivial.
| n.Unlock() | ||
| if nlFd == -1 { | ||
| // The netlink socket got closed, simply exit to not leak this goroutine | ||
| return |
There was a problem hiding this comment.
Shouldnt we retain this logic for all cases ?
In case the file descriptor of the netlink socket is closed the recvfrom is not returning. This may create deadlock conditions. The current solution is to make sure that all the netlink socket used have a proper timeout set on them to have the possibility to return Added test to emulate the watchMiss condition Signed-off-by: Flavio Crisciani <flavio.crisciani@docker.com>
- needed the methods to set the proper timeout Signed-off-by: Flavio Crisciani <flavio.crisciani@docker.com>
fc7323d to
45ea903Compare
Add netlink timeouts to all the netlink operations
Vendored netlink library to pick the required methods (temporary my branch)