Uh oh!
There was an error while loading. Please reload this page.
Reimplement router.redundant.vrrp.interval setting - #1486
Conversation
4084784 to
f40dbc2Compareremibergsma
commented
Apr 12, 2016
@swill force pushed |
swill
commented
Apr 12, 2016
I am not sure what is going on with Jenkins and Travis now days, but they are being very flaky. Maybe the added load of people being more active is messing with them, but it is annoying. What should we do about this? Do I just keep asking people to force push until everything clears? |
f40dbc2 to
5219be1Compareremibergsma
commented
Apr 30, 2016
@swill force pushed |
5219be1 to
0ae6103Compareremibergsma
commented
May 1, 2016
Also force pushed this one again. |
yadvr
commented
May 2, 2016
@remibergsma can do a push -f again, Travis failed |
swill
commented
May 7, 2016
CI RESULTSAssociated Uploads
Uploads will be available until Comment created by |
0ae6103 to
1235e9bCompareremibergsma
commented
May 12, 2016
Force pushed. |
swill
commented
May 12, 2016
@remibergsma sorry to do this to you again. can you force push again. I have pushed fixes to jenkins and travis this morning so hopefully we can get this all green. I need some code review on this one too. Thx... |
1235e9b to
3d46696CompareIt defaults to 1, which is hardcoded in the template: ./cosmic/cosmic-core/systemvm/patches/debian/config/opt/cloud/templates/keepalived.conf.templ As non-VPC redundant routers use this setting, I think it makes sense to use it for rVPCs as well. We also need a change to pickup the cmd_line parameter and use it in the Python code that configures the router.
3d46696 to
9c0eee4Compareremibergsma
commented
May 13, 2016
Resolved conflict |
swill
commented
May 13, 2016
This one is looking pretty good. Can we get some code review on it so we can move it forward. Thanks... |
DaanHoogland
commented
May 13, 2016
LGTM, tag:easypr |
swill
commented
May 13, 2016
Ok, we need one more review on this. tag:needsreview tag:needreview |
swill
commented
May 16, 2016
This one is ready to merge. I would love another code review on this one as it is green and the CI is clean... |
GabrielBrascher
commented
May 16, 2016
Based on code review and the documentation cited by @remibergsma, the code LGTM. |
Reimplement router.redundant.vrrp.interval settingGlobal setting `router.redundant.vrrp.interval` is not used any more and it is now set to a hardcoded 1. This results in a failover from master->backup when the backup doesn't hear from the master in ~3.6sec. This is a bit too tight, as we've seen failovers during live migrations. We could reproduce it in about half of the cases. Setting this to setting to 2 (tested it by hardcoding it in the systemvms) gives twice as much time and we didn't see issues any more. Instead of updating the hardcoded setting from 1 to 2, I reimplemented the global setting by sending it to the router with the cmd_line, as the non-VPC router also does. Background: Why is the maximum failover time in the example 3.6 seconds? This comes from the advertisement interval and the skew time. The default advertisement interval is 1 second (configurable in keepalived.conf). The skew time helps to keep everyone from trying to transition at once. It is a number between 0 and 1, based on the formula (256 - priority) / 256 As defined in the RFC, the backup must receive an advertisement from the master every (3 * advert_int) + skew_time seconds. If it doesn't hear anything from the master, it takes over. With a backup router priority of 100 (as in the example), the failover will happen at most 3.6 seconds after the master goes down. Source: http://www.hollenback.net/KeepalivedForNetworkReliability * pr/1486: Configure rVPC for router.redundant.vrrp.interval advert_int setting Have rVPCs use the router.redundant.vrrp.interval setting Signed-off-by: Will Stevens <williamstevens@gmail.com>
Global setting
router.redundant.vrrp.intervalis not used any more and it is now set to a hardcoded 1.This results in a failover from master->backup when the backup doesn't hear from the master in ~3.6sec. This is a bit too tight, as we've seen failovers during live migrations. We could reproduce it in about half of the cases. Setting this to setting to 2 (tested it by hardcoding it in the systemvms) gives twice as much time and we didn't see issues any more. Instead of updating the hardcoded setting from 1 to 2, I reimplemented the global setting by sending it to the router with the cmd_line, as the non-VPC router also does.
Background:
Why is the maximum failover time in the example 3.6 seconds? This comes from the advertisement interval and the skew time. The default advertisement interval is 1 second (configurable in keepalived.conf). The skew time helps to keep everyone from trying to transition at once. It is a number between 0 and 1, based on the formula (256 - priority) / 256
As defined in the RFC, the backup must receive an advertisement from the master every (3 * advert_int) + skew_time seconds. If it doesn't hear anything from the master, it takes over. With a backup router priority of 100 (as in the example), the failover will happen at most 3.6 seconds after the master goes down.
Source: http://www.hollenback.net/KeepalivedForNetworkReliability