Skip to content

VR: Fix Redundant VRouter guest network on wrong interface - #3847

Merged
DaanHoogland merged 8 commits into
apache:4.13from
ustcweizhou:4.14.0-vpc-vr-issues
Feb 29, 2020
Merged

VR: Fix Redundant VRouter guest network on wrong interface#3847
DaanHoogland merged 8 commits into
apache:4.13from
ustcweizhou:4.14.0-vpc-vr-issues

Conversation

@weizhouapache

Copy link
Copy Markdown
Member

Description

This PR aims to fix the issues on KVM environments with linux bridge and OVS if there are multiple public IPs associated to a network or vpc.

fixes: #3179

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

  • Tested the following network on KVM with linux bridge and KVM with OVS.

  • networks with single VR

  • networks with redundant VRs

Actions

  1. Create network offering with single VR, and enable it

  2. create isolated network with the network offering

  3. create a vm in the network.
    verify the available nics in VR should be "eth0,eth1,eth2"
    verify the IPs in VR. eth0 -> guest nic, eth2 -> source nat IP

  4. get a free public ip, assign to network, and create port forwarding rules (ssh) to the vm
    verify the available nics in VR should be "eth0,eth1,eth2"
    verify the IPs in VR. eth0 -> guest nic, eth2 -> source nat IP and new ip

  5. remove the port forwarding rule, and release the new ip
    verify the available nics in VR should be "eth0,eth1,eth2"
    verify the IPs in VR. eth0 -> guest nic IP, eth2 -> source nat IP

  6. create new public ip range 1

  7. get a free ip in new ip range, assign to network, and create port forwarding rules (ssh) to the vm
    verify the available nics in VR should be "eth0,eth1,eth2,eth3"
    verify the IPs in VR. eth0 -> guest nic, eth2 -> source nat IP, eth3 -> new ip 1

  8. get a free ip in new ip range, assign to network, and create port forwarding rules (ssh) to the vm
    verify the available nics in VR should be "eth0,eth1,eth2,eth3"
    verify the IPs in VR. eth0 -> guest nic, eth2 -> source nat IP, eth3 -> new ip 1, new ip 2,

  9. get a free ip in new ip range, assign to network, and create port forwarding rules (ssh) to the vm
    verify the available nics in VR should be "eth0,eth1,eth2,eth3"
    verify the IPs in VR. eth0 -> guest nic, eth2 -> source nat IP, eth3 -> new ip 1, new ip 2, new ip 3

  10. release new ip 2
    verify the available nics in VR should be "eth0,eth1,eth2,eth3"
    verify the IPs in VR. eth0 -> guest nic, eth2 -> source nat IP, eth3 -> new ip 1, new ip 3

  11. release new ip 1
    verify the available nics in VR should be "eth0,eth1,eth2,eth3"
    verify the IPs in VR. eth0 -> guest nic, eth2 -> source nat IP, eth3 -> new ip 3

  12. create new public ip range 2

  13. get a free ip 4 in new ip range 2, assign to network, and create port forwarding rules (ssh) to the vm
    verify the available nics in VR should be "eth0,eth1,eth2,eth3,eth4,"
    verify the IPs in VR. eth0 -> guest nic, eth2 -> source nat IP, eth3 -> new ip 3, eth4 -> new ip 4

  14. get a free ip 5 in new ip range 2, assign to network, and create port forwarding rules (ssh) to the vm
    verify the available nics in VR should be "eth0,eth1,eth2,eth3,eth4,"
    verify the IPs in VR. eth0 -> guest nic, eth2 -> source nat IP, eth3 -> new ip 3, eth4 -> new ip 4/5

  15. get a free ip 6 in new ip range 2, assign to network, and create port forwarding rules (ssh) to the vm
    verify the available nics in VR should be "eth0,eth1,eth2,eth3,eth4,"
    verify the IPs in VR. eth0 -> guest nic, eth2 -> source nat IP, eth3 -> new ip 3, eth4 -> new ip 4/5/6

  16. release new ip 5
    verify the available nics in VR should be "eth0,eth1,eth2,eth3,eth4,"
    verify the IPs in VR. eth0 -> guest nic, eth2 -> source nat IP, eth3 -> new ip 3, eth4 -> new ip 4/6

  17. release new ip 4
    verify the available nics in VR should be "eth0,eth1,eth2,eth3,eth4,"
    verify the IPs in VR. eth0 -> guest nic, eth2 -> source nat IP, eth3 -> new ip 3, eth4 -> new ip 6

  18. release new ip 3
    verify the available nics in VR should be "eth0,eth1,eth2,eth4,"
    verify the IPs in VR. eth0 -> guest nic, eth2 -> source nat IP, eth4 -> new ip 6

  19. restart network
    verify the available nics in VR should be "eth0,eth1,eth2,eth4,"
    verify the IPs in VR. eth0 -> guest nic, eth2 -> source nat IP, eth4 -> new ip 6

  20. restart network with cleanup
    verify the available nics in VR should be "eth0,eth1,eth2,eth3,"
    verify the IPs in VR. eth0 -> guest nic, eth2 -> source nat IP, eth3 -> new ip 6

  21. restart network with cleanup, makeredundant=true
    verify the available nics in VR should be "eth0,eth1,eth2,eth3,"
    verify the IPs in VR. eth0 -> guest nic, eth2 -> source nat IP, eth3 -> new ip 6

  • Tested the following vpcs on KVM with linux bridge and KVM with OVS.

  • vpc with single VR

  • vpc with redundant VRs

Actions

  1. get vpc offering with single VR

  2. create a vpc with the vpc offering
    verify the available nics in VR should be "eth0,eth1"
    verify the IPs in VR. eth0 -> control nic, eth1 -> source nat IP

  3. create a tier in the vpc, and create a vm in the tier.
    verify the available nics in VR should be "eth0,eth1,eth2"
    verify the IPs in VR. eth0 -> control nic, eth1 -> source nat IP, eth2 -> tier 1

  4. get a free public ip, assign to network, and create port forwarding rules (ssh) to the vm
    verify the available nics in VR should be "eth0,eth1,eth2"
    verify the IPs in VR. eth0 -> control nic, eth1 -> source nat IP, eth2 -> tier 1

  5. remove the port forwarding rule, and release the new ip
    verify the available nics in VR should be "eth0,eth1,eth2"
    verify the IPs in VR. eth0 -> control nic, eth1 -> source nat IP, eth2 -> tier 1

  6. create new public ip range 1

  7. get a free ip in new ip range, assign to network, and create port forwarding rules (ssh) to the vm
    verify the available nics in VR should be "eth0,eth1,eth2,eth3"
    verify the IPs in VR. eth1 -> source nat IP, eth2 -> tier 1, eth3 -> new ip 1

  8. get a free ip in new ip range, assign to network, and create port forwarding rules (ssh) to the vm
    verify the available nics in VR should be "eth0,eth1,eth2,eth3"
    verify the IPs in VR. eth1 -> source nat IP, eth2 -> tier 1, eth3 -> new ip 1/2

  9. get a free ip in new ip range, assign to network, and create port forwarding rules (ssh) to the vm
    verify the available nics in VR should be "eth0,eth1,eth2,eth3"
    verify the IPs in VR. eth1 -> source nat IP, eth2 -> tier 1, eth3 -> new ip 1/2/3

  10. release new ip 2
    verify the available nics in VR should be "eth0,eth1,eth2,eth3"
    verify the IPs in VR. eth1 -> source nat IP, eth2 -> tier 1, eth3 -> new ip 1/3

  11. release new ip 1
    verify the available nics in VR should be "eth0,eth1,eth2,eth3"
    verify the IPs in VR. eth1 -> source nat IP, eth2 -> tier 1, eth3 -> new ip 3

  12. create a tier 2 in the vpc, and create a vm 2 in the tier2.
    verify the available nics in VR should be "eth0,eth1,eth2,eth3,eth4"
    verify the IPs in VR. eth1 -> source nat IP, eth2 -> tier 1, eth3 -> new ip 3, eth4 -> tier 2

  13. create new public ip range 2

  14. get a free ip 4 in new ip range 2, assign to network, and create port forwarding rules (ssh) to the vm
    verify the available nics in VR should be "eth0,eth1,eth2,eth3,eth4,eth5,"
    verify the IPs in VR. eth1 -> source nat IP, eth2 -> tier 1, eth3 -> new ip 3, eth4 -> tier 2, eth5 -> new ip 4

  15. get a free ip 5 in new ip range 2, assign to network, and create port forwarding rules (ssh) to the vm
    verify the available nics in VR should be "eth0,eth1,eth2,eth3,eth4,eth5,"
    verify the IPs in VR. eth1 -> source nat IP, eth2 -> tier 1, eth3 -> new ip 3, eth4 -> tier 2, eth5 -> new ip 4/5

  16. get a free ip 6 in new ip range 2, assign to network, and create port forwarding rules (ssh) to the vm
    verify the available nics in VR should be "eth0,eth1,eth2,eth3,eth4,eth5,"
    verify the IPs in VR. eth1 -> source nat IP, eth2 -> tier 1, eth3 -> new ip 3, eth4 -> tier 2, eth5 -> new ip 4/5/6

  17. release new ip 5
    verify the available nics in VR should be "eth0,eth1,eth2,eth3,eth4,eth5,"
    verify the IPs in VR. eth1 -> source nat IP, eth2 -> tier 1, eth3 -> new ip 3, eth4 -> tier 2, eth5 -> new ip 4/6

  18. release new ip 4
    verify the available nics in VR should be "eth0,eth1,eth2,eth3,eth4,eth5,"
    verify the IPs in VR. eth1 -> source nat IP, eth2 -> tier 1, eth3 -> new ip 3, eth4 -> tier 2, eth5 -> new ip 6

  19. release new ip 3
    verify the available nics in VR should be "eth0,eth1,eth2,eth4,eth5,"
    verify the IPs in VR. eth1 -> source nat IP, eth2 -> tier 1, eth4 -> tier 2, eth5 -> new ip 6

  20. restart tier1
    verify the available nics in VR should be "eth0,eth1,eth2,eth4,eth5,"
    verify the IPs in VR. eth1 -> source nat IP, eth2 -> tier 1, eth4 -> tier 2, eth5 -> new ip 6

  21. restart tier2
    verify the available nics in VR should be "eth0,eth1,eth2,eth4,eth5,"
    verify the IPs in VR. eth1 -> source nat IP, eth2 -> tier 1, eth4 -> tier 2, eth5 -> new ip 6

  22. restart VPC
    verify the available nics in VR should be "eth0,eth1,eth2,eth4,eth5,"
    verify the IPs in VR. eth1 -> source nat IP, eth2 -> tier 1, eth4 -> tier 2, eth5 -> new ip 6

  23. restart VPC with cleanup
    verify the available nics in VR should be "eth0,eth1,eth2,eth3,eth4,"
    verify the IPs in VR. eth1 -> source nat IP, eth2 -> new ip 6, eth3 -> tier 1, eth4 -> tier 2

  24. restart VPC with cleanup, makeredundant=true
    verify the available nics in VR should be "eth0,eth1,eth2,eth3,eth4,"
    verify the IPs in VR. eth1 -> source nat IP, eth2 -> new ip 6, eth3 -> tier 1, eth4 -> tier 2

@weizhouapache

Copy link
Copy Markdown
MemberAuthor

@svenvogel@DennisKonrad
rebased with latest master. please test it.

@weizhouapache

Copy link
Copy Markdown
MemberAuthor

@DaanHoogland @rhtyd could you please kick travis test ?

@yadvryadvr added this to the 4.14.0.0 milestone Jan 28, 2020
@yadvr

Copy link
Copy Markdown
Member

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-677

@yadvr

Copy link
Copy Markdown
Member

@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

Trillian test result (tid-828)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 461 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3847-t828-kvm-centos7.zip
Smoke tests completed. 0 look OK, 1 have error(s)
Only failed tests results shown below:

TestResultTime (s)Test File

@svenvogel

Copy link
Copy Markdown
Contributor

@weizhouapache thanks. we will test it and come back fastly.

@yadvryadvr closed this Jan 29, 2020
@yadvryadvr reopened this Jan 29, 2020

@DaanHooglandDaanHoogland left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code lgtm, style remarks only.

Comment on lines +673 to +684
Long guestNetworkId = null;
final List<NicVO> nics = _nicDao.listByVmId(router.getId());
for (final NicVO nic : nics) {
final NetworkVO nw = _networkDao.findById(nic.getNetworkId());
if (nw.getTrafficType() == TrafficType.Guest) {
guestNetworkId = nw.getId();
break;
}
}

Map<String, Boolean> vlanLastIp = getVlanLastIpMap(router.getVpcId(), guestNetworkId);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could this be a aeparate method? getVlanLastIpMapForGuestNetwork();?

and than later

Comment on lines +744 to +745
Boolean lastIp = vlanLastIp.get(vlanTagKey);
if (lastIp == null) {
cmd.setAccessDetail(NetworkElementCommand.NETWORK_PUB_LAST_IP, "true");
} else {
cmd.setAccessDetail(NetworkElementCommand.NETWORK_PUB_LAST_IP, "false");
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here useAccessDetailsAccordingToVlanLastIpForGuestNetwork(); (long names just to be descriptive

@yadvr

Copy link
Copy Markdown
Member

I'll wait for your testing @svenvogel and then review/test. Thanks.

for (final InterfaceDef pluggedNic : pluggedNics) {
final String pluggedVlan = pluggedNic.getBrName();
macAddressToNicNum.put(pluggedNic.getMacAddress(), devNum);
devNum++;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicated lines (1797 - 1805=1838 - 1847). Can you please extract them into a method?

Comment threadsystemvm/debian/etc/sysctl.conf Outdated
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0

# Prmote secondary ip to be primary if primary IP is removed

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a typo at line 30

@DaanHooglandDaanHoogland modified the milestones: 4.14.0.0, 4.13.1.0Feb 3, 2020
@DaanHoogland

Copy link
Copy Markdown
Contributor

@weizhouapache , as the issue (#3179) is marked for 4.13.1 can you rebase, please?

@DennisKonrad

Copy link
Copy Markdown
Contributor

@weizhouapache @rhtyd I'm testing this since last week and as of now the PR looks very solid. I'll come back in some days and add the review when I'm done

@weizhouapache

Copy link
Copy Markdown
MemberAuthor

@DaanHoogland@GabrielBrascher thanks for review.
I will rebase with 4.13 and test it.

@blueorangutan

Copy link
Copy Markdown

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@apacheapache deleted a comment from blueorangutanFeb 20, 2020
@yadvr

Copy link
Copy Markdown
Member

@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

Trillian test result (tid-1065)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 25984 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3847-t1065-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Smoke tests completed. 76 look OK, 1 have error(s)
Only failed tests results shown below:

TestResultTime (s)Test File
test_02_vpc_privategw_static_routesFailure168.16test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanupFailure170.77test_privategw_acl.py
test_04_rvpc_privategw_static_routesFailure229.63test_privategw_acl.py

@blueorangutan

Copy link
Copy Markdown

Trillian test result (tid-1066)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 26123 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3847-t1066-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Smoke tests completed. 76 look OK, 1 have error(s)
Only failed tests results shown below:

TestResultTime (s)Test File
test_02_vpc_privategw_static_routesFailure170.89test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanupFailure168.77test_privategw_acl.py
test_04_rvpc_privategw_static_routesFailure239.88test_privategw_acl.py

@DaanHoogland

Copy link
Copy Markdown
Contributor

@weizhouapache@andrijapanicsb @rhtyd @DennisKonrad I did not have a chance to test xenserver and won't this week. Are we ok, to merge this as is? regression on xenserver shows ok ^^.

@svenvogel

svenvogel commented Feb 20, 2020 via email

Copy link
Copy Markdown
Contributor

@DaanHoogland

Copy link
Copy Markdown
Contributor

thanks @weizhouapache for the extensive added integration test coverage. You put them in a package called leaseweb. Can you explain why? (to a wider audience)

@weizhouapache

Copy link
Copy Markdown
MemberAuthor

thanks @weizhouapache for the extensive added integration test coverage. You put them in a package called leaseweb. Can you explain why? (to a wider audience)

@DaanHoogland moved to test/integration/component/

@yadvr

Copy link
Copy Markdown
Member

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-966

@yadvr

Copy link
Copy Markdown
Member

@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

Trillian test result (tid-1144)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 29590 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3847-t1144-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_nic.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Smoke tests completed. 75 look OK, 2 have error(s)
Only failed tests results shown below:

TestResultTime (s)Test File
test_01_nicError60.73test_nic.py
test_02_vpc_privategw_static_routesFailure182.90test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanupFailure189.94test_privategw_acl.py
test_04_rvpc_privategw_static_routesFailure278.09test_privategw_acl.py

@DaanHoogland

Copy link
Copy Markdown
Contributor

@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

Trillian test result (tid-1158)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 27010 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3847-t1158-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_nic.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Smoke tests completed. 75 look OK, 2 have error(s)
Only failed tests results shown below:

TestResultTime (s)Test File
test_01_nicError50.21test_nic.py
test_02_vpc_privategw_static_routesFailure177.43test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanupFailure172.22test_privategw_acl.py
test_04_rvpc_privategw_static_routesFailure229.26test_privategw_acl.py

@DaanHoogland

Copy link
Copy Markdown
Contributor

as in #3553 (comment) this nic error is new. We might have merged something that causes it, but it disappeared after applying insanity there.
@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

Trillian test result (tid-1159)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 27673 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3847-t1159-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_iso.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Smoke tests completed. 74 look OK, 3 have error(s)
Only failed tests results shown below:

TestResultTime (s)Test File
test_04_extract_IsoFailure1.07test_iso.py
test_02_vpc_privategw_static_routesFailure169.93test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanupFailure170.06test_privategw_acl.py
test_04_rvpc_privategw_static_routesFailure224.93test_privategw_acl.py
test_04_extract_templateFailure1.06test_templates.py

@DaanHoogland

Copy link
Copy Markdown
Contributor

ok, other errors without recompile, merging

@DaanHoogland
DaanHoogland merged commit 313e21a into apache:4.13Feb 29, 2020
DaanHoogland added a commit that referenced this pull request Feb 29, 2020
* 4.13:
VR: Fix Redundant VRouter guest network on wrong interface (#3847)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Redundant VRouter guest network on wrong interface

9 participants

@weizhouapache@yadvr@blueorangutan@svenvogel@DaanHoogland@DennisKonrad@andrijapanicsb@GabrielBrascher@ustcweizhou