server: Do NOT cleanup dhcp and dns when stop a vm by ustcweizhou · Pull Request #3627 · apache/cloudstack

ustcweizhou · 2019-10-08T08:00:03Z

Description

According comment in PR #3608, dhcp and dns entries are cleaned up only when a VM is expunged.
Revert part of commit 8fb388e.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

According comment in PR apache#3608, dhcp and dns entries are cleaned up only when a VM is expunged. Revert part of commit 8fb388e.

ustcweizhou · 2019-10-08T08:00:26Z

@rhtyd

yadvr · 2019-10-08T08:23:10Z

@blueorangutan package

blueorangutan · 2019-10-08T08:24:07Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-10-08T08:46:08Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-311

yadvr · 2019-10-08T09:06:09Z

@blueorangutan test

blueorangutan · 2019-10-08T09:07:08Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

yadvr · 2019-10-10T11:36:35Z

@blueorangutan test

blueorangutan · 2019-10-10T11:37:17Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

anuragaw

Haven't tested manually but change seems in line with PR description and added fix for removing entries only on VM expunge.

andrijapanicsb · 2019-10-10T11:51:56Z

This absolutely requires manual testing (that also means #3608).

No merging until tested thoroughly, please!
(I'll test myself as well)

yadvr · 2019-10-10T11:59:32Z

We'll wait for your test confirmation @andrijapanicsb

andrijapanicsb · 2019-10-10T21:07:38Z

In short - NOTHING IS CLEANED at all, whatsoever...

Test:
ENV built based from packages built from this PR (VMware 6.5u, CenOS7 mgmt) - clean install.

Created Isolated network (R21 in examples below)
Created Shared network (R22 in examples below)

Created a "vm1" (and some more VMs, but irrelevant) and joined vm1 to:

Isolated network (default one) , IP 10.1.1.101
Shared network (additional network), IP 1.1.1.4

While VM running, inside R21 (Isolated)

root@r-21-VM:~# grep -ir "10\.1\.1\.101" /var/; grep -ir "10\.1\.1\.101" /etc/
/var/www/html/userdata/10.1.1.101/.htaccess:Allow from 10.1.1.101
/var/www/html/metadata/10.1.1.101/local-ipv4:10.1.1.101
/var/www/html/metadata/10.1.1.101/.htaccess:Allow from 10.1.1.101
/var/cache/cloud/passwords-10.1.1.1:10.1.1.101=saved_password
1","default_gateway":"10.1.1.1","default_entry":true,"remove":false,"type":"dhcpentry"}
/var/cache/cloud/processed/VR-df1e8224-8c07-4e8d-9006-191c1f48224a.cfg:{"vm_ip_address":"10.1.1.101","vm_metadata":[["userdata","user-data",null],["metadata","service-offering","Small Instance"],["metadata","availability-zone","pr3627-t408-vmware-65u2"],["metadata","local-ipv4","10.1.1.101"],["metadata","local-hostname","vm1"],["metadata","public-ipv4","10.1.35.64"],["metadata","public-hostname","10.1.35.64"],["metadata","instance-id","75c43c29-a101-458f-81c6-8fe6e7546890"],["metadata","vm-id","75c43c29-a101-458f-81c6-8fe6e7546890"],["metadata","public-keys",null],["metadata","cloud-identifier","CloudStack-{f36badac-df80-40f2-a1f2-ab3a9ea41259}"]],"type":"vmdata"}
/var/lib/misc/dnsmasq.leases:1573462858 02:00:2d:24:00:0e 10.1.1.101 vm1 *
/var/log/cloud.log:2019-10-10 20:52:55,009 INFO Processing metadata for 10.1.1.101
/var/log/cloud.log:2019-10-10 20:53:08,194 INFO Processing metadata for 10.1.1.101
/var/log/cloud.log:2019-10-10 20:59:14,675 configure.py process:330 Processing metadata for 10.1.1.101
/var/log/dnsmasq.log:Oct 10 21:00:58 dnsmasq-dhcp[2826]: DHCPDISCOVER(eth0) 10.1.1.101 02:00:2d:24:00:0e
/var/log/dnsmasq.log:Oct 10 21:00:58 dnsmasq-dhcp[2826]: DHCPOFFER(eth0) 10.1.1.101 02:00:2d:24:00:0e
/var/log/dnsmasq.log:Oct 10 21:00:58 dnsmasq-dhcp[2826]: DHCPREQUEST(eth0) 10.1.1.101 02:00:2d:24:00:0e
/var/log/dnsmasq.log:Oct 10 21:00:58 dnsmasq-dhcp[2826]: DHCPACK(eth0) 10.1.1.101 02:00:2d:24:00:0e vm1
/var/log/messages:Oct 10 20:59:13 systemvm passwd_server_ip.py: serve_password: password saved for VM IP 10.1.1.101
/etc/dhcphosts.txt:02:00:2d:24:00:0e,10.1.1.101,vm1,756h
/etc/cloudstack/vmdata.json: "10.1.1.101": [
/etc/cloudstack/vmdata.json: "10.1.1.101"
/etc/cloudstack/vmpassword.json: "10.1.1.101": "saved_password",
/etc/cloudstack/dhcpentry.json: "10.1.1.101": {
/etc/cloudstack/dhcpentry.json: "ipv4_address": "10.1.1.101",
/etc/hosts:10.1.1.101 vm1

vm1 expunged, inside R21 (Isolated)

root@r-21-VM:~# grep -ir "10\.1\.1\.101" /var/; grep -ir "10\.1\.1\.101" /etc/
/var/www/html/userdata/10.1.1.101/.htaccess:Allow from 10.1.1.101
/var/www/html/metadata/10.1.1.101/local-ipv4:10.1.1.101
/var/www/html/metadata/10.1.1.101/.htaccess:Allow from 10.1.1.101
/var/cache/cloud/passwords-10.1.1.1:10.1.1.101=saved_password
/var/cache/cloud/processed/VR-df1e8224-8c07-4e8d-9006-191c1f48224a.cfg:{"host_name":"vm1","mac_address":"02:00:2d:24:00:0e","ipv4_address":"10.1.1.101","ipv6_duid":"00:03:00:01:02:00:2d:24:00:0e","dns_addresses":"10.1.1.1","default_gateway":"10.1.1.1","default_entry":true,"remove":false,"type":"dhcpentry"}
/var/cache/cloud/processed/VR-df1e8224-8c07-4e8d-9006-191c1f48224a.cfg:{"vm_ip_address":"10.1.1.101","vm_metadata":[["userdata","user-data",null],["metadata","service-offering","Small Instance"],["metadata","availability-zone","pr3627-t408-vmware-65u2"],["metadata","local-ipv4","10.1.1.101"],["metadata","local-hostname","vm1"],["metadata","public-ipv4","10.1.35.64"],["metadata","public-hostname","10.1.35.64"],["metadata","instance-id","75c43c29-a101-458f-81c6-8fe6e7546890"],["metadata","vm-id","75c43c29-a101-458f-81c6-8fe6e7546890"],["metadata","public-keys",null],["metadata","cloud-identifier","CloudStack-{f36badac-df80-40f2-a1f2-ab3a9ea41259}"]],"type":"vmdata"}
/var/lib/misc/dnsmasq.leases:1573462858 02:00:2d:24:00:0e 10.1.1.101 vm1 *
/var/log/cloud.log:2019-10-10 20:52:55,009 INFO Processing metadata for 10.1.1.101
/var/log/cloud.log:2019-10-10 20:53:08,194 INFO Processing metadata for 10.1.1.101
/var/log/cloud.log:2019-10-10 20:59:14,675 configure.py process:330 Processing metadata for 10.1.1.101
/var/log/dnsmasq.log:Oct 10 21:00:58 dnsmasq-dhcp[2826]: DHCPDISCOVER(eth0) 10.1.1.101 02:00:2d:24:00:0e
/var/log/dnsmasq.log:Oct 10 21:00:58 dnsmasq-dhcp[2826]: DHCPOFFER(eth0) 10.1.1.101 02:00:2d:24:00:0e
/var/log/dnsmasq.log:Oct 10 21:00:58 dnsmasq-dhcp[2826]: DHCPREQUEST(eth0) 10.1.1.101 02:00:2d:24:00:0e
/var/log/dnsmasq.log:Oct 10 21:00:58 dnsmasq-dhcp[2826]: DHCPACK(eth0) 10.1.1.101 02:00:2d:24:00:0e vm1
/var/log/messages:Oct 10 20:59:13 systemvm passwd_server_ip.py: serve_password: password saved for VM IP 10.1.1.101
/etc/dhcphosts.txt:02:00:2d:24:00:0e,10.1.1.101,vm1,756h
/etc/cloudstack/vmdata.json: "10.1.1.101": [
/etc/cloudstack/vmdata.json: "10.1.1.101"
/etc/cloudstack/vmpassword.json: "10.1.1.101": "saved_password",
/etc/cloudstack/dhcpentry.json: "10.1.1.101": {
/etc/cloudstack/dhcpentry.json: "ipv4_address": "10.1.1.101",
/etc/hosts:10.1.1.101 vm1

Identical results are for R22 (Shared network) - zero difference in grep results (not sharing, for brevity)

andrijapanicsb · 2019-10-10T21:08:41Z

Additional thing I saw - when you restart network with cleanup - dhcp.leases file is completely empty (probably some more issues in other dns and dhcp related files)

andrijapanicsb · 2019-10-10T21:28:27Z

Tested with a VM attached just to Isolated network - same results.

blueorangutan · 2019-10-10T22:02:28Z

Trillian test result (tid-406)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 35774 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3627-t406-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Smoke tests completed. 76 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_05_rvpc_multi_tiers	`Failure`	414.14	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	443.62	test_vpc_redundant.py

ustcweizhou · 2019-10-15T08:14:49Z

@andrijapanicsb thanks a lot for testing.
the issue is not caused by this PR, but PR 3608 which has been merged.
in finalizeExpunge part, nics has already been removed so no rules will be applied in virtual routers.
we should move it to _networkMgr.cleanupNics

I will test it and update this PR if it is tested ok.

ustcweizhou · 2019-10-15T09:33:50Z

@andrijapanicsb I have updated this PR and tested ok. could you please test it ?

Before expunging vm

root@r-39-VM:~# cat /var/lib/misc/dnsmasq.leases
1573859468 1e:00:33:00:00:09 10.11.120.232 vm1 *
root@r-39-VM:~# cat /etc/dhcphosts.txt
1e:00:33:00:00:09,10.11.120.232,vm1,758h

nothing changes after stopping vm
after expunging vm

root@r-39-VM:~# cat /var/lib/misc/dnsmasq.leases
root@r-39-VM:~# cat /etc/dhcphosts.txt

yadvr · 2019-10-15T11:02:22Z

@andrijapanicsb I did not review your test, the test should be to check the dnsmasq related config file and even maybe the leases file.

andrijapanicsb · 2019-10-15T11:11:12Z

Thx @ustcweizhou @rhtyd - will test is as soon as possible.

blueorangutan · 2019-10-15T15:45:23Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-315

andrijapanicsb

All right - this seems to be good - I can confirm what @ustcweizhou shared.
I can see in logs proper the request to remove DHCP entries from the VR, later removing NICs of the VM and finally wiping the VM.
(another issue which I noticed, seems to be solved now with this PR)

Just a notice, on Network Restart with Clean UP:

/var/lib/misc/dnsmasq.leases file is wiped (same in 4.11.2 - i.e. before issues in 4.11.3)

This means that DHCP client will get DHCPNAK on its DHCPREQUEST to extend the lease - and will later fail back to the "initialization process" i.e. start from scratch with DHCPDISCOVER - so will again get it's IP address from the /etc/dhcphosts.txt file.

Once all tests pass, I'm OK for this to be merged.

LGTM

ustcweizhou · 2019-10-16T20:00:18Z

@andrijapanicsb that's good, thanks for testing.
@rhtyd can you set up a trillian test ?

andrijapanicsb · 2019-10-17T08:48:13Z

@blueorangutan package

blueorangutan · 2019-10-17T08:48:49Z

@andrijapanicsb a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-10-17T09:10:05Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-317

yadvr · 2019-10-17T09:21:13Z

@ustcweizhou @andrijapanicsb - have you tested the case that when a VM is stopped, its rules are not removed from the VR? I'll kick the test
@blueorangutan test

blueorangutan · 2019-10-17T09:21:48Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

andrijapanicsb · 2019-10-17T09:22:50Z

yes @rhtyd , tested pretty thoroughly. Also tested when VR is stopped (and VM expunge automatically kicks in, etc). Looks good to me. Was even restarting networks with and without cleanup - all good.

blueorangutan · 2019-10-17T19:22:04Z

Trillian test result (tid-413)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 34026 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3627-t413-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 76 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_05_rvpc_multi_tiers	`Failure`	433.28	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	462.09	test_vpc_redundant.py

andrijapanicsb · 2019-11-21T23:30:58Z

@weizhouapache @rhtyd I think I see a bit of a problem with this one (somewhat edge case).

VM1 has default network NET1
VM1 is attached to NET2 as an additional network - DNS and DHCP configs are created for this in the VR of NET2 (/etc/hosts and /etc/dhcphosts.txt
We later detach VM1 from NET2 (and potentially never again attach it) - garbage is left forever in VR of NET2 (this consumes IP if nothing else).

Can you think of any other consequence here?
I see that when attaching AGAIN VM1 to NET2, a NEW config (new IP) is used in DNS and DHCP config, while the OLD IP is still present (in /etc/hosts and /etc/dhcphosts.txt) - so this will make issues for sure.

andrijapanicsb · 2019-11-21T23:42:55Z

So... when an additional network is detached from the VM, and we later stop/delete/expunge that VM - nothing happens, since we only clean DHCP/DNS from the VRs/networks to which the VM is still attached.

Possibly a correct fix would be to ALSO do the same cleanup (that we otherwise do only when VM is expunging) when DETACHING from an ADDITIONAL network. Well, when detaching from ANY network - since after VM is detached, its reference(s) to its previous network(s) is lost and will never be later cleaned up (dhcp/dns config) when VM is destroyed

Makes sense @ustcweizhou @rhtyd /CC @PaulAngus ?

yadvr · 2019-11-22T08:22:11Z

This PR is closed, we might lose the discussion @andrijapanicsb - can you create a ticket with your concerns? (i.e. to cleanup on vm nic remove/detach?)

weizhouapache · 2019-11-22T08:36:05Z

@andrijapanicsb make sense. we need to remove dhcp/dns entries in VR when remove a nic from vm.

andrijapanicsb · 2019-11-22T10:22:06Z

To be continued via: #3706

thx both!

server: Do NOT cleanup dhcp and dns when stop a vm
cecb713
According comment in PR apache#3608, dhcp and dns entries are cleaned up only when a VM is expunged. Revert part of commit 8fb388e.

ustcweizhou changed the base branch from master to 4.13October 8, 2019 08:00

yadvr approved these changes Oct 8, 2019
View reviewed changes

svenvogel added type:bug component:virtual-router labels Oct 9, 2019

yadvr requested review from andrijapanicsb and nvazquez October 10, 2019 11:36

nvazquez approved these changes Oct 10, 2019
View reviewed changes

anuragaw approved these changes Oct 10, 2019
View reviewed changes

andrijapanicsb changed the title ~~server: Do NOT cleanup dhcp and dns when stop a vm~~[do not merge] server: Do NOT cleanup dhcp and dns when stop a vmOct 10, 2019

server: cleanup dns/dhcp entries in removeNic instead of finalizeExpunge
06946d7

andrijapanicsb approved these changes Oct 16, 2019
View reviewed changes

yadvr merged commit cc27095 into apache:4.13Oct 17, 2019

yadvr modified the milestones: 4.13.1.0, 4.14.0.0 Oct 17, 2019

yadvr changed the title ~~[do not merge] server: Do NOT cleanup dhcp and dns when stop a vm~~server: Do NOT cleanup dhcp and dns when stop a vmOct 17, 2019

andrijapanicsb mentioned this pull request Oct 20, 2019
Multi-homed VMs stuck in expunging state #3645
Closed

This was referenced Oct 29, 2019
VR Loses Instance IP Address #3550
Closed
Increase DHCP lease time to infinite #3662
Merged

andrijapanicsb mentioned this pull request Nov 22, 2019
Remove dhcp/dns configuration from VR, when VM is being DETACHED from a network #3706
Closed

Uh oh!

Conversation

ustcweizhou commented Oct 8, 2019

Description

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

ustcweizhou commented Oct 8, 2019

Uh oh!

yadvr commented Oct 8, 2019

Uh oh!

blueorangutan commented Oct 8, 2019

Uh oh!

blueorangutan commented Oct 8, 2019

Uh oh!

yadvr commented Oct 8, 2019

Uh oh!

blueorangutan commented Oct 8, 2019

Uh oh!

yadvr commented Oct 10, 2019

Uh oh!

blueorangutan commented Oct 10, 2019

Uh oh!

anuragaw left a comment

Choose a reason for hiding this comment

Uh oh!

andrijapanicsb commented Oct 10, 2019

Uh oh!

yadvr commented Oct 10, 2019

Uh oh!

andrijapanicsb commented Oct 10, 2019

Uh oh!

andrijapanicsb commented Oct 10, 2019

Uh oh!

andrijapanicsb commented Oct 10, 2019

Uh oh!

blueorangutan commented Oct 10, 2019

Uh oh!

ustcweizhou commented Oct 15, 2019

Uh oh!

ustcweizhou commented Oct 15, 2019

Uh oh!

yadvr commented Oct 15, 2019

Uh oh!

andrijapanicsb commented Oct 15, 2019

Uh oh!

blueorangutan commented Oct 15, 2019

Uh oh!

andrijapanicsb left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ustcweizhou commented Oct 16, 2019

Uh oh!

andrijapanicsb commented Oct 17, 2019

Uh oh!

blueorangutan commented Oct 17, 2019

Uh oh!

blueorangutan commented Oct 17, 2019

Uh oh!

yadvr commented Oct 17, 2019

Uh oh!

blueorangutan commented Oct 17, 2019

Uh oh!

andrijapanicsb commented Oct 17, 2019

Uh oh!

blueorangutan commented Oct 17, 2019

Uh oh!

andrijapanicsb commented Nov 21, 2019

Uh oh!

andrijapanicsb commented Nov 21, 2019

Uh oh!

yadvr commented Nov 22, 2019

Uh oh!

weizhouapache commented Nov 22, 2019

Uh oh!

andrijapanicsb commented Nov 22, 2019

Uh oh!

Reviewers

andrijapanicsb left a comment •
edited
Loading