[4.7] CLOUDSTACK-8715: qemu-guest-agent support on KVM

[ustcweizhou]

@wido can you check if these codes are useful for you?

[wido]

Overall this seems OK, but I personally think that we can enable it by default on all KVM guests. It doesn't hurt to add the port.

We can detect if the port is open or not. Also, we don't need external Python code to send commands, we can do this directly from Java.

[ustcweizhou]

The virtio console is a bridge between hypervisor and vm instance. I am not sure if customers feel good/friendly with this.

I donot know if libvirt-java 0.5.1 supports qemu guest agent. If yes, we can send commands/data directly from java. It is another way for implementation.
python script and java have their own benifits.

[DaanHoogland]

ran the regression set with some f

nosetests --with-marvin --marvin-config=${marvinCfg} -s -a tags=advanced,required_hardware=true \
component/test_vpc_redundant.py \
component/test_routers_iptables_default_policy.py \
component/test_routers_network_ops.py \
component/test_vpc_router_nics.py \
component/test_password_server.py \
component/test_router_dhcphosts.py \
smoke/test_loadbalance.py \
smoke/test_internal_lb.py \
smoke/test_ssvm.py \
smoke/test_network.py

results:

Create a redundant VPC with two networks with two VMs in each network ... === TestName: test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | Status : SUCCESS ===
ok
Create a redundant VPC with two networks with two VMs in each network and check default routes ... === TestName: test_02_redundant_VPC_default_routes | Status : SUCCESS ===
ok
Test iptables default INPUT/FORWARD policy on RouterVM ... === TestName: test_02_routervm_iptables_policies | Status : SUCCESS ===
ok
Test iptables default INPUT/FORWARD policies on VPC router ... === TestName: test_01_single_VPC_iptables_policies | Status : SUCCESS ===
ok
Stop existing router, add a PF rule and check we can access the VM ... === TestName: test_isolate_network_FW_PF_default_routes | Status : SUCCESS ===
ok
Test redundant router internals ... === TestName: test_RVR_Network_FW_PF_SSH_default_routes | Status : SUCCESS ===
ok
Create a VPC with two networks with one VM in each network and test nics after destroy ... === TestName: test_01_VPC_nics_after_destroy | Status : SUCCESS ===
ok
Create a VPC with two networks with one VM in each network and test default routes ... === TestName: test_02_VPC_default_routes | Status : SUCCESS ===
ok
Check the password file in the Router VM ... === TestName: test_isolate_network_password_server | Status : SUCCESS ===
ok
Check that the /etc/dhcphosts.txt doesn't contain duplicate IPs ... === TestName: test_router_dhcphosts | Status : SUCCESS ===
ok
Test to create Load balancing rule with source NAT ... === TestName: test_01_create_lb_rule_src_nat | Status : SUCCESS ===
ok
Test to create Load balancing rule with non source NAT ... === TestName: test_02_create_lb_rule_non_nat | Status : SUCCESS ===
ok
Test for assign & removing load balancing rule ... === TestName: test_assign_and_removal_lb | Status : SUCCESS ===
ok
=== TestName: test_assign_and_removal_lb | Status : EXCEPTION ===
ERROR
Test SSVM Internals ... === TestName: test_03_ssvm_internals | Status : FAILED ===
FAIL
Test CPVM Internals ... === TestName: test_04_cpvm_internals | Status : SUCCESS ===
ok
Test stop SSVM ... === TestName: test_05_stop_ssvm | Status : SUCCESS ===
ok
Test stop CPVM ... === TestName: test_06_stop_cpvm | Status : SUCCESS ===
ok
Test reboot SSVM ... === TestName: test_07_reboot_ssvm | Status : SUCCESS ===
ok
Test reboot CPVM ... === TestName: test_08_reboot_cpvm | Status : SUCCESS ===
ok
Test destroy SSVM ... === TestName: test_09_destroy_ssvm | Status : SUCCESS ===
ok
Test destroy CPVM ... === TestName: test_10_destroy_cpvm | Status : SUCCESS ===
ok
Test for port forwarding on source NAT ... === TestName: test_01_port_fwd_on_src_nat | Status : SUCCESS ===
ok
Test for port forwarding on non source NAT ... === TestName: test_02_port_fwd_on_non_src_nat | Status : SUCCESS ===
ok
Test for reboot router ... === TestName: test_reboot_router | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_1_static_nat_rule | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_2_nat_rule | Status : SUCCESS ===
ok
Test for Router rules for network rules on acquired public IP ... === TestName: test_network_rules_acquired_public_ip_3_Load_Balancer_Rule | Status : SUCCESS ===
ok
======================================================================
ERROR: test suite for <class 'integration.smoke.test_internal_lb.TestInternalLb'>
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/nose/suite.py", line 209, in run
self.setUp()
File "/usr/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/nose/suite.py", line 292, in setUp
self.setupContext(ancestor)
File "/usr/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/nose/suite.py", line 315, in setupContext
try_run(context, names)
File "/usr/lib/python2.7/site-packages/nose-1.3.7-py2.7.egg/nose/util.py", line 471, in try_run
return func()
File "/data/git/cs1/cloudstack/test/integration/smoke/test_internal_lb.py", line 263, in setUpClass
cls.template.download(cls.apiclient)
File "/usr/lib/python2.7/site-packages/marvin/lib/base.py", line 1255, in download
elif 'Downloaded' in template.status:
TypeError: argument of type 'NoneType' is not iterable
-------------------- >> begin captured logging << --------------------
test_assign_and_removal_lb (integration.smoke.test_loadbalance.TestLoadBalance): DEBUG: Payload: {'signature': '0lBnC8qpCUtsGx9fAJLR+YjtGEw=', 'apiKey': u'Oae1-v8LPa8gLweImVlDyX2VZWEHzZxU-8hniEaGI8AsuRfQkz75pkNhvBT7QWDP-D84Xq7TdfVozWK3rqOrkw', 'command': 'deleteAccount', 'id': u'34e7f428-f031-4761-bb56-5e49f73f6627', 'response': 'json'}
<snip/>
======================================================================
FAIL: Test SSVM Internals
----------------------------------------------------------------------
Traceback (most recent call last):
File "/data/git/cs1/cloudstack/test/integration/smoke/test_ssvm.py", line 434, in test_03_ssvm_internals
"Check for Errors in tests"
AssertionError: Check for Errors in tests
-------------------- >> begin captured logging << --------------------
test_assign_and_removal_lb (integration.smoke.test_loadbalance.TestLoadBalance): DEBUG: Payload: {'signature': '0lBnC8qpCUtsGx9fAJLR+YjtGEw=', 'apiKey': u'Oae1-v8LPa8gLweImVlDyX2VZWEH
<snip/>
----------------------------------------------------------------------
Ran 27 tests in 13577.248s
FAILED (errors=1, failures=1)

and

nosetests --with-marvin --marvin-config=${marvinCfg} -s -a tags=advanced,required_hardware=false \
smoke/test_routers.py \
smoke/test_network_acl.py \
smoke/test_privategw_acl.py \
smoke/test_reset_vm_on_reboot.py \
smoke/test_vm_life_cycle.py \
smoke/test_vpc_vpn.py \
smoke/test_service_offerings.py \
smoke/test_network.py \
component/test_vpc_offerings.py \
component/test_vpc_routers.py

results:

Test router internal advanced zone ... === TestName: test_02_router_internal_adv | Status : SUCCESS ===
ok
Test restart network ... === TestName: test_03_restart_network_cleanup | Status : SUCCESS ===
ok
Test router basic setup ... === TestName: test_05_router_basic | Status : SUCCESS ===
ok
Test router advanced setup ... === TestName: test_06_router_advanced | Status : SUCCESS ===
ok
Test stop router ... === TestName: test_07_stop_router | Status : SUCCESS ===
ok
Test start router ... === TestName: test_08_start_router | Status : SUCCESS ===
ok
Test reboot router ... === TestName: test_09_reboot_router | Status : SUCCESS ===
ok
test_privategw_acl (integration.smoke.test_privategw_acl.TestPrivateGwACL) ... === TestName: test_privategw_acl | Status : SUCCESS ===
ok
Test reset virtual machine on reboot ... === TestName: test_01_reset_vm_on_reboot | Status : SUCCESS ===
ok
Test advanced zone virtual router ... === TestName: test_advZoneVirtualRouter | Status : SUCCESS ===
ok
Test Deploy Virtual Machine ... === TestName: test_deploy_vm | Status : SUCCESS ===
ok
Test Multiple Deploy Virtual Machine ... === TestName: test_deploy_vm_multiple | Status : SUCCESS ===
ok
Test Stop Virtual Machine ... === TestName: test_01_stop_vm | Status : SUCCESS ===
ok
Test Start Virtual Machine ... === TestName: test_02_start_vm | Status : SUCCESS ===
ok
Test Reboot Virtual Machine ... === TestName: test_03_reboot_vm | Status : SUCCESS ===
ok
Test destroy Virtual Machine ... === TestName: test_06_destroy_vm | Status : SUCCESS ===
ok
Test recover Virtual Machine ... === TestName: test_07_restore_vm | Status : SUCCESS ===
ok
Test migrate VM ... === TestName: test_08_migrate_vm | Status : SUCCESS ===
ok
Test destroy(expunge) Virtual Machine ... === TestName: test_09_expunge_vm | Status : SUCCESS ===
ok
Test Remote Access VPN in VPC ... === TestName: test_vpc_remote_access_vpn | Status : SUCCESS ===
ok
Test VPN in VPC ... === TestName: test_vpc_site2site_vpn | Status : SUCCESS ===
ok
Test to create service offering ... === TestName: test_01_create_service_offering | Status : SUCCESS ===
ok
Test to update existing service offering ... === TestName: test_02_edit_service_offering | Status : SUCCESS ===
ok
Test to delete service offering ... === TestName: test_03_delete_service_offering | Status : SUCCESS ===
ok
Test for delete account ... === TestName: test_delete_account | Status : SUCCESS ===
ok
Test for Associate/Disassociate public IP address for admin account ... === TestName: test_public_ip_admin_account | Status : SUCCESS ===
ok
Test for Associate/Disassociate public IP address for user account ... === TestName: test_public_ip_user_account | Status : SUCCESS ===
ok
Test for release public IP address ... === TestName: test_releaseIP | Status : SUCCESS ===
ok
Test create VPC offering ... === TestName: test_01_create_vpc_offering | Status : SUCCESS ===
ok
Test VPC offering without load balancing service ... === TestName: test_03_vpc_off_without_lb | Status : SUCCESS ===
ok
Test VPC offering without static NAT service ... === TestName: test_04_vpc_off_without_static_nat | Status : SUCCESS ===
ok
Test VPC offering without port forwarding service ... === TestName: test_05_vpc_off_without_pf | Status : SUCCESS ===
ok
Test VPC offering with invalid services ... === TestName: test_06_vpc_off_invalid_services | Status : SUCCESS ===
ok
Test update VPC offering ... === TestName: test_07_update_vpc_off | Status : SUCCESS ===
ok
Test list VPC offering ... === TestName: test_08_list_vpc_off | Status : SUCCESS ===
ok
test_09_create_redundant_vpc_offering (integration.component.test_vpc_offerings.TestVPCOffering) ... === TestName: test_09_create_redundant_vpc_offering | Status : SUCCESS ===
ok
Test start/stop of router after addition of one guest network ... === TestName: test_01_start_stop_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test reboot of router after addition of one guest network ... === TestName: test_02_reboot_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test to change service offering of router after addition of one guest network ... === TestName: test_04_chg_srv_off_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test destroy of router after addition of one guest network ... === TestName: test_05_destroy_router_after_addition_of_one_guest_network | Status : SUCCESS ===
ok
Test to stop and start router after creation of VPC ... === TestName: test_01_stop_start_router_after_creating_vpc | Status : SUCCESS ===
ok
Test to reboot the router after creating a VPC ... === TestName: test_02_reboot_router_after_creating_vpc | Status : SUCCESS ===
ok
Tests to change service offering of the Router after ... === TestName: test_04_change_service_offerring_vpc | Status : SUCCESS ===
ok
Test to destroy the router after creating a VPC ... === TestName: test_05_destroy_router_after_creating_vpc | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 44 tests in 12621.592s
OK

The two failures seem not related so I will run them again.

[wido]

@ustcweizhou I don't see the difference personally, as a admin you already have full access to the Instance if you want to.

But this PR conflicts with mine, they can't be merged both :)

[sspans]

Consistent Snapshots are not possible without freeze/thaw via the guest-agent.
I think most people want good snapshots of their virtual machines.

That said it's quite possible to disable the agent on particularly sensitive VM's if one feels the need,
the channel will still be open but nobody will be listening

[wido]

@sspans Indeed. Users can stop/disable the agent or even block specific commands if they wish to. As CloudStack we don't control what's in the Instance, so it's the user who decides if he/she wants the Agent or not.

[wido]

So I wrote #985 which adds a channel by default to all guests. The question is, how do we proceed. Which one goes in?

[ustcweizhou]

@wido this PR already includes your commits in #985

[wido]

@ustcweizhou Ah, I see indeed.

So the discussion is:

• Do we enable the port by default? I would say yes. (Less code!)
• Do we enable the port for SSVMs? I would say yes (Easier control over SSVMs)
• Do we need Python? I would say no, this can be done from Java

[ustcweizhou]

@wido@sspans
To be honest, this code was written by me about two years ago , which is not used for now. I created the PR because you guys can judge this code are useful or not. I have no comments if we revert some of the commits (or even all).

1. for the default port, if most of you agree, I will revert the related commits.
2. I agree. I did not remove it in the commits. However, I change the 'vmName'.vport as the first virtio port, because cloud-early-config get the publickey and boot args from /dev/vport0p1.
3. rom my point of view, python is easy for testing, the data send/receive are already implemented. If you want to replace it with Java, that's very good. for now, python works fine in my testing, I even use it for guest agent command testing. btw, do you want replace patachviasocket.pl with java ? I replace it with the python script in the PR.

[wido]

@ustcweizhou I looked at the code more closely and it looks good.

However, I would still vote for having the Agent port always enabled. This saves a lot of code and a lot of testing as well.

I would also remove the Python code and write a Java Util which directly talks to the local Unix socket.

So, we always enable a Guest Agent port and when we want to talk to the Instance we send a ping. If it responds, it responds. If not, we assume there is no Agent in there.

[ustcweizhou]

@wido thanks for your review. good to hear that.
There are 5 commits by me in this PR. You can cherry-pick the commits you need to your branch and continue your work :)

The function checkGuestAgentSync in the commit is used to verify if the guest agent works.

[wido]

I'm going to take a look at this via @borisroman today.

I'm also working on a patch for libvirt-java, since using those bindings you can also send Qemu Guest Agent commands, even easier :)

[NuxRo]

Great stuff, +1 for having the channel enabled by default.

[wido]

I've just send a patch upstream to libvirt-java so that we can use Libvirt for sending commands to Instances: https://www.redhat.com/archives/libvir-list/2015-November/msg00247.html

A small piece of example Java code: https://gist.github.com/wido/30457cbe66e3965f00de

I will work on my PR #985 to get this all imported.

[wido]

@ustcweizhou Could you close this PR? #985 has some new patches which use libvirt. Some of your code also went in there :)

[ustcweizhou]

@wido sure , closed.