CLOUDSTACK-8715: Add VirtIO channel to all Instances for the Qemu Gue…

[wido]

…st Agent

This commit adds a additional VirtIO channel with the name 'org.qemu.guest_agent.0'
to all Instances.

With the Qemu Guest Agent the Hypervisor gains more control over the Instance if
these tools are present inside the Instance, for example:

• Power control
• Flushing filesystems

In the future this should allow safer snapshots on KVM since we can instruct the
Instance to flush the filesystems prior to snapshotting the disk.

More information: http://wiki.qemu.org/Features/QAPI/GuestAgent

[remibergsma]

@wido Thanks for working on this, awesome!

I was testing this PR and found that the SSVM and CPVM systemvms do not start properly. This is the error:

2015-10-28 12:36:30,145 WARN [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-4:null) Li
bvirtException org.libvirt.LibvirtException: internal error: process exited while connecting to monitor: 2015-10-28T12:36:2
9.904910Z qemu-kvm: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/v-273-VM.org.qemu.guest_agent.0,server,nowait: Failed to bind socket: Permission denied
2015-10-28T12:36:29.904949Z qemu-kvm: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/v-273-VM.org.qemu.guest_agent.0,server,nowait: chardev: opening backend "socket" failed
at org.libvirt.ErrorHandler.processError(Unknown Source)
at org.libvirt.Connect.processError(Unknown Source)
at org.libvirt.Connect.processError(Unknown Source)
at org.libvirt.Connect.domainCreateXML(Unknown Source)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1292)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:82)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:46)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:75)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1316)
at com.cloud.agent.Agent.processRequest(Agent.java:518)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:823)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Here is the channel dir:

[root@kvm1 channel]# ls -al /var/lib/libvirt/qemu/channel/
total 12
drwxr-x---. 3 qemu qemu 4096 Oct 28 09:55 .
drwxr-x---. 7 root root 4096 Oct 28 12:39 ..
drwxr-x---. 2 qemu qemu 4096 Sep 15 14:03 target

The target folder was empty.

I've installed the rpm package that I've built from this branch. The agent seems to run as root:

root 9815 0.4 3.1 4815960 255916 ? Sl 10:17 0:41 /usr/lib/jvm/jre/bin/java -Xms256m -Xmx2048 <cut>

It was tested on a KVM hypervisor on CentOS 7. Ping me if you need more details.

[wido]

@remibergsma Hmm, was SELinux enabled on that system? Can't see any reason why it wouldn't work. All the directories exist.

Do you by any chance have the XML the Agent generated for the SSVM for example?

[remibergsma]

Hi @wido :

SELinux is off:

[root@kvm1 channel]# getenforce Permissive

The generated XML:

2015-10-28 13:02:00,216 DEBUG [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-5:null) st
arting s-324-VM: <domain type='kvm'>
<name>s-324-VM</name>
<uuid>acdaffdf-9676-42eb-a386-f8aee0aaccca</uuid>
<description>Debian GNU/Linux 5.0 (64-bit)</description>
<cpu></cpu><sysinfo type='smbios'>
<system>
<entry name='manufacturer'>Apache Software Foundation</entry>
<entry name='product'>CloudStack KVM Hypervisor</entry>
<entry name='uuid'>acdaffdf-9676-42eb-a386-f8aee0aaccca</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='pc'>hvm</type>
<boot dev='cdrom'/>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<on_reboot>restart</on_reboot>
<on_poweroff>destroy</on_poweroff>
<on_crash>destroy</on_crash>
<memory>524288</memory>
<devices>
<memballoon model='none'/>
</devices>
<vcpu>1</vcpu>
<features>
<pae/>
<apic/>
<acpi/>
</features>
<cputune>
<shares>500</shares>
</cputune>
<clock offset='utc'>
<timer name='kvmclock' >
</timer>
</clock>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<interface type='bridge'>
<source bridge='cloud0'/>
<mac address='0e:00:a9:fe:00:66'/>
<model type='virtio'/>
<rom bar='off' file=''/></interface>
<interface type='bridge'>
<source bridge='cloudbr0'/>
<mac address='06:03:40:00:00:08'/>
<model type='virtio'/>
<rom bar='off' file=''/></interface>
<interface type='bridge'>
<source bridge='breth0-50'/>
<mac address='06:ce:f8:00:00:17'/>
<model type='virtio'/>
<rom bar='off' file=''/></interface>
<interface type='bridge'>
<source bridge='cloudbr0'/>
<mac address='06:16:8c:00:00:12'/>
<model type='virtio'/>
<rom bar='off' file=''/></interface>
<console type='pty'>
<target port='0'/>
</console>
<disk device='disk' type='file'>
<driver name='qemu' type='qcow2' cache='none' />
<source file='/mnt/812ea6a3-7ad0-30f4-9cab-01e3f2985b98/544cbd54-7cfc-423c-b956-c0135efa059f'/>
<target dev='vda' bus='virtio'/>
</disk>
<disk device='cdrom' type='file'>
<driver name='qemu' type='raw' cache='none' />
<source file='/usr/share/cloudstack-common/vms/systemvm.iso'/>
<target dev='hdc' bus='ide'/>
</disk>
<serial type='pty'>
<target port='0'/>
</serial>
<graphics type='vnc' autoport='yes' listen='192.168.22.21' passwd='NYq46aTBPtud7Nj4D62C9Q'/>
<channel type='unix'>
<source mode='bind'/>
<address type='virtio-serial'/>
<target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
</channel>
<channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/s-324-VM.agent'/>
<address type='virtio-serial'/>
<target type='virtio' name='s-324-VM.vport'/>
</channel>
<input type='tablet' bus='usb'/>
</devices>
</domain>

[wido]

@remibergsma Ok, that is odd.

A proper XML should look like this:

 <channel type='unix'>
<source mode='bind' path='/var/lib/libvirt/qemu/channel/target/man.org.qemu.guest_agent.0'/>
<target type='virtio' name='org.qemu.guest_agent.0'/>
<alias name='channel1'/>
<address type='virtio-serial' controller='0' bus='0' port='2'/>
</channel>

path is not required, that is computed by libvirt. But the state connected is probably the issue. I'll fix that.

[wido]

@borisroman Could you give this PR a spin?

[remibergsma]

@wido I will also test it again today.

[ustcweizhou]

I have same issue with Remi,
there are two virtio-serial in the vm definition.

[ustcweizhou]

succeed after the following change:

diff --git a/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java b/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingReso
index 4ce1889..59b0c3c 100755
--- a/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java
+++ b/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java
@@ -1994,13 +1994,13 @@ public class LibvirtComputingResource extends ServerResourceBase implements Serv
final SerialDef serial = new SerialDef("pty", null, (short)0);
devices.addDevice(serial);
- /* Add a VirtIO channel for the Qemu Guest Agent tools */
- devices.addDevice(new ChannelDef("org.qemu.guest_agent.0", ChannelDef.ChannelType.UNIX));
-
/* Add a VirtIO channel for SystemVMs for communication and provisioning */
if (vmTO.getType() != VirtualMachine.Type.User) {
devices.addDevice(new ChannelDef(vmTO.getName() + ".vport", ChannelDef.ChannelType.UNIX,
"/var/lib/libvirt/qemu/" + vmTO.getName() + ".agent"));
+ } else {
+ /* Add a VirtIO channel for the Qemu Guest Agent tools */
+ devices.addDevice(new ChannelDef("org.qemu.guest_agent.0", ChannelDef.ChannelType.UNIX));
}
final VideoDef videoCard = new VideoDef(_videoHw, _videoRam);

[wido]

Hmm, ok. That should be allowed. Since we want the Guest tools also to be supported on SSVMs to control them properly.

[ustcweizhou]

that is good idea. then we need to rebuild systemvm template with qemu-guest-agent

[ustcweizhou]

by the way, for user vms, I think it is better to add the field in vm_instance/user_vm or zone/cluster/global setting to determine if the vm is qemu guest agent enabled.

[ustcweizhou]

uservm can not start

2015-10-29 21:04:38,017 WARN resource.wrapper.LibvirtStartCommandWrapper LibvirtException
org.libvirt.LibvirtException: internal error Missing source path attribute for char device

[wido]

@ustcweizhou Which version of libvirt are you using?

If you use libvirt 1.0.6 or newer, you can omit the path='...' attribute of the  element, and libvirt will manage things automatically on your behalf.

[ustcweizhou]

@wido ja, you got it. The issue happened on a host running with Ubuntu 12.04 (QEMU 1.2.1 and libvirt 0.9.13)
There is no issue on Ubuntu 14.04 (QEMU 2.0.0 and libvirt 1.2.2)

[ustcweizhou]

@wido by the way, I just remember I have implemented some codes for qemu-guest-agent support , based on cloudstack 4.2.0 maybe.
It is not fully tested. I will share that with you if you need (maybe create a pull request to your github branch)

[wido]

@ustcweizhou@remibergsma I just pushed a new version of the commit.

On Ubuntu AppArmor needs to be disabled since the default profile for libvirt doesn't allow writing into /var/lib/libvirt/qemu. This is however already the case with the SSVM.

This could be fixed by adding this to '/etc/apparmor.d/abstractions/libvirt-qemu':

/var/lib/libvirt/qemu/channel/target/* rw,

[wido]

I justed pushed a new commit which uses libvirt-java for the communication with the Guest Agent.

The build will fail since this patch for libvirt-java has not been accepted upstream:

• https://www.redhat.com/archives/libvir-list/2015-November/msg00247.html
• wido/libvirt-java@7e38a53

[ustcweizhou]

@wido nice, I will test it.
does this need the change on libvirt-java ?

[wido]

@ustcweizhou Yes, it does. See the patches I posted above.

[wido]

Still working on this one. However, I need to get stuff into libvirt-java upstream. Working on that here: https://github.com/wido/libvirt-java/commits/qemu-guest-command

[NuxRo]

Ok, godspeed.
With a bit of luck perhaps we'll see this somewhere in 4.7.x?

[nislim]

Any progress on getting the patches into upstream libvirt-java? @wido

[wido]

@nislim No, not really. The people at libvirt-java aren't the fastest. Thinking about forking it into CloudStack itself.

[wido]

I just rebased the commit against master, but still waiting on libvirt-java.

Really think we should fork libvirt-java to get this fixed.

[yadvr]

this needs to go in tools/appliance/definitions/systemvmtemplate

[yadvr]

@wido thanks for the feature, can you rebase against latest master and push -f; update on the state of the PR

We should have this feature.

tag:needlove

[wido]

@rhtyd Done! Rebased against master

[swill]

Does this require a change to the system VM template? I am pulling this in for testing now...

[wido]

@swill Yes, it does. It install the Qemu Quest Agent inside the SSVM.

It doesn't hurt to do that already, it is just that the libvirt-java bindings don't allow us to talk to the VMs.

[wido]

Thinking about closing this PR and coming up with two new ones.

One which only adds the channel and installs Qemu Guest Agent in the SSVM.

[swill]

This failed during compiling:

[INFO] Apache CloudStack Plugin - Hypervisor KVM ......... FAILURE [11.036s]

Here is the error

[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ cloud-engine-storage-volume ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /data/git/cs1/cloudstack/dist/rpmbuild/BUILD/cloudstack-4.9.0-SNAPSHOT/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java:[3425,30] error: cannot find symbol
[INFO] 1 error
[INFO] -------------------------------------------------------------

[wido]

Closing this one in favor of #1545