IoT/ARM64 support: allow cloudstack-agent on Raspberry Pi 4 (armv8) to use kvm acceleration by yadvr · Pull Request #3644 · apache/cloudstack

yadvr · 2019-10-20T09:08:14Z

KVM is supported on arm64 Linux (https://www.linux-kvm.org/page/Processor_support#ARM:). For a small (IoT) platform such as the new Raspberry Pi 4 that uses armv8 processor (cortex-a72) it's possible to run Linux host with /dev/kvm accleration.

This PR is from a fun weekend project where:

I set up a Raspberry Pi 4 - 4GB RAM model with 4 CPU cores @ 1.5Ghz, 128GB SD samsung evo plus card
Installed Ubuntu 19.10 raspi3 base image: http://cdimage.ubuntu.com/releases/19.10/release/ubuntu-19.10-preinstalled-server-arm64+raspi3.img.xz
Build a custom Linux 5.3 kernel with KVM enabled, deb here: http://dl.rohityadav.cloud/cloudstack-rpi/kernel-19.10/ and install the linux-image and linux-module
Then install/setup CloudStack on it (fix some issues around jna, by manually installing newer libjna-java to /usr/share/cloudstack-agent/lib)
Since the host processor is not x86_64, I had to build a new arm64 (or aarch64) systemvmtemplate: http://dl.rohityadav.cloud/cloudstack-rpi/systemvmtemplate/

I could finally get a 4.13 CloudStack + Adv zone/networking to run on it and deployed a KVM based Ubuntu 19.10 environment and NFS storage. Deployed a test vm with isolated network, VR works as expected. Console proxy works as well, for this tested against arm64 openstack Debian 9/10 templates.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

KVM is supported on arm64 Linux (https://www.linux-kvm.org/page/Processor_support#ARM:). For a small (IoT) platform such as the new Raspberry Pi 4 that uses armv8 processor (cortex-a72) it's possible to run Linux host with `/dev/kvm` accleration. This adds support for IoT IaaS in CloudStack. This PR is from a fun weekend project where: - I set up a Raspberry Pi 4 - 4GB RAM model with 4 CPU cores @ 1.5Ghz, 128GB SD samsung evo plus card - Installed Ubuntu 19.10 raspi3 base image: http://cdimage.ubuntu.com/releases/19.10/release/ubuntu-19.10-preinstalled-server-arm64+raspi3.img.xz - Build a custom Linux 5.3 kernel with KVM enabled, deb here: http://dl.rohityadav.cloud/cloudstack-rpi/kernel-19.10/ and install the linux-image and linux-module - Then install/setup CloudStack on it (fix some issues around jna, by manually installing newer libjna-java to /usr/share/cloudstack-agent/lib) - Since the host processor is not x86_64, I had to build a new arm64 (or aarch64) systemvmtemplate: http://dl.rohityadav.cloud/cloudstack-rpi/systemvmtemplate/ I could finally get a 4.13 CloudStack + Adv zone/networking to run on it and deployed a KVM based Ubuntu 19.10 environment and NFS storage. Deployed a test vm with isolated network, VR works as expected. Console proxy works as well, for this tested against arm64 openstack Debian 9/10 templates. Limitation: on my aarch64 env, it did not support IDE, therefore all default bus type for volumes are SCSI by default. With VIRTIO it fails sometimes. Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

yadvr · 2019-10-20T11:57:37Z

@blueorangutan package

blueorangutan · 2019-10-20T11:58:02Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-10-20T12:16:03Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-320

yadvr · 2019-10-20T12:18:23Z

@blueorangutan test

blueorangutan · 2019-10-20T12:19:03Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

yadvr · 2019-10-20T17:49:15Z

@blueorangutan test

blueorangutan · 2019-10-20T17:50:03Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2019-10-21T04:41:05Z

Trillian test result (tid-417)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 37651 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3644-t417-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 77 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

yadvr · 2019-10-30T09:25:13Z

Tested with two RPi4, live VM migration works too.

GabrielBrascher

Code LGTM

DaanHoogland · 2019-11-07T18:23:57Z

looks good @rhtyd Jenkins failed so closing to rekick.

DaanHoogland · 2019-11-07T18:24:11Z

go jenkins go

yadvr · 2019-11-11T16:29:34Z

Thanks @DaanHoogland @GabrielBrascher Travis is green and trillian smoke tests is all pass, I'll merge this.

yadvr · 2019-11-13T09:25:02Z

Update note: Ubuntu kernel linux-raspi2 - 5.3.0-1012.14 will now ship with KVM enabled for arm64 by default: https://bugs.launchpad.net/ubuntu/+source/linux-raspi2/+bug/1783961

…o use kvm acceleration (apache#3644) KVM is supported on arm64 Linux (https://www.linux-kvm.org/page/Processor_support#ARM:). For a small (IoT) platform such as the new Raspberry Pi 4 that uses armv8 processor (cortex-a72) it's possible to run Linux host with `/dev/kvm` accleration. This adds support for IoT IaaS in CloudStack. This PR is from a fun weekend project where: - I set up a Raspberry Pi 4 - 4GB RAM model with 4 CPU cores @ 1.5Ghz, 128GB SD samsung evo plus card - Installed Ubuntu 19.10 raspi3 base image: http://cdimage.ubuntu.com/releases/19.10/release/ubuntu-19.10-preinstalled-server-arm64+raspi3.img.xz - Build a custom Linux 5.3 kernel with KVM enabled, deb here: http://dl.rohityadav.cloud/cloudstack-rpi/kernel-19.10/ and install the linux-image and linux-module - Then install/setup CloudStack on it (fix some issues around jna, by manually installing newer libjna-java to /usr/share/cloudstack-agent/lib) - Since the host processor is not x86_64, I had to build a new arm64 (or aarch64) systemvmtemplate: http://dl.rohityadav.cloud/cloudstack-rpi/systemvmtemplate/ I could finally get a 4.13 CloudStack + Adv zone/networking to run on it and deployed a KVM based Ubuntu 19.10 environment and NFS storage. Deployed a test vm with isolated network, VR works as expected. Console proxy works as well, for this tested against arm64 openstack Debian 9/10 templates. I raised the issue of enabling KVM in upstream Ubuntu arm64 build: https://bugs.launchpad.net/ubuntu/+source/linux-raspi2/+bug/1783961 Ubuntu kernel team has come back and future arm64 releases may have KVM enabled by default. Limitation: on my aarch64 env, it did not support IDE, therefore all default bus type for volumes are SCSI by default. With VIRTIO it fails sometimes. Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

yadvr force-pushed the 4.13.0.0-arm64-rpi branch from 62e0c22 to 993df6dCompare October 20, 2019 09:08

yadvr requested review from DaanHoogland, GabrielBrascher, PaulAngus, andrijapanicsb, kiwiflyer, nathanejohnson, nvazquez, resmo, svenvogel and wido October 20, 2019 09:09

yadvr force-pushed the 4.13.0.0-arm64-rpi branch from 4ea5220 to 01b679eCompare October 20, 2019 10:00

apache deleted a comment from blueorangutan Oct 20, 2019

kvm: fix unit test failure
8f33a88
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

apache deleted a comment from blueorangutan Oct 20, 2019

resmo removed their request for review October 20, 2019 13:46

apache deleted a comment from blueorangutan Oct 20, 2019

yadvr changed the title ~~[WIP] IoT support: allow cloudstack-agent on Raspberry Pi 3/4 (arm64) to use kvm acceleration~~IoT/ARM64 support: allow cloudstack-agent on Raspberry Pi 4 (armv8) to use kvm accelerationOct 22, 2019

yadvr added this to the 4.14.0.0 milestone Oct 22, 2019

yadvr added type:new-feature component:kvm labels Oct 22, 2019

GabrielBrascher approved these changes Oct 31, 2019
View reviewed changes

DaanHoogland closed this Nov 7, 2019

DaanHoogland reopened this Nov 7, 2019

yadvr merged commit 524b995 into apache:masterNov 11, 2019

Uh oh!

Conversation

yadvr commented Oct 20, 2019

Types of changes

Screenshots (if appropriate):

Uh oh!

yadvr commented Oct 20, 2019

Uh oh!

blueorangutan commented Oct 20, 2019

Uh oh!

blueorangutan commented Oct 20, 2019

Uh oh!

yadvr commented Oct 20, 2019

Uh oh!

blueorangutan commented Oct 20, 2019

Uh oh!

yadvr commented Oct 20, 2019

Uh oh!

blueorangutan commented Oct 20, 2019

Uh oh!

blueorangutan commented Oct 21, 2019

Uh oh!

yadvr commented Oct 30, 2019

Uh oh!

GabrielBrascher left a comment

Choose a reason for hiding this comment

Uh oh!

DaanHoogland commented Nov 7, 2019

Uh oh!

DaanHoogland commented Nov 7, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yadvr commented Nov 11, 2019

Uh oh!

yadvr commented Nov 13, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

DaanHoogland commented Nov 7, 2019•
edited
Loading