[4.16] systemvm: configurable root disk size based on new systemvm template definition by ustcweizhou · Pull Request #3900 · apache/cloudstack

ustcweizhou · 2020-02-20T09:49:24Z

Description

The official systemvm template has very few free spaces, it would be better to make the disk size configuration.

Main changes in systemvm template
(1) remove /var and /tmp partitions
(2) move / partition to the last.
(3) install cloud-guest-utils to support "growpart"
(4) install haproxy 1.8 from stretch-backports instead of haproxy 1.7

other changes
(3) add global setting systemvm.root.disk.size
(4) when systemvm starts, allocate all free spaces on disk to the extend partition, root partition (/dev/vda6) and file system.

This applies on all systemvms (cpvm,ssvm,virtual routers, etc)

This fixes#3455

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

build new systemvm template
register new systemvm template
change type from "USER" to "SYSTEM"
change global setting systemvm.root.disk.size to 3GB
destroy ssvm or cpvm
new ssvm/cpvm will have 3GB disk (default 1.5GB) and 2.5 GB free space (default 1GB) on filesystem.

old VR

root@r-269-VM:~# df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 103748 0 103748 0% /dev
tmpfs 47352 23480 23872 50% /run
/dev/vda5 2086316 1209892 860040 59% /
tmpfs 118372 0 118372 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 118372 0 118372 0% /sys/fs/cgroup
/dev/vda1 93207 46230 46977 50% /boot
tmpfs 23672 0 23672 0% /run/user/0

new VR

root@r-270-VM:~# df
Filesystem 1K-blocks Used Available Use% Mounted on
udev 103688 0 103688 0% /dev
tmpfs 47352 5556 41796 12% /run
/dev/vda6 2687084 970912 1699788 37% /
tmpfs 118372 0 118372 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 118372 0 118372 0% /sys/fs/cgroup
/dev/vda1 93207 46361 46846 50% /boot
tmpfs 23672 0 23672 0% /run/user/0

ustcweizhou · 2020-02-20T09:50:53Z

@rhtyd this has some small conflicts with your PR #3797

weizhouapache · 2020-02-20T10:07:12Z

@rhtyd
if the specified value is small than template size, then it will not be used. "-1" and "0" makes no difference actually.

 Long size = _tmpltMgr.getTemplateSize(template.getId(), vm.getDataCenterId());
if (rootDisksize != null) {
rootDisksize = rootDisksize * 1024 * 1024 * 1024;
if (rootDisksize > size) {
s_logger.debug("Using root disk size of " + rootDisksize + " Bytes for volume " + name);
size = rootDisksize;
} else {
s_logger.debug("Using root disk size of " + size + " Bytes for volume " + name + "since specified root disk size of " + rootDisksize + " Bytes is smaller than template");
}
}

https://github.com/apache/cloudstack/blob/master/engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/VolumeOrchestrator.java#L713-L722

DaanHoogland · 2020-02-20T13:12:04Z

hm, I think you want milestone 4.14 for this one as well, right @weizhouapache ?

weizhouapache · 2020-02-20T15:26:09Z

hm, I think you want milestone 4.14 for this one as well, right @weizhouapache ?

@DaanHoogland I expect to add to 4.14. However, it require to build new systemvm template (and test it).
4.15 is also good for me.

yadvr · 2020-02-21T07:25:03Z

Thanks per your comment assigned against 4.15 @weizhouapache

yadvr · 2020-06-12T05:03:43Z

@ustcweizhou do we want this for master/4.15+? Can you fix conflicts?

PaulAngus · 2020-07-10T11:02:00Z

bump @ustcweizhou - can you fix conflicts so we can progress into testing please.

DaanHoogland

code looks good, we need this integration tested on the new release (which will happen automatically 🤞 )

weizhouapache · 2020-09-26T08:26:55Z

@rhtyd @DaanHoogland any chance to get this into 4.15 ?

yadvr · 2020-09-29T05:19:47Z


+ Long rootDiskSize = rootDiskOfferingInfo.getSize();
+ if (vm.getType().isUsedBySystem() && SystemVmRootDiskSize.value() != null) {
+ rootDiskSize = SystemVmRootDiskSize.value();


@ustcweizhou should there be a check for > 0 ?

@rhtyd added check.

yadvr · 2020-09-29T05:20:54Z

+ if which growpart > /dev/null; then
+ growpart /dev/vda 2
+ growpart /dev/vda 6
+ resize2fs /dev/vda6


@ustcweizhou have you tested this for KVM, and non-KVM hypervisors? It's possible the partitions are not called (hard-coded) vda; instead of this can be find/use partitions by mount point names or other valid/static identifiers?

This is screenshot of cpvm on xenserver for example:

Screenshot of partitions of cpvm on kvm:

Screenshot of partitions of cpvm, on vmware:

@ustcweizhou have you tested this for KVM, and non-KVM hypervisors? It's possible the partitions are not called (hard-coded) vda; instead of this can be find/use partitions by mount point names or other valid/static identifiers?
@rhtyd good point. thanks for review

yadvr · 2020-09-29T05:30:15Z

 echo "" > /root/.ssh/known_hosts

+ if which growpart > /dev/null; then
+ growpart /dev/vda 2


I checked the 4.15 systemvmtemplate, growpart is not available but resize2fs is available. Can we use fdisk instead of growpart to resize partition instead, or some other alternative? This would save from creating a new systemvmtemplate as we get closer to the 4.15 release?

@rhtyd root partition must be the last partition (it is linux swap for now), so this feature requires new systemvm template. let's move to 4.16.
I will work on other hypervisors except kvm.

yadvr · 2020-09-29T05:36:40Z

@ustcweizhou I've left some review comments, please see. As we're getting closer to 4.15 release I would like to avoid anything that could add any regression, usually a systemvmtemplate change could potentially add regressions.

A new Debian 10.6 was released over the weekend so if we build a systemvmtemplate now there may be several differences due to pkg/kernel changes against the one we've been testing for months - http://download.cloudstack.org/systemvm/4.15/

weizhouapache · 2020-09-29T11:09:15Z

@ustcweizhou I've left some review comments, please see. As we're getting closer to 4.15 release I would like to avoid anything that could add any regression, usually a systemvmtemplate change could potentially add regressions.
A new Debian 10.6 was released over the weekend so if we build a systemvmtemplate now there may be several differences due to pkg/kernel changes against the one we've been testing for months - http://download.cloudstack.org/systemvm/4.15/

@rhtyd ok, then let's not merge this into 4.15
I will fix the issues, hopefully this can be merged into 4.16

old code, needs reconsidering

weizhouapache · 2021-01-28T07:57:47Z


+ Long rootDiskSize = rootDiskOfferingInfo.getSize();
+ if (vm.getType().isUsedBySystem() && SystemVmRootDiskSize.value() != null) {
+ rootDiskSize = SystemVmRootDiskSize.value();


@rhtyd added check.

weizhouapache · 2021-01-28T07:59:13Z

ignore my approval ...

yadvr · 2021-03-23T08:16:51Z

@weizhouapache I think @Pearl1594 has already included these changes in her #4329 PR

weizhouapache · 2021-03-23T08:39:20Z

@weizhouapache I think @Pearl1594 has already included these changes in her #4329 PR

@rhtyd good. I closed this ticket then.

weizhouapache changed the title ~~4.14 systemvm root disk size~~systemvm: configurable root disk size based on new systemvm template definitionFeb 20, 2020

svenvogel added type:new-feature component:systemvm-template labels Feb 20, 2020

yadvr added this to the 4.15.0.0 milestone Feb 21, 2020

DaanHoogland previously approved these changes Jul 29, 2020
View reviewed changes

yadvr removed this from the 4.15.0.0 milestone Jul 30, 2020

yadvr reviewed Sep 29, 2020
View reviewed changes

yadvr added this to the 4.16.0.0 milestone Oct 7, 2020

weizhouapache changed the title ~~systemvm: configurable root disk size based on new systemvm template definition~~[DO NOT MERGE] systemvm: configurable root disk size based on new systemvm template definitionOct 30, 2020

ACSGitBot added the type:new_feature label Nov 3, 2020

PaulAngus removed the type:new_feature label Nov 3, 2020

ACSGitBot added the type:new_feature label Nov 8, 2020

DaanHoogland removed the type:new_feature label Nov 10, 2020

DaanHoogland added status:needs-work status:needs-CI status:needs-testing labels Nov 20, 2020

systemvm: configurable root disk size
45828e9

ustcweizhou force-pushed the 4.14-systemvm-root-disk-size branch from 112c8ab to 45828e9Compare January 17, 2021 09:23

weizhouapache changed the title ~~[DO NOT MERGE] systemvm: configurable root disk size based on new systemvm template definition~~[4.16] systemvm: configurable root disk size based on new systemvm template definitionJan 19, 2021

systemvm: check systemvm root disk size
9c31a9d

Pearl1594 mentioned this pull request Jan 28, 2021
Adding AutoScaling for cks + CKS CoreOS EOL update + systemvmtemplate improvements #4329
Merged
5 tasks

weizhouapache approved these changes Jan 28, 2021
View reviewed changes

weizhouapache closed this Mar 23, 2021

weizhouapache mentioned this pull request Jun 4, 2021
systemvmtemplate: use latest LTS kernel from buster-ports #5073
Merged

Pearl1594 mentioned this pull request Aug 9, 2021
Configurable disk size of systemvm #3455
Closed

DaanHoogland removed the status:needs-testing label Jan 28, 2026

Uh oh!

Conversation

ustcweizhou commented Feb 20, 2020• edited by weizhouapache Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

ustcweizhou commented Feb 20, 2020

Uh oh!

weizhouapache commented Feb 20, 2020

Uh oh!

DaanHoogland commented Feb 20, 2020

Uh oh!

weizhouapache commented Feb 20, 2020

Uh oh!

yadvr commented Feb 21, 2020

Uh oh!

yadvr commented Jun 12, 2020

Uh oh!

PaulAngus commented Jul 10, 2020

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

weizhouapache commented Sep 26, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yadvr commented Sep 29, 2020

Uh oh!

weizhouapache commented Sep 29, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

weizhouapache commented Jan 28, 2021

Uh oh!

yadvr commented Mar 23, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

weizhouapache commented Mar 23, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

ustcweizhou commented Feb 20, 2020•
edited by weizhouapache
Loading

yadvr commented Mar 23, 2021•
edited
Loading