VM stats history visualization by joseflauzino · Pull Request #6401 · apache/cloudstack

joseflauzino · 2022-05-20T11:38:31Z

Description

PR #5984 recently implemented the listVirtualMachinesUsageHistory API, which provides historical data of VM stats. However, the current ACS UI only shows VM stats from the most recent data collection.

This PR implements new features in the UI which allow ACS users to view the historical stats of each VM individually.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Screenshots:

Charts:

Possible Filtering:

How Has This Been Tested?

I went to Compute > Instances > any-vm-instance > Statistics and was able to visualize all the statistics charts for the selected VM.
Also, I filtered by different period combinations and the data was presented correctly.

acs-robot · 2022-05-20T11:39:34Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-05-20T11:40:03Z

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

blueorangutan · 2022-05-20T11:45:21Z

UI build: ✔️
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6401 (SL-JID-1605)

DaanHoogland · 2022-05-20T15:26:12Z

@blueorangutan package

blueorangutan · 2022-05-20T15:27:03Z

@DaanHoogland a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

yadvr · 2022-05-20T15:31:26Z

Looks fantastic, looking forward to reviewing/testing this.

nvazquez · 2022-05-20T15:33:05Z

Wow - looks very nice @joseflauzino thanks!

blueorangutan · 2022-05-20T16:16:39Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 3459

DaanHoogland · 2022-05-20T17:31:48Z

@blueorangutan test keepEnv

blueorangutan · 2022-05-20T17:33:03Z

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2022-05-21T05:18:53Z

Trillian test result (tid-4223)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 41093 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6401-t4223-kvm-centos7.zip
Smoke tests completed. 97 look OK, 0 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File

DaanHoogland · 2022-05-23T07:18:40Z

@joseflauzino, two minor points

I build and deployed the code from this PR but don´t get any data from this weekend, dispite deploying some VMs in the nested env on saterday: This is because the default value for vm.stats.max.retention.time is 1. Does it make sense to set it for instance to 5, so there is always something to see?
when refreshing the default tab is displayed again and after re-opening the statistics tab the configurations (all dataor a start date/end date pair) are reset/forgotten.

in general tests good and looks good,
nice thanks

DaanHoogland

clgtm
tests ok

joseflauzino · 2022-05-23T17:14:13Z

@DaanHoogland thanks for your review and testing.

1. I build and deployed the code from this PR but don´t get any data from this weekend, dispite deploying some VMs in the nested env on saterday: This is because the default value for `vm.stats.max.retention.time` is 1. Does it make sense to set it for instance to 5, so there is always something to see?

Both vm.stats.max.retention.time and vm.stats.interval were set to 1 min by default on purpose (in PR #5984, not in the current PR). The reason is that it is common to find clouds with a considerable amount of VM instances (e.g. hundreds or even over 1k). So the amount of data generated even in a short 5 min period can be very large and the impact can also be large if it is not something planned by the cloud operators. So I think it is more prudent that we leave the default values of these parameters in a way that generates the minimum amount of data possible. It is then up to the cloud administrators/operators to decide whether or not to increase the amount of stats data to be kept in the database.

2. when refreshing the default tab is displayed again and after re-opening the statistics tab the configurations (`all data`or a `start date`/`end date` pair) are reset/forgotten.

This was also an intentional detail. The idea is to have the user open the filtering modal and click the OK button whenever they want to get updated data for the selected period. The reason is that during our tests there were several times when users clicked the refresh button and it looked like the data was not updated. But what happened was that the selected period was in the past, so the data would always be the same. The way it is now is that the user will always check the selected period before getting updated data. My intention is to leave it this way for now and according to the feedback we get from users we could think of other ways to improve the user experience.

DaanHoogland · 2022-05-23T17:51:46Z

@joseflauzino
Ad 1. Ok makes sense
Ad 2. Makes sense as well, can we then in this situation change the text on the button or add a note next to it instructing like wise?

acs-robot · 2022-06-03T19:30:59Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-06-03T19:32:02Z

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

blueorangutan · 2022-06-03T19:38:45Z

UI build: ✔️
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6401 (SL-JID-1674)

joseflauzino · 2022-06-03T19:44:52Z

@DaanHoogland I have just committed a change that applies your suggestion related to the submit button text.

DaanHoogland · 2022-06-07T09:38:56Z

@DaanHoogland I have just committed a change that applies your suggestion related to the submit button text.

looks ok, I can imagine the novice won´t be clear on this, but letś go for it!

github-actions · 2022-06-23T14:11:03Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

acs-robot · 2022-08-08T10:19:05Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-08-08T10:20:03Z

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

blueorangutan · 2022-08-08T10:25:20Z

UI build: ✔️
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6401 (SL-JID-2103)

shwstppr · 2022-08-08T10:26:22Z

@blueorangutan package

blueorangutan · 2022-08-08T10:27:03Z

@shwstppr a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-08-08T11:05:58Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 3939

shwstppr

@joseflauzino found a couple of issues while testing this time,

While filtering stats, API is called with hidden fields. eg: First filter using start, end dates Then select All data. Check API call, it still contains start/enddate param even though they are hidden in UI.
Time in filter UI is shown in the local timezone and the same is used in the API call without any timezone. Does listVirtualMachinesUsageHistory API convert it to server time? I guess not and it won't list stats correctly in many cases

acs-robot · 2022-08-08T14:24:41Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-08-08T14:25:03Z

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

joseflauzino · 2022-08-08T14:27:37Z

@shwstppr

While filtering stats, API is called with hidden fields. eg: First filter using start, end dates Then select All data. Check API call, it still contains start/enddate param even though they are hidden in UI.

Good point. I fixed this bug now.

Time in filter UI is shown in the local timezone and the same is used in the API call without any timezone. Does listVirtualMachinesUsageHistory API convert it to server time? I guess not and it won't list stats correctly in many cases

The timestamps come in server timezone, but the listVirtualMachinesUsageHistory API does not do a conversion or anything like that. In fact, the API receives the date fields without timezone. But the reason for this goes beyond the API itself:

Currently, the backend only supports dates in the yyy-MM-dd or yyy-MM-dd HH:mm:ss format, so there is no way to pass dates with timezone to the APIs (the related code is in the ParamProcessWorker class).
I think it would be interesting to implement support for timezone date objects in the ACS APIs. However, this is beyond the scope of this PR, so my suggestion is to map this to another issue.

blueorangutan · 2022-08-08T14:30:09Z

UI build: ✔️
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6401 (SL-JID-2106)

shwstppr · 2022-08-09T07:00:36Z

@joseflauzino sorry if I didn't explain myself better. I'm not saying API should accept timezone but maybe we can call API with timezone that the server expects. My issue currently is,

In UI, I'm using IST timezone (UTC+0530)
Server is storing stats with UTC timezone, API returns the times with the same.
I deploy a VM at 1000hr IST. Server starts collecting/storing stats for the VM but they are stored with time starting 0430hr UTC
Now in UI at 1100hr IST I want to see stats. UI allows me to filter stats between 1000-1100hr IST. When I make this selection I see nothing because API is called with something like startDate=2022-08-09+10:00:00&endDate=2022-08-09+11:00:00.
As the server is behind by 0530h. I'll never see stats for the current time. But if I call API with all available data it will show the stats.

In short, I think while calling the API it is needed to convert the selected time back

joseflauzino · 2022-08-11T14:27:13Z

@shwstppr

... maybe we can call API with timezone that the server expects.

To do that, before calling the API, we would have to convert the client timezone (from the browser) to the management server timezone, right?

In this case, my question is: in the UI context (on the client), how can we find out the management server timezone before calling the API?

Is there a ready-made mechanism to do this? If there is, please let me know and then I will implement this timezone conversion.

If not, I think we can address this in another issue and then implement this kind of timezone conversion in this and other views that do similar filtering (API calls passing date objects as parameters).

shwstppr · 2022-08-12T04:41:22Z

@joseflauzino maybe the reverse of how we convert server time into local time, https://github.com/apache/cloudstack/blob/main/ui/src/utils/plugins.js#L294-L301

yadvr · 2022-08-17T08:34:17Z

ping @joseflauzino are you able to address the comment - I think we're very close to merging this.

…tats-history-visualization

acs-robot · 2022-08-25T13:30:49Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-08-25T13:31:03Z

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

blueorangutan · 2022-08-25T13:42:37Z

UI build: ✔️
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6401 (SL-JID-2225)

sonarqubecloud · 2022-08-25T14:22:17Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

codecov · 2022-08-25T14:29:26Z

Codecov Report

Merging #6401 (889ea8d) into main (f76b6c6) will not change coverage.
The diff coverage is n/a.

@@ Coverage Diff @@## main #6401 +/- ##
=========================================
Coverage 5.87% 5.87% Complexity 3934 3934 =========================================
Files 2454 2454 Lines 242682 242682 Branches 37980 37980 =========================================
Hits 14261 14261 Misses 226842 226842 Partials 1579 1579

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

shwstppr · 2022-08-25T19:20:21Z

@blueorangutan package

blueorangutan · 2022-08-25T19:21:03Z

@shwstppr a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-08-25T19:58:53Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 4061

shwstppr

LGTM

yadvr · 2022-08-29T16:47:13Z

Good stuff @joseflauzino !

* Add VM stats view feature * Add dynamic label to the submit button of the VM stats filtering modal * Improve CPU info message * Reduce identation * Remove missed line * Fix timestamp when 'Use local timezone' is selected * Fix bug when filtering for all data * Support for toggle 'Use local timezone' button Co-authored-by: joseflauzino <jose@scclouds.com.br> Co-authored-by: Daniel Augusto Veronezi Salvador <38945620+GutoVeronezi@users.noreply.github.com>

Add VM stats view feature
f956658

DaanHoogland added this to the 4.18.0.0 milestone May 20, 2022

nvazquez added component:UI type:new-feature labels May 20, 2022

DaanHoogland approved these changes May 23, 2022
View reviewed changes

Add dynamic label to the submit button of the VM stats filtering modal
ab4e1b9

nvazquez force-pushed the main branch from c56220f to 0f4c3afCompare June 6, 2022 15:14

GutoVeronezi reviewed Jun 21, 2022
View reviewed changes

Comment threadui/public/locales/en.json Outdated
Comment threadui/src/components/view/StatsTab.vue Outdated
Comment threadui/src/components/view/StatsTab.vue Outdated

joseflauzino added 2 commits June 23, 2022 11:06

Improve CPU info message
46fe0bc

Reduce identation
5108fde

shwstppr requested changes Aug 8, 2022
View reviewed changes

Fix bug when filtering for all data
560407a

joseflauzino added 2 commits August 25, 2022 10:25

Support for toggle 'Use local timezone' button
12787ba

Merge branch 'main' ofhttps://github.com/apache/cloudstackinto vm-s…
889ea8d
…tats-history-visualization

shwstppr approved these changes Aug 29, 2022
View reviewed changes

DaanHoogland merged commit 9f7e0cc into apache:mainAug 29, 2022

Uh oh!

Conversation

joseflauzino commented May 20, 2022

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Screenshots:

How Has This Been Tested?

Uh oh!

acs-robot commented May 20, 2022

Uh oh!

blueorangutan commented May 20, 2022

Uh oh!

blueorangutan commented May 20, 2022

Uh oh!

DaanHoogland commented May 20, 2022

Uh oh!

blueorangutan commented May 20, 2022

Uh oh!

yadvr commented May 20, 2022

Uh oh!

nvazquez commented May 20, 2022

Uh oh!

blueorangutan commented May 20, 2022

Uh oh!

DaanHoogland commented May 20, 2022

Uh oh!

blueorangutan commented May 20, 2022

Uh oh!

blueorangutan commented May 21, 2022

Uh oh!

DaanHoogland commented May 23, 2022

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

joseflauzino commented May 23, 2022

Uh oh!

DaanHoogland commented May 23, 2022

Uh oh!

acs-robot commented Jun 3, 2022

Uh oh!

blueorangutan commented Jun 3, 2022

Uh oh!

blueorangutan commented Jun 3, 2022

Uh oh!

joseflauzino commented Jun 3, 2022

Uh oh!

DaanHoogland commented Jun 7, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actionsBot commented Jun 23, 2022

Uh oh!

acs-robot commented Aug 8, 2022

Uh oh!

blueorangutan commented Aug 8, 2022

Uh oh!

blueorangutan commented Aug 8, 2022

Uh oh!

shwstppr commented Aug 8, 2022

Uh oh!

blueorangutan commented Aug 8, 2022

Uh oh!

blueorangutan commented Aug 8, 2022

Uh oh!

shwstppr left a comment

Choose a reason for hiding this comment

Uh oh!

acs-robot commented Aug 8, 2022

Uh oh!

blueorangutan commented Aug 8, 2022

Uh oh!

joseflauzino commented Aug 8, 2022

Uh oh!

blueorangutan commented Aug 8, 2022

Uh oh!

shwstppr commented Aug 9, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

shwstppr commented Aug 9, 2022•
edited
Loading