api,server: purge expunged resources

[shwstppr]

Description

Feature spec: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Purge+Expunged+Resources

This PR introduces the functionality of purging removed DB entries for CloudStack entities (currently only for VirtualMachine). There would be three mechanisms for purging removed resources:

• Background task - CloudStack will run a background task which runs at a defined interval. Other parameters for this task can be controlled with new global settings.
• API - New admin-only API purgeExpungedResources. It will allow passing the following parameters - resourcetype, batchsize, startdate, enddate. Currently, API is not supported in the UI.
• Config for service offering. Service offerings can be created with purgeresources parameter which would allow purging resources immediately on expunge.

Following new global settings have been added:

• expunged.resources.purge.enabled: Default: false. Whether to run a background task to purge the DB records of the expunged resources.
• expunged.resources.purge.resources: Default: (empty). A comma-separated list of resource types that will be considered by the background task to purge the DB records of the expunged resources. Currently only VirtualMachine is supported. An empty value will result in considering all resource types for purging.
• expunged.resources.purge.interval: Default: 86400. Interval (in seconds) for the background task to purge the DB records of the expunged resources.
• expunged.resources.purge.delay: Default: 300. Initial delay (in seconds) to start the background task to purge the DB records of the expunged resources task.
• expunged.resources.purge.batch.size: Default: 50. Batch size to be used during purging of the DB records of the expunged resources.
• expunged.resources.purge.start.time: Default: (empty). Start time to be used by the background task to purge the DB records of the expunged resources. Use format yyyy-MM-dd or yyyy-MM-dd HH:mm:ss.
• expunged.resources.purge.keep.past.days: Default: 30. The number of days in the past from the execution time of the background task to purge the DB records of the expunged resources for which the expunged resources must not be purged. To enable purging DB records of the expunged resource till the execution of the background task, set the value to zero.
• expunged.resource.purge.job.delay: Default: 180. Delay (in seconds) to execute the purging of the DB records of an expunged resource initiated by the configuration in the offering. Minimum value should be 180 seconds and if a lower value is set then the minimum value will be used.

Documentation PR: apache/cloudstack-documentation#397

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• build/CI

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

API usage:

> purge expungedresources startdate=2024-04-15 enddate=2024-04-20 resourcetype=VirtualMachine
{
"purgeexpungedresourcesresponse": {
"resourcecount": 6
}
}

Verified new smoke test,

[root@pr8999-t6925-kvm-centos7-marvin marvin]# nosetests --with-xunit --xunit-file=results.xml --with-marvin --marvin-config=./pr8999-t6925-kvm-centos7-advanced-cfg -s -a tags=advanced --hypervisor=KVM tests/smoke/test_purge_expunged_vms.py
/usr/local/lib/python3.6/site-packages/paramiko/transport.py:32: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography. The next release of cryptography will remove support for Python 3.6.
from cryptography.hazmat.backends import default_backend
==== Marvin Init Started ====
=== Marvin Parse Config Successful ===
=== Marvin Setting TestData Successful===
==== Log Folder Path: /marvin/MarvinLogs/May_01_2024_08_09_34_TNMKB9 All logs will be available here ====
=== Marvin Init Logging Successful===
==== Marvin Init Successful ====
=== TestName: test_01_purge_expunged_api_vm_start_date | Status : SUCCESS ===
=== TestName: test_02_purge_expunged_api_vm_end_date | Status : SUCCESS ===
=== TestName: test_03_purge_expunged_api_vm_start_end_date | Status : SUCCESS ===
=== TestName: test_04_purge_expunged_api_vm_no_date | Status : SUCCESS ===
=== TestName: test_05_purge_expunged_vm_service_offering | Status : SUCCESS ===
{'mgtSvrIp': '10.1.33.150', 'port': 8080, 'user': 'root', 'passwd': 'P@ssword123', 'hypervisor': 'kvm', 'useHttps': 'False', 'certCAPath': 'NA', 'certPath': 'NA', 'apiKey': 'YZDWI356FRnLh8vEHROd_ZUVD5U8JeGG-mObb7wPhyL5OQqi6df3KCmF5AlVWxPsZvsvNY2-z9BbyqSZGPaBCg', 'securityKey': 'U4LP9-ZdTuRl4_hJl3x2KN6nTNZCYAWhcMXmFKBSBYTNoUghjj3Pb5-0s2hN_tVG0l8oK6jDrmp953NOP59GAg'}
====Trying SSH Connection: Host:10.1.33.150 User:root Port:22 RetryCnt:60===
===SSH to Host 10.1.33.150 port : 22 SUCCESSFUL===
{Cmd: service cloudstack-management stop via Host: 10.1.33.150} {returns: ['Redirecting to /bin/systemctl stop cloudstack-management.service']}
{Cmd: service cloudstack-management start via Host: 10.1.33.150} {returns: ['Redirecting to /bin/systemctl start cloudstack-management.service']}
====Trying SSH Connection: Host:10.1.35.104 User:root Port:22 RetryCnt:60===
====Trying SSH Connection: Host:10.1.35.104 User:root Port:22 RetryCnt:60===
===SSH to Host 10.1.35.104 port : 22 SUCCESSFUL===
===SSH to Host 10.1.35.104 port : 22 SUCCESSFUL===
{Cmd: service cloudstack-management stop via Host: 10.1.35.104} {returns: ['Redirecting to /bin/systemctl stop cloudstack-management.service']}
{Cmd: service cloudstack-management start via Host: 10.1.35.104} {returns: ['Redirecting to /bin/systemctl start cloudstack-management.service']}
{'mgtSvrIp': '10.1.33.150', 'port': 8080, 'user': 'root', 'passwd': 'P@ssword123', 'hypervisor': 'kvm', 'useHttps': 'False', 'certCAPath': 'NA', 'certPath': 'NA', 'apiKey': 'YZDWI356FRnLh8vEHROd_ZUVD5U8JeGG-mObb7wPhyL5OQqi6df3KCmF5AlVWxPsZvsvNY2-z9BbyqSZGPaBCg', 'securityKey': 'U4LP9-ZdTuRl4_hJl3x2KN6nTNZCYAWhcMXmFKBSBYTNoUghjj3Pb5-0s2hN_tVG0l8oK6jDrmp953NOP59GAg'}
====Trying SSH Connection: Host:10.1.33.150 User:root Port:22 RetryCnt:60===
===SSH to Host 10.1.33.150 port : 22 SUCCESSFUL===
{Cmd: service cloudstack-management stop via Host: 10.1.33.150} {returns: ['Redirecting to /bin/systemctl stop cloudstack-management.service']}
{Cmd: service cloudstack-management start via Host: 10.1.33.150} {returns: ['Redirecting to /bin/systemctl start cloudstack-management.service']}
====Trying SSH Connection: Host:10.1.35.104 User:root Port:22 RetryCnt:60===
====Trying SSH Connection: Host:10.1.35.104 User:root Port:22 RetryCnt:60===
===SSH to Host 10.1.35.104 port : 22 SUCCESSFUL===
===SSH to Host 10.1.35.104 port : 22 SUCCESSFUL===
{Cmd: service cloudstack-management stop via Host: 10.1.35.104} {returns: ['Redirecting to /bin/systemctl stop cloudstack-management.service']}
{Cmd: service cloudstack-management start via Host: 10.1.35.104} {returns: ['Redirecting to /bin/systemctl start cloudstack-management.service']}
=== TestName: test_06_purge_expunged_vm_background_task | Status : SUCCESS ===

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 55.68581% with 378 lines in your changes missing coverage. Please review.

Project coverage is 15.42%. Comparing base (a9caee3) to head (93edde5).
Report is 15 commits behind head on main.

Files	Patch %	Lines
...loudstack/resource/ResourceCleanupServiceImpl.java	50.51%	220 Missing and 20 partials ⚠️
...c/main/java/com/cloud/utils/db/GenericDaoBase.java	0.00%	35 Missing ⚠️
.../cloud/configuration/ConfigurationManagerImpl.java	0.00%	25 Missing ⚠️
...udstack/resourcedetail/ResourceDetailsDaoBase.java	0.00%	10 Missing ⚠️
...mand/admin/resource/PurgeExpungedResourcesCmd.java	72.72%	9 Missing ⚠️
...tack/engine/orchestration/NetworkOrchestrator.java	0.00%	9 Missing ⚠️
...in/java/com/cloud/storage/dao/SnapshotDaoImpl.java	0.00%	9 Missing ⚠️
...a/com/cloud/vm/snapshot/dao/VMSnapshotDaoImpl.java	55.00%	9 Missing ⚠️
...om/cloud/network/lb/dao/ElasticLbVmMapDaoImpl.java	0.00%	9 Missing ⚠️
.../main/java/com/cloud/vm/dao/VMInstanceDaoImpl.java	75.00%	1 Missing and 3 partials ⚠️
... and 8 more

Additional details and impacted files

@@ Coverage Diff @@## main #8999 +/- ##
============================================
+ Coverage 15.32% 15.42% +0.10% - Complexity 11686 11808 +122 
============================================
Files 5459 5464 +5 Lines 477298 478139 +841 Branches 60202 61057 +855 ============================================
+ Hits 73135 73765 +630 - Misses 396084 396247 +163 - Partials 8079 8127 +48 

Flag	Coverage Δ
uitests	4.19% <ø> (ø)
unittests	16.18% <55.68%> (+0.11%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✖️ debian ✔️ suse15. SL-JID 9464

[shwstppr]

@blueorangutan test

[blueorangutan]

@shwstppr a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian Build Failed (tid-10080)

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9471

[blueorangutan]

[SF] Trillian Build Failed (tid-10089)

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9490

[shwstppr]

@blueorangutan test

[blueorangutan]

@shwstppr a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[github-actions]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✖️ el7 ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 9995

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9996

[shwstppr]

@blueorangutan test

[blueorangutan]

@shwstppr a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[borisstoyanov]

LGTM, manually tested it

[yadvr]

LGTM - waiting on smoketests for merging this

[shwstppr]

@blueorangutan test rocky8 kvm-rocky8

[blueorangutan]

@shwstppr a [SL] Trillian-Jenkins test job (rocky8 mgmt + kvm-rocky8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-10511)
Environment: kvm-rocky8 (x2), Advanced Networking with Mgmt server r8
Total time taken: 49664 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8999-t10511-kvm-rocky8.zip
Smoke tests completed. 135 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File