Flexible tags for hosts and storage pools

[JoaoJandre]

Description

Currently when setting tags for a resource, a host for example, offerings with those tags will be directed towards that resource. However, offerings without any tags may also be directed towards it. Making it so even after tagging a resource, with the intention of making it exclusive to certain types of offerings, this exclusiveness might be ignored.

Furthermore, the current tag system only allows the user to inform a simple list of tags, without the possibility to create more complex rules, like checking if the offering has certain pairs of tags.

To address the described problems, we propose to create a new feature to allow users to use JS to create more complex tagging rules in the following APIs:

• updateHost
• createStoragePool
• updateStoragePool

For this, we will use the JS interpreter introduced in #5909. We will introduce a new preset variable, called "tags", which can be used to refer to the offering's tags list that will be paired with the resource. A simple example of a rule using this preset variable is:
tags[0] == cereal && tags[1] == milk

When setting a resource with this tag rule, only offerings whose first tag is cereal and second tag is milk will be accepted, offerings without tags will not be accepted. Although, this simple rule does not care for the rest of the offering's tags, so an offering with a third tag will also be accepted, no matter what the tag is.

To allow users to keep using the current tagging system without any hassle, we propose to introduce a new parameter to the affected APIs to inform if ACS should treat them the classic way, or to try to interpret it a rule in JS.

In order to support this behavior, we need to store which way the tags should be interpreted by ACS. We propose to add a new column called is_tag_a_rule in the following tables:

• host_tags
• storage_pool_tags

Furthermore, those same tables will have their tag column type changed from varchar(255) to text, to allow rules that have more than 255 characters.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

Unit tests were created, and some were updated. Moreover, manual tests were performed:
Regarding creating and updating storagepools (SP)

No.	Test	Result	Expected behavior? (Y/N)
1	Create zoneWide SP with flexible tag	SP successfully created	Y
2	Create clusterWide SP with flexible tag	SP successfully created	Y
3	Create SP datastore Cluster (vmware) with flexible tag	SP successfully created	Y
4	Update SP (all three types) with flexible tag	Success	Y

With the following situation in the environment:
Storage 1 with rule tags[0] == 'milk' and Storage 2 with rule tags[0] == 'cereal'

No.	Test	Result	Expected behavior? (Y/N)
1	Create volume using offering tagged milk	Volume allocated to storage 1	Y
2	Create volume using offering tagged cereal	Volume allocated to storage 2	Y
3	Create volume using untagged offering	Error trying to allocate volume	Y
3	Create volume using offering tagged banana	Error trying to allocate volume	Y

With the following situation in the environment:
Storage 1 with rule tags[0] == 'milk' and Storage 2 with tag cereal

No.	Test	Result	Expected behavior? (Y/N)
1	Create volume using offering tagged milk	Volume allocated to storage 1	Y
2	Create volume using offering tagged cereal	Volume allocated to storage 2	Y
3	Create volume using untagged offering	Volume allocated to storage 2	Y
4	With the volume of test 1, resize the volume, without changing the offering	Success	Y
5	With the volume of test 1, resize the volume, changing the offering to another with the same tag	Success	Y
6	With the volume of test 1, resize the volume, changing the offering to another one with a different tag	Exception	Y
7	With the volume of test 2, tests 4, 5 and 6 were redone	Same results	Y

Regarding Hosts:

No.	Test	Result	Expected behavior? (Y/N)
1	Removing a host and making that host rediscover mgmt	Success	Y
2	Call the updateHost API to update a host tag to a flexible tag	Changed tag	Y

Regarding deploying VMs:
With the following situation in the environment:
Host 1 with rule tags[0] == 'milk', Host 2 and Host 3 with rule tags[0] == cereal

No.	Test	Result	Expected behavior? (Y/N)
1	VM Deploy with offering tagged milk	VM allocated to host 1	Y
2	VM Deploy with offering tagged cereal	VM allocated to host 2/3	Y
3	VM Deploy with untagged offering	Exception	Y
4	Vm Deploy with offering tagged banana	Exception	Y

With the following situation in the environment:
Host 1 with rule tags[0] == 'milk', Host 2 and Host 3 with tag cereal

No.	Test	Result	Expected behavior? (Y/N)
1	VM Deploy with offering tagged milk	VM allocated to host 1	Y
2	VM Deploy with offering tagged cereal	VM allocated to host 2/3	Y
3	VM Deploy with untagged offering	VM allocated to host 2/3	Y
4	VM Deploy with offering tagged banana	Exception	Y

Regarding quota activation rules:

No.	Test	Result	Expected behavior? (Y/N)
1	Create tariff using activation rule value.host.isTagARule	rule activated correctly	Y

[DaanHoogland]

@JoaoJandre , I haven´t got my head around how exactly this would work yet but it seems promising.
@andrijapanicsb Can you have a look?

[yadvr]

@JoaoJandre could a global setting and its use in the deployment planner be enough to make the deployment and related lifecycles are strictly or non-strictly allowed the by deployment planner?

[codecov]

Codecov Report

Attention: 135 lines in your changes are missing coverage. Please review.

Comparison is base (c7ed4ca) 18.33% compared to head (76e4aae) 29.17%.
Report is 5 commits behind head on main.

Files	Patch %	Lines
.../src/main/java/com/cloud/host/dao/HostDaoImpl.java	53.84%	20 Missing and 4 partials ⚠️
...loudstack/utils/jsinterpreter/TagAsRuleHelper.java	0.00%	16 Missing ⚠️
...ain/java/com/cloud/storage/StorageManagerImpl.java	13.33%	13 Missing ⚠️
.../cloud/configuration/ConfigurationManagerImpl.java	40.00%	10 Missing and 2 partials ⚠️
...om/cloud/api/query/dao/StoragePoolJoinDaoImpl.java	78.37%	6 Missing and 2 partials ⚠️
.../agent/manager/allocator/impl/RandomAllocator.java	0.00%	7 Missing ⚠️
.../main/java/com/cloud/storage/StoragePoolTagVO.java	45.45%	6 Missing ⚠️
.../com/cloud/storage/dao/StoragePoolTagsDaoImpl.java	40.00%	6 Missing ⚠️
...che/cloudstack/metrics/PrometheusExporterImpl.java	0.00%	6 Missing ⚠️
...gent/manager/allocator/impl/FirstFitAllocator.java	20.00%	3 Missing and 1 partial ⚠️
... and 16 more

Additional details and impacted files

@@ Coverage Diff @@## main #7489 +/- ##
=============================================
+ Coverage 18.33% 29.17% +10.84% - Complexity 17808 31127 +13319 
=============================================
Files 5059 5195 +136 Lines 344296 366930 +22634 Branches 49577 53735 +4158 =============================================
+ Hits 63132 107065 +43933 + Misses 272093 245223 -26870 - Partials 9071 14642 +5571 

Flag	Coverage Δ
simulator-marvin-tests	25.16% <41.56%> (+5.60%)	⬆️
uitests	4.45% <ø> (-0.04%)	⬇️
unit-tests	14.82% <28.32%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[JoaoJandre]

@JoaoJandre could a global setting and its use in the deployment planner be enough to make the deployment and related lifecycles are strictly or non-strictly allowed the by deployment planner?

This would be the simplest solution; however, it would be limited as well. A setting like that would either make all hosts accept or not accept offerings without tags (the same with storage pools). The solution proposed in this PR makes the hosts and storage tags much more flexible; for instance, you could have hosts that accept untagged offerings, while others do not. Furthermore, being able to write a rule in JS opens space for operator creativity, and they would have a much finer control/understanding over their cloud environment.

Also, the current behavior is kept; therefore, the use of tag as rules is optional.

[andrijapanicsb]

Sounds interesting - as long as we keep the original behaviour a default (backward compatibility), I think this would be an interesting feature. It does add complexity (as with anything else), so proper documentation should be in place.

[github-actions]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[DaanHoogland]

Sounds interesting - as long as we keep the original behaviour a default (backward compatibility), I think this would be an interesting feature. It does add complexity (as with anything else), so proper documentation should be in place.

@JoaoJandre , can you reply this comment, please? As I read this the old behaviour is intact without change. But is there any risk that a user may think they use an old fashion tag but end up creating a dynamic one?

[JoaoJandre]

Sounds interesting - as long as we keep the original behaviour a default (backward compatibility), I think this would be an interesting feature. It does add complexity (as with anything else), so proper documentation should be in place.

@JoaoJandre , can you reply this comment, please? As I read this the old behaviour is intact without change. But is there any risk that a user may think they use an old fashion tag but end up creating a dynamic one?

Sure, as I stated on #7489 (comment), the current behavior is kept; therefore, the use of tag as rules is optional. Users have to explicitly inform the isTagARule parameter as true when calling the APIs to make the tags be interpreted as rules.

[DaanHoogland]

@blueorangutan package

[blueorangutan]

@DaanHoogland a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[DaanHoogland]

good code , but needs good functional verification

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6180

[DaanHoogland]

@blueorangutan test matrix

[blueorangutan]

@DaanHoogland a [SF] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-6627)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 41078 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7489-t6627-xenserver-71.zip
Smoke tests completed. 110 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[blueorangutan]

[SF] Trillian test result (tid-6628)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 41071 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7489-t6628-vmware-67u3.zip
Smoke tests completed. 85 look OK, 1 have errors, 24 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_08_arping_in_ssvm	Failure	60.66	test_diagnostics.py
test_09_arping_in_cpvm	Failure	60.75	test_diagnostics.py
all_test_safe_shutdown	Skipped	---	test_safe_shutdown.py
all_test_metrics_api	Skipped	---	test_metrics_api.py
all_test_outofbandmanagement	Skipped	---	test_outofbandmanagement.py
all_test_outofbandmanagement_nestedplugin	Skipped	---	test_outofbandmanagement_nestedplugin.py
all_test_routers_iptables_default_policy	Skipped	---	test_routers_iptables_default_policy.py
all_test_secondary_storage	Skipped	---	test_secondary_storage.py
all_test_service_offerings	Skipped	---	test_service_offerings.py
all_test_storage_policy	Skipped	---	test_storage_policy.py
all_test_templates	Skipped	---	test_templates.py
all_test_update_security_group	Skipped	---	test_update_security_group.py
all_test_usage_events	Skipped	---	test_usage_events.py
all_test_vm_autoscaling	Skipped	---	test_vm_autoscaling.py
all_test_vm_deployment_planner	Skipped	---	test_vm_deployment_planner.py
all_test_vm_life_cycle	Skipped	---	test_vm_life_cycle.py
all_test_vm_lifecycle_unmanage_import	Skipped	---	test_vm_lifecycle_unmanage_import.py
all_test_vm_snapshot_kvm	Skipped	---	test_vm_snapshot_kvm.py
all_test_vm_snapshots	Skipped	---	test_vm_snapshots.py
all_test_volumes	Skipped	---	test_volumes.py
all_test_vpc_ipv6	Skipped	---	test_vpc_ipv6.py
all_test_vpc_redundant	Skipped	---	test_vpc_redundant.py
all_test_vpc_router_nics	Skipped	---	test_vpc_router_nics.py
all_test_vpc_vpn	Skipped	---	test_vpc_vpn.py
all_test_host_maintenance	Skipped	---	test_host_maintenance.py
all_test_hostha_kvm	Skipped	---	test_hostha_kvm.py

[blueorangutan]

[SF] Trillian test result (tid-6629)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 44219 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7489-t6629-kvm-centos7.zip
Smoke tests completed. 110 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[DaanHoogland]

@JoaoJandre can you solve the conflicts?

[github-actions]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[JoaoJandre]

@JoaoJandre can you solve the conflicts?

@DaanHoogland I guess the conflicts solved themselves :D

[shwstppr]

@JoaoJandre do we need to move storage_pool_view to the view file now as per #7417?

[JoaoJandre]

@shwstppr done

[shwstppr]

@blueorangutan package

[blueorangutan]

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[DaanHoogland]

@blueorangutan test

[blueorangutan]

@DaanHoogland a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[DaanHoogland]

from the backend:

Smoke tests completed. 117 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_08_migrate_vm	Error	42.83	test_vm_life_cycle.py

github api rate limit hit, but I think we are ok with this one, @shwstppr.

[GutoVeronezi]

@blueorangutan package

[blueorangutan]

@GutoVeronezi a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 7860

[DaanHoogland]

@shwstppr@GutoVeronezi are we good with this one?

[GutoVeronezi]

@shwstppr@GutoVeronezi are we good with this one?

LGTM, nothing more from my side.

[shwstppr]

@DaanHoogland I'm good as it already has the required approvals.

@JoaoJandre should we add some notes in the documentation to highlight the new behaviour?

[DaanHoogland]

@DaanHoogland I'm good as it already has the required approvals.

@JoaoJandre should we add some notes in the documentation to highlight the new behaviour?

@JoaoJandre I added an issue for documentation and assigned you (#8283 )

merging

[JoaoJandre]

@DaanHoogland I'm good as it already has the required approvals.
@JoaoJandre should we add some notes in the documentation to highlight the new behaviour?

@JoaoJandre I added an issue for documentation and assigned you (#8283 )

merging

@DaanHoogland Okay, I'll add it to my tasks and do it in the following days.