Normalize encryption on global configurations values by BryanMLima · Pull Request #6812 · apache/cloudstack

BryanMLima · 2022-10-07T20:57:38Z

Description

ACS encrypts the configuration values from accounts and domains, e.g. AccountDetailVO and DomainDetailVO while plain values are stored in the DB on data center and cluster scopes, e.g. DataCenterDetailVO and ClusterDetailVO. However, there are many different configuration values that make no sense to encrypt. The only ones that should be encrypted are in the secure/hidden category. Furthermore, there are use cases when retrieving the value on a DB view is needed, which is impossible with the current process in the account and domain scopes. Moreover, the response while updating a configuration value now returns the encrypted value for the categories mentioned above.

This PR aims to normalize the encryption on these four scopes; for this, all values that are not in the secure/hidden category are not encrypted. Furthermore, an unused mock file was removed due to a requirement to add a mock method to this class that was not even used.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Screenshots (if appropriate):

Domain details table before changes:

Domain details table after changes:

Update configuration value response after these changes:

These images are just an example, these changes are for all four tables mentioned above.

How Has This Been Tested?

This was tested in a local lab generating a new version for the .deb file of CloudStack. The lab upgraded successfully and the DB script ran smoothly. There were custom configuration values for all four scopes in the DB, the image in the screenshot section presents the result of the upgrade in the domain_details table.

DaanHoogland

looks generally good. one concern.

acs-robot · 2022-10-11T17:03:20Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-10-11T17:04:03Z

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

blueorangutan · 2022-10-11T17:09:52Z

UI build: ✔️
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6812 (SL-JID-2485)

codecov · 2022-10-11T17:56:58Z

Codecov Report

Merging #6812 (88d32c7) into main (d25521e) will increase coverage by 6.47%.
Report is 6 commits behind head on main.
The diff coverage is 28.57%.

@@ Coverage Diff @@## main #6812 +/- ##
============================================
+ Coverage 22.71% 29.18% +6.47% - Complexity 22296 30425 +8129 
============================================
Files 4966 5100 +134 Lines 336376 358386 +22010 Branches 48340 52310 +3970 ============================================
+ Hits 76416 104608 +28192 + Misses 248869 239406 -9463 - Partials 11091 14372 +3281

Flag	Coverage Δ
simulator-marvin-tests	`25.20% <28.57%> (+1.01%)`	⬆️
uitests	`4.87% <ø> (ø)`
unit-tests	`14.42% <0.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...src/main/java/com/cloud/domain/DomainDetailVO.java	`76.92% <ø> (ø)`
...ava/com/cloud/upgrade/dao/Upgrade41810to41900.java	`3.00% <0.00%> (-0.09%)`	⬇️
.../src/main/java/com/cloud/user/AccountDetailVO.java	`69.23% <ø> (ø)`
.../cloud/utils/crypt/EncryptionSecretKeyChanger.java	`0.00% <0.00%> (ø)`
.../cloud/configuration/ConfigurationManagerImpl.java	`42.98% <57.14%> (+6.15%)`	⬆️
...udstack/api/command/admin/config/UpdateCfgCmd.java	`76.19% <88.23%> (-1.59%)`	⬇️
...udstack/framework/config/impl/ConfigurationVO.java	`79.16% <100.00%> (ø)`

... and 1267 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

DaanHoogland

clgtm

DaanHoogland · 2022-10-12T18:40:13Z

@blueorangutan package

blueorangutan · 2022-10-12T18:41:03Z

@DaanHoogland a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-10-12T20:22:15Z

Packaging result: ✖️ el7 ✔️ el8 ✖️ debian ✔️ suse15. SL-JID 4436

sureshanaparti · 2022-10-13T01:34:29Z

@blueorangutan package

blueorangutan · 2022-10-13T01:35:03Z

@sureshanaparti a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-10-13T02:19:19Z

Packaging result: ✖️ el7 ✖️ el8 ✖️ debian ✖️ suse15. SL-JID 4442

blueorangutan · 2022-10-13T08:04:41Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 4448

DaanHoogland · 2022-10-13T08:07:55Z

@blueorangutan test

blueorangutan · 2022-10-13T08:09:03Z

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2022-10-13T08:26:34Z

Trillian Build Failed (tid-5118)

acs-robot · 2022-10-13T20:31:22Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-10-13T20:32:03Z

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

acs-robot · 2022-10-13T20:34:15Z

Found UI changes, kicking a new UI QA build
@blueorangutan ui

blueorangutan · 2022-10-13T20:35:03Z

@acs-robot a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

blueorangutan · 2022-10-13T20:37:56Z

UI build: ✔️
Live QA URL: http://qa.cloudstack.cloud:8080/client/pr/6812 (SL-JID-2498)

BryanMLima · 2023-09-22T21:00:40Z

@blueorangutan package

blueorangutan · 2023-09-22T21:02:03Z

@BryanMLima a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-09-22T21:53:12Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 7088

DaanHoogland · 2023-09-25T08:04:46Z

@blueorangutan test matrix

blueorangutan · 2023-09-25T08:06:03Z

@DaanHoogland a [SF] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2023-09-25T20:25:30Z

[SF] Trillian test result (tid-7707)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 43049 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6812-t7707-xenserver-71.zip
Smoke tests completed. 112 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Failure`	428.79	test_vpc_redundant.py

blueorangutan · 2023-09-25T21:35:38Z

[SF] Trillian test result (tid-7709)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 47138 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6812-t7709-kvm-centos7.zip
Smoke tests completed. 110 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_10_reboot_cpvm_forced	`Error`	6.25	test_ssvm.py
test_08_upgrade_kubernetes_ha_cluster	`Failure`	732.28	test_kubernetes_clusters.py
test_01_vpc_site2site_vpn	`Error`	273.77	test_vpc_vpn.py

blueorangutan · 2023-09-26T04:50:36Z

[SF] Trillian test result (tid-7708)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 73236 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6812-t7708-vmware-67u3.zip
Smoke tests completed. 111 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_list_vms_metrics_admin	`Error`	3616.48	test_metrics_api.py
test_list_vms_metrics_history	`Error`	7.72	test_metrics_api.py
test_list_volumes_metrics_history	`Error`	3616.05	test_metrics_api.py
test_01_deploy_vm_on_specific_host	`Error`	3602.67	test_vm_deployment_planner.py
test_02_deploy_vm_on_specific_cluster	`Error`	18.72	test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_pod	`Error`	3601.80	test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster	`Error`	1.35	test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_pod	`Error`	4.40	test_vm_deployment_planner.py

DaanHoogland · 2023-09-26T06:49:26Z

insanity:
@blueorangutan test matrix

blueorangutan · 2023-09-26T06:50:03Z

@DaanHoogland a [SF] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2023-09-26T21:49:59Z

[SF] Trillian test result (tid-7714)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 52619 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6812-t7714-xenserver-71.zip
Smoke tests completed. 112 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_invalid_upgrade_kubernetes_cluster	`Failure`	3611.96	test_kubernetes_clusters.py
test_02_upgrade_kubernetes_cluster	`Failure`	3604.53	test_kubernetes_clusters.py
test_03_deploy_and_scale_kubernetes_cluster	`Failure`	0.04	test_kubernetes_clusters.py
test_04_autoscale_kubernetes_cluster	`Failure`	0.03	test_kubernetes_clusters.py
test_05_basic_lifecycle_kubernetes_cluster	`Failure`	0.03	test_kubernetes_clusters.py
test_06_delete_kubernetes_cluster	`Failure`	0.04	test_kubernetes_clusters.py
test_07_deploy_kubernetes_ha_cluster	`Failure`	0.03	test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_cluster	`Failure`	0.04	test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster	`Failure`	0.03	test_kubernetes_clusters.py
test_10_vpc_tier_kubernetes_cluster	`Failure`	49.75	test_kubernetes_clusters.py
test_11_test_unmanaged_cluster_lifecycle	`Error`	1.22	test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown	`Error`	70.78	test_kubernetes_clusters.py

blueorangutan · 2023-09-26T22:48:42Z

[SF] Trillian test result (tid-7716)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 56022 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6812-t7716-kvm-centos7.zip
Smoke tests completed. 112 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_invalid_upgrade_kubernetes_cluster	`Failure`	3612.22	test_kubernetes_clusters.py
test_02_upgrade_kubernetes_cluster	`Failure`	3621.63	test_kubernetes_clusters.py
test_03_deploy_and_scale_kubernetes_cluster	`Failure`	0.05	test_kubernetes_clusters.py
test_04_autoscale_kubernetes_cluster	`Failure`	0.06	test_kubernetes_clusters.py
test_05_basic_lifecycle_kubernetes_cluster	`Failure`	0.04	test_kubernetes_clusters.py
test_06_delete_kubernetes_cluster	`Failure`	0.04	test_kubernetes_clusters.py
test_07_deploy_kubernetes_ha_cluster	`Failure`	0.03	test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_cluster	`Failure`	0.04	test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster	`Failure`	0.05	test_kubernetes_clusters.py
test_10_vpc_tier_kubernetes_cluster	`Failure`	46.71	test_kubernetes_clusters.py
test_11_test_unmanaged_cluster_lifecycle	`Error`	1.21	test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown	`Error`	54.45	test_kubernetes_clusters.py

blueorangutan · 2023-09-27T03:58:05Z

[SF] Trillian test result (tid-7715)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 74644 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6812-t7715-vmware-67u3.zip
Smoke tests completed. 110 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_snapshot_usage	`Error`	3603.10	test_usage.py
test_02_upgrade_kubernetes_cluster	`Failure`	588.89	test_kubernetes_clusters.py
test_10_vpc_tier_kubernetes_cluster	`Failure`	3735.93	test_kubernetes_clusters.py
test_10_vpc_tier_kubernetes_cluster	`Error`	3736.06	test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown	`Error`	180.63	test_kubernetes_clusters.py
test_01_snapshot_root_disk	`Error`	361.92	test_snapshots.py
test_02_list_snapshots_with_removed_data_store	`Error`	64.32	test_snapshots.py

DaanHoogland · 2023-09-27T07:28:00Z

@blueorangutan LLtest rocky8 vmware-67u3

blueorangutan · 2023-09-27T07:30:03Z

@DaanHoogland a [LL] Trillian-Jenkins test job (rocky8 mgmt + vmware-67u3) has been kicked to run smoke tests

blueorangutan · 2023-09-27T07:40:42Z

[LL] Trillian Build Failed (tid-6748)

blueorangutan · 2023-09-27T21:27:56Z

[LL]Trillian test result (tid-6749)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server a8
Total time taken: 46495 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr6812-t6749-vmware-67u3.zip
Smoke tests completed. 113 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

DaanHoogland · 2023-09-28T07:13:54Z

@weizhouapache @sureshanaparti is this alright in your view now?

weizhouapache · 2023-09-28T08:44:00Z

@weizhouapache @sureshanaparti is this alright in your view now?

lgtm, not tested.

DaanHoogland · 2023-09-28T09:30:35Z

lgtm, not tested.

ok, I think we need an upgrade test.

DaanHoogland · 2023-09-28T14:44:22Z

Did an upgrade from 4.18.1. I think we must take our chances and deal with whatever... merging

BryanMLima · 2023-09-28T15:21:16Z

Thanks, @DaanHoogland, though, I was already preparing a cake for the one-year anniversary.
Jokes aside, I will keep an eye in the mailing listing and issues, should a problem related to this PR arises.

boring-cyborgBot added component:api component:vpc labels Oct 7, 2022

DaanHoogland reviewed Oct 11, 2022
View reviewed changes

Comment threadserver/src/main/java/com/cloud/configuration/ConfigurationManagerImpl.java Outdated

DaanHoogland added the status:needs-testing label Oct 11, 2022

RodrigoDLopez approved these changes Oct 11, 2022
View reviewed changes

DaanHoogland previously approved these changes Oct 12, 2022
View reviewed changes

sureshanaparti reviewed Oct 13, 2022
View reviewed changes