CLOUDSTACK-9386: DS template copies don’t get deleted in VMware ESXi with multiple clusters and zone wide storage by nvazquez · Pull Request #1560 · apache/cloudstack

nvazquez · 2016-05-24T19:34:45Z

JIRA TICKET: https://issues.apache.org/jira/browse/CLOUDSTACK-9386

Introduction

In some production environments with multiple clusters it was noticed that unused templates were consuming too much storage. It was discovered that template cleanup was not deleting marked templates on ESXi.

Description of the problem

Suppose we have multiple clusters (c1, c2,...,cN) on a data center and template T from which we deploy vms on c1.
Suppose now that we expunge those vms, and there's no other vm instance from template T, so this was the actual workflow:

CloudStack marks template for cleanup after storage.cleanup.interval seconds, by setting marked_for_gc = 1 on template_spool_ref table, for that template.
After another storage.cleanup.interval seconds a DestroyCommand will be sent, to delete template from primary storage
On VmwareResource, command is processed, and it first picks up a random cluster, say ci != c1 to look for vm template (using volume's path) and destroy it. But, as template was on c1 it cannot be found, so it won't be deleted. Entry on template_spool_ref is deleted but not the actual template on hypervisor side.

Proposed solution

We propose a way to attack problem shown in point 3, by not picking up a random cluster to look for vm but using vSphere data center. This way we make sure vm template will be deleted in every case, and not depending on random cluster selection

rafaelweingartner · 2016-05-24T21:37:30Z

Hi @nvazquez,

Thanks for the PR. Great work as always, especially on documenting the problem.

I think we can improve it a little bit. What about extracting lines 5480-5485 to a method? Then, those java comments can become java docs (which I believe are much better) and also you would be able to write test cases.

Also, I think we should change the “throw new Exception()” to a more specific exception. What about the CloudRuntimeException or another one?

nvazquez · 2016-05-25T16:01:03Z

Hi @rafaelweingartner thanks for your comments!
I extracted to a method and added test cases for this new method. Also changed Exception to CloudRuntimeException when data center can't be found.

rafaelweingartner · 2016-05-25T16:25:16Z

Great job, thanks @nvazquez

rafaelweingartner · 2016-05-25T16:38:00Z

+ s_logger.error(msg);
+ throw new CloudRuntimeException(msg);
+ }
+ VirtualMachineMO vmMo = dcMo.findVm(vol.getPath());


What about using here: return dcMo.findVm(vol.getPath());
then, we could save 1 line of code ;)

My bad, I pushed and fixed it :)

bvbharatk · 2016-06-11T17:36:41Z

ACS CI BVT Run

Sumarry:
Build Number 185
Hypervisor xenserver
NetworkType Advanced
Passed=73
Failed=0
Skipped=3

Link to logs Folder (search by build_no):https://www.dropbox.com/sh/yj3wnzbceo9uef2/AAB6u-Iap-xztdm6jHX9SjPja?dl=0

Failed tests:

Skipped tests:
test_vm_nic_adapter_vmxnet3
test_static_role_account_acls
test_deploy_vgpu_enabled_vm

Passed test suits:
test_deploy_vm_with_userdata.py
test_affinity_groups_projects.py
test_portable_publicip.py
test_vpc_vpn.py
test_over_provisioning.py
test_global_settings.py
test_scale_vm.py
test_service_offerings.py
test_routers_iptables_default_policy.py
test_routers.py
test_reset_vm_on_reboot.py
test_snapshots.py
test_deploy_vms_with_varied_deploymentplanners.py
test_login.py
test_list_ids_parameter.py
test_public_ip_range.py
test_multipleips_per_nic.py
test_regions.py
test_affinity_groups.py
test_network_acl.py
test_pvlan.py
test_volumes.py
test_nic.py
test_deploy_vm_root_resize.py
test_resource_detail.py
test_secondary_storage.py
test_vm_life_cycle.py
test_disk_offerings.py

nvazquez · 2016-08-04T17:01:01Z

Hi guys,

This pull request it's been opened for a while, @rafaelweingartner, @bvbharatk you've reviewed it before, could you give your blessing to this PR?

CC: @swill

serg38 · 2016-08-16T16:48:52Z

@rafaelweingartner @jburwell @rhtyd @koushik-das Can you review this PR?

rafaelweingartner · 2016-08-23T18:00:54Z

LGTM from me here ;)
thanks @serg38 and @nvazquez for the hard work.

serg38 · 2016-08-29T22:29:39Z

@rhtyd @jburwell Can this be merged?

yadvr · 2016-08-30T05:00:13Z

LGTM (code/logic seems alright, did not run any tests).

nvazquez · 2016-09-01T17:31:22Z

I rebased master branch, I'll post test results for this PR

nvazquez · 2016-09-02T16:30:29Z

@jburwell @rhtyd I post test results:

[root@ussarlabcsmgt41 cloudstack]# cat /tmp//MarvinLogs/test_volumes_SAGNPP/results.txt
test DeployVM in anti-affinity groups for project ... === TestName: test_DeployVmAntiAffinityGroup_in_project | Status : SUCCESS ===
ok
test DeployVM in anti-affinity groups ... === TestName: test_DeployVmAntiAffinityGroup | Status : SUCCESS ===
ok
Test Deploy Virtual Machine ... SKIP: Skipping test because suitable hypervisor/host not present
Test Deploy Virtual Machine from ISO ... === TestName: test_deploy_vm_from_iso | Status : SUCCESS ===
ok
Test deploy virtual machine with root resize ... === TestName: test_00_deploy_vm_root_resize | Status : SUCCESS ===
ok
Test proper failure to deploy virtual machine with rootdisksize of 0 ... === TestName: test_01_deploy_vm_root_resize | Status : SUCCESS ===
ok
Test proper failure to deploy virtual machine with rootdisksize less than template size ... === TestName: test_02_deploy_vm_root_resize | Status : SUCCESS ===
ok
Test to deploy vm with a first fit offering ... === TestName: test_deployvm_firstfit | Status : SUCCESS ===
ok
Test deploy VMs using user concentrated planner ... === TestName: test_deployvm_userconcentrated | Status : SUCCESS ===
ok
Test deploy VMs using user dispersion planner ... === TestName: test_deployvm_userdispersing | Status : SUCCESS ===
ok
Test userdata as GET, size > 2k ... === TestName: test_deployvm_userdata | Status : SUCCESS ===
ok
Test userdata as POST, size > 2k ... === TestName: test_deployvm_userdata_post | Status : SUCCESS ===
ok
Test to create disk offering ... === TestName: test_01_create_disk_offering | Status : SUCCESS ===
ok
Test to create a sparse type disk offering ... === TestName: test_02_create_sparse_type_disk_offering | Status : SUCCESS ===
ok
Test to create a sparse type disk offering ... === TestName: test_04_create_fat_type_disk_offering | Status : SUCCESS ===
ok
Test to update existing disk offering ... === TestName: test_02_edit_disk_offering | Status : SUCCESS ===
ok
Test to delete disk offering ... === TestName: test_03_delete_disk_offering | Status : SUCCESS ===
ok
Test to ensure 4 default roles cannot be deleted ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Test to check role, role permissions and account life cycles ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Test for role-rule enforcement in case of multiple mgmt servers ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Test to ensure role in use cannot be deleted ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests normal lifecycle operations for roles ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests role update ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests that default four roles exist ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests role update ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests role update when role is in use by an account ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests concurrent order updation of role permission ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests creation of role permission ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests deletion of role permission ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests listing of default role's permission ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests order updation of role permission ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
test update configuration setting at zone level scope ... === TestName: test_UpdateConfigParamWithScope | Status : SUCCESS ===
ok
Test guest vlan range dedication ... === TestName: test_dedicateGuestVlanRange | Status : SUCCESS ===
ok
Test create public & private ISO ... === TestName: test_01_create_iso | Status : SUCCESS ===
ok
Test Edit ISO ... === TestName: test_02_edit_iso | Status : SUCCESS ===
ok
Test delete ISO ... === TestName: test_03_delete_iso | Status : SUCCESS ===
ok
Test for extract ISO ... === TestName: test_04_extract_Iso | Status : SUCCESS ===
ok
Update & Test for ISO permissions ... === TestName: test_05_iso_permissions | Status : SUCCESS ===
ok
Test for copy ISO from one zone to another ... SKIP: Not enough zones available to perform copy template
Test delete ISO ... === TestName: test_07_list_default_iso | Status : SUCCESS ===
ok
Test listing Volumes using 'ids' parameter ... === TestName: test_01_list_volumes | Status : SUCCESS ===
ok
Test listing Templates using 'ids' parameter ... === TestName: test_02_list_templates | Status : SUCCESS ===
ok
Test listing Snapshots using 'ids' parameter ... === TestName: test_03_list_snapshots | Status : SUCCESS ===
ok
Test to create Load balancing rule with source NAT ... === TestName: test_01_create_lb_rule_src_nat | Status : SUCCESS ===
ok
Test to create Load balancing rule with non source NAT ... === TestName: test_02_create_lb_rule_non_nat | Status : SUCCESS ===
ok
Test for assign & removing load balancing rule ... === TestName: test_assign_and_removal_lb | Status : SUCCESS ===
ok
Tests that SAML users are not allowed CloudStack local log in ... === TestName: login_test_saml_user | Status : SUCCESS ===
ok
test_nic_secondaryip_add_remove (integration.smoke.test_multipleips_per_nic.TestDeployVM) ... === TestName: test_nic_secondaryip_add_remove | Status : SUCCESS ===
ok
Test network ACL lists and items in VPC ... === TestName: test_network_acl | Status : SUCCESS ===
ok
# 1. Register a template for VMware with nicAdapter vmxnet3 ... SKIP: VCenter API Integration Remaining
Test to add and update added nic to a virtual machine ... === TestName: test_01_nic | Status : SUCCESS ===
ok
Test to update a physical network and extend its vlan ... === TestName: test_extendPhysicalNetworkVlan | Status : SUCCESS ===
ok
test update configuration setting at storage scope ... === TestName: test_UpdateStorageOverProvisioningFactor | Status : SUCCESS ===
ok
test_create_pvlan_network (integration.smoke.test_pvlan.TestPVLAN) ... === TestName: test_create_pvlan_network | Status : SUCCESS ===
ok
Test for create region ... === TestName: test_createRegion | Status : SUCCESS ===
ok
Test reset virtual machine on reboot ... === TestName: test_01_reset_vm_on_reboot | Status : SUCCESS ===
ok
Test volume detail ... === TestName: test_01_updatevolumedetail | Status : SUCCESS ===
ok
Test scale virtual machine ... SKIP: Skipping scale VM operation because VMware tools are not installed on the VM
Test to create service offering ... === TestName: test_01_create_service_offering | Status : SUCCESS ===
ok
Test to update existing service offering ... === TestName: test_02_edit_service_offering | Status : SUCCESS ===
ok
Test to delete service offering ... === TestName: test_03_delete_service_offering | Status : SUCCESS ===
ok
Test to change service to a small capacity ... === TestName: test_04_change_offering_small | Status : SUCCESS ===
ok
Test List secondary storage VMs ... === TestName: test_01_list_sec_storage_vm | Status : SUCCESS ===
ok
Test List console proxy VMs ... === TestName: test_02_list_cpvm_vm | Status : SUCCESS ===
ok
Test SSVM Internals ... === TestName: test_03_ssvm_internals | Status : SUCCESS ===
ok
Test CPVM Internals ... === TestName: test_04_cpvm_internals | Status : SUCCESS ===
ok
Test stop SSVM ... === TestName: test_05_stop_ssvm | Status : SUCCESS ===
ok
Test stop CPVM ... === TestName: test_06_stop_cpvm | Status : SUCCESS ===
ok
Test reboot SSVM ... === TestName: test_07_reboot_ssvm | Status : SUCCESS ===
ok
Test reboot CPVM ... === TestName: test_08_reboot_cpvm | Status : SUCCESS ===
ok
Test destroy SSVM ... === TestName: test_09_destroy_ssvm | Status : SUCCESS ===
ok
Test destroy CPVM ... === TestName: test_10_destroy_cpvm | Status : SUCCESS ===
ok
Tests allowed APIs for common account types ... === TestName: test_static_role_account_acls | Status : SUCCESS ===
ok
Test create public & private template ... === TestName: test_01_create_template | Status : SUCCESS ===
ok
Test when createTemplate is used to create templates having the same name all of them get ... === TestName: test_CreateTemplateWithDuplicateName | Status : SUCCESS ===
ok
Test Edit template ... === TestName: test_02_edit_template | Status : SUCCESS ===
ok
Test delete template ... === TestName: test_03_delete_template | Status : SUCCESS ===
ok
Test for extract template ... === TestName: test_04_extract_template | Status : SUCCESS ===
ok
Update & Test for template permissions ... === TestName: test_05_template_permissions | Status : SUCCESS ===
ok
Test for copy template from one zone to another ... SKIP: Not enough zones available to perform copy template
Test only public templates are visible to normal user ... === TestName: test_07_list_public_templates | Status : SUCCESS ===
ok
Test System templates are not visible to normal user ... === TestName: test_08_list_system_templates | Status : SUCCESS ===
ok
Check events in usage_events table when VM creation fails ... === TestName: test_01_positive_tests_usage | Status : SUCCESS ===
ok
Test advanced zone virtual router ... === TestName: test_advZoneVirtualRouter | Status : SUCCESS ===
ok
Tests for basic zone virtual router ... === TestName: test_basicZoneVirtualRouter | Status : SUCCESS ===
ok
Test Deploy Virtual Machine ... === TestName: test_deploy_vm | Status : SUCCESS ===
ok
Test Multiple Deploy Virtual Machine ... === TestName: test_deploy_vm_multiple | Status : SUCCESS ===
ok
Test Stop Virtual Machine ... === TestName: test_01_stop_vm | Status : SUCCESS ===
ok
Test Start Virtual Machine ... === TestName: test_02_start_vm | Status : SUCCESS ===
ok
Test Reboot Virtual Machine ... === TestName: test_03_reboot_vm | Status : SUCCESS ===
ok
Test destroy Virtual Machine ... === TestName: test_06_destroy_vm | Status : SUCCESS ===
ok
Test recover Virtual Machine ... === TestName: test_07_restore_vm | Status : SUCCESS ===
ok
Test migrate VM ... === TestName: test_08_migrate_vm | Status : SUCCESS ===
ok
Test destroy(expunge) Virtual Machine ... === TestName: test_09_expunge_vm | Status : SUCCESS ===
ok
Test for attach and detach ISO to virtual machine ... === TestName: test_10_attachAndDetach_iso | Status : SUCCESS ===
ok
Test Volume creation for all Disk Offerings (incl. custom) ... === TestName: test_01_create_volume | Status : SUCCESS ===
ok
Attach a created Volume to a Running VM ... === TestName: test_02_attach_volume | Status : SUCCESS ===
ok
Download a Volume attached to a VM ... === TestName: test_03_download_attached_volume | Status : SUCCESS ===
ok
Delete a Volume attached to a VM ... === TestName: test_04_delete_attached_volume | Status : SUCCESS ===
ok
Detach a Volume attached to a VM ... === TestName: test_05_detach_volume | Status : SUCCESS ===
ok
Download a Volume unattached to an VM ... === TestName: test_06_download_detached_volume | Status : SUCCESS ===
ok
Test resize (negative) non-existent volume ... SKIP: Resize Volume is unsupported on VmWare and Hyper-V
Test resize a volume ... SKIP: Resize Volume is unsupported on VmWare and Hyper-V
Delete a Volume unattached to an VM ... === TestName: test_09_delete_detached_volume | Status : SUCCESS ===
ok
----------------------------------------------------------------------
Ran 104 tests in 9979.601s
OK (SKIP=21)

serg38 · 2016-09-11T15:12:25Z

LGTM for testing. Re-ran smoke tests:

test DeployVM in anti-affinity groups for project ... === TestName: test_DeployVmAntiAffinityGroup_in_project | Status : SUCCESS ===
ok
test DeployVM in anti-affinity groups ... === TestName: test_DeployVmAntiAffinityGroup | Status : SUCCESS ===
ok
Test Deploy Virtual Machine ... SKIP: Skipping test because suitable hypervisor/host not present
Test Deploy Virtual Machine from ISO ... === TestName: test_deploy_vm_from_iso | Status : SUCCESS ===
ok
Test deploy virtual machine with root resize ... === TestName: test_00_deploy_vm_root_resize | Status : SUCCESS ===
ok
Test proper failure to deploy virtual machine with rootdisksize of 0 ... === TestName: test_01_deploy_vm_root_resize | Status : SUCCESS ===
ok
Test proper failure to deploy virtual machine with rootdisksize less than template size ... === TestName: test_02_deploy_vm_root_resize | Status : SUCCESS ===
ok
Test to deploy vm with a first fit offering ... === TestName: test_deployvm_firstfit | Status : SUCCESS ===
ok
Test deploy VMs using user concentrated planner ... === TestName: test_deployvm_userconcentrated | Status : SUCCESS ===
ok
Test deploy VMs using user dispersion planner ... === TestName: test_deployvm_userdispersing | Status : SUCCESS ===
ok
Test userdata as GET, size > 2k ... === TestName: test_deployvm_userdata | Status : SUCCESS ===
ok
Test userdata as POST, size > 2k ... === TestName: test_deployvm_userdata_post | Status : SUCCESS ===
ok
Test to create disk offering ... === TestName: test_01_create_disk_offering | Status : SUCCESS ===
ok
Test to create a sparse type disk offering ... === TestName: test_02_create_sparse_type_disk_offering | Status : SUCCESS ===
ok
Test to create a sparse type disk offering ... === TestName: test_04_create_fat_type_disk_offering | Status : SUCCESS ===
ok
Test to update existing disk offering ... === TestName: test_02_edit_disk_offering | Status : SUCCESS ===
ok
Test to delete disk offering ... === TestName: test_03_delete_disk_offering | Status : SUCCESS ===
ok
Test to ensure 4 default roles cannot be deleted ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Test to check role, role permissions and account life cycles ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Test for role-rule enforcement in case of multiple mgmt servers ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Test to ensure role in use cannot be deleted ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests normal lifecycle operations for roles ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests role update ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests that default four roles exist ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests role update ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests role update when role is in use by an account ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests concurrent order updation of role permission ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests creation of role permission ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests deletion of role permission ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests listing of default role's permission ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests order updation of role permission ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
test update configuration setting at zone level scope ... === TestName: test_UpdateConfigParamWithScope | Status : SUCCESS ===
ok
Test guest vlan range dedication ... === TestName: test_dedicateGuestVlanRange | Status : SUCCESS ===
ok
Test create public & private ISO ... === TestName: test_01_create_iso | Status : SUCCESS ===
ok
Test Edit ISO ... === TestName: test_02_edit_iso | Status : SUCCESS ===
ok
Test delete ISO ... === TestName: test_03_delete_iso | Status : SUCCESS ===
ok
Test for extract ISO ... === TestName: test_04_extract_Iso | Status : SUCCESS ===
ok
Update & Test for ISO permissions ... === TestName: test_05_iso_permissions | Status : SUCCESS ===
ok
Test for copy ISO from one zone to another ... SKIP: Not enough zones available to perform copy template
Test delete ISO ... === TestName: test_07_list_default_iso | Status : SUCCESS ===
ok
Test listing Volumes using 'ids' parameter ... === TestName: test_01_list_volumes | Status : SUCCESS ===
ok
Test listing Templates using 'ids' parameter ... === TestName: test_02_list_templates | Status : SUCCESS ===
ok
Test listing Snapshots using 'ids' parameter ... === TestName: test_03_list_snapshots | Status : SUCCESS ===
ok
Test to create Load balancing rule with source NAT ... === TestName: test_01_create_lb_rule_src_nat | Status : SUCCESS ===
ok
Test to create Load balancing rule with non source NAT ... === TestName: test_02_create_lb_rule_non_nat | Status : SUCCESS ===
ok
Test for assign & removing load balancing rule ... === TestName: test_assign_and_removal_lb | Status : SUCCESS ===
ok
Tests that SAML users are not allowed CloudStack local log in ... === TestName: login_test_saml_user | Status : SUCCESS ===
ok
test_nic_secondaryip_add_remove (integration.smoke.test_multipleips_per_nic.TestDeployVM) ... === TestName: test_nic_secondaryip_add_remove | Status : SUCCESS ===
ok
Test network ACL lists and items in VPC ... === TestName: test_network_acl | Status : SUCCESS ===
ok
Register a template for VMware with nicAdapter vmxnet3 ... SKIP: VCenter API Integration Remaining
Test to add and update added nic to a virtual machine ... === TestName: test_01_nic | Status : SUCCESS ===
ok
Test to update a physical network and extend its vlan ... === TestName: test_extendPhysicalNetworkVlan | Status : SUCCESS ===
ok
test update configuration setting at storage scope ... === TestName: test_UpdateStorageOverProvisioningFactor | Status : SUCCESS ===
ok
test_create_pvlan_network (integration.smoke.test_pvlan.TestPVLAN) ... === TestName: test_create_pvlan_network | Status : SUCCESS ===
ok
Test for create region ... === TestName: test_createRegion | Status : SUCCESS ===
ok
Test reset virtual machine on reboot ... === TestName: test_01_reset_vm_on_reboot | Status : SUCCESS ===
ok
Test volume detail ... === TestName: test_01_updatevolumedetail | Status : SUCCESS ===
ok
Test scale virtual machine ... SKIP: Skipping scale VM operation because VMware tools are not installed on the VM
Test to create service offering ... === TestName: test_01_create_service_offering | Status : SUCCESS ===
ok
Test to update existing service offering ... === TestName: test_02_edit_service_offering | Status : SUCCESS ===
ok
Test to delete service offering ... === TestName: test_03_delete_service_offering | Status : SUCCESS ===
ok
Test to change service to a small capacity ... === TestName: test_04_change_offering_small | Status : SUCCESS ===
ok
Test List secondary storage VMs ... === TestName: test_01_list_sec_storage_vm | Status : SUCCESS ===
ok
Test List console proxy VMs ... === TestName: test_02_list_cpvm_vm | Status : SUCCESS ===
ok
Test SSVM Internals ... === TestName: test_03_ssvm_internals | Status : SUCCESS ===
ok
Test CPVM Internals ... === TestName: test_04_cpvm_internals | Status : SUCCESS ===
ok
Test stop SSVM ... === TestName: test_05_stop_ssvm | Status : SUCCESS ===
ok
Test stop CPVM ... === TestName: test_06_stop_cpvm | Status : SUCCESS ===
ok
Test reboot SSVM ... === TestName: test_07_reboot_ssvm | Status : SUCCESS ===
ok
Test reboot CPVM ... === TestName: test_08_reboot_cpvm | Status : SUCCESS ===
ok
Test destroy SSVM ... === TestName: test_09_destroy_ssvm | Status : SUCCESS ===
ok
Test destroy CPVM ... === TestName: test_10_destroy_cpvm | Status : SUCCESS ===
ok
Tests allowed APIs for common account types ... === TestName: test_static_role_account_acls | Status : SUCCESS ===
ok
Test create public & private template ... === TestName: test_01_create_template | Status : SUCCESS ===
ok
Test when createTemplate is used to create templates having the same name all of them get ... === TestName: test_CreateTemplateWithDuplicateName | Status : SUCCESS ===
ok
Test Edit template ... === TestName: test_02_edit_template | Status : SUCCESS ===
ok
Test delete template ... === TestName: test_03_delete_template | Status : SUCCESS ===
ok
Test for extract template ... === TestName: test_04_extract_template | Status : SUCCESS ===
ok
Update & Test for template permissions ... === TestName: test_05_template_permissions | Status : SUCCESS ===
ok
Test for copy template from one zone to another ... SKIP: Not enough zones available to perform copy template
Test only public templates are visible to normal user ... === TestName: test_07_list_public_templates | Status : SUCCESS ===
ok
Test System templates are not visible to normal user ... === TestName: test_08_list_system_templates | Status : SUCCESS ===
ok
Check events in usage_events table when VM creation fails ... === TestName: test_01_positive_tests_usage | Status : SUCCESS ===
ok
Test advanced zone virtual router ... === TestName: test_advZoneVirtualRouter | Status : SUCCESS ===
ok
Tests for basic zone virtual router ... === TestName: test_basicZoneVirtualRouter | Status : SUCCESS ===
ok
Test Deploy Virtual Machine ... === TestName: test_deploy_vm | Status : SUCCESS ===
ok
Test Multiple Deploy Virtual Machine ... === TestName: test_deploy_vm_multiple | Status : SUCCESS ===
ok
Test Stop Virtual Machine ... === TestName: test_01_stop_vm | Status : SUCCESS ===
ok
Test Start Virtual Machine ... === TestName: test_02_start_vm | Status : SUCCESS ===
ok
Test Reboot Virtual Machine ... === TestName: test_03_reboot_vm | Status : SUCCESS ===
ok
Test destroy Virtual Machine ... === TestName: test_06_destroy_vm | Status : SUCCESS ===
ok
Test recover Virtual Machine ... === TestName: test_07_restore_vm | Status : SUCCESS ===
ok
Test migrate VM ... === TestName: test_08_migrate_vm | Status : SUCCESS ===
ok
Test destroy(expunge) Virtual Machine ... === TestName: test_09_expunge_vm | Status : SUCCESS ===
ok
Test for attach and detach ISO to virtual machine ... === TestName: test_10_attachAndDetach_iso | Status : SUCCESS ===
ok
Test Volume creation for all Disk Offerings (incl. custom) ... === TestName: test_01_create_volume | Status : SUCCESS ===
ok
Attach a created Volume to a Running VM ... === TestName: test_02_attach_volume | Status : SUCCESS ===
ok
Download a Volume attached to a VM ... === TestName: test_03_download_attached_volume | Status : SUCCESS ===
ok
Delete a Volume attached to a VM ... === TestName: test_04_delete_attached_volume | Status : SUCCESS ===
ok
Detach a Volume attached to a VM ... === TestName: test_05_detach_volume | Status : SUCCESS ===
ok
Download a Volume unattached to an VM ... === TestName: test_06_download_detached_volume | Status : SUCCESS ===
ok
Test resize (negative) non-existent volume ... SKIP: Resize Volume is unsupported on VmWare and Hyper-V
Test resize a volume ... SKIP: Resize Volume is unsupported on VmWare and Hyper-V
Delete a Volume unattached to an VM ... === TestName: test_09_delete_detached_volume | Status : SUCCESS ===
ok

Ran 104 tests in 9821.371s

OK (SKIP=21)

serg38 · 2016-09-11T15:16:16Z

@rafaelweingartner @jburwell This PR seems to be ready for merging as well

jburwell · 2016-09-11T15:18:43Z

@nvazquez this PR has merge conflicts that must be resolved before we can merge it. @serg38 after @nvazquez has resolved the merge conflicts, could you please re-execute your tests to ensure that the conflict resolution does not introduce any issues?

nvazquez · 2016-09-11T22:36:36Z

Great, thanks @jburwell!

… cluster

serg38 · 2016-09-12T20:29:17Z

@rafaelweingartner @jburwell @karuturi Integration tests passed after merge conflict resolution

test DeployVM in anti-affinity groups for project ... === TestName: test_DeployVmAntiAffinityGroup_in_project | Status : SUCCESS ===
test DeployVM in anti-affinity groups ... === TestName: test_DeployVmAntiAffinityGroup | Status : SUCCESS ===
Test Deploy Virtual Machine ... SKIP: Skipping test because suitable hypervisor/host not present
Test Deploy Virtual Machine from ISO ... === TestName: test_deploy_vm_from_iso | Status : SUCCESS ===
Test deploy virtual machine with root resize ... === TestName: test_00_deploy_vm_root_resize | Status : SUCCESS ===
Test proper failure to deploy virtual machine with rootdisksize of 0 ... === TestName: test_01_deploy_vm_root_resize | Status : SUCCESS ===
Test proper failure to deploy virtual machine with rootdisksize less than template size ... === TestName: test_02_deploy_vm_root_resize | Status : SUCCESS ===
Test to deploy vm with a first fit offering ... === TestName: test_deployvm_firstfit | Status : SUCCESS ===
Test deploy VMs using user concentrated planner ... === TestName: test_deployvm_userconcentrated | Status : SUCCESS ===
Test deploy VMs using user dispersion planner ... === TestName: test_deployvm_userdispersing | Status : SUCCESS ===
Test userdata as GET, size > 2k ... === TestName: test_deployvm_userdata | Status : SUCCESS ===
Test userdata as POST, size > 2k ... === TestName: test_deployvm_userdata_post | Status : SUCCESS ===
Test to create disk offering ... === TestName: test_01_create_disk_offering | Status : SUCCESS ===
Test to create a sparse type disk offering ... === TestName: test_02_create_sparse_type_disk_offering | Status : SUCCESS ===
Test to create a sparse type disk offering ... === TestName: test_04_create_fat_type_disk_offering | Status : SUCCESS ===
Test to update existing disk offering ... === TestName: test_02_edit_disk_offering | Status : SUCCESS ===
Test to delete disk offering ... === TestName: test_03_delete_disk_offering | Status : SUCCESS ===
Test to ensure 4 default roles cannot be deleted ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Test to check role, role permissions and account life cycles ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Test for role-rule enforcement in case of multiple mgmt servers ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Test to ensure role in use cannot be deleted ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests normal lifecycle operations for roles ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests role update ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests that default four roles exist ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests role update ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests role update when role is in use by an account ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests concurrent order updation of role permission ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests creation of role permission ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests deletion of role permission ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests listing of default role's permission ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
Tests order updation of role permission ... SKIP: Dynamic Role-Based API checker not enabled, skipping test
test update configuration setting at zone level scope ... === TestName: test_UpdateConfigParamWithScope | Status : SUCCESS ===
Test guest vlan range dedication ... === TestName: test_dedicateGuestVlanRange | Status : SUCCESS ===
Test create public & private ISO ... === TestName: test_01_create_iso | Status : SUCCESS ===
Test Edit ISO ... === TestName: test_02_edit_iso | Status : SUCCESS ===
Test delete ISO ... === TestName: test_03_delete_iso | Status : SUCCESS ===
Test for extract ISO ... === TestName: test_04_extract_Iso | Status : SUCCESS ===
Update & Test for ISO permissions ... === TestName: test_05_iso_permissions | Status : SUCCESS ===
Test for copy ISO from one zone to another ... SKIP: Not enough zones available to perform copy template
Test delete ISO ... === TestName: test_07_list_default_iso | Status : SUCCESS ===
Test listing Volumes using 'ids' parameter ... === TestName: test_01_list_volumes | Status : SUCCESS ===
Test listing Templates using 'ids' parameter ... === TestName: test_02_list_templates | Status : SUCCESS ===
Test listing Snapshots using 'ids' parameter ... === TestName: test_03_list_snapshots | Status : SUCCESS ===
Test to create Load balancing rule with source NAT ... === TestName: test_01_create_lb_rule_src_nat | Status : SUCCESS ===
Test to create Load balancing rule with non source NAT ... === TestName: test_02_create_lb_rule_non_nat | Status : SUCCESS ===
Test for assign & removing load balancing rule ... === TestName: test_assign_and_removal_lb | Status : SUCCESS ===
Tests that SAML users are not allowed CloudStack local log in ... === TestName: login_test_saml_user | Status : SUCCESS ===
test_nic_secondaryip_add_remove (integration.smoke.test_multipleips_per_nic.TestDeployVM) ... === TestName: test_nic_secondaryip_add_remove | Status : SUCCESS ===
Test network ACL lists and items in VPC ... === TestName: test_network_acl | Status : SUCCESS ===

1. Register a template for VMware with nicAdapter vmxnet3 ... SKIP: VCenter API Integration Remaining

Test to add and update added nic to a virtual machine ... === TestName: test_01_nic | Status : SUCCESS ===
Test to update a physical network and extend its vlan ... === TestName: test_extendPhysicalNetworkVlan | Status : SUCCESS ===
test update configuration setting at storage scope ... === TestName: test_UpdateStorageOverProvisioningFactor | Status : SUCCESS ===
test_create_pvlan_network (integration.smoke.test_pvlan.TestPVLAN) ... === TestName: test_create_pvlan_network | Status : SUCCESS ===
Test for create region ... === TestName: test_createRegion | Status : SUCCESS ===
Test reset virtual machine on reboot ... === TestName: test_01_reset_vm_on_reboot | Status : SUCCESS ===
Test volume detail ... === TestName: test_01_updatevolumedetail | Status : SUCCESS ===
Test scale virtual machine ... SKIP: Skipping scale VM operation because VMware tools are not installed on the VM
Test to create service offering ... === TestName: test_01_create_service_offering | Status : SUCCESS ===
Test to update existing service offering ... === TestName: test_02_edit_service_offering | Status : SUCCESS ===
Test to delete service offering ... === TestName: test_03_delete_service_offering | Status : SUCCESS ===
Test to change service to a small capacity ... === TestName: test_04_change_offering_small | Status : SUCCESS ===
Test List secondary storage VMs ... === TestName: test_01_list_sec_storage_vm | Status : SUCCESS ===
Test List console proxy VMs ... === TestName: test_02_list_cpvm_vm | Status : SUCCESS ===
Test SSVM Internals ... === TestName: test_03_ssvm_internals | Status : SUCCESS ===
Test CPVM Internals ... === TestName: test_04_cpvm_internals | Status : SUCCESS ===
Test stop SSVM ... === TestName: test_05_stop_ssvm | Status : SUCCESS ===
Test stop CPVM ... === TestName: test_06_stop_cpvm | Status : SUCCESS ===
Test reboot SSVM ... === TestName: test_07_reboot_ssvm | Status : SUCCESS ===
Test reboot CPVM ... === TestName: test_08_reboot_cpvm | Status : SUCCESS ===
Test destroy SSVM ... === TestName: test_09_destroy_ssvm | Status : SUCCESS ===
Test destroy CPVM ... === TestName: test_10_destroy_cpvm | Status : SUCCESS ===
Tests allowed APIs for common account types ... === TestName: test_static_role_account_acls | Status : SUCCESS ===
Test create public & private template ... === TestName: test_01_create_template | Status : SUCCESS ===
Test when createTemplate is used to create templates having the same name all of them get ... === TestName: test_CreateTemplateWithDuplicateName | Status : SUCCESS ===
Test Edit template ... === TestName: test_02_edit_template | Status : SUCCESS ===
Test delete template ... === TestName: test_03_delete_template | Status : SUCCESS ===
Test for extract template ... === TestName: test_04_extract_template | Status : SUCCESS ===
Update & Test for template permissions ... === TestName: test_05_template_permissions | Status : SUCCESS ===
Test for copy template from one zone to another ... SKIP: Not enough zones available to perform copy template
Test only public templates are visible to normal user ... === TestName: test_07_list_public_templates | Status : SUCCESS ===
Test System templates are not visible to normal user ... === TestName: test_08_list_system_templates | Status : SUCCESS ===
Check events in usage_events table when VM creation fails ... === TestName: test_01_positive_tests_usage | Status : SUCCESS ===
Test advanced zone virtual router ... === TestName: test_advZoneVirtualRouter | Status : SUCCESS ===
Tests for basic zone virtual router ... === TestName: test_basicZoneVirtualRouter | Status : SUCCESS ===
Test Deploy Virtual Machine ... === TestName: test_deploy_vm | Status : SUCCESS ===
Test Multiple Deploy Virtual Machine ... === TestName: test_deploy_vm_multiple | Status : SUCCESS ===
Test Stop Virtual Machine ... === TestName: test_01_stop_vm | Status : SUCCESS ===
Test Start Virtual Machine ... === TestName: test_02_start_vm | Status : SUCCESS ===
Test Reboot Virtual Machine ... === TestName: test_03_reboot_vm | Status : SUCCESS ===
Test destroy Virtual Machine ... === TestName: test_06_destroy_vm | Status : SUCCESS ===
Test recover Virtual Machine ... === TestName: test_07_restore_vm | Status : SUCCESS ===
Test migrate VM ... === TestName: test_08_migrate_vm | Status : SUCCESS ===
Test destroy(expunge) Virtual Machine ... === TestName: test_09_expunge_vm | Status : SUCCESS ===
Test for attach and detach ISO to virtual machine ... === TestName: test_10_attachAndDetach_iso | Status : SUCCESS ===
Test Volume creation for all Disk Offerings (incl. custom) ... === TestName: test_01_create_volume | Status : SUCCESS ===
Attach a created Volume to a Running VM ... === TestName: test_02_attach_volume | Status : SUCCESS ===
Download a Volume attached to a VM ... === TestName: test_03_download_attached_volume | Status : SUCCESS ===
Delete a Volume attached to a VM ... === TestName: test_04_delete_attached_volume | Status : SUCCESS ===
Detach a Volume attached to a VM ... === TestName: test_05_detach_volume | Status : SUCCESS ===
Download a Volume unattached to an VM ... === TestName: test_06_download_detached_volume | Status : SUCCESS ===
Test resize (negative) non-existent volume ... SKIP: Resize Volume is unsupported on VmWare and Hyper-V
Test resize a volume ... SKIP: Resize Volume is unsupported on VmWare and Hyper-V
Delete a Volume unattached to an VM ... === TestName: test_09_delete_detached_volume | Status : SUCCESS ===

Ran 104 tests in 10873.015s

OK (SKIP=21)

rafaelweingartner · 2016-09-12T22:19:42Z

@serg38 great work.
It seems that we already have reviewed this one, and executed tests.
I will merge this PR tomorrow if no one objects.

jburwell · 2016-09-13T03:04:47Z

@serg38 the git-pr script is not working properly for me. I will debug further tomorrow.

@rafaelweingartner do you mind doing the merge honors?

CLOUDSTACK-9386: DS template copies dont get deleted in VMware ESXi with multiple clusters and zone wide storageJIRA TICKET: https://issues.apache.org/jira/browse/CLOUDSTACK-9386 ### Introduction In some production environments with multiple clusters it was noticed that unused templates were consuming too much storage. It was discovered that template cleanup was not deleting marked templates on ESXi. ### Description of the problem Suppose we have multiple clusters `(c1, c2,...,cN)` on a data center and template `T` from which we deploy vms on `c1.` Suppose now that we expunge those vms, and there's no other vm instance from template `T,` so this was the actual workflow: 1. CloudStack marks template for cleanup after `storage.cleanup.interval` seconds, by setting `marked_for_gc = 1` on `template_spool_ref` table, for that template. 2. After another `storage.cleanup.interval` seconds a `DestroyCommand` will be sent, to delete template from primary storage 3. On `VmwareResource`, command is processed, and it first picks up a random cluster, say `ci != c1` to look for vm template (using volume's path) and destroy it. But, as template was on `c1` it cannot be found, so it won't be deleted. Entry on `template_spool_ref` is deleted but not the actual template on hypervisor side. ### Proposed solution We propose a way to attack problem shown in point 3, by not picking up a random cluster to look for vm but using vSphere data center. This way we make sure vm template will be deleted in every case, and not depending on random cluster selection * pr/1560: CLOUDSTACK-9386: Find vm on datacenter instead of randomly choosing a cluster Signed-off-by: Koushik Das <koushik@apache.org>

rafaelweingartner · 2016-09-13T12:50:55Z

@jburwell someone else already merged ;)

jburwell · 2016-09-18T13:53:41Z

@serg38 @nvazquez I just noticed that this bug was merged to master, but it should also be included in LTS My thinking is that we use the following steps to move it to the 4.9 branch and pull it forward:

Rollback this commit from master
Re-submit this PR against the 4.9 release branch
Test the new PR on 4.9
Merge to new PR to the 4.9 branch
Forward merge this PR to master

Does this make sense? If so, @serg38 would you be able to quickly re-test it against 4.9? If we make this change, I would like it to be included in the upcoming 4.9.1.0 and 4.10.0.0 releases.

nvazquez · 2016-09-19T13:58:40Z

Thanks @jburwell, I've opened new PR (1676) against 4.9 release branch

CLOUDSTACK-9502: DS template copies dont get deleted in VMware ESXi with multiple clusters and zone wide storage (include CLOUDSTACK-9386 into 4.9 release branch)Include #1560 into 4.9 release branch * pr/1676: CLOUDSTACK-9502: DS template copies don’t get deleted in VMware ESXi with multiple clusters and zone wide storage Signed-off-by: John Burwell <meaux@cockamamy.net>

Correção de erro de _log_ ao realizar _backup_ Closesapache#1560 See merge request scclouds/scclouds!1168

nvazquez force-pushed the gcbug branch from 656bba0 to 3c98746Compare May 25, 2016 15:58

rafaelweingartner reviewed May 25, 2016
View reviewed changes

nvazquez force-pushed the gcbug branch from 3c98746 to fe9c52fCompare May 25, 2016 17:35

nvazquez changed the title ~~CLOUDSTACK-9368: DS template copies don’t get deleted in VMware ESXi with multiple clusters and zone wide storage~~CLOUDSTACK-9386: DS template copies don’t get deleted in VMware ESXi with multiple clusters and zone wide storageMay 26, 2016

nvazquez force-pushed the gcbug branch from fe9c52f to e850546Compare May 30, 2016 15:26

nvazquez force-pushed the gcbug branch from e850546 to 66c1c19Compare June 20, 2016 18:55

nvazquez force-pushed the gcbug branch from 66c1c19 to 07a8893Compare August 4, 2016 16:55

nvazquez force-pushed the gcbug branch from 07a8893 to 6224f16Compare August 4, 2016 19:16

nvazquez force-pushed the gcbug branch from 6224f16 to e8c74e7Compare September 1, 2016 17:12

nvazquez force-pushed the gcbug branch from e8c74e7 to 6bdacbeCompare September 1, 2016 18:17

nvazquez force-pushed the gcbug branch from 6bdacbe to 5b54960Compare September 11, 2016 22:34

CLOUDSTACK-9386: Find vm on datacenter instead of randomly choosing a…
1384d74
… cluster

nvazquez force-pushed the gcbug branch from 5b54960 to 1384d74Compare September 11, 2016 23:56

asfgit merged commit 1384d74 into apache:masterSep 13, 2016

nvazquez mentioned this pull request Sep 19, 2016
CLOUDSTACK-9502: DS template copies don’t get deleted in VMware ESXi with multiple clusters and zone wide storage (include CLOUDSTACK-9386 into 4.9 release branch) #1676
Merged

bernardodemarco pushed a commit to scclouds/cloudstack that referenced this pull request Jul 16, 2025
Merge branch '1560-Corrigir-log' into '4.20.0.0-scclouds'
6e180b4
Correção de erro de _log_ ao realizar _backup_ Closesapache#1560 See merge request scclouds/scclouds!1168

Uh oh!

Conversation

nvazquez commented May 24, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Introduction

Description of the problem

Proposed solution

Uh oh!

rafaelweingartner commented May 24, 2016

Uh oh!

nvazquez commented May 25, 2016

Uh oh!

rafaelweingartner commented May 25, 2016

Uh oh!

rafaelweingartnerMay 25, 2016

Choose a reason for hiding this comment

Uh oh!

nvazquezMay 25, 2016

Choose a reason for hiding this comment

Uh oh!

bvbharatk commented Jun 11, 2016

ACS CI BVT Run

Uh oh!

nvazquez commented Aug 4, 2016

Uh oh!

serg38 commented Aug 16, 2016

Uh oh!

rafaelweingartner commented Aug 23, 2016

Uh oh!

serg38 commented Aug 29, 2016

Uh oh!

yadvr commented Aug 30, 2016

Uh oh!

nvazquez commented Sep 1, 2016

Uh oh!

nvazquez commented Sep 2, 2016

Uh oh!

serg38 commented Sep 11, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serg38 commented Sep 11, 2016

Uh oh!

jburwell commented Sep 11, 2016

Uh oh!

nvazquez commented Sep 11, 2016

Uh oh!

serg38 commented Sep 12, 2016

1. Register a template for VMware with nicAdapter vmxnet3 ... SKIP: VCenter API Integration Remaining

Uh oh!

rafaelweingartner commented Sep 12, 2016

Uh oh!

jburwell commented Sep 13, 2016

Uh oh!

rafaelweingartner commented Sep 13, 2016

Uh oh!

jburwell commented Sep 18, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nvazquez commented Sep 19, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

nvazquez commented May 24, 2016•
edited
Loading

serg38 commented Sep 11, 2016•
edited
Loading

jburwell commented Sep 18, 2016•
edited
Loading