Skip to content

CLOUDSTACK-8415 [VMware] SSVM shutdown during snapshot operation results in disks to be left behind - #2090

Closed
sureshanaparti wants to merge 1 commit into
masterfrom
unknown repository
Closed

CLOUDSTACK-8415 [VMware] SSVM shutdown during snapshot operation results in disks to be left behind#2090
sureshanaparti wants to merge 1 commit into
masterfrom
unknown repository

Conversation

@sureshanaparti

@sureshanapartisureshanaparti commented May 9, 2017

Copy link
Copy Markdown
Contributor

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)

In VMware environment, SSVM shutdown during snapshot operation leaves behind partial, unusable disks in secondary storage and VM snapshots.

Fix: Enhance storage garbage collector to delete the partial disks created in secondary storage and VM snapshots while expunging 'ERROR' snapshot.

This closes#540

@sureshanaparti

Copy link
Copy Markdown
ContributorAuthor

@DaanHoogland@wilderrodrigues Please review the changes.

@sureshanaparti

Copy link
Copy Markdown
ContributorAuthor

@borisstoyanov Can you please kick off VMware trillian tests

@borisstoyanov

Copy link
Copy Markdown
Contributor

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-705

@borisstoyanov

Copy link
Copy Markdown
Contributor

@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@borisstoyanov

Copy link
Copy Markdown
Contributor

@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@DaanHooglandDaanHoogland left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the functionality is good but shouldn't be in here. It is vmware specific and we should move away from closely tighing storage to hypervisors.

for (SnapshotVO snapshotVO : snapshots) {
try {
List<SnapshotDataStoreVO> storeRefs = _snapshotStoreDao.findBySnapshotId(snapshotVO.getId());
boolean isVMware = snapshotVO.getHypervisorType().equals(HypervisorType.VMware);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please try to move this logic into the vmware plugin. this code should remain generic.
opinions @mike-tutkowski@swill @rhtyd ?

@mike-tutkowskimike-tutkowskiMay 15, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I believe your comment makes sense, @DaanHoogland, it appears this file has hypervisor-specific logic sprinkled all around it.

Perhaps it would be better to allow this here and for you to write up a bug noting that it would be ideal for this file to be hypervisor agnostic?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DaanHoogland@mike-tutkowski Partial disks are left behind in VMware env when SSVM is shutdown/destroyed. So, DeleteSnapshotsDirCommand for Image store and DeleteVMSnapshotCommand for Primary store (to remove worker VM snapshot) has been triggered in VMware env here to cleanup the partial disks created in secondary storage and VM snapshots in primary respectively. The VMware storage manager would cleanup the worker VM snapshot from DeleteVMSnapshotCommand. No VMware specific code here, except sending these commands for cleanup.

}
}
if (removeSnapshot) {
_snapshotDao.expunge(snapshotVO.getId());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this method is already to long.
please factor out the new part into a seperate method.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DaanHoogland moved to a private method removeSnapshot()

@blueorangutan

Copy link
Copy Markdown

Trillian test result (tid-1075)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 39573 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2090-t1075-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Test completed. 49 look ok, 2 have error(s)

TestResultTime (s)Test File
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACLFailure435.10test_vpc_redundant.py
test_04_rvpc_privategw_static_routesFailure421.56test_privategw_acl.py
test_01_vpc_site2site_vpnSuccess215.62test_vpc_vpn.py
test_01_vpc_remote_access_vpnSuccess96.39test_vpc_vpn.py
test_01_redundant_vpc_site2site_vpnSuccess301.30test_vpc_vpn.py
test_02_VPC_default_routesSuccess326.88test_vpc_router_nics.py
test_01_VPC_nics_after_destroySuccess680.39test_vpc_router_nics.py
test_05_rvpc_multi_tiersSuccess558.21test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nicsSuccess1493.79test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routersSuccess656.46test_vpc_redundant.py
test_02_redundant_VPC_default_routesSuccess903.85test_vpc_redundant.py
test_09_delete_detached_volumeSuccess156.78test_volumes.py
test_08_resize_volumeSuccess156.56test_volumes.py
test_07_resize_failSuccess161.62test_volumes.py
test_06_download_detached_volumeSuccess156.46test_volumes.py
test_05_detach_volumeSuccess150.82test_volumes.py
test_04_delete_attached_volumeSuccess151.32test_volumes.py
test_03_download_attached_volumeSuccess156.47test_volumes.py
test_02_attach_volumeSuccess90.92test_volumes.py
test_01_create_volumeSuccess742.38test_volumes.py
test_03_delete_vm_snapshotsSuccess275.23test_vm_snapshots.py
test_02_revert_vm_snapshotsSuccess105.85test_vm_snapshots.py
test_01_create_vm_snapshotsSuccess169.13test_vm_snapshots.py
test_deploy_vm_multipleSuccess368.62test_vm_life_cycle.py
test_deploy_vmSuccess0.03test_vm_life_cycle.py
test_advZoneVirtualRouterSuccess0.03test_vm_life_cycle.py
test_10_attachAndDetach_isoSuccess26.93test_vm_life_cycle.py
test_09_expunge_vmSuccess125.27test_vm_life_cycle.py
test_08_migrate_vmSuccess30.97test_vm_life_cycle.py
test_07_restore_vmSuccess0.14test_vm_life_cycle.py
test_06_destroy_vmSuccess125.89test_vm_life_cycle.py
test_03_reboot_vmSuccess125.96test_vm_life_cycle.py
test_02_start_vmSuccess10.20test_vm_life_cycle.py
test_01_stop_vmSuccess35.37test_vm_life_cycle.py
test_CreateTemplateWithDuplicateNameSuccess161.47test_templates.py
test_08_list_system_templatesSuccess0.03test_templates.py
test_07_list_public_templatesSuccess0.04test_templates.py
test_05_template_permissionsSuccess0.06test_templates.py
test_04_extract_templateSuccess5.24test_templates.py
test_03_delete_templateSuccess5.11test_templates.py
test_02_edit_templateSuccess90.15test_templates.py
test_01_create_templateSuccess110.95test_templates.py
test_10_destroy_cpvmSuccess197.34test_ssvm.py
test_09_destroy_ssvmSuccess193.91test_ssvm.py
test_08_reboot_cpvmSuccess132.06test_ssvm.py
test_07_reboot_ssvmSuccess134.63test_ssvm.py
test_06_stop_cpvmSuccess162.21test_ssvm.py
test_05_stop_ssvmSuccess163.98test_ssvm.py
test_04_cpvm_internalsSuccess1.33test_ssvm.py
test_03_ssvm_internalsSuccess4.83test_ssvm.py
test_02_list_cpvm_vmSuccess0.16test_ssvm.py
test_01_list_sec_storage_vmSuccess0.15test_ssvm.py
test_02_list_snapshots_with_removed_data_storeSuccess203.14test_snapshots.py
test_01_snapshot_root_diskSuccess16.42test_snapshots.py
test_04_change_offering_smallSuccess240.82test_service_offerings.py
test_03_delete_service_offeringSuccess0.04test_service_offerings.py
test_02_edit_service_offeringSuccess0.08test_service_offerings.py
test_01_create_service_offeringSuccess0.13test_service_offerings.py
test_02_sys_template_readySuccess0.15test_secondary_storage.py
test_01_sys_vm_startSuccess0.21test_secondary_storage.py
test_09_reboot_routerSuccess45.38test_routers.py
test_08_start_routerSuccess35.34test_routers.py
test_07_stop_routerSuccess10.17test_routers.py
test_06_router_advancedSuccess0.06test_routers.py
test_05_router_basicSuccess0.05test_routers.py
test_04_restart_network_wo_cleanupSuccess5.79test_routers.py
test_03_restart_network_cleanupSuccess95.80test_routers.py
test_02_router_internal_advSuccess1.14test_routers.py
test_01_router_internal_basicSuccess0.59test_routers.py
test_router_dns_guestipquerySuccess80.15test_router_dns.py
test_router_dns_externalipquerySuccess0.11test_router_dns.py
test_router_dhcphostsSuccess344.34test_router_dhcphosts.py
test_router_dhcp_optsSuccess22.34test_router_dhcphosts.py
test_01_updatevolumedetailSuccess0.12test_resource_detail.py
test_01_reset_vm_on_rebootSuccess130.95test_reset_vm_on_reboot.py
test_createRegionSuccess0.05test_regions.py
test_create_pvlan_networkSuccess5.24test_pvlan.py
test_dedicatePublicIpRangeSuccess0.57test_public_ip_range.py
test_03_vpc_privategw_restart_vpc_cleanupSuccess601.40test_privategw_acl.py
test_02_vpc_privategw_static_routesSuccess446.13test_privategw_acl.py
test_01_vpc_privategw_aclSuccess117.83test_privategw_acl.py
test_03_migration_options_storage_tagsSuccess161.68test_primary_storage.py
test_02_edit_primary_storage_tagsSuccess0.10test_primary_storage.py
test_01_primary_storage_nfsSuccess35.91test_primary_storage.py
test_01_deploy_vms_storage_tagsSuccess31.24test_primary_storage.py
test_createPortablePublicIPRangeSuccess15.23test_portable_publicip.py
test_createPortablePublicIPAcquireSuccess15.67test_portable_publicip.py
test_isolate_network_password_serverSuccess92.03test_password_server.py
test_UpdateStorageOverProvisioningFactorSuccess0.16test_over_provisioning.py
test_oobm_zchange_passwordSuccess30.82test_outofbandmanagement.py
test_oobm_multiple_mgmt_server_ownershipSuccess16.49test_outofbandmanagement.py
test_oobm_issue_power_statusSuccess10.45test_outofbandmanagement.py
test_oobm_issue_power_softSuccess15.39test_outofbandmanagement.py
test_oobm_issue_power_resetSuccess10.37test_outofbandmanagement.py
test_oobm_issue_power_onSuccess10.44test_outofbandmanagement.py
test_oobm_issue_power_offSuccess15.37test_outofbandmanagement.py
test_oobm_issue_power_cycleSuccess10.36test_outofbandmanagement.py
test_oobm_enabledisable_across_clusterzonesSuccess87.74test_outofbandmanagement.py
test_oobm_enable_feature_validSuccess0.16test_outofbandmanagement.py
test_oobm_enable_feature_invalidSuccess5.15test_outofbandmanagement.py
test_oobm_disable_feature_validSuccess5.21test_outofbandmanagement.py
test_oobm_disable_feature_invalidSuccess0.12test_outofbandmanagement.py
test_oobm_configure_invalid_driverSuccess0.09test_outofbandmanagement.py
test_oobm_configure_default_driverSuccess0.10test_outofbandmanagement.py
test_oobm_background_powerstate_syncSuccess23.46test_outofbandmanagement.py
test_extendPhysicalNetworkVlanSuccess15.39test_non_contigiousvlan.py
test_01_nicSuccess459.79test_nic.py
test_releaseIPSuccess318.47test_network.py
test_reboot_routerSuccess459.06test_network.py
test_public_ip_user_accountSuccess10.25test_network.py
test_public_ip_admin_accountSuccess40.28test_network.py
test_network_rules_acquired_public_ip_3_Load_Balancer_RuleSuccess66.98test_network.py
test_network_rules_acquired_public_ip_2_nat_ruleSuccess62.03test_network.py
test_network_rules_acquired_public_ip_1_static_nat_ruleSuccess123.88test_network.py
test_delete_accountSuccess328.19test_network.py
test_02_port_fwd_on_non_src_natSuccess55.67test_network.py
test_01_port_fwd_on_src_natSuccess111.83test_network.py
test_nic_secondaryip_add_removeSuccess257.87test_multipleips_per_nic.py
test_list_zones_metricsSuccess0.34test_metrics_api.py
test_list_volumes_metricsSuccess5.55test_metrics_api.py
test_list_vms_metricsSuccess191.77test_metrics_api.py
test_list_pstorage_metricsSuccess0.33test_metrics_api.py
test_list_infrastructure_metricsSuccess0.48test_metrics_api.py
test_list_hosts_metricsSuccess0.40test_metrics_api.py
test_list_clusters_metricsSuccess0.35test_metrics_api.py
login_test_saml_userSuccess19.30test_login.py
test_assign_and_removal_lbSuccess145.90test_loadbalance.py
test_02_create_lb_rule_non_natSuccess187.79test_loadbalance.py
test_01_create_lb_rule_src_natSuccess218.63test_loadbalance.py
test_03_list_snapshotsSuccess0.09test_list_ids_parameter.py
test_02_list_templatesSuccess0.04test_list_ids_parameter.py
test_01_list_volumesSuccess0.03test_list_ids_parameter.py
test_07_list_default_isoSuccess0.06test_iso.py
test_05_iso_permissionsSuccess0.08test_iso.py
test_04_extract_IsoSuccess5.16test_iso.py
test_03_delete_isoSuccess95.20test_iso.py
test_02_edit_isoSuccess0.10test_iso.py
test_01_create_isoSuccess21.02test_iso.py
test_04_rvpc_internallb_haproxy_stats_on_all_interfacesSuccess209.04test_internal_lb.py
test_03_vpc_internallb_haproxy_stats_on_all_interfacesSuccess172.84test_internal_lb.py
test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80Success510.36test_internal_lb.py
test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80Success480.96test_internal_lb.py
test_dedicateGuestVlanRangeSuccess10.51test_guest_vlan_range.py
test_UpdateConfigParamWithScopeSuccess0.15test_global_settings.py
test_rolepermission_lifecycle_updateSuccess6.17test_dynamicroles.py
test_rolepermission_lifecycle_listSuccess6.00test_dynamicroles.py
test_rolepermission_lifecycle_deleteSuccess5.88test_dynamicroles.py
test_rolepermission_lifecycle_createSuccess5.93test_dynamicroles.py
test_rolepermission_lifecycle_concurrent_updatesSuccess6.03test_dynamicroles.py
test_role_lifecycle_update_role_inuseSuccess5.92test_dynamicroles.py
test_role_lifecycle_updateSuccess5.99test_dynamicroles.py
test_role_lifecycle_listSuccess5.90test_dynamicroles.py
test_role_lifecycle_deleteSuccess6.01test_dynamicroles.py
test_role_lifecycle_createSuccess5.94test_dynamicroles.py
test_role_inuse_deletionSuccess5.94test_dynamicroles.py
test_role_account_acls_multiple_mgmt_serversSuccess8.18test_dynamicroles.py
test_role_account_aclsSuccess8.39test_dynamicroles.py
test_default_role_deletionSuccess6.11test_dynamicroles.py
test_04_create_fat_type_disk_offeringSuccess0.10test_disk_offerings.py
test_03_delete_disk_offeringSuccess0.04test_disk_offerings.py
test_02_edit_disk_offeringSuccess0.05test_disk_offerings.py
test_02_create_sparse_type_disk_offeringSuccess0.08test_disk_offerings.py
test_01_create_disk_offeringSuccess0.12test_disk_offerings.py
test_deployvm_userdispersingSuccess25.79test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_userconcentratedSuccess60.90test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_firstfitSuccess60.74test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_userdata_postSuccess10.40test_deploy_vm_with_userdata.py
test_deployvm_userdataSuccess80.95test_deploy_vm_with_userdata.py
test_02_deploy_vm_root_resizeSuccess0.09test_deploy_vm_root_resize.py
test_01_deploy_vm_root_resizeSuccess0.07test_deploy_vm_root_resize.py
test_00_deploy_vm_root_resizeSuccess247.02test_deploy_vm_root_resize.py
test_deploy_vm_from_isoSuccess212.48test_deploy_vm_iso.py
test_06_verify_guest_lspci_againSuccess40.45test_deploy_virtio_scsi_vm.py
test_05_change_vm_ostype_restartSuccess10.95test_deploy_virtio_scsi_vm.py
test_04_verify_guest_lspciSuccess40.51test_deploy_virtio_scsi_vm.py
test_03_verify_libvirt_attach_diskSuccess10.82test_deploy_virtio_scsi_vm.py
test_02_verify_libvirt_after_restartSuccess131.60test_deploy_virtio_scsi_vm.py
test_01_verify_libvirtSuccess0.56test_deploy_virtio_scsi_vm.py
test_DeployVmAntiAffinityGroupSuccess65.98test_affinity_groups.py
test_change_service_offering_for_vm_with_snapshotsSkipped0.00test_vm_snapshots.py
test_06_copy_templateSkipped0.00test_templates.py
test_static_role_account_aclsSkipped0.03test_staticroles.py
test_11_ss_nfs_version_on_ssvmSkipped0.02test_ssvm.py
test_01_scale_vmSkipped0.00test_scale_vm.py
test_01_primary_storage_iscsiSkipped0.07test_primary_storage.py
test_nested_virtualization_vmwareSkipped0.00test_nested_virtualization.py
test_06_copy_isoSkipped0.00test_iso.py
test_deploy_vgpu_enabled_vmSkipped0.04test_deploy_vgpu_enabled_vm.py
test_3d_gpu_supportSkipped0.04test_deploy_vgpu_enabled_vm.py

@sureshanaparti

Copy link
Copy Markdown
ContributorAuthor

@DaanHoogland Can you check the changes.

@sureshanaparti

Copy link
Copy Markdown
ContributorAuthor

@DaanHoogland Can you check and confirm the changes.

@cloudmonger

Copy link
Copy Markdown

ACS CI BVT Run

Sumarry:
Build Number 841
Hypervisor xenserver
NetworkType Advanced
Passed=107
Failed=6
Skipped=12

Link to logs Folder (search by build_no):https://www.dropbox.com/sh/r2si930m8xxzavs/AAAzNrnoF1fC3auFrvsKo_8-a?dl=0

Failed tests:

  • test_deploy_vm_iso.py

  • test_deploy_vm_from_iso Failed

  • test_routers_network_ops.py

  • test_01_isolate_network_FW_PF_default_routes_egress_true Failing since 29 runs

  • test_02_isolate_network_FW_PF_default_routes_egress_false Failing since 29 runs

  • test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true Failing since 29 runs

  • test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false Failing since 29 runs

  • test_03_RVR_Network_check_router_state Failed

Skipped tests:
test_vm_nic_adapter_vmxnet3
test_01_verify_libvirt
test_02_verify_libvirt_after_restart
test_03_verify_libvirt_attach_disk
test_04_verify_guest_lspci
test_05_change_vm_ostype_restart
test_06_verify_guest_lspci_again
test_static_role_account_acls
test_11_ss_nfs_version_on_ssvm
test_nested_virtualization_vmware
test_3d_gpu_support
test_deploy_vgpu_enabled_vm

Passed test suits:
test_deploy_vm_with_userdata.py
test_affinity_groups_projects.py
test_portable_publicip.py
test_vm_snapshots.py
test_over_provisioning.py
test_global_settings.py
test_scale_vm.py
test_service_offerings.py
test_routers_iptables_default_policy.py
test_loadbalance.py
test_routers.py
test_reset_vm_on_reboot.py
test_deploy_vms_with_varied_deploymentplanners.py
test_network.py
test_router_dns.py
test_non_contigiousvlan.py
test_login.py
test_list_ids_parameter.py
test_public_ip_range.py
test_multipleips_per_nic.py
test_metrics_api.py
test_regions.py
test_affinity_groups.py
test_network_acl.py
test_pvlan.py
test_volumes.py
test_nic.py
test_deploy_vm_root_resize.py
test_resource_detail.py
test_secondary_storage.py
test_vm_life_cycle.py
test_disk_offerings.py

@sureshanaparti

Copy link
Copy Markdown
ContributorAuthor

ping @DaanHoogland

@yadvryadvr added this to the 4.11 milestone Dec 10, 2017
@yadvryadvr removed this from the 4.11 milestone Jan 4, 2018
@yadvr

Copy link
Copy Markdown
Member

ping @sureshanaparti can you fix the conflicts?

@DaanHoogland

Copy link
Copy Markdown
Contributor

@sureshanaparti sorry for totally missing this. Yes this looks like what I asked for. please rebase as @rhtyd asked so we can test and merge.

@DaanHoogland

Copy link
Copy Markdown
Contributor

@rhtyd @PaulAngus@andrijapanicsb is this still relevant?

@Humbedooh
Humbedooh deleted the branch apache:masterJune 11, 2021 09:50
@yadvr

Copy link
Copy Markdown
Member

Hi, the PR got auto-closed due to no known source repository or some other Github-specific issue during the recent renaming of the default branch to main. If you wish kindly submit a new PR.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

11 participants

@sureshanaparti@borisstoyanov@blueorangutan@cloudmonger@yadvr@DaanHoogland@mike-tutkowski@resmo@Humbedooh@PaulAngus@ACSGitBot