CLOUDSTACK-8308 automation test for Delta Snapshot

[DaanHoogland]

#384 and #720 tried to submit this.

[DaanHoogland]

@blueorangutan package

[blueorangutan]

@DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-698

[DaanHoogland]

@blueorangutan test

[blueorangutan]

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[blueorangutan]

Trillian test result (tid-1059)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 33558 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2061-t1059-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Test completed. 50 look ok, 1 have error(s)

Test	Result	Time (s)	Test File
test_04_rvpc_privategw_static_routes	Failure	350.76	test_privategw_acl.py
test_01_vpc_site2site_vpn	Success	150.11	test_vpc_vpn.py
test_01_vpc_remote_access_vpn	Success	56.11	test_vpc_vpn.py
test_01_redundant_vpc_site2site_vpn	Success	230.55	test_vpc_vpn.py
test_02_VPC_default_routes	Success	259.02	test_vpc_router_nics.py
test_01_VPC_nics_after_destroy	Success	505.74	test_vpc_router_nics.py
test_05_rvpc_multi_tiers	Success	520.80	test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics	Success	1267.49	test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	Success	537.90	test_vpc_redundant.py
test_02_redundant_VPC_default_routes	Success	738.98	test_vpc_redundant.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	Success	1279.49	test_vpc_redundant.py
test_09_delete_detached_volume	Success	156.59	test_volumes.py
test_08_resize_volume	Success	151.43	test_volumes.py
test_07_resize_fail	Success	156.43	test_volumes.py
test_06_download_detached_volume	Success	156.33	test_volumes.py
test_05_detach_volume	Success	145.80	test_volumes.py
test_04_delete_attached_volume	Success	151.63	test_volumes.py
test_03_download_attached_volume	Success	156.30	test_volumes.py
test_02_attach_volume	Success	79.04	test_volumes.py
test_01_create_volume	Success	622.58	test_volumes.py
test_03_delete_vm_snapshots	Success	275.19	test_vm_snapshots.py
test_02_revert_vm_snapshots	Success	100.63	test_vm_snapshots.py
test_01_create_vm_snapshots	Success	133.62	test_vm_snapshots.py
test_deploy_vm_multiple	Success	252.64	test_vm_life_cycle.py
test_deploy_vm	Success	0.03	test_vm_life_cycle.py
test_advZoneVirtualRouter	Success	0.03	test_vm_life_cycle.py
test_10_attachAndDetach_iso	Success	26.66	test_vm_life_cycle.py
test_09_expunge_vm	Success	125.22	test_vm_life_cycle.py
test_08_migrate_vm	Success	30.86	test_vm_life_cycle.py
test_07_restore_vm	Success	0.12	test_vm_life_cycle.py
test_06_destroy_vm	Success	125.83	test_vm_life_cycle.py
test_03_reboot_vm	Success	125.85	test_vm_life_cycle.py
test_02_start_vm	Success	10.17	test_vm_life_cycle.py
test_01_stop_vm	Success	35.31	test_vm_life_cycle.py
test_CreateTemplateWithDuplicateName	Success	70.68	test_templates.py
test_08_list_system_templates	Success	0.03	test_templates.py
test_07_list_public_templates	Success	0.04	test_templates.py
test_05_template_permissions	Success	0.05	test_templates.py
test_04_extract_template	Success	5.13	test_templates.py
test_03_delete_template	Success	5.11	test_templates.py
test_02_edit_template	Success	90.11	test_templates.py
test_01_create_template	Success	45.46	test_templates.py
test_10_destroy_cpvm	Success	131.33	test_ssvm.py
test_09_destroy_ssvm	Success	163.19	test_ssvm.py
test_08_reboot_cpvm	Success	131.54	test_ssvm.py
test_07_reboot_ssvm	Success	103.06	test_ssvm.py
test_06_stop_cpvm	Success	161.78	test_ssvm.py
test_05_stop_ssvm	Success	133.15	test_ssvm.py
test_04_cpvm_internals	Success	1.20	test_ssvm.py
test_03_ssvm_internals	Success	2.82	test_ssvm.py
test_02_list_cpvm_vm	Success	0.12	test_ssvm.py
test_01_list_sec_storage_vm	Success	0.13	test_ssvm.py
test_02_list_snapshots_with_removed_data_store	Success	86.74	test_snapshots.py
test_01_snapshot_root_disk	Success	11.29	test_snapshots.py
test_04_change_offering_small	Success	209.56	test_service_offerings.py
test_03_delete_service_offering	Success	0.04	test_service_offerings.py
test_02_edit_service_offering	Success	0.08	test_service_offerings.py
test_01_create_service_offering	Success	0.11	test_service_offerings.py
test_02_sys_template_ready	Success	0.13	test_secondary_storage.py
test_01_sys_vm_start	Success	0.18	test_secondary_storage.py
test_09_reboot_router	Success	40.34	test_routers.py
test_08_start_router	Success	30.28	test_routers.py
test_07_stop_router	Success	10.17	test_routers.py
test_06_router_advanced	Success	0.06	test_routers.py
test_05_router_basic	Success	0.04	test_routers.py
test_04_restart_network_wo_cleanup	Success	5.70	test_routers.py
test_03_restart_network_cleanup	Success	60.51	test_routers.py
test_02_router_internal_adv	Success	1.04	test_routers.py
test_01_router_internal_basic	Success	0.57	test_routers.py
test_router_dns_guestipquery	Success	76.77	test_router_dns.py
test_router_dns_externalipquery	Success	0.07	test_router_dns.py
test_router_dhcphosts	Success	237.20	test_router_dhcphosts.py
test_router_dhcp_opts	Success	21.66	test_router_dhcphosts.py
test_01_updatevolumedetail	Success	0.10	test_resource_detail.py
test_01_reset_vm_on_reboot	Success	130.89	test_reset_vm_on_reboot.py
test_createRegion	Success	0.04	test_regions.py
test_create_pvlan_network	Success	5.22	test_pvlan.py
test_dedicatePublicIpRange	Success	0.43	test_public_ip_range.py
test_03_vpc_privategw_restart_vpc_cleanup	Success	465.06	test_privategw_acl.py
test_02_vpc_privategw_static_routes	Success	370.26	test_privategw_acl.py
test_01_vpc_privategw_acl	Success	92.30	test_privategw_acl.py
test_03_migration_options_storage_tags	Success	161.68	test_primary_storage.py
test_02_edit_primary_storage_tags	Success	0.10	test_primary_storage.py
test_01_primary_storage_nfs	Success	35.77	test_primary_storage.py
test_01_deploy_vms_storage_tags	Success	31.26	test_primary_storage.py
test_createPortablePublicIPRange	Success	15.19	test_portable_publicip.py
test_createPortablePublicIPAcquire	Success	15.50	test_portable_publicip.py
test_isolate_network_password_server	Success	89.18	test_password_server.py
test_UpdateStorageOverProvisioningFactor	Success	0.13	test_over_provisioning.py
test_oobm_zchange_password	Success	30.75	test_outofbandmanagement.py
test_oobm_multiple_mgmt_server_ownership	Success	16.42	test_outofbandmanagement.py
test_oobm_issue_power_status	Success	20.39	test_outofbandmanagement.py
test_oobm_issue_power_soft	Success	15.33	test_outofbandmanagement.py
test_oobm_issue_power_reset	Success	15.36	test_outofbandmanagement.py
test_oobm_issue_power_on	Success	15.36	test_outofbandmanagement.py
test_oobm_issue_power_off	Success	15.33	test_outofbandmanagement.py
test_oobm_issue_power_cycle	Success	15.38	test_outofbandmanagement.py
test_oobm_enabledisable_across_clusterzones	Success	92.78	test_outofbandmanagement.py
test_oobm_enable_feature_valid	Success	5.17	test_outofbandmanagement.py
test_oobm_enable_feature_invalid	Success	0.11	test_outofbandmanagement.py
test_oobm_disable_feature_valid	Success	0.16	test_outofbandmanagement.py
test_oobm_disable_feature_invalid	Success	0.11	test_outofbandmanagement.py
test_oobm_configure_invalid_driver	Success	0.09	test_outofbandmanagement.py
test_oobm_configure_default_driver	Success	0.08	test_outofbandmanagement.py
test_oobm_background_powerstate_sync	Success	23.41	test_outofbandmanagement.py
test_extendPhysicalNetworkVlan	Success	15.32	test_non_contigiousvlan.py
test_01_nic	Success	429.31	test_nic.py
test_releaseIP	Success	187.54	test_network.py
test_reboot_router	Success	393.51	test_network.py
test_public_ip_user_account	Success	10.28	test_network.py
test_public_ip_admin_account	Success	40.27	test_network.py
test_network_rules_acquired_public_ip_3_Load_Balancer_Rule	Success	66.84	test_network.py
test_network_rules_acquired_public_ip_2_nat_rule	Success	61.85	test_network.py
test_network_rules_acquired_public_ip_1_static_nat_rule	Success	123.76	test_network.py
test_delete_account	Success	257.73	test_network.py
test_02_port_fwd_on_non_src_nat	Success	55.65	test_network.py
test_01_port_fwd_on_src_nat	Success	111.77	test_network.py
test_nic_secondaryip_add_remove	Success	222.78	test_multipleips_per_nic.py
test_list_zones_metrics	Success	0.33	test_metrics_api.py
test_list_volumes_metrics	Success	5.49	test_metrics_api.py
test_list_vms_metrics	Success	186.68	test_metrics_api.py
test_list_pstorage_metrics	Success	0.27	test_metrics_api.py
test_list_infrastructure_metrics	Success	0.47	test_metrics_api.py
test_list_hosts_metrics	Success	0.39	test_metrics_api.py
test_list_clusters_metrics	Success	0.36	test_metrics_api.py
login_test_saml_user	Success	19.28	test_login.py
test_assign_and_removal_lb	Success	133.44	test_loadbalance.py
test_02_create_lb_rule_non_nat	Success	187.18	test_loadbalance.py
test_01_create_lb_rule_src_nat	Success	207.64	test_loadbalance.py
test_03_list_snapshots	Success	0.09	test_list_ids_parameter.py
test_02_list_templates	Success	0.05	test_list_ids_parameter.py
test_01_list_volumes	Success	0.03	test_list_ids_parameter.py
test_07_list_default_iso	Success	0.07	test_iso.py
test_05_iso_permissions	Success	0.08	test_iso.py
test_04_extract_Iso	Success	5.15	test_iso.py
test_03_delete_iso	Success	95.21	test_iso.py
test_02_edit_iso	Success	0.08	test_iso.py
test_01_create_iso	Success	21.00	test_iso.py
test_04_rvpc_internallb_haproxy_stats_on_all_interfaces	Success	182.78	test_internal_lb.py
test_03_vpc_internallb_haproxy_stats_on_all_interfaces	Success	117.81	test_internal_lb.py
test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80	Success	511.11	test_internal_lb.py
test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80	Success	439.41	test_internal_lb.py
test_dedicateGuestVlanRange	Success	10.26	test_guest_vlan_range.py
test_UpdateConfigParamWithScope	Success	0.13	test_global_settings.py
test_rolepermission_lifecycle_update	Success	6.19	test_dynamicroles.py
test_rolepermission_lifecycle_list	Success	6.02	test_dynamicroles.py
test_rolepermission_lifecycle_delete	Success	5.87	test_dynamicroles.py
test_rolepermission_lifecycle_create	Success	5.96	test_dynamicroles.py
test_rolepermission_lifecycle_concurrent_updates	Success	6.04	test_dynamicroles.py
test_role_lifecycle_update_role_inuse	Success	5.92	test_dynamicroles.py
test_role_lifecycle_update	Success	11.01	test_dynamicroles.py
test_role_lifecycle_list	Success	5.90	test_dynamicroles.py
test_role_lifecycle_delete	Success	10.96	test_dynamicroles.py
test_role_lifecycle_create	Success	5.90	test_dynamicroles.py
test_role_inuse_deletion	Success	5.90	test_dynamicroles.py
test_role_account_acls_multiple_mgmt_servers	Success	8.22	test_dynamicroles.py
test_role_account_acls	Success	8.39	test_dynamicroles.py
test_default_role_deletion	Success	6.01	test_dynamicroles.py
test_04_create_fat_type_disk_offering	Success	0.07	test_disk_offerings.py
test_03_delete_disk_offering	Success	0.04	test_disk_offerings.py
test_02_edit_disk_offering	Success	0.05	test_disk_offerings.py
test_02_create_sparse_type_disk_offering	Success	0.08	test_disk_offerings.py
test_01_create_disk_offering	Success	0.11	test_disk_offerings.py
test_deployvm_userdispersing	Success	20.62	test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_userconcentrated	Success	20.58	test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_firstfit	Success	90.83	test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_userdata_post	Success	30.63	test_deploy_vm_with_userdata.py
test_deployvm_userdata	Success	45.75	test_deploy_vm_with_userdata.py
test_02_deploy_vm_root_resize	Success	6.02	test_deploy_vm_root_resize.py
test_01_deploy_vm_root_resize	Success	6.00	test_deploy_vm_root_resize.py
test_00_deploy_vm_root_resize	Success	222.56	test_deploy_vm_root_resize.py
test_deploy_vm_from_iso	Success	212.52	test_deploy_vm_iso.py
test_06_verify_guest_lspci_again	Success	40.48	test_deploy_virtio_scsi_vm.py
test_05_change_vm_ostype_restart	Success	11.05	test_deploy_virtio_scsi_vm.py
test_04_verify_guest_lspci	Success	40.52	test_deploy_virtio_scsi_vm.py
test_03_verify_libvirt_attach_disk	Success	10.75	test_deploy_virtio_scsi_vm.py
test_02_verify_libvirt_after_restart	Success	131.51	test_deploy_virtio_scsi_vm.py
test_01_verify_libvirt	Success	0.58	test_deploy_virtio_scsi_vm.py
test_DeployVmAntiAffinityGroup	Success	65.94	test_affinity_groups.py
test_change_service_offering_for_vm_with_snapshots	Skipped	0.00	test_vm_snapshots.py
test_06_copy_template	Skipped	0.00	test_templates.py
test_static_role_account_acls	Skipped	0.02	test_staticroles.py
test_11_ss_nfs_version_on_ssvm	Skipped	0.02	test_ssvm.py
test_01_scale_vm	Skipped	0.00	test_scale_vm.py
test_01_primary_storage_iscsi	Skipped	0.04	test_primary_storage.py
test_nested_virtualization_vmware	Skipped	0.00	test_nested_virtualization.py
test_06_copy_iso	Skipped	0.00	test_iso.py
test_deploy_vgpu_enabled_vm	Skipped	0.03	test_deploy_vgpu_enabled_vm.py
test_3d_gpu_support	Skipped	0.03	test_deploy_vgpu_enabled_vm.py

[DaanHoogland]

@karuturi this is only integration tests so I think I can be merged without hindering the release.
@borisstoyanov can you review (maybe @borisroman or @remibergsma too??)

[borisstoyanov]

LGTM @DaanHoogland, do you think we can move this to smoketests?

[DaanHoogland]

@borisstoyanov That would mean we want to run them on every PR as regression check. Is that your intention?

[borisstoyanov]

@DaanHoogland yes, what's your thoughts on that?

[cloudmonger]

ACS CI BVT Run

Sumarry:
Build Number 823
Hypervisor xenserver
NetworkType Advanced
Passed=105
Failed=6
Skipped=12

Link to logs Folder (search by build_no):https://www.dropbox.com/sh/r2si930m8xxzavs/AAAzNrnoF1fC3auFrvsKo_8-a?dl=0

Failed tests:

• test_list_ids_parameter.py

• ContextSuite context=TestListIdsParams>:setup Failing since 29 runs

• test_volumes.py

• test_06_download_detached_volume Failed

• test_routers_network_ops.py

• test_01_isolate_network_FW_PF_default_routes_egress_true Failing since 16 runs

• test_02_isolate_network_FW_PF_default_routes_egress_false Failing since 16 runs

• test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true Failing since 16 runs

• test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false Failing since 16 runs

Skipped tests:
test_vm_nic_adapter_vmxnet3
test_01_verify_libvirt
test_02_verify_libvirt_after_restart
test_03_verify_libvirt_attach_disk
test_04_verify_guest_lspci
test_05_change_vm_ostype_restart
test_06_verify_guest_lspci_again
test_static_role_account_acls
test_11_ss_nfs_version_on_ssvm
test_nested_virtualization_vmware
test_3d_gpu_support
test_deploy_vgpu_enabled_vm

Passed test suits:
test_deploy_vm_with_userdata.py
test_affinity_groups_projects.py
test_portable_publicip.py
test_vm_snapshots.py
test_over_provisioning.py
test_global_settings.py
test_scale_vm.py
test_service_offerings.py
test_routers_iptables_default_policy.py
test_loadbalance.py
test_routers.py
test_reset_vm_on_reboot.py
test_deploy_vms_with_varied_deploymentplanners.py
test_network.py
test_router_dns.py
test_non_contigiousvlan.py
test_login.py
test_deploy_vm_iso.py
test_public_ip_range.py
test_multipleips_per_nic.py
test_metrics_api.py
test_regions.py
test_affinity_groups.py
test_network_acl.py
test_pvlan.py
test_nic.py
test_deploy_vm_root_resize.py
test_resource_detail.py
test_secondary_storage.py
test_vm_life_cycle.py
test_disk_offerings.py

[yadvr]

@borisstoyanov@DaanHoogland any update on this, we can accept if we can show this test actually works

[DaanHoogland]

@borisstoyanov can you run with this?

[borisstoyanov]

yes, LGTM, it's OK if it stays in testpaths/

[DaanHoogland]

fine, regression tests on release will do. this is isolated enough from other functionality.
merge, @rhtyd @borisstoyanov ?