improvement: systemvm slow operations by romain-dartigues · Pull Request #1604 · apache/cloudstack

romain-dartigues · 2016-07-05T13:17:47Z

On our setup we had issues with vrouters when a large number of VM were created at once.

The vrouters try to arping the gateway (to refresh the ARP cache I suppose), but in some case it take too much time and the vrouters got killed for being unresponsive.

The following patches attempt to solve this by:

a96ab18, 8908371 defensive fix, maybe a bug?
fce10c5 do not block on arping command

This patch has been running for a month on our test environment and seems to solve the initial problem.

An operation (starting 220 VM on XenServer, shared storage) which took 46 mn, now take 4 mn (without errors).

syed · 2016-07-05T15:28:09Z

 dp['gateway'] = self.qFile.data['cmd_line']['localgw']
 else:
- dp['gateway'] = 'None'
+ dp['gateway'] = self.qFile.data['cmd_line'].get('gateway', 'None')


there could be a possibility of having an ipv6 gateway, in that case, we would be looking at ip6gateway.

syed · 2016-07-05T15:34:51Z

@romain-dartigues do we know why the arping is failing? Ideally, if the vif is configured correctly, the arping should go through properly. If the arping fails, do we still know that the connectivity is correct/establised?

romain-dartigues · 2016-07-05T20:40:23Z

@syed

there could be a possibility of having an ipv6 gateway

Right, but IPv6 does not use ARP but NDP (RFC 4861).

It seems that a ping6 -c1 ff02::1%eth0 (interface name is required) would have the same effect (to refresh the cache); so we could do it when no IPv4 GW is found (self reminder: ARP is L2, IP is L3).

do we know why the arping is failing?

Apparently because the file "/etc/cloudstack/ips.json" is generated with a gateway at "None" even when the gateway is present in the input "cmd_line.json" (but my memory could fail me at this time of the day).

One month ago we found the "problem" and wanted a fix; trying to understand why it did happen made me start #1575 (now #1603).

I though it would be good (at least for me) to get a better understanding of the code; doing hundreds of arping everytime is probably inefficient as each of them is supposed to refresh the ARP table of all equipment on the way...

syed · 2016-07-06T13:48:16Z

Thanks @romain-dartigues based on your comments the code LGTM

syed · 2016-07-06T13:49:13Z

we do have to remember that arping is ipv4 only so we need to fix this for ipv6 too

yadvr · 2016-08-06T08:51:07Z

@blueorangutan package

blueorangutan · 2016-08-06T08:52:04Z

@rhtyd a Trillian-Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2016-08-06T10:08:45Z

Packaging result: ✔centos6 ✔centos7 ✖debian repo: http://packages.shapeblue.com/cloudstack/pr/1604

jburwell · 2016-09-06T17:46:16Z

 cmd = "arping -c 1 -I %s -A -U -s %s %s" % (
 self.dev, self.address['public_ip'], self.address['gateway'])
- CsHelper.execute(cmd)
+ CsHelper.execute(cmd, wait=False)


Flag arguments are an anti-pattern. Please consider introducing and executeAndWait function instead.

yadvr · 2016-10-21T05:21:34Z

@romain-dartigues can you fix the merge conflict, squash the changes, open a JIRA issue referencing that in the commit message.
Pl. rebase against latest 4.9, also edit the PR's base branch against 4.9?

romain-dartigues · 2016-10-26T06:35:21Z

@rhtyd < JIRA issue #9568

Rebase done, change merge request target done.

yadvr · 2016-10-26T07:33:16Z

@murali-reddy can you help review this one?
@romain-dartigues thanks, can you also squash your commits to a single commit, and amend the git commit message to reflect the JIRA ID in the format: CLOUDSTACK-9569: <one line summary>.

yadvr · 2016-11-20T08:09:35Z

@romain-dartigues @murali-reddy ping

@blueorangutan package

blueorangutan · 2016-11-20T08:10:03Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2016-11-20T18:27:41Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-208

romain-dartigues · 2016-11-21T07:56:06Z

@rhtyd pong (is it OK like that? please advise)

yadvr · 2016-11-21T08:07:14Z

Thanks @romain-dartigues that looks good. I'll kick a test round soon.
@blueorangutan package

blueorangutan · 2016-11-21T08:08:03Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2016-11-21T08:47:15Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-227

yadvr · 2016-11-21T11:17:52Z

@blueorangutan test

blueorangutan · 2016-11-21T11:18:03Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2016-11-21T18:54:20Z

Trillian test result (tid-379)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 27371 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr1604-t379-kvm-centos7.zip
Test completed. 38 look ok, 9 have error(s)

Test	Result	Time (s)	Test File
test_02_VPC_default_routes	`Failure`	167.18	test_vpc_router_nics.py
test_02_redundant_VPC_default_routes	`Failure`	280.77	test_vpc_redundant.py
test_router_dhcphosts	`Failure`	86.53	test_router_dhcphosts.py
test_04_rvpc_privategw_static_routes	`Failure`	173.41	test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	82.54	test_privategw_acl.py
test_02_vpc_privategw_static_routes	`Failure`	87.52	test_privategw_acl.py
test_01_vpc_privategw_acl	`Failure`	52.19	test_privategw_acl.py
test_isolate_network_password_server	`Failure`	57.22	test_password_server.py
test_oobm_zchange_password	`Failure`	20.61	test_outofbandmanagement.py
test_01_vpc_remote_access_vpn	`Error`	0.11	test_vpc_vpn.py
ContextSuite context=TestRouterDHCPHosts>:teardown	`Error`	136.98	test_router_dhcphosts.py
test_extendPhysicalNetworkVlan	`Error`	0.04	test_non_contigiousvlan.py
test_network_rules_acquired_public_ip_3_Load_Balancer_Rule	`Error`	741.36	test_network.py
test_network_rules_acquired_public_ip_2_nat_rule	`Error`	735.36	test_network.py
test_network_rules_acquired_public_ip_1_static_nat_rule	`Error`	728.33	test_network.py
ContextSuite context=TestDedicateGuestVlanRange>:setup	`Error`	0.00	test_guest_vlan_range.py
test_01_vpc_site2site_vpn	Success	206.09	test_vpc_vpn.py
test_01_redundant_vpc_site2site_vpn	Success	271.57	test_vpc_vpn.py
test_01_VPC_nics_after_destroy	Success	511.57	test_vpc_router_nics.py
test_05_rvpc_multi_tiers	Success	482.12	test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics	Success	1515.49	test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	Success	563.78	test_vpc_redundant.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	Success	1082.83	test_vpc_redundant.py
test_09_delete_detached_volume	Success	16.02	test_volumes.py
test_08_resize_volume	Success	15.48	test_volumes.py
test_07_resize_fail	Success	20.61	test_volumes.py
test_06_download_detached_volume	Success	15.29	test_volumes.py
test_05_detach_volume	Success	100.25	test_volumes.py
test_04_delete_attached_volume	Success	10.24	test_volumes.py
test_03_download_attached_volume	Success	15.30	test_volumes.py
test_02_attach_volume	Success	44.87	test_volumes.py
test_01_create_volume	Success	617.40	test_volumes.py
test_deploy_vm_multiple	Success	263.73	test_vm_life_cycle.py
test_deploy_vm	Success	0.03	test_vm_life_cycle.py
test_advZoneVirtualRouter	Success	0.02	test_vm_life_cycle.py
test_10_attachAndDetach_iso	Success	71.71	test_vm_life_cycle.py
test_09_expunge_vm	Success	125.16	test_vm_life_cycle.py
test_08_migrate_vm	Success	36.06	test_vm_life_cycle.py
test_07_restore_vm	Success	0.12	test_vm_life_cycle.py
test_06_destroy_vm	Success	126.05	test_vm_life_cycle.py
test_03_reboot_vm	Success	125.93	test_vm_life_cycle.py
test_02_start_vm	Success	10.18	test_vm_life_cycle.py
test_01_stop_vm	Success	40.36	test_vm_life_cycle.py
test_CreateTemplateWithDuplicateName	Success	90.81	test_templates.py
test_08_list_system_templates	Success	0.04	test_templates.py
test_07_list_public_templates	Success	0.04	test_templates.py
test_05_template_permissions	Success	0.06	test_templates.py
test_04_extract_template	Success	5.13	test_templates.py
test_03_delete_template	Success	5.12	test_templates.py
test_02_edit_template	Success	90.11	test_templates.py
test_01_create_template	Success	70.64	test_templates.py
test_10_destroy_cpvm	Success	161.75	test_ssvm.py
test_09_destroy_ssvm	Success	133.75	test_ssvm.py
test_08_reboot_cpvm	Success	131.52	test_ssvm.py
test_07_reboot_ssvm	Success	103.11	test_ssvm.py
test_06_stop_cpvm	Success	131.74	test_ssvm.py
test_05_stop_ssvm	Success	133.24	test_ssvm.py
test_04_cpvm_internals	Success	1.31	test_ssvm.py
test_03_ssvm_internals	Success	3.06	test_ssvm.py
test_02_list_cpvm_vm	Success	0.11	test_ssvm.py
test_01_list_sec_storage_vm	Success	0.12	test_ssvm.py
test_01_snapshot_root_disk	Success	46.38	test_snapshots.py
test_04_change_offering_small	Success	234.95	test_service_offerings.py
test_03_delete_service_offering	Success	0.15	test_service_offerings.py
test_02_edit_service_offering	Success	0.14	test_service_offerings.py
test_01_create_service_offering	Success	0.16	test_service_offerings.py
test_02_sys_template_ready	Success	0.16	test_secondary_storage.py
test_01_sys_vm_start	Success	0.23	test_secondary_storage.py
test_09_reboot_router	Success	35.53	test_routers.py
test_08_start_router	Success	25.46	test_routers.py
test_07_stop_router	Success	10.27	test_routers.py
test_06_router_advanced	Success	0.14	test_routers.py
test_05_router_basic	Success	0.04	test_routers.py
test_04_restart_network_wo_cleanup	Success	5.74	test_routers.py
test_03_restart_network_cleanup	Success	60.92	test_routers.py
test_02_router_internal_adv	Success	1.20	test_routers.py
test_01_router_internal_basic	Success	0.66	test_routers.py
test_01_updatevolumedetail	Success	5.12	test_resource_detail.py
test_01_reset_vm_on_reboot	Success	166.35	test_reset_vm_on_reboot.py
test_createRegion	Success	0.04	test_regions.py
test_create_pvlan_network	Success	5.20	test_pvlan.py
test_dedicatePublicIpRange	Success	0.41	test_public_ip_range.py
test_01_primary_storage_nfs	Success	35.76	test_primary_storage.py
test_createPortablePublicIPRange	Success	10.27	test_portable_publicip.py
test_createPortablePublicIPAcquire	Success	15.46	test_portable_publicip.py
test_UpdateStorageOverProvisioningFactor	Success	0.14	test_over_provisioning.py
test_oobm_multiple_mgmt_server_ownership	Success	11.99	test_outofbandmanagement.py
test_oobm_issue_power_status	Success	10.54	test_outofbandmanagement.py
test_oobm_issue_power_soft	Success	15.52	test_outofbandmanagement.py
test_oobm_issue_power_reset	Success	16.02	test_outofbandmanagement.py
test_oobm_issue_power_on	Success	15.52	test_outofbandmanagement.py
test_oobm_issue_power_off	Success	15.50	test_outofbandmanagement.py
test_oobm_issue_power_cycle	Success	15.52	test_outofbandmanagement.py
test_oobm_enabledisable_across_clusterzones	Success	62.92	test_outofbandmanagement.py
test_oobm_enable_feature_valid	Success	5.20	test_outofbandmanagement.py
test_oobm_enable_feature_invalid	Success	5.24	test_outofbandmanagement.py
test_oobm_disable_feature_valid	Success	5.34	test_outofbandmanagement.py
test_oobm_disable_feature_invalid	Success	0.14	test_outofbandmanagement.py
test_oobm_configure_invalid_driver	Success	0.10	test_outofbandmanagement.py
test_oobm_configure_default_driver	Success	0.11	test_outofbandmanagement.py
test_oobm_background_powerstate_sync	Success	29.57	test_outofbandmanagement.py
test_01_nic	Success	551.40	test_nic.py
test_releaseIP	Success	249.11	test_network.py
test_reboot_router	Success	445.02	test_network.py
test_public_ip_user_account	Success	10.36	test_network.py
test_public_ip_admin_account	Success	40.40	test_network.py
test_delete_account	Success	329.24	test_network.py
test_02_port_fwd_on_non_src_nat	Success	55.65	test_network.py
test_01_port_fwd_on_src_nat	Success	111.72	test_network.py
test_nic_secondaryip_add_remove	Success	203.59	test_multipleips_per_nic.py
login_test_saml_user	Success	25.55	test_login.py
test_assign_and_removal_lb	Success	133.76	test_loadbalance.py
test_02_create_lb_rule_non_nat	Success	187.23	test_loadbalance.py
test_01_create_lb_rule_src_nat	Success	217.90	test_loadbalance.py
test_03_list_snapshots	Success	0.09	test_list_ids_parameter.py
test_02_list_templates	Success	0.04	test_list_ids_parameter.py
test_01_list_volumes	Success	0.03	test_list_ids_parameter.py
test_07_list_default_iso	Success	0.06	test_iso.py
test_05_iso_permissions	Success	0.06	test_iso.py
test_04_extract_Iso	Success	5.30	test_iso.py
test_03_delete_iso	Success	95.20	test_iso.py
test_02_edit_iso	Success	0.08	test_iso.py
test_01_create_iso	Success	67.66	test_iso.py
test_04_rvpc_internallb_haproxy_stats_on_all_interfaces	Success	243.40	test_internal_lb.py
test_03_vpc_internallb_haproxy_stats_on_all_interfaces	Success	193.55	test_internal_lb.py
test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80	Success	557.92	test_internal_lb.py
test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80	Success	432.08	test_internal_lb.py
test_UpdateConfigParamWithScope	Success	0.12	test_global_settings.py
test_rolepermission_lifecycle_update	Success	7.21	test_dynamicroles.py
test_rolepermission_lifecycle_list	Success	7.27	test_dynamicroles.py
test_rolepermission_lifecycle_delete	Success	7.03	test_dynamicroles.py
test_rolepermission_lifecycle_create	Success	6.94	test_dynamicroles.py
test_rolepermission_lifecycle_concurrent_updates	Success	7.08	test_dynamicroles.py
test_role_lifecycle_update_role_inuse	Success	7.33	test_dynamicroles.py
test_role_lifecycle_update	Success	12.05	test_dynamicroles.py
test_role_lifecycle_list	Success	7.34	test_dynamicroles.py
test_role_lifecycle_delete	Success	12.29	test_dynamicroles.py
test_role_lifecycle_create	Success	6.94	test_dynamicroles.py
test_role_inuse_deletion	Success	7.28	test_dynamicroles.py
test_role_account_acls_multiple_mgmt_servers	Success	9.21	test_dynamicroles.py
test_role_account_acls	Success	9.06	test_dynamicroles.py
test_default_role_deletion	Success	7.13	test_dynamicroles.py
test_04_create_fat_type_disk_offering	Success	0.10	test_disk_offerings.py
test_03_delete_disk_offering	Success	0.04	test_disk_offerings.py
test_02_edit_disk_offering	Success	0.05	test_disk_offerings.py
test_02_create_sparse_type_disk_offering	Success	0.06	test_disk_offerings.py
test_01_create_disk_offering	Success	0.10	test_disk_offerings.py
test_deployvm_userdispersing	Success	20.58	test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_userconcentrated	Success	76.42	test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_firstfit	Success	70.71	test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_userdata_post	Success	5.41	test_deploy_vm_with_userdata.py
test_deployvm_userdata	Success	45.66	test_deploy_vm_with_userdata.py
test_02_deploy_vm_root_resize	Success	6.88	test_deploy_vm_root_resize.py
test_01_deploy_vm_root_resize	Success	6.89	test_deploy_vm_root_resize.py
test_00_deploy_vm_root_resize	Success	284.27	test_deploy_vm_root_resize.py
test_deploy_vm_from_iso	Success	258.75	test_deploy_vm_iso.py
test_DeployVmAntiAffinityGroup	Success	76.33	test_affinity_groups.py
test_03_delete_vm_snapshots	Skipped	0.00	test_vm_snapshots.py
test_02_revert_vm_snapshots	Skipped	0.00	test_vm_snapshots.py
test_01_test_vm_volume_snapshot	Skipped	0.00	test_vm_snapshots.py
test_01_create_vm_snapshots	Skipped	0.00	test_vm_snapshots.py
test_06_copy_template	Skipped	0.00	test_templates.py
test_static_role_account_acls	Skipped	0.02	test_staticroles.py
test_01_scale_vm	Skipped	0.00	test_scale_vm.py
test_01_primary_storage_iscsi	Skipped	0.03	test_primary_storage.py
test_06_copy_iso	Skipped	0.00	test_iso.py
test_deploy_vgpu_enabled_vm	Skipped	0.00	test_deploy_vgpu_enabled_vm.py

romain-dartigues · 2016-11-22T07:55:16Z

I'll need help to understand what gone wrong.

I just peeked at pr1604-t379-kvm-centos7/test_vpc_router_nics_CC22OE/results.txt and:
TestVPCNics: DEBUG: SSH result: [u'PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.', u'64 bytes from 8.8.8.8: icmp_seq=1 ttl=52 time=15.6 ms', u'64 bytes from 8.8.8.8: icmp_seq=2 ttl=52 time=15.8 ms', u'64 bytes from 8.8.8.8: icmp_seq=3 ttl=52 time=15.7 ms', u'', u'--- 8.8.8.8 ping statistics ---', u'3 packets transmitted, 3 received, 0% packet loss, time 1999ms', u'rtt min/avg/max/mdev = 15.617/15.723/15.804/0.164 ms']; COUNT is ==> 0

But " 0% packet loss" is in the string.

yadvr · 2016-11-23T09:24:18Z

@romain-dartigues I'll have a look at the failure, meanwhile can you see my comments against the changes https://github.com/apache/cloudstack/pull/1604/files

romain-dartigues · 2016-11-23T12:58:26Z

@rhtyd < separate function done.

yadvr · 2016-11-23T13:03:32Z

okay @romain-dartigues I'll kick some tests
@blueorangutan package

blueorangutan · 2016-11-23T13:04:03Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2016-11-23T13:53:35Z

Packaging result: ✖centos6 ✖centos7 ✖debian. JID-238

yadvr · 2016-11-24T07:07:55Z

@blueorangutan package

blueorangutan · 2016-11-24T07:08:03Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2016-11-24T08:23:05Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-243

yadvr · 2016-11-24T08:38:37Z

@murali-reddy @abhinandanprateek can you help review this, thanks
@blueorangutan test

blueorangutan · 2016-11-24T08:39:05Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2016-11-24T18:22:35Z

Trillian test result (tid-425)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 35008 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr1604-t425-kvm-centos7.zip
Test completed. 34 look ok, 14 have error(s)

Test	Result	Time (s)	Test File
test_02_VPC_default_routes	`Failure`	767.42	test_vpc_router_nics.py
test_01_VPC_nics_after_destroy	`Failure`	822.53	test_vpc_router_nics.py
test_05_rvpc_multi_tiers	`Failure`	349.19	test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics	`Failure`	282.51	test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	`Failure`	313.05	test_vpc_redundant.py
test_02_redundant_VPC_default_routes	`Failure`	865.30	test_vpc_redundant.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Failure`	374.96	test_vpc_redundant.py
test_02_attach_volume	`Failure`	668.60	test_volumes.py
test_01_create_volume	`Failure`	683.83	test_volumes.py
test_10_attachAndDetach_iso	`Failure`	684.04	test_vm_life_cycle.py
test_04_change_offering_small	`Failure`	794.67	test_service_offerings.py
test_router_dns_guestipquery	`Failure`	277.46	test_router_dns.py
test_router_dhcphosts	`Failure`	188.70	test_router_dhcphosts.py
test_router_dhcp_opts	`Failure`	21.41	test_router_dhcphosts.py
test_04_rvpc_privategw_static_routes	`Failure`	1025.28	test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	949.88	test_privategw_acl.py
test_02_vpc_privategw_static_routes	`Failure`	964.88	test_privategw_acl.py
test_isolate_network_password_server	`Failure`	188.95	test_password_server.py
test_reboot_router	`Failure`	450.27	test_network.py
test_network_rules_acquired_public_ip_3_Load_Balancer_Rule	`Failure`	831.90	test_network.py
test_network_rules_acquired_public_ip_2_nat_rule	`Failure`	679.46	test_network.py
test_network_rules_acquired_public_ip_1_static_nat_rule	`Failure`	675.76	test_network.py
test_02_port_fwd_on_non_src_nat	`Failure`	679.11	test_network.py
test_01_port_fwd_on_src_nat	`Failure`	673.51	test_network.py
test_assign_and_removal_lb	`Failure`	110.43	test_loadbalance.py
test_02_create_lb_rule_non_nat	`Failure`	110.48	test_loadbalance.py
test_01_create_lb_rule_src_nat	`Failure`	110.56	test_loadbalance.py
test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80	`Failure`	300.53	test_internal_lb.py
test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80	`Failure`	255.38	test_internal_lb.py
test_01_vpc_site2site_vpn	`Error`	301.23	test_vpc_vpn.py
test_01_redundant_vpc_site2site_vpn	`Error`	386.36	test_vpc_vpn.py
test_05_rvpc_multi_tiers	`Error`	410.12	test_vpc_redundant.py
test_04_extract_template	`Error`	5.10	test_templates.py
test_03_delete_template	`Error`	5.14	test_templates.py
test_01_create_template	`Error`	35.40	test_templates.py
ContextSuite context=TestRouterDHCPHosts>:teardown	`Error`	239.16	test_router_dhcphosts.py
test_04_rvpc_internallb_haproxy_stats_on_all_interfaces	`Error`	265.55	test_internal_lb.py
test_03_vpc_internallb_haproxy_stats_on_all_interfaces	`Error`	296.40	test_internal_lb.py
test_01_vpc_remote_access_vpn	Success	66.14	test_vpc_vpn.py
test_09_delete_detached_volume	Success	15.46	test_volumes.py
test_08_resize_volume	Success	15.42	test_volumes.py
test_07_resize_fail	Success	20.47	test_volumes.py
test_06_download_detached_volume	Success	15.31	test_volumes.py
test_05_detach_volume	Success	100.22	test_volumes.py
test_04_delete_attached_volume	Success	10.23	test_volumes.py
test_03_download_attached_volume	Success	15.28	test_volumes.py
test_deploy_vm_multiple	Success	268.62	test_vm_life_cycle.py
test_deploy_vm	Success	0.03	test_vm_life_cycle.py
test_advZoneVirtualRouter	Success	0.02	test_vm_life_cycle.py
test_09_expunge_vm	Success	125.22	test_vm_life_cycle.py
test_08_migrate_vm	Success	35.90	test_vm_life_cycle.py
test_07_restore_vm	Success	0.14	test_vm_life_cycle.py
test_06_destroy_vm	Success	125.83	test_vm_life_cycle.py
test_03_reboot_vm	Success	125.89	test_vm_life_cycle.py
test_02_start_vm	Success	5.14	test_vm_life_cycle.py
test_01_stop_vm	Success	125.85	test_vm_life_cycle.py
test_CreateTemplateWithDuplicateName	Success	106.52	test_templates.py
test_08_list_system_templates	Success	0.03	test_templates.py
test_07_list_public_templates	Success	0.04	test_templates.py
test_05_template_permissions	Success	0.06	test_templates.py
test_02_edit_template	Success	90.21	test_templates.py
test_10_destroy_cpvm	Success	161.65	test_ssvm.py
test_09_destroy_ssvm	Success	198.70	test_ssvm.py
test_08_reboot_cpvm	Success	161.76	test_ssvm.py
test_07_reboot_ssvm	Success	133.54	test_ssvm.py
test_06_stop_cpvm	Success	166.83	test_ssvm.py
test_05_stop_ssvm	Success	163.83	test_ssvm.py
test_04_cpvm_internals	Success	1.18	test_ssvm.py
test_03_ssvm_internals	Success	3.27	test_ssvm.py
test_02_list_cpvm_vm	Success	0.16	test_ssvm.py
test_01_list_sec_storage_vm	Success	0.14	test_ssvm.py
test_01_snapshot_root_disk	Success	11.26	test_snapshots.py
test_03_delete_service_offering	Success	0.04	test_service_offerings.py
test_02_edit_service_offering	Success	0.10	test_service_offerings.py
test_01_create_service_offering	Success	0.12	test_service_offerings.py
test_02_sys_template_ready	Success	0.12	test_secondary_storage.py
test_01_sys_vm_start	Success	0.18	test_secondary_storage.py
test_09_reboot_router	Success	35.32	test_routers.py
test_08_start_router	Success	25.32	test_routers.py
test_07_stop_router	Success	10.17	test_routers.py
test_06_router_advanced	Success	0.06	test_routers.py
test_05_router_basic	Success	0.04	test_routers.py
test_04_restart_network_wo_cleanup	Success	5.72	test_routers.py
test_03_restart_network_cleanup	Success	80.75	test_routers.py
test_02_router_internal_adv	Success	1.18	test_routers.py
test_01_router_internal_basic	Success	0.70	test_routers.py
test_router_dns_externalipquery	Success	0.09	test_router_dns.py
test_01_updatevolumedetail	Success	0.09	test_resource_detail.py
test_01_reset_vm_on_reboot	Success	131.00	test_reset_vm_on_reboot.py
test_createRegion	Success	0.07	test_regions.py
test_create_pvlan_network	Success	5.24	test_pvlan.py
test_dedicatePublicIpRange	Success	0.46	test_public_ip_range.py
test_01_vpc_privategw_acl	Success	103.37	test_privategw_acl.py
test_01_primary_storage_nfs	Success	36.07	test_primary_storage.py
test_createPortablePublicIPRange	Success	15.21	test_portable_publicip.py
test_createPortablePublicIPAcquire	Success	15.50	test_portable_publicip.py
test_UpdateStorageOverProvisioningFactor	Success	0.13	test_over_provisioning.py
test_oobm_zchange_password	Success	21.02	test_outofbandmanagement.py
test_oobm_multiple_mgmt_server_ownership	Success	16.51	test_outofbandmanagement.py
test_oobm_issue_power_status	Success	10.51	test_outofbandmanagement.py
test_oobm_issue_power_soft	Success	15.52	test_outofbandmanagement.py
test_oobm_issue_power_reset	Success	15.53	test_outofbandmanagement.py
test_oobm_issue_power_on	Success	15.52	test_outofbandmanagement.py
test_oobm_issue_power_off	Success	15.51	test_outofbandmanagement.py
test_oobm_issue_power_cycle	Success	15.52	test_outofbandmanagement.py
test_oobm_enabledisable_across_clusterzones	Success	62.56	test_outofbandmanagement.py
test_oobm_enable_feature_valid	Success	5.22	test_outofbandmanagement.py
test_oobm_enable_feature_invalid	Success	0.14	test_outofbandmanagement.py
test_oobm_disable_feature_valid	Success	0.21	test_outofbandmanagement.py
test_oobm_disable_feature_invalid	Success	0.16	test_outofbandmanagement.py
test_oobm_configure_invalid_driver	Success	0.11	test_outofbandmanagement.py
test_oobm_configure_default_driver	Success	0.11	test_outofbandmanagement.py
test_oobm_background_powerstate_sync	Success	23.91	test_outofbandmanagement.py
test_extendPhysicalNetworkVlan	Success	15.38	test_non_contigiousvlan.py
test_01_nic	Success	385.13	test_nic.py
test_releaseIP	Success	324.78	test_network.py
test_public_ip_user_account	Success	10.29	test_network.py
test_public_ip_admin_account	Success	40.30	test_network.py
test_delete_account	Success	299.09	test_network.py
test_nic_secondaryip_add_remove	Success	233.68	test_multipleips_per_nic.py
login_test_saml_user	Success	24.97	test_login.py
test_03_list_snapshots	Success	0.08	test_list_ids_parameter.py
test_02_list_templates	Success	0.04	test_list_ids_parameter.py
test_01_list_volumes	Success	0.03	test_list_ids_parameter.py
test_07_list_default_iso	Success	0.06	test_iso.py
test_05_iso_permissions	Success	0.08	test_iso.py
test_04_extract_Iso	Success	5.20	test_iso.py
test_03_delete_iso	Success	95.19	test_iso.py
test_02_edit_iso	Success	0.07	test_iso.py
test_01_create_iso	Success	21.98	test_iso.py
test_dedicateGuestVlanRange	Success	10.40	test_guest_vlan_range.py
test_UpdateConfigParamWithScope	Success	0.25	test_global_settings.py
test_rolepermission_lifecycle_update	Success	7.13	test_dynamicroles.py
test_rolepermission_lifecycle_list	Success	6.91	test_dynamicroles.py
test_rolepermission_lifecycle_delete	Success	6.74	test_dynamicroles.py
test_rolepermission_lifecycle_create	Success	6.82	test_dynamicroles.py
test_rolepermission_lifecycle_concurrent_updates	Success	6.91	test_dynamicroles.py
test_role_lifecycle_update_role_inuse	Success	7.02	test_dynamicroles.py
test_role_lifecycle_update	Success	11.93	test_dynamicroles.py
test_role_lifecycle_list	Success	6.83	test_dynamicroles.py
test_role_lifecycle_delete	Success	7.01	test_dynamicroles.py
test_role_lifecycle_create	Success	6.80	test_dynamicroles.py
test_role_inuse_deletion	Success	6.79	test_dynamicroles.py
test_role_account_acls_multiple_mgmt_servers	Success	9.33	test_dynamicroles.py
test_role_account_acls	Success	9.37	test_dynamicroles.py
test_default_role_deletion	Success	6.90	test_dynamicroles.py
test_04_create_fat_type_disk_offering	Success	0.07	test_disk_offerings.py
test_03_delete_disk_offering	Success	0.04	test_disk_offerings.py
test_02_edit_disk_offering	Success	0.05	test_disk_offerings.py
test_02_create_sparse_type_disk_offering	Success	0.07	test_disk_offerings.py
test_01_create_disk_offering	Success	0.10	test_disk_offerings.py
test_deployvm_userdispersing	Success	10.52	test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_userconcentrated	Success	20.59	test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_firstfit	Success	126.14	test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_userdata_post	Success	10.40	test_deploy_vm_with_userdata.py
test_deployvm_userdata	Success	171.96	test_deploy_vm_with_userdata.py
test_02_deploy_vm_root_resize	Success	7.45	test_deploy_vm_root_resize.py
test_01_deploy_vm_root_resize	Success	6.87	test_deploy_vm_root_resize.py
test_00_deploy_vm_root_resize	Success	304.04	test_deploy_vm_root_resize.py
test_deploy_vm_from_iso	Success	243.90	test_deploy_vm_iso.py
test_DeployVmAntiAffinityGroup	Success	181.81	test_affinity_groups.py
test_03_delete_vm_snapshots	Skipped	0.00	test_vm_snapshots.py
test_02_revert_vm_snapshots	Skipped	0.00	test_vm_snapshots.py
test_01_test_vm_volume_snapshot	Skipped	0.00	test_vm_snapshots.py
test_01_create_vm_snapshots	Skipped	0.00	test_vm_snapshots.py
test_06_copy_template	Skipped	0.00	test_templates.py
test_static_role_account_acls	Skipped	0.02	test_staticroles.py
test_01_scale_vm	Skipped	0.00	test_scale_vm.py
test_01_primary_storage_iscsi	Skipped	0.04	test_primary_storage.py
test_06_copy_iso	Skipped	0.00	test_iso.py
test_deploy_vgpu_enabled_vm	Skipped	0.01	test_deploy_vgpu_enabled_vm.py

murali-reddy · 2016-12-05T10:53:55Z

@romain-dartigues we fixed issue where ips.json has wrong gateway due to error in parsing the cmd_line.json. its committed in 4.8 and above [1]. Do you still see (or can you please verify) performance hit due to arpping to wrong gateway on 4.8/4.9/master?

[1] 93ac134

romain-dartigues · 2016-12-05T20:21:13Z

@murali-reddy I'm not in position to run tests anymore
@dpassante could you take the lead for it?

yadvr · 2017-07-23T11:12:57Z

Several test failures, unless the PR author can fix, this is on hold.

yadvr · 2017-12-28T10:55:08Z

@romain-dartigues @dpassante any update on this?

romain-dartigues · 2018-01-13T17:01:50Z

No comment, since it does not seems required anymore I close this request.

syed reviewed Jul 5, 2016
View reviewed changes

jburwell reviewed Sep 6, 2016
View reviewed changes

romain-dartigues force-pushed the systemvm-none-gateway branch from fce10c5 to 9c47e97Compare October 26, 2016 06:53

romain-dartigues changed the base branch from master to 4.9October 26, 2016 06:54

romain-dartigues force-pushed the systemvm-none-gateway branch 2 times, most recently from 21f4e6d to 67bfc7cCompare October 26, 2016 10:33

romain-dartigues force-pushed the systemvm-none-gateway branch from 67bfc7c to ebe6537Compare November 21, 2016 07:55

CLOUDSTACK-9568: slow operations on system VM
a0d0c8d

romain-dartigues force-pushed the systemvm-none-gateway branch from ebe6537 to a0d0c8dCompare November 23, 2016 12:48

romain-dartigues closed this Jan 13, 2018

Uh oh!

Conversation

romain-dartigues commented Jul 5, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

syedJul 5, 2016

Choose a reason for hiding this comment

Uh oh!

syed commented Jul 5, 2016

Uh oh!

romain-dartigues commented Jul 5, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

syed commented Jul 6, 2016

Uh oh!

syed commented Jul 6, 2016

Uh oh!

yadvr commented Aug 6, 2016

Uh oh!

blueorangutan commented Aug 6, 2016

Uh oh!

blueorangutan commented Aug 6, 2016

Uh oh!

jburwellSep 6, 2016

Choose a reason for hiding this comment

Uh oh!

yadvr commented Oct 21, 2016

Uh oh!

romain-dartigues commented Oct 26, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yadvr commented Oct 26, 2016

Uh oh!

yadvr commented Nov 20, 2016

Uh oh!

blueorangutan commented Nov 20, 2016

Uh oh!

blueorangutan commented Nov 20, 2016

Uh oh!

romain-dartigues commented Nov 21, 2016

Uh oh!

yadvr commented Nov 21, 2016

Uh oh!

blueorangutan commented Nov 21, 2016

Uh oh!

blueorangutan commented Nov 21, 2016

Uh oh!

yadvr commented Nov 21, 2016

Uh oh!

blueorangutan commented Nov 21, 2016

Uh oh!

blueorangutan commented Nov 21, 2016

Uh oh!

romain-dartigues commented Nov 22, 2016

Uh oh!

yadvr commented Nov 23, 2016

Uh oh!

romain-dartigues commented Nov 23, 2016

Uh oh!

yadvr commented Nov 23, 2016

Uh oh!

blueorangutan commented Nov 23, 2016

Uh oh!

blueorangutan commented Nov 23, 2016

Uh oh!

yadvr commented Nov 24, 2016

Uh oh!

blueorangutan commented Nov 24, 2016

Uh oh!

blueorangutan commented Nov 24, 2016

Uh oh!

yadvr commented Nov 24, 2016

Uh oh!

blueorangutan commented Nov 24, 2016

Uh oh!

blueorangutan commented Nov 24, 2016

Uh oh!

murali-reddy commented Dec 5, 2016

Uh oh!

romain-dartigues commented Dec 5, 2016

romain-dartigues commented Jul 5, 2016•
edited
Loading

romain-dartigues commented Jul 5, 2016•
edited
Loading

romain-dartigues commented Oct 26, 2016•
edited
Loading