Skip to content

CLOUDSTACK-9317: Enable/disable static NAT associates only relevant IPs. - #1623

Closed
ProjectMoon wants to merge 1 commit into
apache:4.8from
greenqloud:pr-static-nat-concurrency-fix-4.8
Closed

CLOUDSTACK-9317: Enable/disable static NAT associates only relevant IPs.#1623
ProjectMoon wants to merge 1 commit into
apache:4.8from
greenqloud:pr-static-nat-concurrency-fix-4.8

Conversation

@ProjectMoon

Copy link
Copy Markdown

New version of #1450, opened against the 4.8 branch.

Original Description
This pull request fixes a concurrency issue when disabling static NAT on a bunch of IPs simultaneously. Under the old behavior, executing multiple disable requests would result in invalid IP associations being sent to the virtual router. This commit changes the behavior to apply an IP association for only the IP being added/released, which means that it is impossible for the virtual router to receive invalid data.

This was tested against a virtual router running on KVM and VMware. It would be nice to have some input how this change could affect redundant routers and other static NAT providers.

@ProjectMoonProjectMoon changed the title Enable/disable static NAT associates only relevant IPs.CLOUDSTACK-9317: Enable/disable static NAT associates only relevant IPs.Aug 2, 2016
@blueorangutan

Copy link
Copy Markdown

RPM packages built and available at: http://packages.shapeblue.com/cloudstack/custom/github-1623.


for (StaticNat snat : staticNats) {
userIps.add(_ipAddressDao.findById(snat.getSourceIpAddressId()));
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Executing queries in a for loop in unnecessarily expensive. Could you please refactor this for loop into a new method on IPAddressDao that retrieves a list of IP addresses for a list of IPs?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes sense, yes. Better to have one database query than many.

@jburwell

Copy link
Copy Markdown
Contributor

@ProjectMoon have you run the test_redundant_router_cleanups, test_redundant_router_services, and test_redundant_router_upgrades test cases for this PR?

Also, is there a JIRA ticket associated with this change?

@ProjectMoon

Copy link
Copy Markdown
Author

JIRA ticket is CLOUDSTACK-9317. Have not run the Marvin tests, but will do so.

@yadvr

yadvr commented Aug 5, 2016

Copy link
Copy Markdown
Member

@blueorangutan kick

@blueorangutan

Copy link
Copy Markdown

A Trillian-Jenkins job has been kicked to build packages and start testing. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result: ✔centos6 ✔centos7 ✔debian repo: http://packages.shapeblue.com/cloudstack/pr/1623

@jburwell

Copy link
Copy Markdown
Contributor

If possible, I would like to get this fix into 4.8.2.0/4.9.1.0, but I will be cutting the RC shortly.

@ProjectMoon could you please amend your commit message to include the JIRA ticket ID and an explanation of the motivation for the change?

@rhtyd @borisstoyanov can we push this PR through Trillian with smoke tests and the following test suites:

  • test_redundant_router_cleanups
  • test_redundant_router_services
  • test_redundant_router_upgrades

@murali-reddy Do you think we need to run any other test VR test cases for proper regression testing?

@blueorangutan

Copy link
Copy Markdown

Trillian test result (trillian-pr1623-38-vmware-55u3-cs48):
Test completed. 37 look ok, 12 have errors

TestResultTime
test_createRegionSuccess0.035
test_DeployVmAntiAffinityGroup_in_projectSuccess267.303
test_01_router_internal_basicSuccess0.606
test_02_router_internal_advSuccess1.027
test_03_restart_network_cleanupSuccess140.944
test_04_restart_network_wo_cleanupSuccess5.607
test_05_router_basicSuccess0.029
test_06_router_advancedSuccess0.046
test_07_stop_routerSuccess25.243
test_08_start_routerSuccess120.828
test_09_reboot_routerSuccess130.835
test_DeployVmAntiAffinityGroupSuccess196.684
test_01_scale_vmSkipped66.441
test_deploy_vgpu_enabled_vmSkipped0.004
test_deploy_vm_from_isoSuccess843.661
test_01_sys_vm_startSuccess0.148
test_02_sys_template_readySuccess0.099
test_00_deploy_vm_root_resizeSuccess6.498
test_01_deploy_vm_root_resizeSuccess6.235
test_02_deploy_vm_root_resizeSuccess6.266
test_deployvm_firstfitSuccess206.520
test_deployvm_userconcentratedSuccess121.187
test_deployvm_userdispersingSuccess55.752
test_01_create_service_offeringSuccess0.108
test_02_edit_service_offeringSuccess0.094
test_03_delete_service_offeringSuccess0.045
test_04_change_offering_smallSuccess97.309
test_deployvm_userdataSuccess161.303
test_deployvm_userdata_postSuccess20.425
test_01_create_disk_offeringSuccess0.113
test_02_create_sparse_type_disk_offeringSuccess0.075
test_04_create_fat_type_disk_offeringSuccess0.076
test_02_edit_disk_offeringSuccess0.048
test_03_delete_disk_offeringSuccess0.042
test_01_snapshot_root_diskSuccess222.219
test_UpdateConfigParamWithScopeSuccess0.165
ContextSuite context=TestDedicateGuestVlanRange>:setupError0.000
test_01_list_sec_storage_vmSuccess0.157
test_02_list_cpvm_vmSuccess0.139
test_03_ssvm_internalsSuccess3.896
test_04_cpvm_internalsSuccess1.199
test_05_stop_ssvmSuccess204.223
test_06_stop_cpvmSuccess181.887
test_07_reboot_ssvmSuccess158.813
test_08_reboot_cpvmSuccess156.708
test_09_destroy_ssvmSuccess233.966
test_10_destroy_cpvmSuccess236.990
test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80Failure441.264
test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80Failure546.859
test_03_vpc_internallb_haproxy_stats_on_all_interfacesError441.481
test_04_rvpc_internallb_haproxy_stats_on_all_interfacesError516.759
test_01_create_templateSuccess146.055
test_CreateTemplateWithDuplicateNameSuccess251.738
test_02_edit_templateSuccess90.149
test_03_delete_templateSuccess5.145
test_04_extract_templateSuccess10.191
test_05_template_permissionsSuccess0.040
test_06_copy_templateSkipped0.000
test_07_list_public_templatesSuccess0.025
test_08_list_system_templatesSuccess0.056
test_01_create_isoSuccess66.332
test_02_edit_isoSuccess0.126
test_03_delete_isoSuccess95.195
test_04_extract_IsoSuccess5.192
test_05_iso_permissionsSuccess0.048
test_06_copy_isoSkipped0.000
test_07_list_default_isoSuccess0.039
test_01_create_lb_rule_src_natSuccess187.843
test_02_create_lb_rule_non_natSuccess187.338
test_assign_and_removal_lbSuccess133.652
login_test_saml_userSuccess22.090
test_nic_secondaryip_add_removeSuccess192.737
test_advZoneVirtualRouterSuccess0.020
test_deploy_vmSuccess0.018
test_deploy_vm_multipleSuccess283.325
test_01_stop_vmSuccess10.132
test_02_start_vmSuccess20.214
test_03_reboot_vmSuccess5.124
test_06_destroy_vmSuccess5.108
test_07_restore_vmSuccess0.102
test_08_migrate_vmSuccess76.170
test_09_expunge_vmSuccess125.236
test_10_attachAndDetach_isoSuccess72.134
test_network_aclSuccess201.902
test_delete_accountSuccess278.007
test_01_port_fwd_on_src_natSuccess111.800
test_02_port_fwd_on_non_src_natSuccess55.628
test_public_ip_admin_accountSuccess40.272
test_public_ip_user_accountSuccess10.259
test_reboot_routerSuccess625.125
test_releaseIPSuccess238.215
test_network_rules_acquired_public_ip_1_static_nat_ruleSuccess124.311
test_network_rules_acquired_public_ip_2_nat_ruleSuccess61.609
test_network_rules_acquired_public_ip_3_Load_Balancer_RuleSuccess66.731
test_01_test_vm_volume_snapshotError156.243
test_01_create_vm_snapshotsSuccess161.736
test_02_revert_vm_snapshotsSuccess194.222
test_03_delete_vm_snapshotsSuccess275.205
test_vm_nic_adapter_vmxnet3Skipped0.000
test_01_nicSuccess868.217
test_01_create_volumeSuccess520.190
test_02_attach_volumeSuccess58.866
test_03_download_attached_volumeSuccess20.328
test_04_delete_attached_volumeSuccess15.258
test_05_detach_volumeSuccess100.311
test_06_download_detached_volumeSuccess60.581
test_07_resize_failSkipped10.258
test_08_resize_volumeSkipped10.193
test_09_delete_detached_volumeSuccess30.919
test_extendPhysicalNetworkVlanError0.022
test_UpdateStorageOverProvisioningFactorSuccess0.123
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACLSuccess1213.876
test_02_redundant_VPC_default_routesSuccess579.689
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routersSuccess759.643
test_04_rvpc_network_garbage_collector_nicsSuccess865.118
test_05_rvpc_multi_tiersSuccess662.118
test_isolate_network_password_serverFailure166.672
test_createPortablePublicIPAcquireSuccess15.658
test_createPortablePublicIPRangeSuccess15.391
test_01_VPC_nics_after_destroySuccess637.734
test_02_VPC_default_routesSuccess393.037
test_01_primary_storage_iscsiSkipped0.028
test_01_primary_storage_nfsSuccess38.355
test_01_vpc_privategw_aclFailure101.960
test_02_vpc_privategw_static_routesFailure207.667
test_03_vpc_privategw_restart_vpc_cleanupFailure242.957
test_04_rvpc_privategw_static_routesFailure434.113
test_01_redundant_vpc_site2site_vpnError692.909
ContextSuite context=TestRVPCSite2SiteVpn>:teardownError698.010
test_01_vpc_remote_access_vpnError0.064
test_01_vpc_site2site_vpnError486.664
test_dedicatePublicIpRangeError0.298
test_create_pvlan_networkSuccess5.216
test_01_quotaError0.033
test_02_quotaError0.029
test_03_quotaError0.028
test_04_quotaError0.029
test_05_quotaError0.027
test_06_quotaError0.032
test_07_quotaError0.031
test_01_reset_vm_on_rebootSuccess25.316
test_01_updatevolumedetailSuccess0.122
test_router_dhcphostsFailure166.785
ContextSuite context=TestRouterDHCPHosts>:teardownError187.075
test_02_routervm_iptables_policiesFailure231.410
test_01_single_VPC_iptables_policiesFailure371.973
test_01_isolate_network_FW_PF_default_routes_egress_trueSuccess258.519
test_02_isolate_network_FW_PF_default_routes_egress_falseSuccess268.553
test_01_RVR_Network_FW_PF_SSH_default_routes_egress_trueFailure410.379
test_02_RVR_Network_FW_PF_SSH_default_routes_egress_falseFailure361.440
test_03_RVR_Network_check_router_stateSuccess369.925

Trillian env - trillian-pr1623-38-vmware-55u3-cs48, Job ID 38
Hypervisor: vmware-55u3 (x2), Advanced Zone
Mgmt host os - 6
Marvin logs at: http://packages.shapeblue.com/cloudstack/pr/1623/trillian/trillian-pr1623-38-vmware-55u3-cs48

@ProjectMoon

Copy link
Copy Markdown
Author

It's been amended and rebased. I still haven't had time to add a new DAO method to the IP address DAO. I can work on that now though. When is the RC being cut?

@jburwell

Copy link
Copy Markdown
Contributor

@ProjectMoon I am trying to cut the 4.8.2.0 RC ASAP. This PR looks like it is not going to make it given the test failures. Do you mind re-pointing it to 4.9? We will make this PR a high priority for 4.9.2.0.

@ProjectMoon

Copy link
Copy Markdown
Author

Yeah. Unfortunately I have not had any time to work on the pull requests lately. When I have time I will reopen it against 4.9.

@jburwell

Copy link
Copy Markdown
Contributor

@ProjectMoon I am extending the date for 4.8 to 25 Sept 2016. Therefore, if we can get the test failure fixed, we can get it into 4.8.2.0, 4.9.1.0, and 4.10.0.0. Also, could you please investigate the Travis and Jenkins build failures?

@yadvr

Copy link
Copy Markdown
Member

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-77

@ProjectMoon

Copy link
Copy Markdown
Author

This one hasn't been rebased in quite a while, so the packages produced will be rather old. Going to rebase this against latest 4.8.

@yadvr

Copy link
Copy Markdown
Member

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-207

The previous behavior would send a list of all IPs with a state of
add/revoke. This causes concurrency issues when multiple IPs were
being added or removed, and could leave networks with wrong IP
addresses assigned to the devices. By assigning and removing only the
IP being changed per-request, the concurrency issue is removed and no
real performance loss occurs.
@ProjectMoon

Copy link
Copy Markdown
Author

Updated to latest 4.8.

@yadvr

Copy link
Copy Markdown
Member

@ProjectMoon thanks
@murali-reddy can you help review this?
@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-231

@yadvr

Copy link
Copy Markdown
Member

@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

Trillian test result (tid-420)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 28011 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr1623-t420-kvm-centos7.zip
Test completed. 42 look ok, 1 have error(s)

TestResultTime (s)Test File
test_05_rvpc_multi_tiersFailure255.12test_vpc_redundant.py
ContextSuite context=TestVPCRedundancy>:teardownError565.30test_vpc_redundant.py
test_01_vpc_site2site_vpnSuccess202.68test_vpc_vpn.py
test_01_vpc_remote_access_vpnSuccess76.67test_vpc_vpn.py
test_01_redundant_vpc_site2site_vpnSuccess299.83test_vpc_vpn.py
test_02_VPC_default_routesSuccess344.28test_vpc_router_nics.py
test_01_VPC_nics_after_destroySuccess604.80test_vpc_router_nics.py
test_04_rvpc_network_garbage_collector_nicsSuccess1490.00test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routersSuccess639.50test_vpc_redundant.py
test_02_redundant_VPC_default_routesSuccess810.54test_vpc_redundant.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACLSuccess1364.84test_vpc_redundant.py
test_09_delete_detached_volumeSuccess16.35test_volumes.py
test_08_resize_volumeSuccess16.01test_volumes.py
test_07_resize_failSuccess21.15test_volumes.py
test_06_download_detached_volumeSuccess15.66test_volumes.py
test_05_detach_volumeSuccess100.31test_volumes.py
test_04_delete_attached_volumeSuccess10.25test_volumes.py
test_03_download_attached_volumeSuccess15.42test_volumes.py
test_02_attach_volumeSuccess74.83test_volumes.py
test_01_create_volumeSuccess721.35test_volumes.py
test_deploy_vm_multipleSuccess366.43test_vm_life_cycle.py
test_deploy_vmSuccess0.03test_vm_life_cycle.py
test_advZoneVirtualRouterSuccess0.02test_vm_life_cycle.py
test_10_attachAndDetach_isoSuccess27.28test_vm_life_cycle.py
test_09_expunge_vmSuccess125.26test_vm_life_cycle.py
test_08_migrate_vmSuccess41.44test_vm_life_cycle.py
test_07_restore_vmSuccess0.26test_vm_life_cycle.py
test_06_destroy_vmSuccess126.82test_vm_life_cycle.py
test_03_reboot_vmSuccess126.19test_vm_life_cycle.py
test_02_start_vmSuccess10.26test_vm_life_cycle.py
test_01_stop_vmSuccess40.48test_vm_life_cycle.py
test_CreateTemplateWithDuplicateNameSuccess116.36test_templates.py
test_08_list_system_templatesSuccess0.05test_templates.py
test_07_list_public_templatesSuccess0.07test_templates.py
test_05_template_permissionsSuccess0.12test_templates.py
test_04_extract_templateSuccess5.26test_templates.py
test_03_delete_templateSuccess5.14test_templates.py
test_02_edit_templateSuccess90.18test_templates.py
test_01_create_templateSuccess156.90test_templates.py
test_10_destroy_cpvmSuccess161.83test_ssvm.py
test_09_destroy_ssvmSuccess163.73test_ssvm.py
test_08_reboot_cpvmSuccess132.03test_ssvm.py
test_07_reboot_ssvmSuccess133.76test_ssvm.py
test_06_stop_cpvmSuccess162.50test_ssvm.py
test_05_stop_ssvmSuccess134.06test_ssvm.py
test_04_cpvm_internalsSuccess1.17test_ssvm.py
test_03_ssvm_internalsSuccess3.38test_ssvm.py
test_02_list_cpvm_vmSuccess0.12test_ssvm.py
test_01_list_sec_storage_vmSuccess0.13test_ssvm.py
test_01_snapshot_root_diskSuccess11.32test_snapshots.py
test_04_change_offering_smallSuccess210.02test_service_offerings.py
test_03_delete_service_offeringSuccess0.04test_service_offerings.py
test_02_edit_service_offeringSuccess0.09test_service_offerings.py
test_01_create_service_offeringSuccess0.29test_service_offerings.py
test_02_sys_template_readySuccess0.12test_secondary_storage.py
test_01_sys_vm_startSuccess0.17test_secondary_storage.py
test_09_reboot_routerSuccess45.57test_routers.py
test_08_start_routerSuccess41.15test_routers.py
test_07_stop_routerSuccess10.62test_routers.py
test_06_router_advancedSuccess0.06test_routers.py
test_05_router_basicSuccess0.06test_routers.py
test_04_restart_network_wo_cleanupSuccess5.89test_routers.py
test_03_restart_network_cleanupSuccess91.34test_routers.py
test_02_router_internal_advSuccess1.15test_routers.py
test_01_router_internal_basicSuccess0.63test_routers.py
test_router_dhcphostsSuccess348.17test_router_dhcphosts.py
test_01_updatevolumedetailSuccess0.10test_resource_detail.py
test_01_reset_vm_on_rebootSuccess171.74test_reset_vm_on_reboot.py
test_createRegionSuccess0.11test_regions.py
test_create_pvlan_networkSuccess5.60test_pvlan.py
test_dedicatePublicIpRangeSuccess1.31test_public_ip_range.py
test_04_rvpc_privategw_static_routesSuccess667.24test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanupSuccess564.57test_privategw_acl.py
test_02_vpc_privategw_static_routesSuccess464.09test_privategw_acl.py
test_01_vpc_privategw_aclSuccess116.21test_privategw_acl.py
test_01_primary_storage_nfsSuccess36.26test_primary_storage.py
test_createPortablePublicIPRangeSuccess15.27test_portable_publicip.py
test_createPortablePublicIPAcquireSuccess15.62test_portable_publicip.py
test_isolate_network_password_serverSuccess87.15test_password_server.py
test_UpdateStorageOverProvisioningFactorSuccess0.17test_over_provisioning.py
test_extendPhysicalNetworkVlanSuccess15.53test_non_contigiousvlan.py
test_01_nicSuccess759.98test_nic.py
test_releaseIPSuccess356.34test_network.py
test_reboot_routerSuccess437.86test_network.py
test_public_ip_user_accountSuccess10.61test_network.py
test_public_ip_admin_accountSuccess40.34test_network.py
test_network_rules_acquired_public_ip_3_Load_Balancer_RuleSuccess67.51test_network.py
test_network_rules_acquired_public_ip_2_nat_ruleSuccess62.15test_network.py
test_network_rules_acquired_public_ip_1_static_nat_ruleSuccess124.43test_network.py
test_delete_accountSuccess345.77test_network.py
test_02_port_fwd_on_non_src_natSuccess56.08test_network.py
test_01_port_fwd_on_src_natSuccess112.51test_network.py
test_nic_secondaryip_add_removeSuccess236.12test_multipleips_per_nic.py
login_test_saml_userSuccess29.14test_login.py
test_assign_and_removal_lbSuccess135.14test_loadbalance.py
test_02_create_lb_rule_non_natSuccess189.02test_loadbalance.py
test_01_create_lb_rule_src_natSuccess210.45test_loadbalance.py
test_07_list_default_isoSuccess0.09test_iso.py
test_05_iso_permissionsSuccess0.09test_iso.py
test_04_extract_IsoSuccess6.41test_iso.py
test_03_delete_isoSuccess95.13test_iso.py
test_02_edit_isoSuccess0.09test_iso.py
test_01_create_isoSuccess42.23test_iso.py
test_04_rvpc_internallb_haproxy_stats_on_all_interfacesSuccess401.55test_internal_lb.py
test_03_vpc_internallb_haproxy_stats_on_all_interfacesSuccess219.33test_internal_lb.py
test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80Success625.53test_internal_lb.py
test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80Success518.04test_internal_lb.py
test_dedicateGuestVlanRangeSuccess10.30test_guest_vlan_range.py
test_UpdateConfigParamWithScopeSuccess0.34test_global_settings.py
test_04_create_fat_type_disk_offeringSuccess0.28test_disk_offerings.py
test_03_delete_disk_offeringSuccess0.16test_disk_offerings.py
test_02_edit_disk_offeringSuccess0.43test_disk_offerings.py
test_02_create_sparse_type_disk_offeringSuccess0.38test_disk_offerings.py
test_01_create_disk_offeringSuccess0.89test_disk_offerings.py
test_deployvm_userdispersingSuccess21.42test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_userconcentratedSuccess45.96test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_firstfitSuccess50.81test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_userdata_postSuccess56.43test_deploy_vm_with_userdata.py
test_deployvm_userdataSuccess56.41test_deploy_vm_with_userdata.py
test_02_deploy_vm_root_resizeSuccess7.61test_deploy_vm_root_resize.py
test_01_deploy_vm_root_resizeSuccess7.14test_deploy_vm_root_resize.py
test_00_deploy_vm_root_resizeSuccess239.89test_deploy_vm_root_resize.py
test_deploy_vm_from_isoSuccess234.26test_deploy_vm_iso.py
test_DeployVmAntiAffinityGroupSuccess97.00test_affinity_groups.py
test_03_delete_vm_snapshotsSkipped0.00test_vm_snapshots.py
test_02_revert_vm_snapshotsSkipped0.00test_vm_snapshots.py
test_01_test_vm_volume_snapshotSkipped0.00test_vm_snapshots.py
test_01_create_vm_snapshotsSkipped0.00test_vm_snapshots.py
test_06_copy_templateSkipped0.00test_templates.py
test_01_scale_vmSkipped0.00test_scale_vm.py
test_01_primary_storage_iscsiSkipped0.04test_primary_storage.py
test_06_copy_isoSkipped0.00test_iso.py
test_deploy_vgpu_enabled_vmSkipped0.01test_deploy_vgpu_enabled_vm.py

@yadvr

Copy link
Copy Markdown
Member

@abhinandanprateek@murali-reddy and others -- can we have review on this, thanks.

@yadvr

Copy link
Copy Markdown
Member

@ProjectMoon

Copy link
Copy Markdown
Author

Closed in favor of #1908

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@ProjectMoon@blueorangutan@jburwell@yadvr