Skip to content

test: fix smoke test failures on xen/vmware - #13628

Draft
weizhouapache wants to merge 6 commits into
apache:mainfrom
weizhouapache:4.23-fix-smoke-test-failures
Draft

test: fix smoke test failures on xen/vmware#13628
weizhouapache wants to merge 6 commits into
apache:mainfrom
weizhouapache:4.23-fix-smoke-test-failures

Conversation

@weizhouapache

Copy link
Copy Markdown
Member

Description

This PR fixes some smoke test failures on vmware or xenserver/xcpng

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • Build/CI
  • Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

CopilotAI review requested due to automatic review settings July 16, 2026 11:34
@boring-cyborgboring-cyborgBot added component:integration-test Python Warning... Python code Ahead! labels Jul 16, 2026
@weizhouapacheweizhouapache added this to the 4.23.0 milestone Jul 16, 2026
@weizhouapache

Copy link
Copy Markdown
MemberAuthor

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@weizhouapacheweizhouapache added this to the 4.23.0 milestone Jul 16, 2026

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR targets integration smoke test stability across hypervisors (notably XenServer/XCP-ng and VMware) by skipping KVM-specific checks/tests when not running on KVM, and by adjusting a few timing/host prerequisite assumptions that can cause intermittent failures.

Changes:

  • Skip KVM-only paths in smoke tests when the detected hypervisor is not KVM.
  • Add additional host-tool prerequisite checks for a VPC source NAT IP update test.
  • Add delays around affinity-group migration steps and make the events-resource test more tolerant around volume attach/detach operations.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

FileDescription
test/integration/smoke/test_nonstrict_affinity_group.pyAdds waits before migration steps in non-strict affinity group tests.
test/integration/smoke/test_network_extension_namespace.pySkips KVM prerequisite checks on non-KVM hypervisors; expands required host tools for one test.
test/integration/smoke/test_kms_lifecycle.pySkips a VM root-disk encryption test on non-KVM hypervisors.
test/integration/smoke/test_events_resource.pyWraps volume attach/detach operations to avoid hard failures in some environments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment threadtest/integration/smoke/test_nonstrict_affinity_group.py
Comment threadtest/integration/smoke/test_nonstrict_affinity_group.py
Comment threadtest/integration/smoke/test_events_resource.py Outdated
@codecov

codecovBot commented Jul 16, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 19.64%. Comparing base (4f11707) to head (a6abcce).
⚠️ Report is 2 commits behind head on main.

Files with missing linesPatch %Lines
...tack/storage/motion/AncientDataMotionStrategy.java0.00%2 Missing and 2 partials ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #13628 +/- ##
============================================
- Coverage 19.65% 19.64% -0.01% + Complexity 19792 19789 -3 
============================================
Files 6368 6368 Lines 574881 574885 +4 Branches 70351 70353 +2 ============================================
- Hits 112970 112961 -9 - Misses 449639 449650 +11 - Partials 12272 12274 +2 
FlagCoverage Δ
uitests3.41% <ø> (ø)
unittests20.92% <0.00%> (-0.01%)⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18593

@weizhouapache

Copy link
Copy Markdown
MemberAuthor

@blueorangutan test ubuntu26 xcpng82 keepEnv

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a [SL] Trillian-Jenkins test job (ubuntu26 mgmt + xcpng82) has been kicked to run smoke tests

@weizhouapache

weizhouapache commented Jul 16, 2026

Copy link
Copy Markdown
MemberAuthor

@blueorangutan test ubuntu26 vmware-80u3e keepEnv

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a [SL] Trillian-Jenkins test job (ubuntu26 mgmt + vmware-80u3e) has been kicked to run smoke tests

CopilotAI review requested due to automatic review settings July 17, 2026 09:41

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

Comment threadtest/integration/smoke/test_network_extension_namespace.py
Comment threadtest/integration/smoke/test_kms_lifecycle.py
Comment threadtest/integration/smoke/test_events_resource.py
@weizhouapache

Copy link
Copy Markdown
MemberAuthor

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18602

@weizhouapache

Copy link
Copy Markdown
MemberAuthor

@blueorangutan test ubuntu26 xcpng82

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a [SL] Trillian-Jenkins test job (ubuntu26 mgmt + xcpng82) has been kicked to run smoke tests

@weizhouapache

Copy link
Copy Markdown
MemberAuthor

@blueorangutan test ubuntu26 vmware-80u3e

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a [SL] Trillian-Jenkins test job (ubuntu26 mgmt + vmware-80u3e) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian Build Failed (tid-7)

@weizhouapache

Copy link
Copy Markdown
MemberAuthor

@blueorangutan test ubuntu26 vmware-80u3

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a [SL] Trillian-Jenkins test job (ubuntu26 mgmt + vmware-80u3) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian Build Failed (tid-9)

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

test/integration/smoke/test_events_resource.py:150

  • This fixed sleep is a brittle way to wait for the newly created volume to become attachable. Volume.create() does not wait for provisioning to complete, so the volume might still be non-Ready after 10s (or be Ready much sooner). Poll for volume.state == 'Ready' with a timeout before calling attach_volume().
 time.sleep(10)

Comment threadtest/integration/smoke/test_vm_life_cycle.py
Comment threadtest/integration/smoke/test_usage.py
@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18686

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian test result (tid-16632)
Environment: xcpng82 (x2), zone: Advanced Networking with Mgmt server ol9
Total time taken: 73150 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13628-t16632-xcpng82.zip
Smoke tests completed. 151 look OK, 5 have errors, 0 did not run
Only failed and skipped tests results shown below:

TestResultTime (s)Test File
ContextSuite context=TestClusterDRS>:setupError0.00test_cluster_drs.py
test_list_system_vms_metrics_historyFailure0.69test_metrics_api.py
test_02_list_snapshots_with_removed_data_storeError13.87test_snapshots.py
test_02_list_snapshots_with_removed_data_storeError13.87test_snapshots.py
test_01_vpn_usageError1.12test_usage.py
test_01_migrate_vm_strict_tags_successError51.68test_vm_strict_host_tags.py

CopilotAI review requested due to automatic review settings July 28, 2026 14:17
@weizhouapache

Copy link
Copy Markdown
MemberAuthor

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (3)

test/integration/smoke/test_network_extension_namespace.py:863

  • In unittest/nose-style tests, skipTest is intended to be called directly (it already raises the SkipTest exception). Using raise self.skipTest(...) is redundant and can confuse readers/tools. Prefer self.skipTest(\"...\").
 if self.hv.lower() != 'kvm':
raise self.skipTest("Skipping test case for non-kvm hypervisor")

test/integration/smoke/test_kms_lifecycle.py:430

  • Same issue as above: skipTest should be called, not raised. Use self.skipTest(\"...\") to skip cleanly.
 if self.hypervisor.lower() != 'kvm':
raise self.skipTest("Skipping test case for non-kvm hypervisor")

test/integration/smoke/test_vm_strict_host_tags.py:535

  • Hard-coded sleep(10) makes the smoke suite slower and can still be flaky across environments (10s may be too short or unnecessarily long). Prefer polling for a condition with a bounded timeout (e.g., VM state == Running/Ready, tools/agent reported, or an API-level 'isready' signal) before attempting migration. This will improve both reliability and runtime; same recommendation applies to the other newly-added time.sleep(10) calls in this PR.
 time.sleep(10) # Wait for VM to boot into OS before migrating the VM
vm.migrate(self.apiclient, self.host_h2.id)

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18697

@DaanHoogland

Copy link
Copy Markdown
Contributor

@blueorangutan test ubuntu26 xcpng82

@blueorangutan

Copy link
Copy Markdown

@DaanHoogland a [SL] Trillian-Jenkins test job (ubuntu26 mgmt + xcpng82) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian Build Failed (tid-14)

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian test result (tid-16642)
Environment: vmware-80u3 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 77858 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13628-t16642-vmware-80u3.zip
Smoke tests completed. 154 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

TestResultTime (s)Test File
test_05_list_volumes_isrecursiveFailure0.04test_list_volumes.py
test_07_list_volumes_listallFailure0.03test_list_volumes.py
test_list_vms_metrics_adminError3620.69test_metrics_api.py
test_list_vms_metrics_historyError4.95test_metrics_api.py
test_list_volumes_metrics_historyError3623.56test_metrics_api.py

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian test result (tid-16649)
Environment: xcpng82 (x2), zone: Advanced Networking with Mgmt server ol9
Total time taken: 78485 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13628-t16649-xcpng82.zip
Smoke tests completed. 154 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

TestResultTime (s)Test File
test_02_list_cpvm_vmFailure0.05test_ssvm.py
test_04_cpvm_internalsFailure0.06test_ssvm.py
test_02_cancel_host_maintenace_with_migration_jobsError0.59test_host_maintenance.py
test_03_cancel_host_maintenace_with_migration_jobs_failureError0.59test_host_maintenance.py

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian test result (tid-16651)
Environment: vmware-80u3 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 134966 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13628-t16651-vmware-80u3.zip
Smoke tests completed. 152 look OK, 4 have errors, 0 did not run
Only failed and skipped tests results shown below:

TestResultTime (s)Test File
test_05_list_volumes_isrecursiveFailure0.03test_list_volumes.py
test_07_list_volumes_listallFailure0.03test_list_volumes.py
test_01_deploy_vm_on_specific_hostError3602.44test_vm_deployment_planner.py
test_02_deploy_vm_on_specific_clusterError1.31test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_podError4.36test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_clusterError4.44test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_podError1.26test_vm_deployment_planner.py
test_03_live_migrate_VM_with_two_data_disksError40.11test_vm_life_cycle.py
test_08_migrate_vmError27.11test_vm_life_cycle.py
test_09_expunge_vmFailure432.04test_vm_life_cycle.py
test_01_migrate_vm_strict_tags_successError3603.92test_vm_strict_host_tags.py
test_02_migrate_vm_strict_tags_failureError9.00test_vm_strict_host_tags.py
ContextSuite context=TestMigrateVMStrictTags>:teardownError10.28test_vm_strict_host_tags.py
test_01_restore_vm_strict_tags_successError25.13test_vm_strict_host_tags.py
test_02_restore_vm_strict_tags_failureError3608.35test_vm_strict_host_tags.py
test_01_scale_vm_strict_tags_successError24.31test_vm_strict_host_tags.py
test_02_scale_vm_strict_tags_failureError3602.36test_vm_strict_host_tags.py
ContextSuite context=TestScaleVMStrictTags>:teardownError3604.89test_vm_strict_host_tags.py
test_01_deploy_vm_on_specific_host_without_strict_tagsError25.40test_vm_strict_host_tags.py
test_02_deploy_vm_on_any_host_without_strict_tagsError3609.18test_vm_strict_host_tags.py
test_03_deploy_vm_on_specific_host_with_strict_tags_successError23.61test_vm_strict_host_tags.py
test_04_deploy_vm_on_any_host_with_strict_tags_successError3610.25test_vm_strict_host_tags.py

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian Build Failed (tid-18)

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian Build Failed (tid-19)

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian Build Failed (tid-20)

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian test result (tid-17)
Environment: xcpng82 (x2), zone: Advanced Networking with Mgmt server u26
Total time taken: 67811 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13628-n17-xcpng82.zip
Smoke tests completed. 155 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

TestResultTime (s)Test File
test_02_cancel_host_maintenace_with_migration_jobsError2090.29test_host_maintenance.py
test_02_cancel_host_maintenace_with_migration_jobsError2090.38test_host_maintenance.py

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian test result (tid-16696)
Environment: vmware-80u3 (x2), zone: Advanced Networking with Mgmt server u24
Total time taken: 88037 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13628-t16696-vmware-80u3.zip
Smoke tests completed. 147 look OK, 9 have errors, 0 did not run
Only failed and skipped tests results shown below:

TestResultTime (s)Test File
test_08_arping_in_ssvmFailure25.08test_diagnostics.py
test_09_arping_in_cpvmFailure9.55test_diagnostics.py
test_deploy_more_vms_than_limit_allowsError193.66test_deploy_vms_in_parallel.py
test_13_test_add_external_nodes_to_clusterFailure1102.87test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardownError1188.22test_kubernetes_clusters.py
test_05_list_volumes_isrecursiveFailure0.07test_list_volumes.py
test_07_list_volumes_listallFailure0.06test_list_volumes.py
test_list_vms_metrics_adminError3640.82test_metrics_api.py
test_list_vms_metrics_historyError6.66test_metrics_api.py
test_list_volumes_metrics_historyError3635.50test_metrics_api.py
test_01_vpc_privategw_aclFailure6.51test_privategw_acl.py
test_02_vpc_privategw_static_routesFailure6.56test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanupFailure6.62test_privategw_acl.py
test_04_rvpc_privategw_static_routesFailure6.37test_privategw_acl.py
test_01_restore_vmError4.63test_restore_vm.py
test_02_restore_vm_with_disk_offeringError7.67test_restore_vm.py
test_03_restore_vm_with_disk_offering_custom_sizeError7.91test_restore_vm.py
test_04_restore_vm_allocated_rootError9.96test_restore_vm.py
ContextSuite context=TestRestoreVM>:teardownError22.64test_restore_vm.py
test_01_deploy_vm_on_specific_hostError4.46test_vm_deployment_planner.py
test_02_deploy_vm_on_specific_clusterError1.37test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_podError4.48test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_clusterError2.45test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_podError1.39test_vm_deployment_planner.py
test_01_migrate_vm_strict_tags_successError9.18test_vm_strict_host_tags.py
test_02_migrate_vm_strict_tags_failureError6.07test_vm_strict_host_tags.py
test_01_restore_vm_strict_tags_successError6.18test_vm_strict_host_tags.py
test_02_restore_vm_strict_tags_failureError6.26test_vm_strict_host_tags.py
test_01_scale_vm_strict_tags_successError9.14test_vm_strict_host_tags.py
test_02_scale_vm_strict_tags_failureError9.28test_vm_strict_host_tags.py
test_01_deploy_vm_on_specific_host_without_strict_tagsError9.05test_vm_strict_host_tags.py
test_02_deploy_vm_on_any_host_without_strict_tagsError12.10test_vm_strict_host_tags.py
test_03_deploy_vm_on_specific_host_with_strict_tags_successError9.04test_vm_strict_host_tags.py
test_04_deploy_vm_on_any_host_with_strict_tags_successError15.16test_vm_strict_host_tags.py

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian test result (tid-23)
Environment: vmware-80u3 (x2), zone: Advanced Networking with Mgmt server u26
Total time taken: 150749 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13628-n23-vmware-80u3.zip
Smoke tests completed. 150 look OK, 6 have errors, 0 did not run
Only failed and skipped tests results shown below:

TestResultTime (s)Test File
test_deploy_more_vms_than_limit_allowsError192.00test_deploy_vms_in_parallel.py
test_08_upgrade_kubernetes_ha_clusterFailure747.89test_kubernetes_clusters.py
test_13_test_add_external_nodes_to_clusterFailure1078.03test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardownError1169.81test_kubernetes_clusters.py
test_05_list_volumes_isrecursiveFailure0.08test_list_volumes.py
test_07_list_volumes_listallFailure0.06test_list_volumes.py
test_01_deploy_vm_on_specific_hostError17.25test_vm_deployment_planner.py
test_02_deploy_vm_on_specific_clusterError3603.00test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_podError4.78test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_clusterError2.69test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_podError2.77test_vm_deployment_planner.py
test_08_migrate_vmError31.83test_vm_life_cycle.py
test_01_migrate_vm_strict_tags_successError3608.28test_vm_strict_host_tags.py
test_02_migrate_vm_strict_tags_failureError3609.54test_vm_strict_host_tags.py
test_01_restore_vm_strict_tags_successError3603.04test_vm_strict_host_tags.py
test_02_restore_vm_strict_tags_failureError7.93test_vm_strict_host_tags.py
ContextSuite context=TestRestoreVMStrictTags>:teardownError10.48test_vm_strict_host_tags.py
test_01_scale_vm_strict_tags_successError20.64test_vm_strict_host_tags.py
test_02_scale_vm_strict_tags_failureError7206.52test_vm_strict_host_tags.py
test_01_deploy_vm_on_specific_host_without_strict_tagsError3602.53test_vm_strict_host_tags.py
test_02_deploy_vm_on_any_host_without_strict_tagsError7.67test_vm_strict_host_tags.py
test_03_deploy_vm_on_specific_host_with_strict_tags_successError8.01test_vm_strict_host_tags.py
test_04_deploy_vm_on_any_host_with_strict_tags_successError7219.92test_vm_strict_host_tags.py
ContextSuite context=TestVMDeploymentPlannerStrictTags>:teardownError8.12test_vm_strict_host_tags.py

@weizhouapache

Copy link
Copy Markdown
MemberAuthor

moving to 4.22.2 milestone as it fixes smoke tests, not functionalities

@weizhouapacheweizhouapache modified the milestones: 4.23.0, 4.22.2Aug 6, 2026
@blueorangutan

Copy link
Copy Markdown

[SF] Trillian test result (tid-25)
Environment: kvm-ol10 (x2), zone: Advanced Networking with Mgmt server ol10
Total time taken: 62835 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13628-n25-kvm-ol10.zip
Smoke tests completed. 155 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

TestResultTime (s)Test File
test_01_ssl_offloading_isolated_networkFailure718.18test_ssl_offloading.py

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@weizhouapache@blueorangutan@DaanHoogland@vladimirpetrov