server: investigate pending HA work when executing in new MS session by shwstppr · Pull Request #10167 · apache/cloudstack

shwstppr · 2025-01-08T11:00:02Z

Description

For HA work items that are created for host state change, checks must be done when execution is called in a new management server session.

A new column, reason, has been added in cloud.op_ha_work table to track the reason for HA work.

When HighAvailabilityManager starts it finds and puts all pending HA work items in Investigating state. During execution of the HA work if it is found in investigating state, checks are done to verify if the work is still valid. If the jobs is found to be invalid it is cancelled.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

For HA work items that are created for host state change, checks must be done when execution is called in a new management server session. A new column, reason, has been added in cloud.op_ha_work table to track the reason for HA work. When HighAvailabilityManager starts it finds and puts all pending HA work items in Investigating state. During execution of the HA work if it is found in investigating state, checks are done to verify if the work is still valid. If the jobs is found to be invalid it is cancelled. Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

codecov · 2025-01-08T11:08:20Z

Codecov Report

Attention: Patch coverage is 71.76471% with 24 lines in your changes missing coverage. Please review.

Project coverage is 16.99%. Comparing base (bd488c4) to head (9c18920).
Report is 38 commits behind head on 4.20.

Files with missing lines	Patch %	Lines
...java/com/cloud/ha/HighAvailabilityManagerImpl.java	74.41%	10 Missing and 1 partial ⚠️
server/src/main/java/com/cloud/ha/HaWorkVO.java	25.00%	6 Missing ⚠️
...n/java/com/cloud/resource/ResourceManagerImpl.java	0.00%	5 Missing ⚠️
...java/com/cloud/agent/manager/AgentManagerImpl.java	0.00%	1 Missing ⚠️
...stack/ha/provider/host/HAAbstractHostProvider.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@ Coverage Diff @@## 4.20 #10167 +/- ##
============================================
+ Coverage 16.07% 16.99% +0.92% - Complexity 12884 12977 +93 
============================================
Files 5639 5245 -394 Lines 494184 461985 -32199 Branches 59924 54191 -5733 ============================================
- Hits 79420 78513 -907 + Misses 405934 374799 -31135 + Partials 8830 8673 -157

Flag	Coverage Δ
uitests	`?`
unittests	`16.99% <71.76%> (+0.07%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

shwstppr · 2025-01-08T11:09:53Z

@blueorangutan package

blueorangutan · 2025-01-08T11:12:05Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

sureshanaparti

clgtm

blueorangutan · 2025-01-08T12:26:50Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12012

shwstppr · 2025-01-08T12:30:21Z

@blueorangutan test

blueorangutan · 2025-01-08T12:32:04Z

@shwstppr a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-01-09T03:59:59Z

[SF] Trillian test result (tid-12060)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 52979 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10167-t12060-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_11_isolated_network_with_dynamic_routed_mode	`Error`	2.28	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Error`	2.50	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Error`	2.50	test_ipv4_routing.py

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr · 2025-01-09T09:49:47Z

@blueorangutan package

blueorangutan · 2025-01-09T09:52:04Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-01-09T11:08:28Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12025

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

sureshanaparti · 2025-01-10T08:00:11Z

@blueorangutan package

blueorangutan · 2025-01-10T08:02:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-01-10T09:17:00Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12041

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr · 2025-01-21T10:48:24Z

@blueorangutan package

blueorangutan · 2025-01-21T10:50:03Z

@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-01-21T12:16:28Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12145

kiranchavala · 2025-01-27T04:27:51Z

@blueorangutan test

blueorangutan · 2025-01-27T04:30:04Z

@kiranchavala a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-01-27T18:48:55Z

[SF] Trillian test result (tid-12189)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 49255 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10167-t12189-kvm-ol8.zip
Smoke tests completed. 139 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_11_isolated_network_with_dynamic_routed_mode	`Error`	0.24	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Error`	2.70	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Error`	2.70	test_ipv4_routing.py
test_02_list_cpvm_vm	`Failure`	0.04	test_ssvm.py
test_04_cpvm_internals	`Failure`	0.04	test_ssvm.py

kiranchavala

LGTM , Tested the scenario of HostMaintenance, HostDown and HostDegraded.

In case of management server restart, found that cloudstack validates the entries in op_ha_work table and takes action based on the host state

Test Case Execution	Result
Test host maintainence scenario	Pass
Test host ha /host down scenario	Pass

* 4.20: server: investigate pending HA work when executing in new MS session (#10167) extra null guard (#10264)

…ession (apache#519) Fixesapache#511 For HA work items that are created for host state change, checks must be done when execution is called in a new management server session. A new column, reason, has been added in cloud.op_ha_work table to track the reason for HA work. When HighAvailabilityManager starts it finds and puts all pending HA work items in Investigating state. During execution of the HA work if it is found in investigating state, checks are done to verify if the work is still valid. If the jobs is found to be invalid it is cancelled. Upstream PR: apache#10167 Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

…pache#10167) For HA work items that are created for host state change, checks must be done when execution is called in a new management server session. A new column, reason, has been added in cloud.op_ha_work table to track the reason for HA work. When HighAvailabilityManager starts it finds and puts all pending HA work items in Investigating state. During execution of the HA work if it is found in investigating state, checks are done to verify if the work is still valid. If the jobs is found to be invalid it is cancelled. Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

boring-cyborgBot added component:agent component:orchestration labels Jan 8, 2025

sureshanaparti approved these changes Jan 8, 2025
View reviewed changes

DaanHoogland added this to the 4.20.1 milestone Jan 8, 2025

fix for one managment server shutdown
e2a6450
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

fix unused aurgument
04913a4
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

fix
9c18920
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

kiranchavala approved these changes Jan 28, 2025
View reviewed changes

yadvr marked this pull request as ready for review January 28, 2025 09:08

yadvr approved these changes Jan 28, 2025
View reviewed changes

yadvr merged commit 33a37da into apache:4.20Jan 28, 2025

yadvr deleted the investigate-pending-ha-420 branch January 28, 2025 09:09

DaanHoogland added a commit that referenced this pull request Jan 28, 2025
Merge release branch 4.20 to main
048649d
* 4.20: server: investigate pending HA work when executing in new MS session (#10167) extra null guard (#10264)

Pearl1594 added this to ACS 4.20.1 Mar 17, 2025

Pearl1594 moved this to Done in ACS 4.20.1 Mar 17, 2025

Uh oh!

Conversation

shwstppr commented Jan 8, 2025

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecovBot commented Jan 8, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

shwstppr commented Jan 8, 2025

Uh oh!

blueorangutan commented Jan 8, 2025

Uh oh!

sureshanaparti left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Jan 8, 2025

Uh oh!

shwstppr commented Jan 8, 2025

Uh oh!

blueorangutan commented Jan 8, 2025

Uh oh!

blueorangutan commented Jan 9, 2025

Uh oh!

shwstppr commented Jan 9, 2025

Uh oh!

blueorangutan commented Jan 9, 2025

Uh oh!

blueorangutan commented Jan 9, 2025

Uh oh!

sureshanaparti commented Jan 10, 2025

Uh oh!

blueorangutan commented Jan 10, 2025

Uh oh!

blueorangutan commented Jan 10, 2025

Uh oh!

shwstppr commented Jan 21, 2025

Uh oh!

blueorangutan commented Jan 21, 2025

Uh oh!

blueorangutan commented Jan 21, 2025

Uh oh!

kiranchavala commented Jan 27, 2025

Uh oh!

blueorangutan commented Jan 27, 2025

Uh oh!

blueorangutan commented Jan 27, 2025

Uh oh!

kiranchavala left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

codecovBot commented Jan 8, 2025•
edited
Loading