Improvement: management server peer states by weizhouapache · Pull Request #9885 · apache/cloudstack

weizhouapache · 2024-11-04T08:35:13Z

Description

This PR improves the peers management servers and provides better peer states based on the mgmt server statistics sync.

Old diagram:

new diagram:

Test results:

Actions	Current	New
Actions	Current	New	stop cloudstack-management	mshost.state is Down immediately, mshost_peer.peer_state is Down after 3mins	Same as current
restart cloudstack-management	mshost.state is Down immediately and Up later. mshost_peer.peer_state is Up during the period. There are some leftover peers with old peer_runid.	mshost.state is Down immediately and Up later. mshost_peer.peer_state is Up during the period. records with old peer_runid is removed.
pkill java	same as "stop cloudstack-management"	Same as current
kill -9 `pid of java process`	mshost.state is still Up. mshost_peer.peer_state is Down after 3 mins.	mshost.state and mshost_peer.peer_state are Down after 3-4 mins.
hard shutdown (echo o > /proc/sysrq-trigger)	same as "kill -9"	mshost.state and mshost_peer.peer_state are Down after 3-4 mins.
hard reset (echo b > /proc/sysrq-trigger)	same as "kill -9"	mshost.state and mshost_peer.peer_state are Down after 3-4 mins.
cannot write to db	same as "kill -9". When link is recovered, exit with code 219	mshost.state and mshost_peer.peer_state are Down after 3-4 mins. When link is recovered, exit with code 219
cannot communicate with other mgmt nodes	nothing changed	Same as current
mgmt link is down	same as "kill -9" as eth0 is also used to connect to db. otherwise it should be same as "cannot communicate with other nodes"	Same as current

New tab "Peers" for each management server on UI.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

weizhouapache · 2024-11-04T08:36:21Z

@blueorangutan package

blueorangutan · 2024-11-04T08:38:04Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

weizhouapache · 2024-11-04T08:46:42Z

@blueorangutan package

codecov · 2024-11-04T08:47:01Z

Codecov Report

Attention: Patch coverage is 0% with 231 lines in your changes missing coverage. Please review.

Project coverage is 15.98%. Comparing base (f7b7339) to head (880c3d9).
Report is 22 commits behind head on 4.20.

Files with missing lines	Patch %	Lines
.../cloud/cluster/ManagementServerHostPeerJoinVO.java	0.00%	60 Missing ⚠️
...ain/java/com/cloud/cluster/ClusterManagerImpl.java	0.00%	50 Missing ⚠️
...api/response/PeerManagementServerNodeResponse.java	0.00%	28 Missing ⚠️
...d/cluster/dao/ManagementServerHostPeerDaoImpl.java	0.00%	27 Missing ⚠️
...ain/java/com/cloud/api/query/QueryManagerImpl.java	0.00%	22 Missing ⚠️
...src/main/java/com/cloud/server/StatsCollector.java	0.00%	12 Missing ⚠️
...udstack/api/response/ManagementServerResponse.java	0.00%	11 Missing ⚠️
...uster/dao/ManagementServerHostPeerJoinDaoImpl.java	0.00%	11 Missing ⚠️
...m/cloud/server/ManagementServerHostStatsEntry.java	0.00%	6 Missing ⚠️
...ack/api/command/admin/management/ListMgmtsCmd.java	0.00%	3 Missing ⚠️
... and 1 more

Additional details and impacted files

@@ Coverage Diff @@## 4.20 #9885 +/- ##
============================================
+ Coverage 15.80% 15.98% +0.17% - Complexity 12586 12772 +186 
============================================
Files 5627 5631 +4 Lines 492294 499767 +7473 Branches 62198 63247 +1049 ============================================
+ Hits 77824 79869 +2045 - Misses 405946 411052 +5106 - Partials 8524 8846 +322

Flag	Coverage Δ
uitests	`4.04% <ø> (-0.01%)`	⬇️
unittests	`16.62% <0.00%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

blueorangutan · 2024-11-04T09:48:34Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11492

weizhouapache · 2024-11-04T09:54:41Z

@blueorangutan package

blueorangutan · 2024-11-04T09:56:06Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-11-04T10:58:52Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11495

weizhouapache · 2024-11-04T11:32:51Z

@blueorangutan test

blueorangutan · 2024-11-04T11:34:03Z

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2024-11-05T01:44:16Z

[SF] Trillian test result (tid-11740)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 48795 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9885-t11740-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

…tServerNodeId

weizhouapache · 2024-11-05T12:07:30Z

@blueorangutan package

blueorangutan · 2024-11-05T12:08:07Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-11-05T13:32:47Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11507

weizhouapache · 2024-11-05T15:42:20Z

@blueorangutan test

blueorangutan · 2024-11-05T15:44:04Z

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2024-11-06T07:11:44Z

[SF] Trillian test result (tid-11747)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 53263 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9885-t11747-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_secure_vm_migration	`Error`	134.23	test_vm_life_cycle.py
test_01_secure_vm_migration	`Error`	134.23	test_vm_life_cycle.py

blueorangutan · 2024-11-27T19:58:06Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-11-27T20:52:03Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✔️ debian ✖️ suse15. SL-JID 11619

weizhouapache · 2024-11-28T08:03:45Z

@blueorangutan package

blueorangutan · 2024-11-28T08:06:13Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-11-28T09:07:02Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 11627

blueorangutan · 2024-11-28T10:17:23Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 11628

blueorangutan · 2024-11-28T13:01:15Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11637

weizhouapache · 2024-11-28T13:02:52Z

@blueorangutan test

blueorangutan · 2024-11-28T13:04:09Z

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2024-11-28T14:06:48Z

[SF] Trillian Build Failed (tid-11812)

yadvr · 2024-11-29T07:31:49Z

@blueorangutan package

blueorangutan · 2024-11-29T07:34:03Z

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

weizhouapache · 2024-11-29T08:09:18Z

@blueorangutan package

blueorangutan · 2024-11-29T08:10:04Z

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-11-29T08:23:21Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 11646

blueorangutan · 2024-11-29T09:40:58Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11649

weizhouapache · 2024-11-29T09:43:26Z

@blueorangutan test

blueorangutan · 2024-11-29T09:44:03Z

@weizhouapache a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2024-11-30T01:18:58Z

[SF] Trillian test result (tid-11819)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 53638 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9885-t11819-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

* Improvement: management server peer states * Update pr9885: consider new mgmt server node which has msId=managementServerNodeId * Update pr9885: update global config description * Update pr9885: update label on UI * framework: Do not update mshost_peer when mgmt server is Up as it will be updated by status update * mgmt: Update state to Up when mgmt server writes heartbeat to db * mgmt: change Service IP to Management IP --------- Co-authored-by: Boris Stoyanov - a.k.a Bobby <bss.stoyanov@gmail.com>

Improvement: management server peer states
861da3b

boring-cyborgBot added component:api component:management-server component:UI labels Nov 4, 2024

weizhouapache modified the milestones: 4.21.0.0, 4.20.1.0 Nov 4, 2024

weizhouapache commented Nov 4, 2024
View reviewed changes

Comment threadapi/src/main/java/org/apache/cloudstack/api/command/admin/management/ListMgmtsCmd.java

weizhouapache force-pushed the 4.20-mgmt-server-peers branch from ca8a951 to 861da3bCompare November 4, 2024 08:39

weizhouapache added 4 commits November 5, 2024 12:47

Update pr9885: consider new mgmt server node which has msId=managemen…
7e8535c
…tServerNodeId

Update pr9885: update global config description
641e841

Update pr9885: update label on UI
d20ac27

Merge remote-tracking branch 'apache/main' into 4.20-mgmt-server-peers
9423dcc

yadvr marked this pull request as ready for review November 29, 2024 07:31

weizhouapache closed this Nov 30, 2024

weizhouapache reopened this Nov 30, 2024

yadvr merged commit 34056d9 into apache:4.20Dec 2, 2024

yadvr removed the status:ready-for-merge label Dec 2, 2024

DaanHoogland deleted the 4.20-mgmt-server-peers branch December 2, 2024 07:55

harikrishna-patnala mentioned this pull request Dec 2, 2024
engine/schema: move SQLs to 4.20.0 to 4.20.1 upgrade #10018
Merged
14 tasks

Pearl1594 added this to ACS 4.20.1 Mar 17, 2025

Pearl1594 moved this to Done in ACS 4.20.1 Mar 17, 2025

weizhouapache mentioned this pull request May 6, 2025
test: fix test_hostha_simulator.py and test_outofbandmanagement.py #10815
Merged
14 tasks

Uh oh!

Conversation

weizhouapache commented Nov 4, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

Uh oh!

weizhouapache commented Nov 4, 2024

Uh oh!

blueorangutan commented Nov 4, 2024

Uh oh!

weizhouapache commented Nov 4, 2024

Uh oh!

codecovBot commented Nov 4, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

blueorangutan commented Nov 4, 2024

Uh oh!

weizhouapache commented Nov 4, 2024

Uh oh!

blueorangutan commented Nov 4, 2024

Uh oh!

blueorangutan commented Nov 4, 2024

Uh oh!

weizhouapache commented Nov 4, 2024

Uh oh!

blueorangutan commented Nov 4, 2024

Uh oh!

blueorangutan commented Nov 5, 2024

Uh oh!

weizhouapache commented Nov 5, 2024

Uh oh!

blueorangutan commented Nov 5, 2024

Uh oh!

blueorangutan commented Nov 5, 2024

Uh oh!

weizhouapache commented Nov 5, 2024

Uh oh!

blueorangutan commented Nov 5, 2024

Uh oh!

blueorangutan commented Nov 6, 2024

Uh oh!

blueorangutan commented Nov 27, 2024

Uh oh!

blueorangutan commented Nov 27, 2024

Uh oh!

weizhouapache commented Nov 28, 2024

Uh oh!

blueorangutan commented Nov 28, 2024

Uh oh!

blueorangutan commented Nov 28, 2024

Uh oh!

blueorangutan commented Nov 28, 2024

Uh oh!

blueorangutan commented Nov 28, 2024

Uh oh!

weizhouapache commented Nov 28, 2024

Uh oh!

blueorangutan commented Nov 28, 2024

Uh oh!

blueorangutan commented Nov 28, 2024

Uh oh!

yadvr commented Nov 29, 2024

Uh oh!

blueorangutan commented Nov 29, 2024

Uh oh!

weizhouapache commented Nov 29, 2024

Uh oh!

blueorangutan commented Nov 29, 2024

Uh oh!

blueorangutan commented Nov 29, 2024

Uh oh!

weizhouapache commented Nov 4, 2024•
edited
Loading

codecovBot commented Nov 4, 2024•
edited
Loading