use plays health in health check - #12385

Merged
alecsavvy merged 2 commits into
mainfrom
as/fix-health
Jun 24, 2025
Merged

use plays health in health check#12385
alecsavvy merged 2 commits into
mainfrom
as/fix-health

Conversation

@alecsavvy

@alecsavvyalecsavvy commented Jun 23, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

Description

Still unsure why this health source can return 0, seems like some kind of race condition. I did find that when it errors the plays health is still reliable. To fix the intermittent issue we'll use the plays latest chain block and fallback to the one we're currently using. Plays block is just fine to use here as it's the same chain as entity manager.

Also updates healthz to route to console as those old grpc rest endpoints don't exist anymore.

bad health check as example, you can see the plays area still has the right value even though other spots have 0:

{
"antiAbuseWalletPubkey": null,
"comms": {
"booted": "2025-06-23T18:16:44Z",
"built": "",
"commit": "",
"errors": null,
"healthy": true,
"host": "https://discoveryprovider3.audius.co",
"is_registered_wallet": true,
"wallet": "0xF2897993951d53a7E3eb2242D6A14D2028140DC8",
"websocket_error": null
},
"data": {
"audius-docker-compose": "63e210459090bef4746c41341409a1f4646f7b5a",
"audius_d_managed": true,
"auto_upgrade_enabled": true,
"block_difference": 5852972,
"challenge_last_event_age_sec": 0,
"core": {
"health": {
"chain_id": "audius-mainnet-alpha-beta",
"indexing_entity_manager": true,
"indexing_plays": true,
"latest_chain_block": 5852972,
"latest_chain_block_ts": 1750716115,
"latest_indexed_block": 5852972
},
"listens": {
"core_block_cutover": 113180,
"is_unhealthy": false,
"latest_chain_slot": 5852972,
"latest_indexed_slot": 5852972,
"slot_diff": 0,
"sol_slot_cutover": 315925000,
"time_diff": 2.152073,
"tx_info": {
"chain_tx": null,
"db_tx": {
"signature": "",
"slot": 5852972,
"timestamp": 1750716115
}
}
}
},
"database_connections": 67,
"database_is_localhost": true,
"database_size": 173582922543,
"db": {
"blockhash": "0f0afc80053e2803e1b147532c9a5af660d946c821c11ba07bf4f62d091bc84e",
"number": 5852972
},
"discovery_provider_healthy": false,
"errors": [
"unhealthy block diff"
],
"filesystem_size": 1065418129408,
"filesystem_used": 567194726400,
"final_poa_block": 31413000,
"git": "cf70425a041ab8b442795dbeb80533cbc5cfdf7c",
"index_eth_age_sec": 125,
"infra_setup": "audius-docker-compose",
"last_scanned_block_for_balance_refresh": 22769967,
"latest_block_num": 0,
"latest_indexed_block_num": 5852972,
"maximum_healthy_block_difference": 100,
"meets_min_requirements": true,
"network": {
...
},
"num_users_in_immediate_balance_refresh_queue": 1,
"num_users_in_lazy_balance_refresh_queue": 2384591,
"number_of_cpus": 48,
"plays": {
"core_block_cutover": 113180,
"is_unhealthy": false,
"latest_chain_slot": 5852972,
"latest_indexed_slot": 5852972,
"slot_diff": 0,
"sol_slot_cutover": 315925000,
"time_diff": 2.152073,
"tx_info": {
"chain_tx": null,
"db_tx": {
"signature": "",
"slot": 5852972,
"timestamp": 1750716115
}
}
},
"received_bytes_per_sec": 0,
"redis_total_memory": 67508137984,
"relay": {
"is_unhealthy": false,
"status": "up"
},
"service": "discovery-node",
"solana_indexers": {
"aggregate_tips": {
"is_healthy": true,
"last_completed_at": 1750716027.550774,
"last_tx": "oU4kgkWr9Ditwq5EaT1mdG1zohkSh5yWra1iXaJc9nvaPrXSFFdi11cuwqWMUZt11CfyqrcioheQntsGsbZq8Z8",
"since_last_completed_at": 91.49335789680481
},
"payment_router": {
"is_healthy": true,
"last_completed_at": 1750716118.83112,
"last_tx": "3i1mGzNrprWcW3PnSRYognDXezFas55fzpP8aCfDSJePg6pc3mWq5LGRheAd8j733rdFyN75GprL2th5i3jB8HDH",
"since_last_completed_at": 0.21285104751586914
},
"reward_manager": {
"is_healthy": true,
"last_completed_at": 1750716118.184873,
"last_tx": "37776jhVY1vYvtSxtDNp4roo75H4iPEDN3w6bP388yWfTVWXao4ZLuPco8X3CVbdAkZDfkrZ8Vr6w6EjC6qjnVHH",
"since_last_completed_at": 0.8589239120483398
},
"spl_token": {
"is_healthy": true,
"last_completed_at": 1750716025.788301,
"last_tx": "4UQP7hvmmyjHAFEz3967rgVFRPbnsLVtNKJwgi35Eix6FVnSBT3Th9dwcq2ymrGoRDCjQTgw4TKmawpFMbqh2VzA",
"since_last_completed_at": 93.25533294677734
},
"user_bank": {
"is_healthy": true,
"last_completed_at": 1750716118.978945,
"last_tx": "4UQP7hvmmyjHAFEz3967rgVFRPbnsLVtNKJwgi35Eix6FVnSBT3Th9dwcq2ymrGoRDCjQTgw4TKmawpFMbqh2VzA",
"since_last_completed_at": 0.06445789337158203
}
},
"total_memory": 67508137984,
"transferred_bytes_per_sec": 0,
"trending_playlists_age_sec": 3555678,
"trending_tracks_age_sec": 2734,
"url": "https://discoveryprovider3.audius.co",
"used_memory": 27486191616,
"user_balances_age_sec": 153,
"version": "0.7.182",
"web": {
"blockhash": null,
"blocknumber": 0
}
},
"latest_chain_block": null,
"latest_chain_slot_plays": 5852972,
"latest_indexed_block": 5852972,
"latest_indexed_slot_plays": 5852972,
"signer": "0xf2897993951d53a7e3eb2242d6a14d2028140dc8",
"success": true,
"version": {
"service": "discovery-node",
"version": "0.7.182"
}
}

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide repro instructions & any configuration.

@alecsavvyalecsavvy self-assigned this Jun 23, 2025
@changeset-bot

changeset-botBot commented Jun 23, 2025•
edited
Loading

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: e58597c

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

raymondjacobson
raymondjacobson approved these changes Jun 23, 2025
View reviewed changes

@raymondjacobsonraymondjacobson left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay as a bandaid. If we weren't rewriting indexing, I'd really dislike this, but it's already so screwed up as is, I'm down with this temp solve

@alecsavvy
alecsavvy enabled auto-merge (squash) June 23, 2025 22:41
@alecsavvy
alecsavvy merged commit d4ecccf into mainJun 24, 2025
@alecsavvy
alecsavvy deleted the as/fix-health branch June 24, 2025 15:53
isaacsolo pushed a commit that referenced this pull request Jun 24, 2025
@alecsavvyisaac
use plays health in health check (#12385)
fc0920d
audius-infra pushed a commit that referenced this pull request Jun 25, 2025
audius-infra
Audius Protocol v0.7.184
16ea1af
[d4ecccf] use plays health in health check (#12385) alecsavvy
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@raymondjacobsonraymondjacobsonraymondjacobson approved these changes

Assignees

@alecsavvyalecsavvy

Labels

size/M

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@alecsavvy@raymondjacobson
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
use plays health in health check - #12385
Merged
alecsavvy merged 2 commits into
mainfrom 
as/fix-health
Jun 24, 2025
Merged
use plays health in health check#12385
alecsavvy merged 2 commits into
mainfrom 
as/fix-health
Conversation
@alecsavvy
@alecsavvyalecsavvy
commented
Jun 23, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
Description
Still unsure why this health source can return 0, seems like some kind of race condition. I did find that when it errors the plays health is still reliable. To fix the intermittent issue we'll use the plays latest chain block and fallback to the one we're currently using. Plays block is just fine to use here as it's the same chain as entity manager.
Also updates healthz to route to console as those old grpc rest endpoints don't exist anymore.
bad health check as example, you can see the plays area still has the right value even though other spots have 0:
{
"antiAbuseWalletPubkey": null,
"comms": {
"booted": "2025-06-23T18:16:44Z",
"built": "",
"commit": "",
"errors": null,
"healthy": true,
"host": "https://discoveryprovider3.audius.co",
"is_registered_wallet": true,
"wallet": "0xF2897993951d53a7E3eb2242D6A14D2028140DC8",
"websocket_error": null
},
"data": {
"audius-docker-compose": "63e210459090bef4746c41341409a1f4646f7b5a",
"audius_d_managed": true,
"auto_upgrade_enabled": true,
"block_difference": 5852972,
"challenge_last_event_age_sec": 0,
"core": {
"health": {
"chain_id": "audius-mainnet-alpha-beta",
"indexing_entity_manager": true,
"indexing_plays": true,
"latest_chain_block": 5852972,
"latest_chain_block_ts": 1750716115,
"latest_indexed_block": 5852972
},
"listens": {
"core_block_cutover": 113180,
"is_unhealthy": false,
"latest_chain_slot": 5852972,
"latest_indexed_slot": 5852972,
"slot_diff": 0,
"sol_slot_cutover": 315925000,
"time_diff": 2.152073,
"tx_info": {
"chain_tx": null,
"db_tx": {
"signature": "",
"slot": 5852972,
"timestamp": 1750716115
}
}
}
},
"database_connections": 67,
"database_is_localhost": true,
"database_size": 173582922543,
"db": {
"blockhash": "0f0afc80053e2803e1b147532c9a5af660d946c821c11ba07bf4f62d091bc84e",
"number": 5852972
},
"discovery_provider_healthy": false,
"errors": [
"unhealthy block diff"
],
"filesystem_size": 1065418129408,
"filesystem_used": 567194726400,
"final_poa_block": 31413000,
"git": "cf70425a041ab8b442795dbeb80533cbc5cfdf7c",
"index_eth_age_sec": 125,
"infra_setup": "audius-docker-compose",
"last_scanned_block_for_balance_refresh": 22769967,
"latest_block_num": 0,
"latest_indexed_block_num": 5852972,
"maximum_healthy_block_difference": 100,
"meets_min_requirements": true,
"network": {
...
},
"num_users_in_immediate_balance_refresh_queue": 1,
"num_users_in_lazy_balance_refresh_queue": 2384591,
"number_of_cpus": 48,
"plays": {
"core_block_cutover": 113180,
"is_unhealthy": false,
"latest_chain_slot": 5852972,
"latest_indexed_slot": 5852972,
"slot_diff": 0,
"sol_slot_cutover": 315925000,
"time_diff": 2.152073,
"tx_info": {
"chain_tx": null,
"db_tx": {
"signature": "",
"slot": 5852972,
"timestamp": 1750716115
}
}
},
"received_bytes_per_sec": 0,
"redis_total_memory": 67508137984,
"relay": {
"is_unhealthy": false,
"status": "up"
},
"service": "discovery-node",
"solana_indexers": {
"aggregate_tips": {
"is_healthy": true,
"last_completed_at": 1750716027.550774,
"last_tx": "oU4kgkWr9Ditwq5EaT1mdG1zohkSh5yWra1iXaJc9nvaPrXSFFdi11cuwqWMUZt11CfyqrcioheQntsGsbZq8Z8",
"since_last_completed_at": 91.49335789680481
},
"payment_router": {
"is_healthy": true,
"last_completed_at": 1750716118.83112,
"last_tx": "3i1mGzNrprWcW3PnSRYognDXezFas55fzpP8aCfDSJePg6pc3mWq5LGRheAd8j733rdFyN75GprL2th5i3jB8HDH",
"since_last_completed_at": 0.21285104751586914
},
"reward_manager": {
"is_healthy": true,
"last_completed_at": 1750716118.184873,
"last_tx": "37776jhVY1vYvtSxtDNp4roo75H4iPEDN3w6bP388yWfTVWXao4ZLuPco8X3CVbdAkZDfkrZ8Vr6w6EjC6qjnVHH",
"since_last_completed_at": 0.8589239120483398
},
"spl_token": {
"is_healthy": true,
"last_completed_at": 1750716025.788301,
"last_tx": "4UQP7hvmmyjHAFEz3967rgVFRPbnsLVtNKJwgi35Eix6FVnSBT3Th9dwcq2ymrGoRDCjQTgw4TKmawpFMbqh2VzA",
"since_last_completed_at": 93.25533294677734
},
"user_bank": {
"is_healthy": true,
"last_completed_at": 1750716118.978945,
"last_tx": "4UQP7hvmmyjHAFEz3967rgVFRPbnsLVtNKJwgi35Eix6FVnSBT3Th9dwcq2ymrGoRDCjQTgw4TKmawpFMbqh2VzA",
"since_last_completed_at": 0.06445789337158203
}
},
"total_memory": 67508137984,
"transferred_bytes_per_sec": 0,
"trending_playlists_age_sec": 3555678,
"trending_tracks_age_sec": 2734,
"url": "https://discoveryprovider3.audius.co",
"used_memory": 27486191616,
"user_balances_age_sec": 153,
"version": "0.7.182",
"web": {
"blockhash": null,
"blocknumber": 0
}
},
"latest_chain_block": null,
"latest_chain_slot_plays": 5852972,
"latest_indexed_block": 5852972,
"latest_indexed_slot_plays": 5852972,
"signer": "0xf2897993951d53a7e3eb2242d6a14d2028140dc8",
"success": true,
"version": {
"service": "discovery-node",
"version": "0.7.182"
}
}

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide repro instructions & any configuration.
@alecsavvyalecsavvy
self-assigned this
Jun 23, 2025
@changeset-bot
changeset-botBot
commented
Jun 23, 2025•

edited

Loading

Copy link


Copy Markdown

⚠️ No Changeset found
Latest commit: e58597c
Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.
This PR includes no changesets
When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types
Click here to learn what changesets are, and how to add one.
Click here if you're a maintainer who wants to add a changeset to this PR
raymondjacobson
raymondjacobson
approved these changes
Jun 23, 2025
View reviewed changes
@raymondjacobsonraymondjacobson
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay as a bandaid. If we weren't rewriting indexing, I'd really dislike this, but it's already so screwed up as is, I'm down with this temp solve
@alecsavvy
alecsavvy
enabled auto-merge (squash)
June 23, 2025 22:41
@alecsavvy
alecsavvy
merged commit d4ecccf
into
mainJun 24, 2025
@alecsavvy
alecsavvy
deleted the

as/fix-health

branch
June 24, 2025 15:53
isaacsolo pushed a commit
that referenced
this pull request
Jun 24, 2025
@alecsavvyisaac
use plays health in health check (#12385)
fc0920d
audius-infra pushed a commit
that referenced
this pull request
Jun 25, 2025
audius-infra
Audius Protocol v0.7.184
16ea1af
[d4ecccf] use plays health in health check (#12385) alecsavvy
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@raymondjacobsonraymondjacobsonraymondjacobson approved these changes

Assignees

@alecsavvyalecsavvy

Labels

size/M

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@alecsavvy@raymondjacobson

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
use plays health in health check - #12385
Merged
alecsavvy merged 2 commits into
mainfrom 
as/fix-health
Jun 24, 2025
Merged
use plays health in health check#12385
alecsavvy merged 2 commits into
mainfrom 
as/fix-health
Conversation
@alecsavvy
@alecsavvyalecsavvy
commented
Jun 23, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
Description
Still unsure why this health source can return 0, seems like some kind of race condition. I did find that when it errors the plays health is still reliable. To fix the intermittent issue we'll use the plays latest chain block and fallback to the one we're currently using. Plays block is just fine to use here as it's the same chain as entity manager.
Also updates healthz to route to console as those old grpc rest endpoints don't exist anymore.
bad health check as example, you can see the plays area still has the right value even though other spots have 0:
{
"antiAbuseWalletPubkey": null,
"comms": {
"booted": "2025-06-23T18:16:44Z",
"built": "",
"commit": "",
"errors": null,
"healthy": true,
"host": "https://discoveryprovider3.audius.co",
"is_registered_wallet": true,
"wallet": "0xF2897993951d53a7E3eb2242D6A14D2028140DC8",
"websocket_error": null
},
"data": {
"audius-docker-compose": "63e210459090bef4746c41341409a1f4646f7b5a",
"audius_d_managed": true,
"auto_upgrade_enabled": true,
"block_difference": 5852972,
"challenge_last_event_age_sec": 0,
"core": {
"health": {
"chain_id": "audius-mainnet-alpha-beta",
"indexing_entity_manager": true,
"indexing_plays": true,
"latest_chain_block": 5852972,
"latest_chain_block_ts": 1750716115,
"latest_indexed_block": 5852972
},
"listens": {
"core_block_cutover": 113180,
"is_unhealthy": false,
"latest_chain_slot": 5852972,
"latest_indexed_slot": 5852972,
"slot_diff": 0,
"sol_slot_cutover": 315925000,
"time_diff": 2.152073,
"tx_info": {
"chain_tx": null,
"db_tx": {
"signature": "",
"slot": 5852972,
"timestamp": 1750716115
}
}
}
},
"database_connections": 67,
"database_is_localhost": true,
"database_size": 173582922543,
"db": {
"blockhash": "0f0afc80053e2803e1b147532c9a5af660d946c821c11ba07bf4f62d091bc84e",
"number": 5852972
},
"discovery_provider_healthy": false,
"errors": [
"unhealthy block diff"
],
"filesystem_size": 1065418129408,
"filesystem_used": 567194726400,
"final_poa_block": 31413000,
"git": "cf70425a041ab8b442795dbeb80533cbc5cfdf7c",
"index_eth_age_sec": 125,
"infra_setup": "audius-docker-compose",
"last_scanned_block_for_balance_refresh": 22769967,
"latest_block_num": 0,
"latest_indexed_block_num": 5852972,
"maximum_healthy_block_difference": 100,
"meets_min_requirements": true,
"network": {
...
},
"num_users_in_immediate_balance_refresh_queue": 1,
"num_users_in_lazy_balance_refresh_queue": 2384591,
"number_of_cpus": 48,
"plays": {
"core_block_cutover": 113180,
"is_unhealthy": false,
"latest_chain_slot": 5852972,
"latest_indexed_slot": 5852972,
"slot_diff": 0,
"sol_slot_cutover": 315925000,
"time_diff": 2.152073,
"tx_info": {
"chain_tx": null,
"db_tx": {
"signature": "",
"slot": 5852972,
"timestamp": 1750716115
}
}
},
"received_bytes_per_sec": 0,
"redis_total_memory": 67508137984,
"relay": {
"is_unhealthy": false,
"status": "up"
},
"service": "discovery-node",
"solana_indexers": {
"aggregate_tips": {
"is_healthy": true,
"last_completed_at": 1750716027.550774,
"last_tx": "oU4kgkWr9Ditwq5EaT1mdG1zohkSh5yWra1iXaJc9nvaPrXSFFdi11cuwqWMUZt11CfyqrcioheQntsGsbZq8Z8",
"since_last_completed_at": 91.49335789680481
},
"payment_router": {
"is_healthy": true,
"last_completed_at": 1750716118.83112,
"last_tx": "3i1mGzNrprWcW3PnSRYognDXezFas55fzpP8aCfDSJePg6pc3mWq5LGRheAd8j733rdFyN75GprL2th5i3jB8HDH",
"since_last_completed_at": 0.21285104751586914
},
"reward_manager": {
"is_healthy": true,
"last_completed_at": 1750716118.184873,
"last_tx": "37776jhVY1vYvtSxtDNp4roo75H4iPEDN3w6bP388yWfTVWXao4ZLuPco8X3CVbdAkZDfkrZ8Vr6w6EjC6qjnVHH",
"since_last_completed_at": 0.8589239120483398
},
"spl_token": {
"is_healthy": true,
"last_completed_at": 1750716025.788301,
"last_tx": "4UQP7hvmmyjHAFEz3967rgVFRPbnsLVtNKJwgi35Eix6FVnSBT3Th9dwcq2ymrGoRDCjQTgw4TKmawpFMbqh2VzA",
"since_last_completed_at": 93.25533294677734
},
"user_bank": {
"is_healthy": true,
"last_completed_at": 1750716118.978945,
"last_tx": "4UQP7hvmmyjHAFEz3967rgVFRPbnsLVtNKJwgi35Eix6FVnSBT3Th9dwcq2ymrGoRDCjQTgw4TKmawpFMbqh2VzA",
"since_last_completed_at": 0.06445789337158203
}
},
"total_memory": 67508137984,
"transferred_bytes_per_sec": 0,
"trending_playlists_age_sec": 3555678,
"trending_tracks_age_sec": 2734,
"url": "https://discoveryprovider3.audius.co",
"used_memory": 27486191616,
"user_balances_age_sec": 153,
"version": "0.7.182",
"web": {
"blockhash": null,
"blocknumber": 0
}
},
"latest_chain_block": null,
"latest_chain_slot_plays": 5852972,
"latest_indexed_block": 5852972,
"latest_indexed_slot_plays": 5852972,
"signer": "0xf2897993951d53a7e3eb2242d6a14d2028140dc8",
"success": true,
"version": {
"service": "discovery-node",
"version": "0.7.182"
}
}

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide repro instructions & any configuration.
@alecsavvyalecsavvy
self-assigned this
Jun 23, 2025
@changeset-bot
changeset-botBot
commented
Jun 23, 2025•

edited

Loading

Copy link


Copy Markdown

⚠️ No Changeset found
Latest commit: e58597c
Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.
This PR includes no changesets
When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types
Click here to learn what changesets are, and how to add one.
Click here if you're a maintainer who wants to add a changeset to this PR
raymondjacobson
raymondjacobson
approved these changes
Jun 23, 2025
View reviewed changes
@raymondjacobsonraymondjacobson
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay as a bandaid. If we weren't rewriting indexing, I'd really dislike this, but it's already so screwed up as is, I'm down with this temp solve
@alecsavvy
alecsavvy
enabled auto-merge (squash)
June 23, 2025 22:41
@alecsavvy
alecsavvy
merged commit d4ecccf
into
mainJun 24, 2025
@alecsavvy
alecsavvy
deleted the

as/fix-health

branch
June 24, 2025 15:53
isaacsolo pushed a commit
that referenced
this pull request
Jun 24, 2025
@alecsavvyisaac
use plays health in health check (#12385)
fc0920d
audius-infra pushed a commit
that referenced
this pull request
Jun 25, 2025
audius-infra
Audius Protocol v0.7.184
16ea1af
[d4ecccf] use plays health in health check (#12385) alecsavvy
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@raymondjacobsonraymondjacobsonraymondjacobson approved these changes

Assignees

@alecsavvyalecsavvy

Labels

size/M

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@alecsavvy@raymondjacobson

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
use plays health in health check - #12385
Merged
alecsavvy merged 2 commits into
mainfrom 
as/fix-health
Jun 24, 2025
Merged
use plays health in health check#12385
alecsavvy merged 2 commits into
mainfrom 
as/fix-health
Conversation
@alecsavvy
@alecsavvyalecsavvy
commented
Jun 23, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
Description
Still unsure why this health source can return 0, seems like some kind of race condition. I did find that when it errors the plays health is still reliable. To fix the intermittent issue we'll use the plays latest chain block and fallback to the one we're currently using. Plays block is just fine to use here as it's the same chain as entity manager.
Also updates healthz to route to console as those old grpc rest endpoints don't exist anymore.
bad health check as example, you can see the plays area still has the right value even though other spots have 0:
{
"antiAbuseWalletPubkey": null,
"comms": {
"booted": "2025-06-23T18:16:44Z",
"built": "",
"commit": "",
"errors": null,
"healthy": true,
"host": "https://discoveryprovider3.audius.co",
"is_registered_wallet": true,
"wallet": "0xF2897993951d53a7E3eb2242D6A14D2028140DC8",
"websocket_error": null
},
"data": {
"audius-docker-compose": "63e210459090bef4746c41341409a1f4646f7b5a",
"audius_d_managed": true,
"auto_upgrade_enabled": true,
"block_difference": 5852972,
"challenge_last_event_age_sec": 0,
"core": {
"health": {
"chain_id": "audius-mainnet-alpha-beta",
"indexing_entity_manager": true,
"indexing_plays": true,
"latest_chain_block": 5852972,
"latest_chain_block_ts": 1750716115,
"latest_indexed_block": 5852972
},
"listens": {
"core_block_cutover": 113180,
"is_unhealthy": false,
"latest_chain_slot": 5852972,
"latest_indexed_slot": 5852972,
"slot_diff": 0,
"sol_slot_cutover": 315925000,
"time_diff": 2.152073,
"tx_info": {
"chain_tx": null,
"db_tx": {
"signature": "",
"slot": 5852972,
"timestamp": 1750716115
}
}
}
},
"database_connections": 67,
"database_is_localhost": true,
"database_size": 173582922543,
"db": {
"blockhash": "0f0afc80053e2803e1b147532c9a5af660d946c821c11ba07bf4f62d091bc84e",
"number": 5852972
},
"discovery_provider_healthy": false,
"errors": [
"unhealthy block diff"
],
"filesystem_size": 1065418129408,
"filesystem_used": 567194726400,
"final_poa_block": 31413000,
"git": "cf70425a041ab8b442795dbeb80533cbc5cfdf7c",
"index_eth_age_sec": 125,
"infra_setup": "audius-docker-compose",
"last_scanned_block_for_balance_refresh": 22769967,
"latest_block_num": 0,
"latest_indexed_block_num": 5852972,
"maximum_healthy_block_difference": 100,
"meets_min_requirements": true,
"network": {
...
},
"num_users_in_immediate_balance_refresh_queue": 1,
"num_users_in_lazy_balance_refresh_queue": 2384591,
"number_of_cpus": 48,
"plays": {
"core_block_cutover": 113180,
"is_unhealthy": false,
"latest_chain_slot": 5852972,
"latest_indexed_slot": 5852972,
"slot_diff": 0,
"sol_slot_cutover": 315925000,
"time_diff": 2.152073,
"tx_info": {
"chain_tx": null,
"db_tx": {
"signature": "",
"slot": 5852972,
"timestamp": 1750716115
}
}
},
"received_bytes_per_sec": 0,
"redis_total_memory": 67508137984,
"relay": {
"is_unhealthy": false,
"status": "up"
},
"service": "discovery-node",
"solana_indexers": {
"aggregate_tips": {
"is_healthy": true,
"last_completed_at": 1750716027.550774,
"last_tx": "oU4kgkWr9Ditwq5EaT1mdG1zohkSh5yWra1iXaJc9nvaPrXSFFdi11cuwqWMUZt11CfyqrcioheQntsGsbZq8Z8",
"since_last_completed_at": 91.49335789680481
},
"payment_router": {
"is_healthy": true,
"last_completed_at": 1750716118.83112,
"last_tx": "3i1mGzNrprWcW3PnSRYognDXezFas55fzpP8aCfDSJePg6pc3mWq5LGRheAd8j733rdFyN75GprL2th5i3jB8HDH",
"since_last_completed_at": 0.21285104751586914
},
"reward_manager": {
"is_healthy": true,
"last_completed_at": 1750716118.184873,
"last_tx": "37776jhVY1vYvtSxtDNp4roo75H4iPEDN3w6bP388yWfTVWXao4ZLuPco8X3CVbdAkZDfkrZ8Vr6w6EjC6qjnVHH",
"since_last_completed_at": 0.8589239120483398
},
"spl_token": {
"is_healthy": true,
"last_completed_at": 1750716025.788301,
"last_tx": "4UQP7hvmmyjHAFEz3967rgVFRPbnsLVtNKJwgi35Eix6FVnSBT3Th9dwcq2ymrGoRDCjQTgw4TKmawpFMbqh2VzA",
"since_last_completed_at": 93.25533294677734
},
"user_bank": {
"is_healthy": true,
"last_completed_at": 1750716118.978945,
"last_tx": "4UQP7hvmmyjHAFEz3967rgVFRPbnsLVtNKJwgi35Eix6FVnSBT3Th9dwcq2ymrGoRDCjQTgw4TKmawpFMbqh2VzA",
"since_last_completed_at": 0.06445789337158203
}
},
"total_memory": 67508137984,
"transferred_bytes_per_sec": 0,
"trending_playlists_age_sec": 3555678,
"trending_tracks_age_sec": 2734,
"url": "https://discoveryprovider3.audius.co",
"used_memory": 27486191616,
"user_balances_age_sec": 153,
"version": "0.7.182",
"web": {
"blockhash": null,
"blocknumber": 0
}
},
"latest_chain_block": null,
"latest_chain_slot_plays": 5852972,
"latest_indexed_block": 5852972,
"latest_indexed_slot_plays": 5852972,
"signer": "0xf2897993951d53a7e3eb2242d6a14d2028140dc8",
"success": true,
"version": {
"service": "discovery-node",
"version": "0.7.182"
}
}

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide repro instructions & any configuration.
@alecsavvyalecsavvy
self-assigned this
Jun 23, 2025
@changeset-bot
changeset-botBot
commented
Jun 23, 2025•

edited

Loading

Copy link


Copy Markdown

⚠️ No Changeset found
Latest commit: e58597c
Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.
This PR includes no changesets
When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types
Click here to learn what changesets are, and how to add one.
Click here if you're a maintainer who wants to add a changeset to this PR
raymondjacobson
raymondjacobson
approved these changes
Jun 23, 2025
View reviewed changes
@raymondjacobsonraymondjacobson
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay as a bandaid. If we weren't rewriting indexing, I'd really dislike this, but it's already so screwed up as is, I'm down with this temp solve
@alecsavvy
alecsavvy
enabled auto-merge (squash)
June 23, 2025 22:41
@alecsavvy
alecsavvy
merged commit d4ecccf
into
mainJun 24, 2025
@alecsavvy
alecsavvy
deleted the

as/fix-health

branch
June 24, 2025 15:53
isaacsolo pushed a commit
that referenced
this pull request
Jun 24, 2025
@alecsavvyisaac
use plays health in health check (#12385)
fc0920d
audius-infra pushed a commit
that referenced
this pull request
Jun 25, 2025
audius-infra
Audius Protocol v0.7.184
16ea1af
[d4ecccf] use plays health in health check (#12385) alecsavvy
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@raymondjacobsonraymondjacobsonraymondjacobson approved these changes

Assignees

@alecsavvyalecsavvy

Labels

size/M

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@alecsavvy@raymondjacobson

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
use plays health in health check - #12385
Merged
alecsavvy merged 2 commits into
mainfrom 
as/fix-health
Jun 24, 2025
Merged
use plays health in health check#12385
alecsavvy merged 2 commits into
mainfrom 
as/fix-health
Conversation
@alecsavvy
@alecsavvyalecsavvy
commented
Jun 23, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
Description
Still unsure why this health source can return 0, seems like some kind of race condition. I did find that when it errors the plays health is still reliable. To fix the intermittent issue we'll use the plays latest chain block and fallback to the one we're currently using. Plays block is just fine to use here as it's the same chain as entity manager.
Also updates healthz to route to console as those old grpc rest endpoints don't exist anymore.
bad health check as example, you can see the plays area still has the right value even though other spots have 0:
{
"antiAbuseWalletPubkey": null,
"comms": {
"booted": "2025-06-23T18:16:44Z",
"built": "",
"commit": "",
"errors": null,
"healthy": true,
"host": "https://discoveryprovider3.audius.co",
"is_registered_wallet": true,
"wallet": "0xF2897993951d53a7E3eb2242D6A14D2028140DC8",
"websocket_error": null
},
"data": {
"audius-docker-compose": "63e210459090bef4746c41341409a1f4646f7b5a",
"audius_d_managed": true,
"auto_upgrade_enabled": true,
"block_difference": 5852972,
"challenge_last_event_age_sec": 0,
"core": {
"health": {
"chain_id": "audius-mainnet-alpha-beta",
"indexing_entity_manager": true,
"indexing_plays": true,
"latest_chain_block": 5852972,
"latest_chain_block_ts": 1750716115,
"latest_indexed_block": 5852972
},
"listens": {
"core_block_cutover": 113180,
"is_unhealthy": false,
"latest_chain_slot": 5852972,
"latest_indexed_slot": 5852972,
"slot_diff": 0,
"sol_slot_cutover": 315925000,
"time_diff": 2.152073,
"tx_info": {
"chain_tx": null,
"db_tx": {
"signature": "",
"slot": 5852972,
"timestamp": 1750716115
}
}
}
},
"database_connections": 67,
"database_is_localhost": true,
"database_size": 173582922543,
"db": {
"blockhash": "0f0afc80053e2803e1b147532c9a5af660d946c821c11ba07bf4f62d091bc84e",
"number": 5852972
},
"discovery_provider_healthy": false,
"errors": [
"unhealthy block diff"
],
"filesystem_size": 1065418129408,
"filesystem_used": 567194726400,
"final_poa_block": 31413000,
"git": "cf70425a041ab8b442795dbeb80533cbc5cfdf7c",
"index_eth_age_sec": 125,
"infra_setup": "audius-docker-compose",
"last_scanned_block_for_balance_refresh": 22769967,
"latest_block_num": 0,
"latest_indexed_block_num": 5852972,
"maximum_healthy_block_difference": 100,
"meets_min_requirements": true,
"network": {
...
},
"num_users_in_immediate_balance_refresh_queue": 1,
"num_users_in_lazy_balance_refresh_queue": 2384591,
"number_of_cpus": 48,
"plays": {
"core_block_cutover": 113180,
"is_unhealthy": false,
"latest_chain_slot": 5852972,
"latest_indexed_slot": 5852972,
"slot_diff": 0,
"sol_slot_cutover": 315925000,
"time_diff": 2.152073,
"tx_info": {
"chain_tx": null,
"db_tx": {
"signature": "",
"slot": 5852972,
"timestamp": 1750716115
}
}
},
"received_bytes_per_sec": 0,
"redis_total_memory": 67508137984,
"relay": {
"is_unhealthy": false,
"status": "up"
},
"service": "discovery-node",
"solana_indexers": {
"aggregate_tips": {
"is_healthy": true,
"last_completed_at": 1750716027.550774,
"last_tx": "oU4kgkWr9Ditwq5EaT1mdG1zohkSh5yWra1iXaJc9nvaPrXSFFdi11cuwqWMUZt11CfyqrcioheQntsGsbZq8Z8",
"since_last_completed_at": 91.49335789680481
},
"payment_router": {
"is_healthy": true,
"last_completed_at": 1750716118.83112,
"last_tx": "3i1mGzNrprWcW3PnSRYognDXezFas55fzpP8aCfDSJePg6pc3mWq5LGRheAd8j733rdFyN75GprL2th5i3jB8HDH",
"since_last_completed_at": 0.21285104751586914
},
"reward_manager": {
"is_healthy": true,
"last_completed_at": 1750716118.184873,
"last_tx": "37776jhVY1vYvtSxtDNp4roo75H4iPEDN3w6bP388yWfTVWXao4ZLuPco8X3CVbdAkZDfkrZ8Vr6w6EjC6qjnVHH",
"since_last_completed_at": 0.8589239120483398
},
"spl_token": {
"is_healthy": true,
"last_completed_at": 1750716025.788301,
"last_tx": "4UQP7hvmmyjHAFEz3967rgVFRPbnsLVtNKJwgi35Eix6FVnSBT3Th9dwcq2ymrGoRDCjQTgw4TKmawpFMbqh2VzA",
"since_last_completed_at": 93.25533294677734
},
"user_bank": {
"is_healthy": true,
"last_completed_at": 1750716118.978945,
"last_tx": "4UQP7hvmmyjHAFEz3967rgVFRPbnsLVtNKJwgi35Eix6FVnSBT3Th9dwcq2ymrGoRDCjQTgw4TKmawpFMbqh2VzA",
"since_last_completed_at": 0.06445789337158203
}
},
"total_memory": 67508137984,
"transferred_bytes_per_sec": 0,
"trending_playlists_age_sec": 3555678,
"trending_tracks_age_sec": 2734,
"url": "https://discoveryprovider3.audius.co",
"used_memory": 27486191616,
"user_balances_age_sec": 153,
"version": "0.7.182",
"web": {
"blockhash": null,
"blocknumber": 0
}
},
"latest_chain_block": null,
"latest_chain_slot_plays": 5852972,
"latest_indexed_block": 5852972,
"latest_indexed_slot_plays": 5852972,
"signer": "0xf2897993951d53a7e3eb2242d6a14d2028140dc8",
"success": true,
"version": {
"service": "discovery-node",
"version": "0.7.182"
}
}

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide repro instructions & any configuration.
@alecsavvyalecsavvy
self-assigned this
Jun 23, 2025
@changeset-bot
changeset-botBot
commented
Jun 23, 2025•

edited

Loading

Copy link


Copy Markdown

⚠️ No Changeset found
Latest commit: e58597c
Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.
This PR includes no changesets
When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types
Click here to learn what changesets are, and how to add one.
Click here if you're a maintainer who wants to add a changeset to this PR
raymondjacobson
raymondjacobson
approved these changes
Jun 23, 2025
View reviewed changes
@raymondjacobsonraymondjacobson
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay as a bandaid. If we weren't rewriting indexing, I'd really dislike this, but it's already so screwed up as is, I'm down with this temp solve
@alecsavvy
alecsavvy
enabled auto-merge (squash)
June 23, 2025 22:41
@alecsavvy
alecsavvy
merged commit d4ecccf
into
mainJun 24, 2025
@alecsavvy
alecsavvy
deleted the

as/fix-health

branch
June 24, 2025 15:53
isaacsolo pushed a commit
that referenced
this pull request
Jun 24, 2025
@alecsavvyisaac
use plays health in health check (#12385)
fc0920d
audius-infra pushed a commit
that referenced
this pull request
Jun 25, 2025
audius-infra
Audius Protocol v0.7.184
16ea1af
[d4ecccf] use plays health in health check (#12385) alecsavvy
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@raymondjacobsonraymondjacobsonraymondjacobson approved these changes

Assignees

@alecsavvyalecsavvy

Labels

size/M

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@alecsavvy@raymondjacobson

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
use plays health in health check - #12385
Merged
alecsavvy merged 2 commits into
mainfrom 
as/fix-health
Jun 24, 2025
Merged
use plays health in health check#12385
alecsavvy merged 2 commits into
mainfrom 
as/fix-health
Conversation
@alecsavvy
@alecsavvyalecsavvy
commented
Jun 23, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
Description
Still unsure why this health source can return 0, seems like some kind of race condition. I did find that when it errors the plays health is still reliable. To fix the intermittent issue we'll use the plays latest chain block and fallback to the one we're currently using. Plays block is just fine to use here as it's the same chain as entity manager.
Also updates healthz to route to console as those old grpc rest endpoints don't exist anymore.
bad health check as example, you can see the plays area still has the right value even though other spots have 0:
{
"antiAbuseWalletPubkey": null,
"comms": {
"booted": "2025-06-23T18:16:44Z",
"built": "",
"commit": "",
"errors": null,
"healthy": true,
"host": "https://discoveryprovider3.audius.co",
"is_registered_wallet": true,
"wallet": "0xF2897993951d53a7E3eb2242D6A14D2028140DC8",
"websocket_error": null
},
"data": {
"audius-docker-compose": "63e210459090bef4746c41341409a1f4646f7b5a",
"audius_d_managed": true,
"auto_upgrade_enabled": true,
"block_difference": 5852972,
"challenge_last_event_age_sec": 0,
"core": {
"health": {
"chain_id": "audius-mainnet-alpha-beta",
"indexing_entity_manager": true,
"indexing_plays": true,
"latest_chain_block": 5852972,
"latest_chain_block_ts": 1750716115,
"latest_indexed_block": 5852972
},
"listens": {
"core_block_cutover": 113180,
"is_unhealthy": false,
"latest_chain_slot": 5852972,
"latest_indexed_slot": 5852972,
"slot_diff": 0,
"sol_slot_cutover": 315925000,
"time_diff": 2.152073,
"tx_info": {
"chain_tx": null,
"db_tx": {
"signature": "",
"slot": 5852972,
"timestamp": 1750716115
}
}
}
},
"database_connections": 67,
"database_is_localhost": true,
"database_size": 173582922543,
"db": {
"blockhash": "0f0afc80053e2803e1b147532c9a5af660d946c821c11ba07bf4f62d091bc84e",
"number": 5852972
},
"discovery_provider_healthy": false,
"errors": [
"unhealthy block diff"
],
"filesystem_size": 1065418129408,
"filesystem_used": 567194726400,
"final_poa_block": 31413000,
"git": "cf70425a041ab8b442795dbeb80533cbc5cfdf7c",
"index_eth_age_sec": 125,
"infra_setup": "audius-docker-compose",
"last_scanned_block_for_balance_refresh": 22769967,
"latest_block_num": 0,
"latest_indexed_block_num": 5852972,
"maximum_healthy_block_difference": 100,
"meets_min_requirements": true,
"network": {
...
},
"num_users_in_immediate_balance_refresh_queue": 1,
"num_users_in_lazy_balance_refresh_queue": 2384591,
"number_of_cpus": 48,
"plays": {
"core_block_cutover": 113180,
"is_unhealthy": false,
"latest_chain_slot": 5852972,
"latest_indexed_slot": 5852972,
"slot_diff": 0,
"sol_slot_cutover": 315925000,
"time_diff": 2.152073,
"tx_info": {
"chain_tx": null,
"db_tx": {
"signature": "",
"slot": 5852972,
"timestamp": 1750716115
}
}
},
"received_bytes_per_sec": 0,
"redis_total_memory": 67508137984,
"relay": {
"is_unhealthy": false,
"status": "up"
},
"service": "discovery-node",
"solana_indexers": {
"aggregate_tips": {
"is_healthy": true,
"last_completed_at": 1750716027.550774,
"last_tx": "oU4kgkWr9Ditwq5EaT1mdG1zohkSh5yWra1iXaJc9nvaPrXSFFdi11cuwqWMUZt11CfyqrcioheQntsGsbZq8Z8",
"since_last_completed_at": 91.49335789680481
},
"payment_router": {
"is_healthy": true,
"last_completed_at": 1750716118.83112,
"last_tx": "3i1mGzNrprWcW3PnSRYognDXezFas55fzpP8aCfDSJePg6pc3mWq5LGRheAd8j733rdFyN75GprL2th5i3jB8HDH",
"since_last_completed_at": 0.21285104751586914
},
"reward_manager": {
"is_healthy": true,
"last_completed_at": 1750716118.184873,
"last_tx": "37776jhVY1vYvtSxtDNp4roo75H4iPEDN3w6bP388yWfTVWXao4ZLuPco8X3CVbdAkZDfkrZ8Vr6w6EjC6qjnVHH",
"since_last_completed_at": 0.8589239120483398
},
"spl_token": {
"is_healthy": true,
"last_completed_at": 1750716025.788301,
"last_tx": "4UQP7hvmmyjHAFEz3967rgVFRPbnsLVtNKJwgi35Eix6FVnSBT3Th9dwcq2ymrGoRDCjQTgw4TKmawpFMbqh2VzA",
"since_last_completed_at": 93.25533294677734
},
"user_bank": {
"is_healthy": true,
"last_completed_at": 1750716118.978945,
"last_tx": "4UQP7hvmmyjHAFEz3967rgVFRPbnsLVtNKJwgi35Eix6FVnSBT3Th9dwcq2ymrGoRDCjQTgw4TKmawpFMbqh2VzA",
"since_last_completed_at": 0.06445789337158203
}
},
"total_memory": 67508137984,
"transferred_bytes_per_sec": 0,
"trending_playlists_age_sec": 3555678,
"trending_tracks_age_sec": 2734,
"url": "https://discoveryprovider3.audius.co",
"used_memory": 27486191616,
"user_balances_age_sec": 153,
"version": "0.7.182",
"web": {
"blockhash": null,
"blocknumber": 0
}
},
"latest_chain_block": null,
"latest_chain_slot_plays": 5852972,
"latest_indexed_block": 5852972,
"latest_indexed_slot_plays": 5852972,
"signer": "0xf2897993951d53a7e3eb2242d6a14d2028140dc8",
"success": true,
"version": {
"service": "discovery-node",
"version": "0.7.182"
}
}

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide repro instructions & any configuration.
@alecsavvyalecsavvy
self-assigned this
Jun 23, 2025
@changeset-bot
changeset-botBot
commented
Jun 23, 2025•

edited

Loading

Copy link


Copy Markdown

⚠️ No Changeset found
Latest commit: e58597c
Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.
This PR includes no changesets
When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types
Click here to learn what changesets are, and how to add one.
Click here if you're a maintainer who wants to add a changeset to this PR
raymondjacobson
raymondjacobson
approved these changes
Jun 23, 2025
View reviewed changes
@raymondjacobsonraymondjacobson
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay as a bandaid. If we weren't rewriting indexing, I'd really dislike this, but it's already so screwed up as is, I'm down with this temp solve
@alecsavvy
alecsavvy
enabled auto-merge (squash)
June 23, 2025 22:41
@alecsavvy
alecsavvy
merged commit d4ecccf
into
mainJun 24, 2025
@alecsavvy
alecsavvy
deleted the

as/fix-health

branch
June 24, 2025 15:53
isaacsolo pushed a commit
that referenced
this pull request
Jun 24, 2025
@alecsavvyisaac
use plays health in health check (#12385)
fc0920d
audius-infra pushed a commit
that referenced
this pull request
Jun 25, 2025
audius-infra
Audius Protocol v0.7.184
16ea1af
[d4ecccf] use plays health in health check (#12385) alecsavvy
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@raymondjacobsonraymondjacobsonraymondjacobson approved these changes

Assignees

@alecsavvyalecsavvy

Labels

size/M

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@alecsavvy@raymondjacobson

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
use plays health in health check - #12385
Merged
alecsavvy merged 2 commits into
mainfrom 
as/fix-health
Jun 24, 2025
Merged
use plays health in health check#12385
alecsavvy merged 2 commits into
mainfrom 
as/fix-health
Conversation
@alecsavvy
@alecsavvyalecsavvy
commented
Jun 23, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
Description
Still unsure why this health source can return 0, seems like some kind of race condition. I did find that when it errors the plays health is still reliable. To fix the intermittent issue we'll use the plays latest chain block and fallback to the one we're currently using. Plays block is just fine to use here as it's the same chain as entity manager.
Also updates healthz to route to console as those old grpc rest endpoints don't exist anymore.
bad health check as example, you can see the plays area still has the right value even though other spots have 0:
{
"antiAbuseWalletPubkey": null,
"comms": {
"booted": "2025-06-23T18:16:44Z",
"built": "",
"commit": "",
"errors": null,
"healthy": true,
"host": "https://discoveryprovider3.audius.co",
"is_registered_wallet": true,
"wallet": "0xF2897993951d53a7E3eb2242D6A14D2028140DC8",
"websocket_error": null
},
"data": {
"audius-docker-compose": "63e210459090bef4746c41341409a1f4646f7b5a",
"audius_d_managed": true,
"auto_upgrade_enabled": true,
"block_difference": 5852972,
"challenge_last_event_age_sec": 0,
"core": {
"health": {
"chain_id": "audius-mainnet-alpha-beta",
"indexing_entity_manager": true,
"indexing_plays": true,
"latest_chain_block": 5852972,
"latest_chain_block_ts": 1750716115,
"latest_indexed_block": 5852972
},
"listens": {
"core_block_cutover": 113180,
"is_unhealthy": false,
"latest_chain_slot": 5852972,
"latest_indexed_slot": 5852972,
"slot_diff": 0,
"sol_slot_cutover": 315925000,
"time_diff": 2.152073,
"tx_info": {
"chain_tx": null,
"db_tx": {
"signature": "",
"slot": 5852972,
"timestamp": 1750716115
}
}
}
},
"database_connections": 67,
"database_is_localhost": true,
"database_size": 173582922543,
"db": {
"blockhash": "0f0afc80053e2803e1b147532c9a5af660d946c821c11ba07bf4f62d091bc84e",
"number": 5852972
},
"discovery_provider_healthy": false,
"errors": [
"unhealthy block diff"
],
"filesystem_size": 1065418129408,
"filesystem_used": 567194726400,
"final_poa_block": 31413000,
"git": "cf70425a041ab8b442795dbeb80533cbc5cfdf7c",
"index_eth_age_sec": 125,
"infra_setup": "audius-docker-compose",
"last_scanned_block_for_balance_refresh": 22769967,
"latest_block_num": 0,
"latest_indexed_block_num": 5852972,
"maximum_healthy_block_difference": 100,
"meets_min_requirements": true,
"network": {
...
},
"num_users_in_immediate_balance_refresh_queue": 1,
"num_users_in_lazy_balance_refresh_queue": 2384591,
"number_of_cpus": 48,
"plays": {
"core_block_cutover": 113180,
"is_unhealthy": false,
"latest_chain_slot": 5852972,
"latest_indexed_slot": 5852972,
"slot_diff": 0,
"sol_slot_cutover": 315925000,
"time_diff": 2.152073,
"tx_info": {
"chain_tx": null,
"db_tx": {
"signature": "",
"slot": 5852972,
"timestamp": 1750716115
}
}
},
"received_bytes_per_sec": 0,
"redis_total_memory": 67508137984,
"relay": {
"is_unhealthy": false,
"status": "up"
},
"service": "discovery-node",
"solana_indexers": {
"aggregate_tips": {
"is_healthy": true,
"last_completed_at": 1750716027.550774,
"last_tx": "oU4kgkWr9Ditwq5EaT1mdG1zohkSh5yWra1iXaJc9nvaPrXSFFdi11cuwqWMUZt11CfyqrcioheQntsGsbZq8Z8",
"since_last_completed_at": 91.49335789680481
},
"payment_router": {
"is_healthy": true,
"last_completed_at": 1750716118.83112,
"last_tx": "3i1mGzNrprWcW3PnSRYognDXezFas55fzpP8aCfDSJePg6pc3mWq5LGRheAd8j733rdFyN75GprL2th5i3jB8HDH",
"since_last_completed_at": 0.21285104751586914
},
"reward_manager": {
"is_healthy": true,
"last_completed_at": 1750716118.184873,
"last_tx": "37776jhVY1vYvtSxtDNp4roo75H4iPEDN3w6bP388yWfTVWXao4ZLuPco8X3CVbdAkZDfkrZ8Vr6w6EjC6qjnVHH",
"since_last_completed_at": 0.8589239120483398
},
"spl_token": {
"is_healthy": true,
"last_completed_at": 1750716025.788301,
"last_tx": "4UQP7hvmmyjHAFEz3967rgVFRPbnsLVtNKJwgi35Eix6FVnSBT3Th9dwcq2ymrGoRDCjQTgw4TKmawpFMbqh2VzA",
"since_last_completed_at": 93.25533294677734
},
"user_bank": {
"is_healthy": true,
"last_completed_at": 1750716118.978945,
"last_tx": "4UQP7hvmmyjHAFEz3967rgVFRPbnsLVtNKJwgi35Eix6FVnSBT3Th9dwcq2ymrGoRDCjQTgw4TKmawpFMbqh2VzA",
"since_last_completed_at": 0.06445789337158203
}
},
"total_memory": 67508137984,
"transferred_bytes_per_sec": 0,
"trending_playlists_age_sec": 3555678,
"trending_tracks_age_sec": 2734,
"url": "https://discoveryprovider3.audius.co",
"used_memory": 27486191616,
"user_balances_age_sec": 153,
"version": "0.7.182",
"web": {
"blockhash": null,
"blocknumber": 0
}
},
"latest_chain_block": null,
"latest_chain_slot_plays": 5852972,
"latest_indexed_block": 5852972,
"latest_indexed_slot_plays": 5852972,
"signer": "0xf2897993951d53a7e3eb2242d6a14d2028140dc8",
"success": true,
"version": {
"service": "discovery-node",
"version": "0.7.182"
}
}

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide repro instructions & any configuration.
@alecsavvyalecsavvy
self-assigned this
Jun 23, 2025
@changeset-bot
changeset-botBot
commented
Jun 23, 2025•

edited

Loading

Copy link


Copy Markdown

⚠️ No Changeset found
Latest commit: e58597c
Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.
This PR includes no changesets
When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types
Click here to learn what changesets are, and how to add one.
Click here if you're a maintainer who wants to add a changeset to this PR
raymondjacobson
raymondjacobson
approved these changes
Jun 23, 2025
View reviewed changes
@raymondjacobsonraymondjacobson
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay as a bandaid. If we weren't rewriting indexing, I'd really dislike this, but it's already so screwed up as is, I'm down with this temp solve
@alecsavvy
alecsavvy
enabled auto-merge (squash)
June 23, 2025 22:41
@alecsavvy
alecsavvy
merged commit d4ecccf
into
mainJun 24, 2025
@alecsavvy
alecsavvy
deleted the

as/fix-health

branch
June 24, 2025 15:53
isaacsolo pushed a commit
that referenced
this pull request
Jun 24, 2025
@alecsavvyisaac
use plays health in health check (#12385)
fc0920d
audius-infra pushed a commit
that referenced
this pull request
Jun 25, 2025
audius-infra
Audius Protocol v0.7.184
16ea1af
[d4ecccf] use plays health in health check (#12385) alecsavvy
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@raymondjacobsonraymondjacobsonraymondjacobson approved these changes

Assignees

@alecsavvyalecsavvy

Labels

size/M

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@alecsavvy@raymondjacobson

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
use plays health in health check - #12385
Merged
alecsavvy merged 2 commits into
mainfrom 
as/fix-health
Jun 24, 2025
Merged
use plays health in health check#12385
alecsavvy merged 2 commits into
mainfrom 
as/fix-health
Conversation
@alecsavvy
@alecsavvyalecsavvy
commented
Jun 23, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
Description
Still unsure why this health source can return 0, seems like some kind of race condition. I did find that when it errors the plays health is still reliable. To fix the intermittent issue we'll use the plays latest chain block and fallback to the one we're currently using. Plays block is just fine to use here as it's the same chain as entity manager.
Also updates healthz to route to console as those old grpc rest endpoints don't exist anymore.
bad health check as example, you can see the plays area still has the right value even though other spots have 0:
{
"antiAbuseWalletPubkey": null,
"comms": {
"booted": "2025-06-23T18:16:44Z",
"built": "",
"commit": "",
"errors": null,
"healthy": true,
"host": "https://discoveryprovider3.audius.co",
"is_registered_wallet": true,
"wallet": "0xF2897993951d53a7E3eb2242D6A14D2028140DC8",
"websocket_error": null
},
"data": {
"audius-docker-compose": "63e210459090bef4746c41341409a1f4646f7b5a",
"audius_d_managed": true,
"auto_upgrade_enabled": true,
"block_difference": 5852972,
"challenge_last_event_age_sec": 0,
"core": {
"health": {
"chain_id": "audius-mainnet-alpha-beta",
"indexing_entity_manager": true,
"indexing_plays": true,
"latest_chain_block": 5852972,
"latest_chain_block_ts": 1750716115,
"latest_indexed_block": 5852972
},
"listens": {
"core_block_cutover": 113180,
"is_unhealthy": false,
"latest_chain_slot": 5852972,
"latest_indexed_slot": 5852972,
"slot_diff": 0,
"sol_slot_cutover": 315925000,
"time_diff": 2.152073,
"tx_info": {
"chain_tx": null,
"db_tx": {
"signature": "",
"slot": 5852972,
"timestamp": 1750716115
}
}
}
},
"database_connections": 67,
"database_is_localhost": true,
"database_size": 173582922543,
"db": {
"blockhash": "0f0afc80053e2803e1b147532c9a5af660d946c821c11ba07bf4f62d091bc84e",
"number": 5852972
},
"discovery_provider_healthy": false,
"errors": [
"unhealthy block diff"
],
"filesystem_size": 1065418129408,
"filesystem_used": 567194726400,
"final_poa_block": 31413000,
"git": "cf70425a041ab8b442795dbeb80533cbc5cfdf7c",
"index_eth_age_sec": 125,
"infra_setup": "audius-docker-compose",
"last_scanned_block_for_balance_refresh": 22769967,
"latest_block_num": 0,
"latest_indexed_block_num": 5852972,
"maximum_healthy_block_difference": 100,
"meets_min_requirements": true,
"network": {
...
},
"num_users_in_immediate_balance_refresh_queue": 1,
"num_users_in_lazy_balance_refresh_queue": 2384591,
"number_of_cpus": 48,
"plays": {
"core_block_cutover": 113180,
"is_unhealthy": false,
"latest_chain_slot": 5852972,
"latest_indexed_slot": 5852972,
"slot_diff": 0,
"sol_slot_cutover": 315925000,
"time_diff": 2.152073,
"tx_info": {
"chain_tx": null,
"db_tx": {
"signature": "",
"slot": 5852972,
"timestamp": 1750716115
}
}
},
"received_bytes_per_sec": 0,
"redis_total_memory": 67508137984,
"relay": {
"is_unhealthy": false,
"status": "up"
},
"service": "discovery-node",
"solana_indexers": {
"aggregate_tips": {
"is_healthy": true,
"last_completed_at": 1750716027.550774,
"last_tx": "oU4kgkWr9Ditwq5EaT1mdG1zohkSh5yWra1iXaJc9nvaPrXSFFdi11cuwqWMUZt11CfyqrcioheQntsGsbZq8Z8",
"since_last_completed_at": 91.49335789680481
},
"payment_router": {
"is_healthy": true,
"last_completed_at": 1750716118.83112,
"last_tx": "3i1mGzNrprWcW3PnSRYognDXezFas55fzpP8aCfDSJePg6pc3mWq5LGRheAd8j733rdFyN75GprL2th5i3jB8HDH",
"since_last_completed_at": 0.21285104751586914
},
"reward_manager": {
"is_healthy": true,
"last_completed_at": 1750716118.184873,
"last_tx": "37776jhVY1vYvtSxtDNp4roo75H4iPEDN3w6bP388yWfTVWXao4ZLuPco8X3CVbdAkZDfkrZ8Vr6w6EjC6qjnVHH",
"since_last_completed_at": 0.8589239120483398
},
"spl_token": {
"is_healthy": true,
"last_completed_at": 1750716025.788301,
"last_tx": "4UQP7hvmmyjHAFEz3967rgVFRPbnsLVtNKJwgi35Eix6FVnSBT3Th9dwcq2ymrGoRDCjQTgw4TKmawpFMbqh2VzA",
"since_last_completed_at": 93.25533294677734
},
"user_bank": {
"is_healthy": true,
"last_completed_at": 1750716118.978945,
"last_tx": "4UQP7hvmmyjHAFEz3967rgVFRPbnsLVtNKJwgi35Eix6FVnSBT3Th9dwcq2ymrGoRDCjQTgw4TKmawpFMbqh2VzA",
"since_last_completed_at": 0.06445789337158203
}
},
"total_memory": 67508137984,
"transferred_bytes_per_sec": 0,
"trending_playlists_age_sec": 3555678,
"trending_tracks_age_sec": 2734,
"url": "https://discoveryprovider3.audius.co",
"used_memory": 27486191616,
"user_balances_age_sec": 153,
"version": "0.7.182",
"web": {
"blockhash": null,
"blocknumber": 0
}
},
"latest_chain_block": null,
"latest_chain_slot_plays": 5852972,
"latest_indexed_block": 5852972,
"latest_indexed_slot_plays": 5852972,
"signer": "0xf2897993951d53a7e3eb2242d6a14d2028140dc8",
"success": true,
"version": {
"service": "discovery-node",
"version": "0.7.182"
}
}

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide repro instructions & any configuration.
@alecsavvyalecsavvy
self-assigned this
Jun 23, 2025
@changeset-bot
changeset-botBot
commented
Jun 23, 2025•

edited

Loading

Copy link


Copy Markdown

⚠️ No Changeset found
Latest commit: e58597c
Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.
This PR includes no changesets
When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types
Click here to learn what changesets are, and how to add one.
Click here if you're a maintainer who wants to add a changeset to this PR
raymondjacobson
raymondjacobson
approved these changes
Jun 23, 2025
View reviewed changes
@raymondjacobsonraymondjacobson
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay as a bandaid. If we weren't rewriting indexing, I'd really dislike this, but it's already so screwed up as is, I'm down with this temp solve
@alecsavvy
alecsavvy
enabled auto-merge (squash)
June 23, 2025 22:41
@alecsavvy
alecsavvy
merged commit d4ecccf
into
mainJun 24, 2025
@alecsavvy
alecsavvy
deleted the

as/fix-health

branch
June 24, 2025 15:53
isaacsolo pushed a commit
that referenced
this pull request
Jun 24, 2025
@alecsavvyisaac
use plays health in health check (#12385)
fc0920d
audius-infra pushed a commit
that referenced
this pull request
Jun 25, 2025
audius-infra
Audius Protocol v0.7.184
16ea1af
[d4ecccf] use plays health in health check (#12385) alecsavvy
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@raymondjacobsonraymondjacobsonraymondjacobson approved these changes

Assignees

@alecsavvyalecsavvy

Labels

size/M

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@alecsavvy@raymondjacobson