Uh oh!
There was an error while loading. Please reload this page.
[CON-301] add prometheus to queues - #3595
Conversation
can we replace the some other queues we should monitor should be the imageProcessingQueue, transcodingQueue, updateReplicaSetQueue. I would take a look at these queues here https://github.com/AudiusProject/audius-protocol/blob/master/creator-node/src/serviceRegistry.js#L51 |
theoilie
left a comment
There was a problem hiding this comment.
good stuff! couple nits/questions:
- I'm wondering why this uses ms instead of seconds like the other prometheus metrics?
- can we remove the prefix label? I don't think we ever change it from "bull"
- it looks like the
JOBS_DURATION_MILLISECONDS_HISTOGRAMtracks job runtime? if so, we should exclude the jobs from the metric that already tracks that to avoid duplication in theSTATE_MACHINE_${jobName}_JOB_DURATION_SECONDS_HISTOGRAMmetric -- see here
|
jonaylor89
commented
Aug 1, 2022
I think for anything state machine related, it only runs one job type |
vicky-g
commented
Aug 1, 2022
let's confirm this just to be sure. if it's just one job, then it's okay to not add job names! |
theoilie
commented
Aug 1, 2022
jonaylor89
commented
Aug 1, 2022
Quick note from a call @vicky-g and I had - skipping adding metrics to TranscodeQueue because adding metrics would require a bit more of a systemic lift. Should be something added later down the line though |
jonaylor89
commented
Aug 2, 2022
jonaylor89
commented
Aug 2, 2022
Or actually, let's keep these for now because I want to see if the labels show anything different. |
theoilie
left a comment
There was a problem hiding this comment.
one small question but LGTM! maybe we should have a Linear card to make sure we clean up the original duplicate duration monitoring metrics for state machine?
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
SidSethi
commented
Aug 2, 2022
btw, all prometheus metrics are meant to be recorded as seconds, or the "base unit" per docs https://prometheus.io/docs/practices/naming/ |
SidSethi
left a comment
There was a problem hiding this comment.
an additional test in prometheus.test.js might be valuable?
Uh oh!
There was an error while loading. Please reload this page.
jonaylor89
commented
Aug 2, 2022
Gotcha, I can change it over to seconds then, |
SidSethi
commented
Aug 2, 2022
tests broken |
## Changelog - 2022-08-04 [be35ce8] INF-210 fix || exit 1 logic (#3635) [Joaquin Casares] - 2022-08-04 [ae4bd30] [C-765] Add native libs entrypoint (#3626) [Dylan Jeffers] - 2022-08-04 [1e04215] [C-739] Patch @openzeppelin/upgrades Initializable.sol fields (#3584) [Dylan Jeffers] - 2022-08-04 [01f924a] Fix add track to playlist notification timestamp (#3629) [Isaac Solo] - 2022-08-03 [fcfdc3e] Remove unused labels (#3628) [Theo Ilie] - 2022-08-03 [2d2fcca] [CON-301] add prometheus to queues (#3595) [Johannes Naylor] - 2022-08-03 [798b7fa] CON-303 Cache CN 200-status /health_check responses for 1sec in nginx (#3624) [Sid Sethi] - 2022-08-03 [b33a56c] Throw error in syncImmediateQueue (#3625) [Dheeraj Manjunath] - 2022-08-03 [1239375] Tests for aggregate table triggers (#3623) [Steve Perkins] - 2022-08-03 [e164665] INF-210 Finishing changes (#3617) [Joaquin Casares] - 2022-08-03 [9b5f4c9] New queue for immediate sync (#3619) [Dheeraj Manjunath] - 2022-08-03 [5cef789] Fix attemptNumber not getting updated + increase queue histories (#3622) [Theo Ilie] - 2022-08-03 [683d545] Related artists index2 (#3621) [Steve Perkins] - 2022-08-03 [c3a0149] Change path due to recent libs change (#3620) [Cheran] - 2022-08-03 [216bf25] Land #21 from docs.audius.org (#3514) [Raymond Jacobson] - 2022-08-02 [54713a7] [C-736] Refactor AudiusLibs to typescript (#3597) [Dylan Jeffers] - 2022-08-02 [54b7db8] Fix runtime includes TypeError (#3178) [Dylan Jeffers] - 2022-08-02 [89e0db2] Bump sdk to v0.0.29 [audius-infra]
## Changelog - 2022-08-04 [2d720a6] Fix bundles (#3639) [Sebastian Klingler] - 2022-08-04 [d2b051d] INF-210 grab VERSION after exiting subshell that bumped version (#3636) [Joaquin Casares] - 2022-08-04 [27433f5] Remove extra done() callback and add sync job result (#3634) [Theo Ilie] - 2022-08-04 [be35ce8] INF-210 fix || exit 1 logic (#3635) [Joaquin Casares] - 2022-08-04 [ae4bd30] [C-765] Add native libs entrypoint (#3626) [Dylan Jeffers] - 2022-08-04 [1e04215] [C-739] Patch @openzeppelin/upgrades Initializable.sol fields (#3584) [Dylan Jeffers] - 2022-08-04 [01f924a] Fix add track to playlist notification timestamp (#3629) [Isaac Solo] - 2022-08-03 [fcfdc3e] Remove unused labels (#3628) [Theo Ilie] - 2022-08-03 [2d2fcca] [CON-301] add prometheus to queues (#3595) [Johannes Naylor] - 2022-08-03 [798b7fa] CON-303 Cache CN 200-status /health_check responses for 1sec in nginx (#3624) [Sid Sethi] - 2022-08-03 [b33a56c] Throw error in syncImmediateQueue (#3625) [Dheeraj Manjunath] - 2022-08-03 [1239375] Tests for aggregate table triggers (#3623) [Steve Perkins] - 2022-08-03 [e164665] INF-210 Finishing changes (#3617) [Joaquin Casares] - 2022-08-03 [9b5f4c9] New queue for immediate sync (#3619) [Dheeraj Manjunath] - 2022-08-03 [5cef789] Fix attemptNumber not getting updated + increase queue histories (#3622) [Theo Ilie] - 2022-08-03 [683d545] Related artists index2 (#3621) [Steve Perkins] - 2022-08-03 [c3a0149] Change path due to recent libs change (#3620) [Cheran] - 2022-08-03 [216bf25] Land #21 from docs.audius.org (#3514) [Raymond Jacobson] - 2022-08-02 [54713a7] [C-736] Refactor AudiusLibs to typescript (#3597) [Dylan Jeffers] - 2022-08-02 [54b7db8] Fix runtime includes TypeError (#3178) [Dylan Jeffers] - 2022-08-02 [89e0db2] Bump sdk to v0.0.29 [audius-infra]
## Changelog - 2022-08-04 [a3556c3] include v as VERSION prefix (#3641) [Joaquin Casares] - 2022-08-04 [7755c57] Update SDK (#3633) [Michael Piazza] - 2022-08-04 [3f6256f] INF-210 rename GIT_TAG to GIT_COMMIT (#3637) [Joaquin Casares] - 2022-08-04 [2d720a6] Fix bundles (#3639) [Sebastian Klingler] - 2022-08-04 [d2b051d] INF-210 grab VERSION after exiting subshell that bumped version (#3636) [Joaquin Casares] - 2022-08-04 [27433f5] Remove extra done() callback and add sync job result (#3634) [Theo Ilie] - 2022-08-04 [be35ce8] INF-210 fix || exit 1 logic (#3635) [Joaquin Casares] - 2022-08-04 [ae4bd30] [C-765] Add native libs entrypoint (#3626) [Dylan Jeffers] - 2022-08-04 [1e04215] [C-739] Patch @openzeppelin/upgrades Initializable.sol fields (#3584) [Dylan Jeffers] - 2022-08-04 [01f924a] Fix add track to playlist notification timestamp (#3629) [Isaac Solo] - 2022-08-03 [fcfdc3e] Remove unused labels (#3628) [Theo Ilie] - 2022-08-03 [2d2fcca] [CON-301] add prometheus to queues (#3595) [Johannes Naylor] - 2022-08-03 [798b7fa] CON-303 Cache CN 200-status /health_check responses for 1sec in nginx (#3624) [Sid Sethi] - 2022-08-03 [b33a56c] Throw error in syncImmediateQueue (#3625) [Dheeraj Manjunath] - 2022-08-03 [1239375] Tests for aggregate table triggers (#3623) [Steve Perkins] - 2022-08-03 [e164665] INF-210 Finishing changes (#3617) [Joaquin Casares] - 2022-08-03 [9b5f4c9] New queue for immediate sync (#3619) [Dheeraj Manjunath] - 2022-08-03 [5cef789] Fix attemptNumber not getting updated + increase queue histories (#3622) [Theo Ilie] - 2022-08-03 [683d545] Related artists index2 (#3621) [Steve Perkins] - 2022-08-03 [c3a0149] Change path due to recent libs change (#3620) [Cheran] - 2022-08-03 [216bf25] Land #21 from docs.audius.org (#3514) [Raymond Jacobson] - 2022-08-02 [54713a7] [C-736] Refactor AudiusLibs to typescript (#3597) [Dylan Jeffers] - 2022-08-02 [54b7db8] Fix runtime includes TypeError (#3178) [Dylan Jeffers] - 2022-08-02 [89e0db2] Bump sdk to v0.0.29 [audius-infra]
[1957c34] Add remove from playlist overflow action (#3621) Kyle Shanks [67e0012] [C-2712] Clear favorites on sign-out (#3620) Dylan Jeffers [97b6b94] [PAY-1421] Fix android reaction offsets (#3611) Michael Piazza [9867571] [PAY-1494] Change chats copy to beginning of message history (#3617) Reed [1b7ca71] Add overflow menu options for track list items on mobile (#3608) Kyle Shanks [810f0df] [PAY-1487] Increase hit slop of mobile dms send button (#3618) Reed [06dae51] [PAY-1470] Remove border radius from track/playlist tiles (#3613) Marcus Pasell [dbdfb84] [PAY-1469] Clear the search after creating a chat (#3616) Marcus Pasell [3f31554] [PAY-1489] Add more space to reacted to messages (#3615) Marcus Pasell [ed4f321] [PAY-1473] Clear input on chat change (#3614) Marcus Pasell [e9683d9] Use emojis instead of 'heart' etc as reactions (#3602) Marcus Pasell [192fd44] [PAY-1486] DMs: Fix purple border around unfurls (#3612) Marcus Pasell [b0c2e4a] Lowercase library restricted handle (#3590) Isaac Solo [a8daa72] Change text to 'Beginning of Conversation' (#3605) Marcus Pasell [332c0c0] Developer app modal - small fixes (#3609) nicoback2 [0e43db1] [PAY-1476] Inbox unavailable copy update (#3607) Reed [d862fd5] [PAY-1484] Allow reactions while keyboard is open on mobile (#3606) Reed [3bd5928] [PAY-944] DMs: Add Mobile Web drawer prompting app download (#3565) Marcus Pasell [5bea105] Wait for libs to init before initting SDK (#3604) Marcus Pasell [f4d28e0] Add option to cache bust node_modules (#3603) Dylan Jeffers [9b165c4] [PAY-1455] Optimize reactions perf (maybe) (#3599) Michael Piazza [0a1a68b] [PAY-1465] Auto-nav to chat only when permitted (#3598) Reed [72e9f92] [C-2665C-2714] Add developer apps settings modal (#3589) Dylan Jeffers [b787344] [PAY-1456] Fix followee-gated create chat crash (#3593) Reed [dbae5bd] Reduce MessageLockedButton size on profile screen (#3597) Reed [097879b] [PAY-1437] Fix drawers sharing state causing crash (#3595) Reed [818b215] [C-2770] Fix issue where playlist library can disappear (#3596) Dylan Jeffers [f3ada88] [PAY-1461] Format follower count in chat search screen (#3594) Reed [eb924e7] [C-2713] Add track metadata fields to new upload form (#3574) Andrew Mendelsohn [be56d1f] Fix mobile textinput borders + labels (#3592) Reed [fc7541b] Update collection list id flow to add the create tile (#3591) Kyle Shanks [dc15518] [PAY-1432] Fix chat playlist artist sometimes undefined (#3585) Reed [399d0b5] [C-2588] Update share modal and drawer to properly add user twitter handle (#3584) Kyle Shanks [9f30855] [C-2761] Add mutations to audius-query (#3581) Dylan Jeffers [597dfe5] [PAY-1448] End the race between SDK and Libs, once and for all (#3588) Marcus Pasell [56914cf] [C-2768] Update InputV2, add TextField/TextAreaField (#3587) Dylan Jeffers [c9358b6] [CON-742] Make users and track actions v2 for v2 users/tracks even when feature flags are disabled (#3582) Theo Ilie [a94f035] audius-query debug flag usage in README (#3586) Reed [bceb3fe] [INF-417] Generate sourcemaps when running web:prod (#3583) Sebastian Klingler [f4ad466] [PAY-1454] Redline mobile header to latest figma design (#3576) Michael Piazza [eec6af0] [PAY-1405] Add padding to bottom of chat user search screen (#3580) Reed [a6c8ece] [PAY-1435] Navigate to chat after unblocking user on mobile (#3578) Reed [ee1b6c3] [PLAT-659] Metamask sign in/up on ACDC (#3563) Raymond Jacobson [fbd4053] [C-2472] Update mobile track list items for playlists to have the overflow menu (#3577) Kyle Shanks [51f44ac] [PAY-1399] Fix Android chat send button color (#3575) Reed [7d440de] [PAY-1442] Fix drawer touchables on android (#3572) Reed [8c90ff6] Allow drag and dropping tracks into chat (#3573) Michael Piazza [f7489e0] Upgrade dapp store version to v1.5.26 (#3504) Dylan Jeffers [6be0c64] [PAY-1440][PAY-1441] Mobile inbox settings UI fixes (#3571) Reed [001007e] [PAY-1434] Dismiss keyboard on create chat screen row press (#3569) Reed [c70e9fe] [PAY-1446] Set MessageLocked icon in profile screen to 0.4 opacity (#3570) Reed [2cc8ad0] [C-2716] Improve notifications on fresh account (#3568) Dylan Jeffers [01f05ba] [C-2474] Generate playlist artwork from track artwork (#3518) Dylan Jeffers [0355c10] [PAY-1325] Remove userbadges from chat unavailable message in chat screen (#3567) Reed [da10bbc] [PAY-1389] CTA on create chat when not permitted (#3566) Reed
Description
This PR add prometheus metrics to many of the queues used for state monitoring and state reconciliation.
Sample metrics
or
Tests
Run locally and tested with
A seed upload-trackand checking the metrics onlocalhost:4000/prometheus_metricsMonitoring - How will this change be monitored? Are there sufficient logs / alerts?
These changes will be monitored with prometheus metrics
and soon grafana panels