Uh oh!
There was an error while loading. Please reload this page.
[CON-152] Add state machine metrics - job runtime length and more - #3329
Conversation
theoilie
commented
Jun 28, 2022
@SidSethi just hoping to get preliminary feedback before adding more metrics, unless you think this is enough to go through by itself 😄 |
SidSethi
left a comment
There was a problem hiding this comment.
generally looks great!! love the dynamic metric creation
only thing i can think of is perhaps a bit more granularity on the error conditions - e.g. instead of "uncaughtError" boolean, a "status" label which is either "success" or an error msg. the issue with this is it can't be arbitrary - there has to be a discrete set of possible values
this seems good as a first pass imo, just to get a general idea, and we can dig into logs when we see errors
imo good to ship as is - i see its in draft mode so lmk when ready for full review
Uh oh!
There was an error while loading. Please reload this page.
theoilie
commented
Jun 28, 2022
ok great thanks for the review! the reason I did "uncaughtError" is because it's generic to each job. are you saying I should standardize each job's output to return an |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
SidSethi
left a comment
There was a problem hiding this comment.
looks great, if tests pass can merge/release 🚀 can you check in on it a bit after release
[eaaf509] [C-2565] Improve mobile app-redirect-popover experience (#3344) Dylan Jeffers [1755765] Update sdk to 2.0.3-beta.18 (#3345) Sebastian Klingler [571c991] [CON-676] Support storage v2 playlists+albums, and bump libs version to 2.0.3-beta.17 (#3314) Theo Ilie [3cca778] [C-2577] Fix mobile web social buttons by removing pull-to-refresh (#3343) Dylan Jeffers [0915ff4] [C-2519] Ignore TOS deeplink (#3341) Dylan Jeffers [39fda1a] Feature flag gates to send metadata directly through chain (#3311) Michelle Brier [80278ce] Upgrade sdk to beta.16, update install script (#3339) Dylan Jeffers [7c3aaa5] [C-2451] Fix infinite buffering in audio player (#3336) Sebastian Klingler [be68f2e] Prevent fetchBlockees/blockers when chat disabled (#3338) Dylan Jeffers [779cfa5] [C-2568] Silence player/error when audio is loaded (#3330) Sebastian Klingler [869504c] HOTFIX track notification navigation (#3335) Dylan Jeffers [f30507c] Dismiss keyboard on mobile chat screen top right press (#3333) Reed [ad7938c] DMs: Don't increment unread if it's your own message (#3331) Marcus Pasell [565f182] Fix keyboard occluding textinput in mobile chat screen (#3332) Reed [a82cf59] DMs: Fix unread indicator not going away (#3329) Marcus Pasell [e47f323] Mobile chat housekeeping (#3288) Reed [47a5e84] [PAY-1207] Use Followers instead of Mutuals on compose chat modal (#3323) Marcus Pasell [03c587d] [C-2393] Cache pods in CI (#3317) Sebastian Klingler [22f4d46] Remove identity subscriptions reads/writes (#3005) Michelle Brier [b9ddae1] [C-2566] Prevent player error on skip (#3327) Dylan Jeffers [ca246f9] DMs: Fix store bug sending messages (#3326) Marcus Pasell [309af58] Update fetchPermission condition (#3324) Dylan Jeffers [a8e301e] [PAY-1199] Mobile inbox settings functional (#3290) Reed [adc7854] [C-2563] Fix demo build (#3322) Sebastian Klingler [d703f6e] Fix search-results saga error due to yield* (#3321) Dylan Jeffers [d940ae6] Fix internal/external audius urls (#3320) Dylan Jeffers [96cd42f] [C-2560] Improve checking for new notifs (#3316) Dylan Jeffers [3130546] I broke main and I'm sorry (#3319) Marcus Pasell [a3062fe] Remove dependency on `download.cid` for determining download eligibility (#3307) Michael Piazza [fe04046] [PAY-901] hotfix: Use SPL amounts for BuyAudio flow (#3318) Marcus Pasell [1c94f06] Fix mobile prem content track headers (#3315) Reed [f87cb04] [PAY-1108] Chats Screen avoids play bar (#3295) Reed [c5450a0] Revert "[PAY-901] hotfix: TransactionDetailsModal now expects wei amounts" (#3313) Marcus Pasell [e3ca9e3] Fix stylelint (#3312) Raymond Jacobson [2031f1a] Debounce reaction for longer on web (#3305) Raymond Jacobson [492fa67] Notification settings improvements (#3264) Michelle Brier [59710d6] Rename sdk_v2 to sdk_discovery_node_selector (#3310) Dylan Jeffers [8519910] [PLAT-863] Fix reaction endpoint when sdk-v2 (#3309) Dylan Jeffers [00ba722] Fix index issue with constructUrl (#3308) Dylan Jeffers [b0b32ff] [PLAT-904] Fix duplicate notifications (#3304) Dylan Jeffers [14c7814] Pass metadata to libs for addTracksToChainAndCnode (#3285) Michelle Brier [75456fd] Add search multimap (#3306) Raymond Jacobson [f8f5f02] [C-2282] Show tracks tab on own profile if all tracks hidden (#3294) Andrew Mendelsohn [49fc834] [PAY-901] hotfix: TransactionDetailsModal now expects wei amounts (#3297) Marcus Pasell [f1366ac] Fix eager fetch (#3302) Raymond Jacobson [4b639ba] [C-2542] Update delete playlist confirmation drawer to conform to new designs (#3303) Kyle Shanks [1ac5ec2] Fix taste maker copy (#3301) Isaac Solo [de48e11] [PAY-1178] Fix new messages on new/unloaded chats (#3298) Marcus Pasell [ea6ed55] Redirect if not allowed ai (#3300) Raymond Jacobson [985a05c] Fix ai attrib upload (#3299) Raymond Jacobson [a586435] [C-2554] Add AI Attribution (#3296) Dylan Jeffers [64dde7f] [CON-674] Make track edit use v2 flow for storage v2 tracks (#3289) Theo Ilie [d38d483] [C-2543, C-2544] Initial updates to mobile playlist management (#3287) Kyle Shanks [ec9dc0e] Fix devnet in wallet authorize for saga phone (#3291) Raymond Jacobson
Description
Commit to add timings (cb6a613)
uncaughtError(true/false) labelCommit with the rest of the changes (b75d8ac)
aggregateReconfigAndPotentialSyncOpsand its tests since it was previously refactored into smaller functionsmanualandrecurringlabels to existingISSUE_SYNC_REQUEST_MONITORING_DURATION_SECONDS_HISTOGRAMmetric so we can track manual vs recurring syncs separatelycNodeEndpointToSpIdMapin find-replica-set-updates job output for better debuggingTests
cNodeEndpointToSpIdMapto output/prometheus_metricsendpointMonitoring - How will this change be monitored? Are there sufficient logs / alerts?
Check the
/prometheus_metricsendpoint and see if metrics for the timing of each job show up with label values that seem reasonable. It should look similar to the screenshots above.