Skip to content

(Content Node) Add dashboard for visualizing Bull queues + change cron to re-queuing on completion/failure - #3055

Merged
theoilie merged 12 commits into
masterfrom
theo-content-bull-board
May 11, 2022
Merged

(Content Node) Add dashboard for visualizing Bull queues + change cron to re-queuing on completion/failure#3055
theoilie merged 12 commits into
masterfrom
theo-content-bull-board

Conversation

@theoilie

@theoilietheoilie commented May 6, 2022•
edited
Loading

Copy link
Copy Markdown
Contributor

Description

  • Add UI to see the status of Content Node's Bull queues in a new /health/bull endpoint
  • Keep the last 500 failed and 500 successful jobs from the state machine queue
  • Re-queue state machine job whenever another job completes/fails instead of adding jobs on a cron

Tests

I let this run locally, made sure the dashboard looks accurate, and verified that memory usage did not increase too much with 500 successful jobs. Also ran on staging over the weekend.

How will this change be monitored? Are there sufficient logs?

  • Search the logs for Failed to initialize bull monitoring UI.
  • Once jobs are faster, keep an eye out for what happens with an extremely large number. The job gets re-added, so its jobId increases at a much, much faster rate. We need to see if this will ever overflow the number or if it will just restart at jobId=1.

@theoilie
theoilieforce-pushed the theo-content-bull-board branch from 906e6dd to a87fc8aCompareMay 7, 2022 00:13
@theoilietheoilie changed the title (Content Node) Add dashboard for visualizing Bull queues(Content Node) Add dashboard for visualizing Bull queues + change cron to re-queuing on completion/failureMay 9, 2022
@theoilie
theoilie marked this pull request as ready for review May 9, 2022 17:38
@theoilie
theoilie requested a review from SidSethiMay 9, 2022 17:38
@theoilie

theoilie commented May 9, 2022

Copy link
Copy Markdown
ContributorAuthor

The dashboards were on many staging CNs over the weekend, and the cron change was/is on CN5 with no new issues. Also just now re-confirmed locally that jobs get re-queued and consumed at the rate they complete at rather than waiting for the cron each time.

SidSethi
SidSethi reviewed May 10, 2022
View reviewed changes
Comment threadcreator-node/src/index.ts Outdated
Comment threadcreator-node/src/snapbackSM/snapbackSM.js Outdated
Comment threadcreator-node/src/snapbackSM/snapbackSM.js
Comment threadcreator-node/src/snapbackSM/snapbackSM.js
Comment threadcreator-node/src/snapbackSM/snapbackSM.js
Comment threadcreator-node/src/app.js Outdated
Comment threadcreator-node/src/app.js Outdated
@theoilie
theoilie requested a review from SidSethiMay 10, 2022 21:39
@theoilietheoilie mentioned this pull request May 10, 2022
Merged
SidSethi
SidSethi approved these changes May 10, 2022
View reviewed changes

@SidSethiSidSethi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

couple nits but otherwise looks great! can merge after addressing

Comment threadcreator-node/src/serviceRegistry.js
Comment threadcreator-node/src/components/healthCheck/healthCheckController.js Outdated
Comment threadcreator-node/src/snapbackSM/snapbackSM.js Outdated
@theoilie
theoilie merged commit 7685838 into masterMay 11, 2022
@theoilie
theoilie deleted the theo-content-bull-board branch May 11, 2022 00:09
sliptype pushed a commit that referenced this pull request Sep 10, 2023
@dylanjeffers
[C-429] Fix edit-profile screen scroll (#3055)
c78c32d
sliptype pushed a commit that referenced this pull request Sep 10, 2023
audius-infra
v1.5.16
aae3bd8
[855cf34] [C-2329] Fix notification navigation (#3060) Dylan Jeffers
[aeccfee] [C-429] Fix edit-profile screen scroll (#3055) Dylan Jeffers
[5a7bacd] Update dapp-store release and docs (#3043) Dylan Jeffers
[5650435] [PAY-1061] - Only show confetting when unlocking from how to unlock section (#3042) Saliou Diallo
[49fe95d] [C-2263C-2283] Add setCurrentUser profile action and call when we focus a profile screen (#3058) Kyle Shanks
[1ca8e12] [C-2286] Update placement of the repeat single track check from play saga to next saga (#3054) Kyle Shanks
[07e42d7] [C-2266] Disabled the PlayBar favorite button for the track owner (#3053) Kyle Shanks
[3f2e1fe] Use wallet instead of ERC wallet (#3048) Michael Piazza
[0838aa6] Fix confirmer lint (#3047) Dylan Jeffers
[da900a7] Confirm if block passed (#3046) Isaac Solo
[1b5efbe] Bump mobile versions (#3045) Sebastian Klingler
[ea2b4c9] [PAY-997][PAY-1012][PAY-1050][PAY-1053][PAY-1066] - Miscellaneous QA fixes (#3035) Saliou Diallo
[65683e5] Fix chat middleware (#3041) Reed
[d401abf] [C-2276] Fix upload track store updates (#3018) Dylan Jeffers
[c77334a] [C-2156] Refactor search-screen ui (#3030) Dylan Jeffers
[0f449e8] [C-2311] Fix notification timestamp logic (#3033) Dylan Jeffers
[e220d43] Revert "Hard code entity manager prod address for now" (#3040) nicoback2
[0209d7e] Hard code entity manager prod address for now (#3039) nicoback2
[ddfddc7] Add nvm install step (#3038) Raymond Jacobson
[683a48b] [PAY-926] DMs: Desktop Inbox Settings tile on Settings Page (#3034) Marcus Pasell
[1a9e5c9] Fix prod ACDC contract address (#3036) Isaac Solo
[bde1f98] [PAY-940] Save permissions in chat permission dialog (#3027) Marcus Pasell
[a049a78] Fix collapsible list typo (#3032) Dylan Jeffers
[2d501e2] Add playback position logic on web for podcasts (#3019) Kyle Shanks
[b99602a] Fix native code changes CI comment C-2305 (#3031) nicoback2
[483b4d8] [C-583] Fix section/flat list generics (#3026) Dylan Jeffers
[efb1b9e] Upgrade mobile app versions for release (#3029) Dylan Jeffers
[170d04d] [C-2220] Upgrade push-notification library (#2994) Dylan Jeffers
[63e731d] [C-2301] Update dapp-store release, Improve docs (#3028) Dylan Jeffers
[ff64e58] [PAY-1037][PAY-1036][PAY-1034] DMs: Font weight, spacing, and reaction button animation (#3025) Marcus Pasell
[542e48b] Add custom mandatory update UI for mandatory OTA updates C-2258 (#3023) nicoback2
[7672b71] Mobile CI Improvements C-2256C-2284C-2285 (#3017) nicoback2
[97fde45] [C-2288] Fix auth error state check (#3024) Sebastian Klingler
[f1cc072] [C-2137] Fix mobile status bar theme (#3022) Dylan Jeffers
[0a5e84a] Re-add chat features to profile screen (#3016) Reed
[5557cd5] Remove upgradeToCreator (#3001) Theo Ilie
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@SidSethiSidSethiSidSethi approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

size/L

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@theoilie@SidSethi