Skip to content

[CON-324] Add orphaned data recovery queue - #3680

Merged
theoilie merged 3 commits into
masterfrom
theo-orphaned-data-queue
Aug 12, 2022
Merged

[CON-324] Add orphaned data recovery queue#3680
theoilie merged 3 commits into
masterfrom
theo-orphaned-data-queue

Conversation

@theoilie

@theoilietheoilie commented Aug 11, 2022

Copy link
Copy Markdown
Contributor

Description

  • Add queue to process jobs for a node to find+reconcile data that's on it but that belongs to a user who doesn't have the node in their replica set
  • Add TODOs for steps of the job (ready to high level review but not implementation details or else this PR would balloon)

Tests

  • Updated tests for StateReconciliationQueue and added skeleton for future tests to be completed once the job processor is implemented
  • Verified that the queue runs locally at the correct rate limit (1 job per minute)

Monitoring - How will this change be monitored? Are there sufficient logs / alerts?

  • Search for the queue's logs that will be added in a followup PR (filter by json.queue:"recover-orphaned-data-queue")
  • Monitor the queue's Bull dashboard (/health/bull/queue/recover-orphaned-data-queue endpoint)

@theoilietheoilie added the content-node Content Node (previously known as Creator Node) label Aug 11, 2022
dmanjunath
dmanjunath reviewed Aug 12, 2022
View reviewed changes
Comment threadcreator-node/src/config.js
SidSethi
SidSethi reviewed Aug 12, 2022
View reviewed changes
Comment threadcreator-node/src/services/stateMachineManager/stateMachineConstants.ts
Comment thread...ode/src/services/stateMachineManager/stateReconciliation/recoverOrphanedData.jobProcessor.ts
Comment thread...ode/src/services/stateMachineManager/stateReconciliation/recoverOrphanedData.jobProcessor.ts
Comment thread...ode/src/services/stateMachineManager/stateReconciliation/recoverOrphanedData.jobProcessor.ts
@theoilie

theoilie commented Aug 12, 2022

Copy link
Copy Markdown
ContributorAuthor

changed the confusing rate limit in tests -- ready for re-review if any other changes are needed!

@theoilie
theoilie requested a review from SidSethiAugust 12, 2022 18:51
@theoilie
theoilie merged commit 1afa5d5 into masterAug 12, 2022
@theoilie
theoilie deleted the theo-orphaned-data-queue branch August 12, 2022 19:15
audius-infra pushed a commit that referenced this pull request Aug 12, 2022
audius-infra
Bump sdk to v0.0.34
4143fbb
## Changelog
- 2022-08-12 [c38a6a8] Update @solana/spl-token to 0.1.8 (#3689) [Marcus Pasell]
- 2022-08-12 [587b38e] CON-327 Miscellaneous secondarySyncFromPrimary() improvements to logic + metric tracking (#3678) [Sid Sethi]
- 2022-08-12 [1afa5d5] [CON-324] Add orphaned data recovery queue (#3680) [Theo Ilie]
- 2022-08-12 [10599f4] Handle zero error code (#3686) [Saliou Diallo]
- 2022-08-12 [2346960] Use abuse error codes in relay (#3683) [Raymond Jacobson]
- 2022-08-12 [c3f578e] populate save_count in feed endpoint (#3684) [Steve Perkins]
- 2022-08-11 [5e6217f] Revert "[PLAT-259] Upgrade EthereumTx version (#3606)" (#3682) [Raymond Jacobson]
- 2022-08-11 [a44aad2] Fix bad notifs (#3681) [Saliou Diallo]
- 2022-08-11 [23a6c3b] [PAY-485][PAY-491] Handle premium content signature (#3610) [Saliou Diallo]
- 2022-08-11 [a478a1d] [PAY-459] Add dethroned notifications on Identity (#3674) [Michael Piazza]
- 2022-08-11 [8fb6084] Add a log before force resync happen (#3677) [vicky :)]
- 2022-08-11 [47f55f4] [PAY-506] Exposes AAO error code (#3662) [Reed]
- 2022-08-11 [4bc58cf] Filter out notifications from bad initiators (#3676) [Saliou Diallo]
- 2022-08-11 [e01c91c] Prevent extra unnecessary reconfig events (#3671) [vicky :)]
- 2022-08-11 [cff11a9] Fix downgrades (#3667) [Joseph Lee]
- 2022-08-11 [424ff50] Add disable flag to logspout (#3672) [Dheeraj Manjunath]
- 2022-08-11 [bac7669] Bump sdk to v0.0.33 [audius-infra]
audius-infra pushed a commit that referenced this pull request Aug 12, 2022
audius-infra
Bump sdk to v0.0.34
82dcf35
## Changelog
- 2022-08-12 [c38a6a8] Update @solana/spl-token to 0.1.8 (#3689) [Marcus Pasell]
- 2022-08-12 [587b38e] CON-327 Miscellaneous secondarySyncFromPrimary() improvements to logic + metric tracking (#3678) [Sid Sethi]
- 2022-08-12 [1afa5d5] [CON-324] Add orphaned data recovery queue (#3680) [Theo Ilie]
- 2022-08-12 [10599f4] Handle zero error code (#3686) [Saliou Diallo]
- 2022-08-12 [2346960] Use abuse error codes in relay (#3683) [Raymond Jacobson]
- 2022-08-12 [c3f578e] populate save_count in feed endpoint (#3684) [Steve Perkins]
- 2022-08-11 [5e6217f] Revert "[PLAT-259] Upgrade EthereumTx version (#3606)" (#3682) [Raymond Jacobson]
- 2022-08-11 [a44aad2] Fix bad notifs (#3681) [Saliou Diallo]
- 2022-08-11 [23a6c3b] [PAY-485][PAY-491] Handle premium content signature (#3610) [Saliou Diallo]
- 2022-08-11 [a478a1d] [PAY-459] Add dethroned notifications on Identity (#3674) [Michael Piazza]
- 2022-08-11 [8fb6084] Add a log before force resync happen (#3677) [vicky :)]
- 2022-08-11 [47f55f4] [PAY-506] Exposes AAO error code (#3662) [Reed]
- 2022-08-11 [4bc58cf] Filter out notifications from bad initiators (#3676) [Saliou Diallo]
- 2022-08-11 [e01c91c] Prevent extra unnecessary reconfig events (#3671) [vicky :)]
- 2022-08-11 [cff11a9] Fix downgrades (#3667) [Joseph Lee]
- 2022-08-11 [424ff50] Add disable flag to logspout (#3672) [Dheeraj Manjunath]
- 2022-08-11 [bac7669] Bump sdk to v0.0.33 [audius-infra]
dmanjunath pushed a commit that referenced this pull request Aug 16, 2022
@theoilie@dmanjunath
[CON-324] Add orphaned data recovery queue (#3680)
659c233
@AudiusProjectAudiusProject deleted a comment from linearBotSep 11, 2023
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

2 more reviewers
@dmanjunathdmanjunathdmanjunath left review comments
@SidSethiSidSethiSidSethi approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

content-nodeContent Node (previously known as Creator Node)size/L

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@theoilie@dmanjunath@SidSethi