[CON-118 + CON-126] (Content Node) Paginate getNodeUsers in snapback

[theoilie]

Description

• Update Content Node to consume the pagination from [CON-118 + CON-126] (Discovery Node) Paginate /v1/full/users/content_node route #3071. This allows the first step of the state machine (getNodeUsers()) to quickly fetch a slice of users instead of having to fetch all users and then manually slice them.
• Expose new config option snapbackUsersPerJob via health check
• Maintain backwards compatibility by performing the manual slicing if getNodeUsers() returns the full list of users instead of a paginated slice

Tests

• Deployed to staging and observed it cycling through the users. For example, the processStateMachineOperation() Decision Tree logs lastProcessedUserId as 0, then 1000, then 2000, etc...
• Tested staging endpoint manually with the prev_user_id_ and max_usersparams to verify that it returns no more thanmax_usersnumber of users and that the first user returned has an ID that's greater thanprev_user_id` (e.g. https://discoveryprovider2.staging.audius.co/v1/full/users/content_node/all?creator_node_endpoint=https://creatornode11.staging.audius.co&prev_user_id=0&max_users=100)
• Verified backwards compatibility by going running locally with an outdated Discovery Node and making sure default slicing behavior worked with greater than max_users being returned from getNodeUsers()

How will this change be monitored? Are there sufficient logs?

• Look through Content Node logs containing processStateMachineOperation or StateMachineQueue and verify that lastProcessedUserId is increasing by snapbackUsersPerJob (from the health check) each time

[vicky-g]

good stuff! i do have some comments / concerns. let's discuss this via voice chat

[theoilie]

thanks @vicky-g! I resolved all your comments related to bull-board here and addressed them in another PR (#3055). also going to break apart the Discovery Node portion into its own PR so no need to re-review til those PRs get through and I rebase this one off of them 😄

[SidSethi]

sweet! generally looks great, couple small comments
also if you could update pr desc with:

• remove part about adding DN pagination since thats in separate PR, instead add reference to that PR
• explicitly call out how you handled back-compat
• explain why this pagination was added
• in Test part, explicitly call out testing of both new and old flows, ideally with some assurance of both pagination & modulo based increments working correctly

[theoilie]

updated PR description with everything requested and made a followup PR (#3102) for the rest of your comments @SidSethi thanks!