CON-327 Miscellaneous secondarySyncFromPrimary() improvements to logic + metric tracking

[SidSethi]

Description

Few improvements to secondarySyncFromPrimary() flow, specifically the forceResync auth from this PR

1. Exit early if forceResync query param is set, but shouldForceResync() verification fails + record metric label value
2. Properly handle deleteAllCNodeUserDataFromDB() errors + record metric label value
3. Add new mode for secondary_sync_from_primary_duration_seconds metric to distinguish force_resync from default
4. Set sync-processing-queue job result on success (this is just adding back old functionality that may have been accidentally removed)

Tests

Existing automated test coverage
Manual steps: A up -> A seed create-user -> check /prometheus_metrics for a secondary to ensure secondary_sync_from_primary_duration_seconds shows mode label -> manually trigger sync with forceResync = true and confirm expected metric values

Monitoring - How will this change be monitored? Are there sufficient logs / alerts?

For secondary_sync_from_primary_duration_seconds prometheus metric, track mode label, and watch for new result label values failure_force_resync_check and failure_delete_db_data
Also in bull dashboard, can now see sync-processing-queue job results

[vicky-g]

can we add to the bio an example of what the metrics would look like if a force resync happened?

[SidSethi]

can we add to the bio an example of what the metrics would look like if a force resync happened?

@vicky-g yep, updated

[vicky-g]

how do we know if a force resync happened successfully \ do we care to know if a force resync happened? like we'll have metrics on the "default" mode, which is a successful sync, but we dont know if in this flow if a force resync happened. maybe we also want to track that flow

edit** nvm, it's in the flow