CON-250 - stateMachine updates to consume primarySyncFromSecondary (Divergent state recovery #3)

[SidSethi]

Description

Update findSyncRequests.jobProcessor to remove the placeholder TODO for primarySyncToSecondary, instead it now correctly processes state when syncMode = MergePrimaryAndSecondary
Update issueSyncRequest.jobProcessor to handle syncMode = MergePrimaryAndSecondary:

• it now calls primarySyncFromSecondary(), then issues sync request to secondary with forceResync = true
• Adds envvar mergePrimaryAndSecondaryEnabled defaulted to true for local, and false for prod

Also:

• lots of test coverage for new findSyncRequests changes
• some code refactor to simplify internal data formats, logging, and metric recording
• update find_sync_request_counts prometheus gauge to record syncMode as separate label

Tests

New test coverage

Monitoring - How will this change be monitored? Are there sufficient logs / alerts?

Logs containing Recorded findSyncRequests from primary
Check /prometheus_metrics for find_sync_request_counts
Logs containing Sync mode is disabled
Logs containing primarySyncFromSecondary failed with error

[SidSethi]

notable changes here:

findSyncsForUser now returns outcomes with syncMode in addition to result
instead of SyncPrimaryFromSecondary having its own log + result type (see prometheus.constants changes), it is now a different label, since all the other result types also apply to that sync mode
most other changes follow on from above, including refactor + simplification of metric recording + logging code

[dmanjunath]

not as much comments as questions. not as familiar with this new job processor code so trying to learn it as i go. i'll do another pass but just left the first round of questions