Add worker reconnection logging to durabletask-go

[Jean Carlos Magalhães (jeanmartins)]

Changes:

• Added consecutiveErrors counter to track connection failures
• Log "reconnected and ready to process work items" when worker recovers from connection errors
• Works for both scenarios: when there are work items to process and when there are none
• Each worker (activity-processor and orchestration-processor) logs its own reconnection status

Benefits:

• Better visibility into worker health and connection status
• Easier debugging of connection issues
• Clear indication when workers successfully reconnect after failures

Testing:

Tested by stopping PostgreSQL container and observing reconnection logs when connection

Evidence:

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds tracking of repeated fetch errors to improve logging when the worker recovers and can process items again.

Changes:

• Introduces an atomic counter on worker to track consecutive fetch failures.
• Adds “reconnected and ready…” log lines when the worker recovers after prior failures.
• Increments the counter on non-context fetch errors and resets it on successful/no-item fetches.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

Suppressed comments (5)

backend/worker.go:120

• This "reconnected" log is currently gated by w.waiting, which appears to represent "no work items currently" rather than "connection was down". As a result, the message can be misleading (e.g., it will fire when the queue becomes non-empty after normal idle waiting, even if there was no connection failure). Consider keying this log off the error-tracking state (e.g., consecutiveErrors > 0) and resetting that state when logging, or changing the wording to reflect resuming from idle rather than reconnection.

 if w.waiting {
w.logger.Infof("%v: reconnected and ready to process work items", w.Name())
}

backend/worker.go:186

• The recovery logging/reset logic is duplicated across multiple branches. This increases the chance of inconsistencies (and makes it harder to adjust behavior later). Consider centralizing this into a small helper (e.g., maybeLogFetchRecovery()) or restructuring so the log/reset happens in exactly one place when a recovery condition is met.

 // Check if we recovered from errors when no work items
if w.consecutiveErrors.Load() > 0 {
w.logger.Infof("%v: reconnected and ready to process work items", w.Name())
w.consecutiveErrors.Store(0)
}

backend/worker.go:204

• The recovery logging/reset logic is duplicated across multiple branches. This increases the chance of inconsistencies (and makes it harder to adjust behavior later). Consider centralizing this into a small helper (e.g., maybeLogFetchRecovery()) or restructuring so the log/reset happens in exactly one place when a recovery condition is met.

 // Check if we recovered from errors when successfully fetching work item
if w.consecutiveErrors.Load() > 0 {
w.logger.Infof("%v: reconnected and ready to process work items", w.Name())
w.consecutiveErrors.Store(0)
}

backend/worker.go:50

• The field name/comment indicate "connection failures", but the counter is incremented on any FetchWorkItem error (not necessarily connection-related). To avoid confusing future maintainers, either rename the field to match current usage (e.g., consecutiveFetchErrors) and update the comment accordingly, or narrow the increment logic to only count errors that actually represent connection failures.

	// consecutiveErrors tracks connection failures for reconnection logging
consecutiveErrors atomic.Int32

backend/worker.go:186

• The Load() > 0 followed by Store(0) is not an atomic "check-and-reset". Since the counter is an atomic.Int32, it’s safer to use a single atomic operation (e.g., Swap(0) and check the returned value) to avoid losing increments that happen between the load and store.

 if w.consecutiveErrors.Load() > 0 {
w.logger.Infof("%v: reconnected and ready to process work items", w.Name())
w.consecutiveErrors.Store(0)
}

[Jean Carlos Magalhães (jeanmartins)]

Hi Chris Gillum (@cgillum) , please review it

[Chris Gillum (cgillum)]

Tomer Rosenthal (@torosent) is there’s another engineer that can review this PR until I get back from vacation?