fix(worker): Fix issues with gracefully shutting down

[brendan-kellam]

This PR resolves a couple of issues:

• During termination, we call worker.close() to terminate any in progress jobs. This takes a gracefulTimeout param (defaults 30 seconds). If there are any jobs still in progress after that duration, there is a bug in groupmq where the job's group lock will not be released (see [bug] Group lock is not released after graceful shutdown, resulting in stalling jobs Openpanel-dev/groupmq#8). This resulted in jobs (especially long running connection jobs) being left in PENDING or IN_PROGRESS indefinitely, and new jobs not being queued until the timeout expires. The current workaround is to explicitly free the locks when calling dispose.
• Improved the graceful shutdown handling in index.ts

Summary by CodeRabbit

Release Notes

Bug Fixes

• Implemented timeout handling for background jobs; timed-out jobs are now properly marked as failed with status updates and metrics.
• Added job state validation to prevent execution of jobs in invalid states.

Improvements

• Enhanced graceful shutdown process with comprehensive error handling and error tracking integration.
• Improved lock management during shutdown to prevent deadlocks.
• Extended process signal handling for more reliable termination.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

The pull request introduces graceful shutdown handling for job queues, implements job state validation before execution, enriches job payloads with metadata, adds a reusable signal listener for shutdown orchestration, and updates method signatures to pass AbortSignal instead of AbortController instances.

Changes

Cohort / File(s)	Summary
Graceful timeout and job validation packages/backend/src/connectionManager.ts, packages/backend/src/repoIndexManager.ts	Added graceful-timeout event handlers that mark timed-out jobs as FAILED with status updates and metric emissions. Implemented job state validation in runJob to fetch and verify job status is PENDING or IN_PROGRESS before execution. Added manual Redis lock cleanup during shutdown to release group locks for in-progress jobs. Extended constructors to store Redis instance as private field for lock management.
Shutdown signal handling packages/backend/src/index.ts	Refactored cleanup flow into a new listenToShutdownSignals function that sequentially disposes managers, disconnects services, and handles errors with Sentry. Replaced hard-coded cleanup with signal handler registration loop over SHUTDOWN_SIGNALS and added uncaught exception/rejection handlers.
Shutdown constants packages/backend/src/constants.ts	Added GROUPMQ_WORKER_STOP_GRACEFUL_TIMEOUT_MS (5 seconds) for worker shutdown timeout and SHUTDOWN_SIGNALS array listing Node.js termination signals.
Parameter type refinement packages/backend/src/repoCompileUtils.ts	Changed compileGithubConfig signature to accept signal: AbortSignal instead of abortController: AbortController, forwarding the signal to getGitHubReposFromConfig.

Sequence Diagram(s)

sequenceDiagram
participant Worker
participant JobQueue
participant DB as Prisma/Redis
participant Handler
rect rgb(200, 220, 255)
note over Worker: Job Lifecycle with Graceful Timeout
end
Worker->>Worker: runJob called
Worker->>DB: Fetch current job status
DB-->>Worker: Job status
alt Status is PENDING or IN_PROGRESS
Worker->>Worker: Execute job
alt Normal Completion
Worker->>Handler: Job success
else Graceful Timeout
Handler->>Worker: onJobGracefulTimeout
Worker->>DB: Mark job as FAILED
Worker->>Handler: Update metrics & log
end
else Status is other
Worker->>Handler: Throw error (skip invalid state)
end

Loading

sequenceDiagram
participant System
participant Index as index.ts
participant Managers as ConnectionManager<br/>RepoIndexManager
participant Services as Prisma<br/>Redis<br/>API<br/>Posthog
participant Signals
rect rgb(255, 220, 200)
note over System: Graceful Shutdown Flow
end
Signals->>Index: SIGTERM/SIGINT/etc.
Index->>Index: listenToShutdownSignals
Index->>Managers: dispose() each manager
par Manager Cleanup
Managers->>Managers: Worker.gracefulStop(timeout)
Managers->>Services: Delete Redis lock keys
Managers->>Managers: Log lock release
end
Index->>Services: Disconnect each service
Services-->>Index: Disconnected
Index->>Index: Exit gracefully
Index->>Signals: Remove handlers

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Areas requiring extra attention:

• Lock cleanup logic in connectionManager.ts and repoIndexManager.ts: Manual Redis key deletion during disposal—verify error handling is robust and doesn't prevent shutdown
• Job state validation logic: Ensure the early-exit pattern for invalid job states doesn't suppress legitimate errors or cause jobs to be lost
• Signal handler guard mechanism: Verify the guard against repeated signal processing works correctly and doesn't prevent graceful shutdown if triggered multiple times
• Payload enrichment impact: Confirm job payload expansion (jobId, connectionId, connectionName, orgId) doesn't break existing job consumers or queue serialization
• Parameter type change in repoCompileUtils.ts: Ensure all call sites pass AbortSignal correctly and the change is backward-compatible with consumers

Possibly related PRs

• db performance improvements and job resilience  #200: Modifies connectionManager.ts queue initialization and Redis-backed queue construction with typed Worker<JobPayload>, sharing the same refactoring patterns for queue and worker type safety
• feat(web): Add force resync buttons for repo & connections #610: Updates repoIndexManager.ts job machinery with visibility and return type changes, overlapping with this PR's modifications to the same file's job handling logic

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: implementing graceful shutdown fixes for workers, which is the core objective of the pull request.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[brendan-kellam]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 18fad64 and d43f35b.

📒 Files selected for processing (5)

• packages/backend/src/connectionManager.ts (9 hunks)
• packages/backend/src/constants.ts (1 hunks)
• packages/backend/src/index.ts (3 hunks)
• packages/backend/src/repoCompileUtils.ts (1 hunks)
• packages/backend/src/repoIndexManager.ts (6 hunks)

🧰 Additional context used

📓 Path-based instructions (1)

**/*

📄 CodeRabbit inference engine (.cursor/rules/style.mdc)

Filenames should always be camelCase. Exception: if there are filenames in the same directory with a format other than camelCase, use that format to keep things consistent.

Files:

• packages/backend/src/constants.ts
• packages/backend/src/repoCompileUtils.ts
• packages/backend/src/index.ts
• packages/backend/src/repoIndexManager.ts
• packages/backend/src/connectionManager.ts

[brendan-kellam]

This PR resolves a couple of issues:

• During termination, we call worker.close() to terminate any in progress jobs. This takes a gracefulTimeout param (defaults 30 seconds). If there are any jobs still in progress after that duration, there is a bug in groupmq where the job's group lock will not be released (see [bug] Group lock is not released after graceful shutdown, resulting in stalling jobs Openpanel-dev/groupmq#8). This resulted in jobs (especially long running connection jobs) being left in PENDING or IN_PROGRESS indefinitely, and new jobs not being queued until the timeout expires. The current workaround is to explicitly free the locks when calling dispose.
• Improved the graceful shutdown handling in index.ts

Summary by CodeRabbit

Release Notes

Bug Fixes

• Implemented timeout handling for background jobs; timed-out jobs are now properly marked as failed with status updates and metrics.
• Added job state validation to prevent execution of jobs in invalid states.

Improvements

• Enhanced graceful shutdown process with comprehensive error handling and error tracking integration.
• Improved lock management during shutdown to prevent deadlocks.
• Extended process signal handling for more reliable termination.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

The pull request introduces graceful shutdown handling for job queues, implements job state validation before execution, enriches job payloads with metadata, adds a reusable signal listener for shutdown orchestration, and updates method signatures to pass AbortSignal instead of AbortController instances.

Changes

Cohort / File(s)	Summary
Graceful timeout and job validation packages/backend/src/connectionManager.ts, packages/backend/src/repoIndexManager.ts	Added graceful-timeout event handlers that mark timed-out jobs as FAILED with status updates and metric emissions. Implemented job state validation in runJob to fetch and verify job status is PENDING or IN_PROGRESS before execution. Added manual Redis lock cleanup during shutdown to release group locks for in-progress jobs. Extended constructors to store Redis instance as private field for lock management.
Shutdown signal handling packages/backend/src/index.ts	Refactored cleanup flow into a new listenToShutdownSignals function that sequentially disposes managers, disconnects services, and handles errors with Sentry. Replaced hard-coded cleanup with signal handler registration loop over SHUTDOWN_SIGNALS and added uncaught exception/rejection handlers.
Shutdown constants packages/backend/src/constants.ts	Added GROUPMQ_WORKER_STOP_GRACEFUL_TIMEOUT_MS (5 seconds) for worker shutdown timeout and SHUTDOWN_SIGNALS array listing Node.js termination signals.
Parameter type refinement packages/backend/src/repoCompileUtils.ts	Changed compileGithubConfig signature to accept signal: AbortSignal instead of abortController: AbortController, forwarding the signal to getGitHubReposFromConfig.

Sequence Diagram(s)

sequenceDiagram
participant Worker
participant JobQueue
participant DB as Prisma/Redis
participant Handler
rect rgb(200, 220, 255)
note over Worker: Job Lifecycle with Graceful Timeout
end
Worker->>Worker: runJob called
Worker->>DB: Fetch current job status
DB-->>Worker: Job status
alt Status is PENDING or IN_PROGRESS
Worker->>Worker: Execute job
alt Normal Completion
Worker->>Handler: Job success
else Graceful Timeout
Handler->>Worker: onJobGracefulTimeout
Worker->>DB: Mark job as FAILED
Worker->>Handler: Update metrics & log
end
else Status is other
Worker->>Handler: Throw error (skip invalid state)
end

Loading

sequenceDiagram
participant System
participant Index as index.ts
participant Managers as ConnectionManager<br/>RepoIndexManager
participant Services as Prisma<br/>Redis<br/>API<br/>Posthog
participant Signals
rect rgb(255, 220, 200)
note over System: Graceful Shutdown Flow
end
Signals->>Index: SIGTERM/SIGINT/etc.
Index->>Index: listenToShutdownSignals
Index->>Managers: dispose() each manager
par Manager Cleanup
Managers->>Managers: Worker.gracefulStop(timeout)
Managers->>Services: Delete Redis lock keys
Managers->>Managers: Log lock release
end
Index->>Services: Disconnect each service
Services-->>Index: Disconnected
Index->>Index: Exit gracefully
Index->>Signals: Remove handlers

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Areas requiring extra attention:

• Lock cleanup logic in connectionManager.ts and repoIndexManager.ts: Manual Redis key deletion during disposal—verify error handling is robust and doesn't prevent shutdown
• Job state validation logic: Ensure the early-exit pattern for invalid job states doesn't suppress legitimate errors or cause jobs to be lost
• Signal handler guard mechanism: Verify the guard against repeated signal processing works correctly and doesn't prevent graceful shutdown if triggered multiple times
• Payload enrichment impact: Confirm job payload expansion (jobId, connectionId, connectionName, orgId) doesn't break existing job consumers or queue serialization
• Parameter type change in repoCompileUtils.ts: Ensure all call sites pass AbortSignal correctly and the change is backward-compatible with consumers

Possibly related PRs

• db performance improvements and job resilience  #200: Modifies connectionManager.ts queue initialization and Redis-backed queue construction with typed Worker<JobPayload>, sharing the same refactoring patterns for queue and worker type safety
• feat(web): Add force resync buttons for repo & connections #610: Updates repoIndexManager.ts job machinery with visibility and return type changes, overlapping with this PR's modifications to the same file's job handling logic

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: implementing graceful shutdown fixes for workers, which is the core objective of the pull request.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[brendan-kellam]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 18fad64 and d43f35b.

📒 Files selected for processing (5)

• packages/backend/src/connectionManager.ts (9 hunks)
• packages/backend/src/constants.ts (1 hunks)
• packages/backend/src/index.ts (3 hunks)
• packages/backend/src/repoCompileUtils.ts (1 hunks)
• packages/backend/src/repoIndexManager.ts (6 hunks)

🧰 Additional context used

📓 Path-based instructions (1)

**/*

📄 CodeRabbit inference engine (.cursor/rules/style.mdc)

Filenames should always be camelCase. Exception: if there are filenames in the same directory with a format other than camelCase, use that format to keep things consistent.

Files:

• packages/backend/src/constants.ts
• packages/backend/src/repoCompileUtils.ts
• packages/backend/src/index.ts
• packages/backend/src/repoIndexManager.ts
• packages/backend/src/connectionManager.ts

[brendan-kellam]

This PR resolves a couple of issues:

• During termination, we call worker.close() to terminate any in progress jobs. This takes a gracefulTimeout param (defaults 30 seconds). If there are any jobs still in progress after that duration, there is a bug in groupmq where the job's group lock will not be released (see [bug] Group lock is not released after graceful shutdown, resulting in stalling jobs Openpanel-dev/groupmq#8). This resulted in jobs (especially long running connection jobs) being left in PENDING or IN_PROGRESS indefinitely, and new jobs not being queued until the timeout expires. The current workaround is to explicitly free the locks when calling dispose.
• Improved the graceful shutdown handling in index.ts

Summary by CodeRabbit

Release Notes

Bug Fixes

• Implemented timeout handling for background jobs; timed-out jobs are now properly marked as failed with status updates and metrics.
• Added job state validation to prevent execution of jobs in invalid states.

Improvements

• Enhanced graceful shutdown process with comprehensive error handling and error tracking integration.
• Improved lock management during shutdown to prevent deadlocks.
• Extended process signal handling for more reliable termination.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

The pull request introduces graceful shutdown handling for job queues, implements job state validation before execution, enriches job payloads with metadata, adds a reusable signal listener for shutdown orchestration, and updates method signatures to pass AbortSignal instead of AbortController instances.

Changes

Cohort / File(s)	Summary
Graceful timeout and job validation packages/backend/src/connectionManager.ts, packages/backend/src/repoIndexManager.ts	Added graceful-timeout event handlers that mark timed-out jobs as FAILED with status updates and metric emissions. Implemented job state validation in runJob to fetch and verify job status is PENDING or IN_PROGRESS before execution. Added manual Redis lock cleanup during shutdown to release group locks for in-progress jobs. Extended constructors to store Redis instance as private field for lock management.
Shutdown signal handling packages/backend/src/index.ts	Refactored cleanup flow into a new listenToShutdownSignals function that sequentially disposes managers, disconnects services, and handles errors with Sentry. Replaced hard-coded cleanup with signal handler registration loop over SHUTDOWN_SIGNALS and added uncaught exception/rejection handlers.
Shutdown constants packages/backend/src/constants.ts	Added GROUPMQ_WORKER_STOP_GRACEFUL_TIMEOUT_MS (5 seconds) for worker shutdown timeout and SHUTDOWN_SIGNALS array listing Node.js termination signals.
Parameter type refinement packages/backend/src/repoCompileUtils.ts	Changed compileGithubConfig signature to accept signal: AbortSignal instead of abortController: AbortController, forwarding the signal to getGitHubReposFromConfig.

Sequence Diagram(s)

sequenceDiagram
participant Worker
participant JobQueue
participant DB as Prisma/Redis
participant Handler
rect rgb(200, 220, 255)
note over Worker: Job Lifecycle with Graceful Timeout
end
Worker->>Worker: runJob called
Worker->>DB: Fetch current job status
DB-->>Worker: Job status
alt Status is PENDING or IN_PROGRESS
Worker->>Worker: Execute job
alt Normal Completion
Worker->>Handler: Job success
else Graceful Timeout
Handler->>Worker: onJobGracefulTimeout
Worker->>DB: Mark job as FAILED
Worker->>Handler: Update metrics & log
end
else Status is other
Worker->>Handler: Throw error (skip invalid state)
end

Loading

sequenceDiagram
participant System
participant Index as index.ts
participant Managers as ConnectionManager<br/>RepoIndexManager
participant Services as Prisma<br/>Redis<br/>API<br/>Posthog
participant Signals
rect rgb(255, 220, 200)
note over System: Graceful Shutdown Flow
end
Signals->>Index: SIGTERM/SIGINT/etc.
Index->>Index: listenToShutdownSignals
Index->>Managers: dispose() each manager
par Manager Cleanup
Managers->>Managers: Worker.gracefulStop(timeout)
Managers->>Services: Delete Redis lock keys
Managers->>Managers: Log lock release
end
Index->>Services: Disconnect each service
Services-->>Index: Disconnected
Index->>Index: Exit gracefully
Index->>Signals: Remove handlers

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Areas requiring extra attention:

• Lock cleanup logic in connectionManager.ts and repoIndexManager.ts: Manual Redis key deletion during disposal—verify error handling is robust and doesn't prevent shutdown
• Job state validation logic: Ensure the early-exit pattern for invalid job states doesn't suppress legitimate errors or cause jobs to be lost
• Signal handler guard mechanism: Verify the guard against repeated signal processing works correctly and doesn't prevent graceful shutdown if triggered multiple times
• Payload enrichment impact: Confirm job payload expansion (jobId, connectionId, connectionName, orgId) doesn't break existing job consumers or queue serialization
• Parameter type change in repoCompileUtils.ts: Ensure all call sites pass AbortSignal correctly and the change is backward-compatible with consumers

Possibly related PRs

• db performance improvements and job resilience  #200: Modifies connectionManager.ts queue initialization and Redis-backed queue construction with typed Worker<JobPayload>, sharing the same refactoring patterns for queue and worker type safety
• feat(web): Add force resync buttons for repo & connections #610: Updates repoIndexManager.ts job machinery with visibility and return type changes, overlapping with this PR's modifications to the same file's job handling logic

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: implementing graceful shutdown fixes for workers, which is the core objective of the pull request.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[brendan-kellam]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 18fad64 and d43f35b.

📒 Files selected for processing (5)

• packages/backend/src/connectionManager.ts (9 hunks)
• packages/backend/src/constants.ts (1 hunks)
• packages/backend/src/index.ts (3 hunks)
• packages/backend/src/repoCompileUtils.ts (1 hunks)
• packages/backend/src/repoIndexManager.ts (6 hunks)

🧰 Additional context used

📓 Path-based instructions (1)

**/*

📄 CodeRabbit inference engine (.cursor/rules/style.mdc)

Filenames should always be camelCase. Exception: if there are filenames in the same directory with a format other than camelCase, use that format to keep things consistent.

Files:

• packages/backend/src/constants.ts
• packages/backend/src/repoCompileUtils.ts
• packages/backend/src/index.ts
• packages/backend/src/repoIndexManager.ts
• packages/backend/src/connectionManager.ts

[brendan-kellam]

This PR resolves a couple of issues:

• During termination, we call worker.close() to terminate any in progress jobs. This takes a gracefulTimeout param (defaults 30 seconds). If there are any jobs still in progress after that duration, there is a bug in groupmq where the job's group lock will not be released (see [bug] Group lock is not released after graceful shutdown, resulting in stalling jobs Openpanel-dev/groupmq#8). This resulted in jobs (especially long running connection jobs) being left in PENDING or IN_PROGRESS indefinitely, and new jobs not being queued until the timeout expires. The current workaround is to explicitly free the locks when calling dispose.
• Improved the graceful shutdown handling in index.ts

Summary by CodeRabbit

Release Notes

Bug Fixes

• Implemented timeout handling for background jobs; timed-out jobs are now properly marked as failed with status updates and metrics.
• Added job state validation to prevent execution of jobs in invalid states.

Improvements

• Enhanced graceful shutdown process with comprehensive error handling and error tracking integration.
• Improved lock management during shutdown to prevent deadlocks.
• Extended process signal handling for more reliable termination.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

The pull request introduces graceful shutdown handling for job queues, implements job state validation before execution, enriches job payloads with metadata, adds a reusable signal listener for shutdown orchestration, and updates method signatures to pass AbortSignal instead of AbortController instances.

Changes

Cohort / File(s)	Summary
Graceful timeout and job validation packages/backend/src/connectionManager.ts, packages/backend/src/repoIndexManager.ts	Added graceful-timeout event handlers that mark timed-out jobs as FAILED with status updates and metric emissions. Implemented job state validation in runJob to fetch and verify job status is PENDING or IN_PROGRESS before execution. Added manual Redis lock cleanup during shutdown to release group locks for in-progress jobs. Extended constructors to store Redis instance as private field for lock management.
Shutdown signal handling packages/backend/src/index.ts	Refactored cleanup flow into a new listenToShutdownSignals function that sequentially disposes managers, disconnects services, and handles errors with Sentry. Replaced hard-coded cleanup with signal handler registration loop over SHUTDOWN_SIGNALS and added uncaught exception/rejection handlers.
Shutdown constants packages/backend/src/constants.ts	Added GROUPMQ_WORKER_STOP_GRACEFUL_TIMEOUT_MS (5 seconds) for worker shutdown timeout and SHUTDOWN_SIGNALS array listing Node.js termination signals.
Parameter type refinement packages/backend/src/repoCompileUtils.ts	Changed compileGithubConfig signature to accept signal: AbortSignal instead of abortController: AbortController, forwarding the signal to getGitHubReposFromConfig.

Sequence Diagram(s)

sequenceDiagram
participant Worker
participant JobQueue
participant DB as Prisma/Redis
participant Handler
rect rgb(200, 220, 255)
note over Worker: Job Lifecycle with Graceful Timeout
end
Worker->>Worker: runJob called
Worker->>DB: Fetch current job status
DB-->>Worker: Job status
alt Status is PENDING or IN_PROGRESS
Worker->>Worker: Execute job
alt Normal Completion
Worker->>Handler: Job success
else Graceful Timeout
Handler->>Worker: onJobGracefulTimeout
Worker->>DB: Mark job as FAILED
Worker->>Handler: Update metrics & log
end
else Status is other
Worker->>Handler: Throw error (skip invalid state)
end

Loading

sequenceDiagram
participant System
participant Index as index.ts
participant Managers as ConnectionManager<br/>RepoIndexManager
participant Services as Prisma<br/>Redis<br/>API<br/>Posthog
participant Signals
rect rgb(255, 220, 200)
note over System: Graceful Shutdown Flow
end
Signals->>Index: SIGTERM/SIGINT/etc.
Index->>Index: listenToShutdownSignals
Index->>Managers: dispose() each manager
par Manager Cleanup
Managers->>Managers: Worker.gracefulStop(timeout)
Managers->>Services: Delete Redis lock keys
Managers->>Managers: Log lock release
end
Index->>Services: Disconnect each service
Services-->>Index: Disconnected
Index->>Index: Exit gracefully
Index->>Signals: Remove handlers

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Areas requiring extra attention:

• Lock cleanup logic in connectionManager.ts and repoIndexManager.ts: Manual Redis key deletion during disposal—verify error handling is robust and doesn't prevent shutdown
• Job state validation logic: Ensure the early-exit pattern for invalid job states doesn't suppress legitimate errors or cause jobs to be lost
• Signal handler guard mechanism: Verify the guard against repeated signal processing works correctly and doesn't prevent graceful shutdown if triggered multiple times
• Payload enrichment impact: Confirm job payload expansion (jobId, connectionId, connectionName, orgId) doesn't break existing job consumers or queue serialization
• Parameter type change in repoCompileUtils.ts: Ensure all call sites pass AbortSignal correctly and the change is backward-compatible with consumers

Possibly related PRs

• db performance improvements and job resilience  #200: Modifies connectionManager.ts queue initialization and Redis-backed queue construction with typed Worker<JobPayload>, sharing the same refactoring patterns for queue and worker type safety
• feat(web): Add force resync buttons for repo & connections #610: Updates repoIndexManager.ts job machinery with visibility and return type changes, overlapping with this PR's modifications to the same file's job handling logic

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: implementing graceful shutdown fixes for workers, which is the core objective of the pull request.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[brendan-kellam]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 18fad64 and d43f35b.

📒 Files selected for processing (5)

• packages/backend/src/connectionManager.ts (9 hunks)
• packages/backend/src/constants.ts (1 hunks)
• packages/backend/src/index.ts (3 hunks)
• packages/backend/src/repoCompileUtils.ts (1 hunks)
• packages/backend/src/repoIndexManager.ts (6 hunks)

🧰 Additional context used

📓 Path-based instructions (1)

**/*

📄 CodeRabbit inference engine (.cursor/rules/style.mdc)

Filenames should always be camelCase. Exception: if there are filenames in the same directory with a format other than camelCase, use that format to keep things consistent.

Files:

• packages/backend/src/constants.ts
• packages/backend/src/repoCompileUtils.ts
• packages/backend/src/index.ts
• packages/backend/src/repoIndexManager.ts
• packages/backend/src/connectionManager.ts

[brendan-kellam]

This PR resolves a couple of issues:

• During termination, we call worker.close() to terminate any in progress jobs. This takes a gracefulTimeout param (defaults 30 seconds). If there are any jobs still in progress after that duration, there is a bug in groupmq where the job's group lock will not be released (see [bug] Group lock is not released after graceful shutdown, resulting in stalling jobs Openpanel-dev/groupmq#8). This resulted in jobs (especially long running connection jobs) being left in PENDING or IN_PROGRESS indefinitely, and new jobs not being queued until the timeout expires. The current workaround is to explicitly free the locks when calling dispose.
• Improved the graceful shutdown handling in index.ts

Summary by CodeRabbit

Release Notes

Bug Fixes

• Implemented timeout handling for background jobs; timed-out jobs are now properly marked as failed with status updates and metrics.
• Added job state validation to prevent execution of jobs in invalid states.

Improvements

• Enhanced graceful shutdown process with comprehensive error handling and error tracking integration.
• Improved lock management during shutdown to prevent deadlocks.
• Extended process signal handling for more reliable termination.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

The pull request introduces graceful shutdown handling for job queues, implements job state validation before execution, enriches job payloads with metadata, adds a reusable signal listener for shutdown orchestration, and updates method signatures to pass AbortSignal instead of AbortController instances.

Changes

Cohort / File(s)	Summary
Graceful timeout and job validation packages/backend/src/connectionManager.ts, packages/backend/src/repoIndexManager.ts	Added graceful-timeout event handlers that mark timed-out jobs as FAILED with status updates and metric emissions. Implemented job state validation in runJob to fetch and verify job status is PENDING or IN_PROGRESS before execution. Added manual Redis lock cleanup during shutdown to release group locks for in-progress jobs. Extended constructors to store Redis instance as private field for lock management.
Shutdown signal handling packages/backend/src/index.ts	Refactored cleanup flow into a new listenToShutdownSignals function that sequentially disposes managers, disconnects services, and handles errors with Sentry. Replaced hard-coded cleanup with signal handler registration loop over SHUTDOWN_SIGNALS and added uncaught exception/rejection handlers.
Shutdown constants packages/backend/src/constants.ts	Added GROUPMQ_WORKER_STOP_GRACEFUL_TIMEOUT_MS (5 seconds) for worker shutdown timeout and SHUTDOWN_SIGNALS array listing Node.js termination signals.
Parameter type refinement packages/backend/src/repoCompileUtils.ts	Changed compileGithubConfig signature to accept signal: AbortSignal instead of abortController: AbortController, forwarding the signal to getGitHubReposFromConfig.

Sequence Diagram(s)

sequenceDiagram
participant Worker
participant JobQueue
participant DB as Prisma/Redis
participant Handler
rect rgb(200, 220, 255)
note over Worker: Job Lifecycle with Graceful Timeout
end
Worker->>Worker: runJob called
Worker->>DB: Fetch current job status
DB-->>Worker: Job status
alt Status is PENDING or IN_PROGRESS
Worker->>Worker: Execute job
alt Normal Completion
Worker->>Handler: Job success
else Graceful Timeout
Handler->>Worker: onJobGracefulTimeout
Worker->>DB: Mark job as FAILED
Worker->>Handler: Update metrics & log
end
else Status is other
Worker->>Handler: Throw error (skip invalid state)
end

Loading

sequenceDiagram
participant System
participant Index as index.ts
participant Managers as ConnectionManager<br/>RepoIndexManager
participant Services as Prisma<br/>Redis<br/>API<br/>Posthog
participant Signals
rect rgb(255, 220, 200)
note over System: Graceful Shutdown Flow
end
Signals->>Index: SIGTERM/SIGINT/etc.
Index->>Index: listenToShutdownSignals
Index->>Managers: dispose() each manager
par Manager Cleanup
Managers->>Managers: Worker.gracefulStop(timeout)
Managers->>Services: Delete Redis lock keys
Managers->>Managers: Log lock release
end
Index->>Services: Disconnect each service
Services-->>Index: Disconnected
Index->>Index: Exit gracefully
Index->>Signals: Remove handlers

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Areas requiring extra attention:

• Lock cleanup logic in connectionManager.ts and repoIndexManager.ts: Manual Redis key deletion during disposal—verify error handling is robust and doesn't prevent shutdown
• Job state validation logic: Ensure the early-exit pattern for invalid job states doesn't suppress legitimate errors or cause jobs to be lost
• Signal handler guard mechanism: Verify the guard against repeated signal processing works correctly and doesn't prevent graceful shutdown if triggered multiple times
• Payload enrichment impact: Confirm job payload expansion (jobId, connectionId, connectionName, orgId) doesn't break existing job consumers or queue serialization
• Parameter type change in repoCompileUtils.ts: Ensure all call sites pass AbortSignal correctly and the change is backward-compatible with consumers

Possibly related PRs

• db performance improvements and job resilience  #200: Modifies connectionManager.ts queue initialization and Redis-backed queue construction with typed Worker<JobPayload>, sharing the same refactoring patterns for queue and worker type safety
• feat(web): Add force resync buttons for repo & connections #610: Updates repoIndexManager.ts job machinery with visibility and return type changes, overlapping with this PR's modifications to the same file's job handling logic

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: implementing graceful shutdown fixes for workers, which is the core objective of the pull request.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[brendan-kellam]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 18fad64 and d43f35b.

📒 Files selected for processing (5)

• packages/backend/src/connectionManager.ts (9 hunks)
• packages/backend/src/constants.ts (1 hunks)
• packages/backend/src/index.ts (3 hunks)
• packages/backend/src/repoCompileUtils.ts (1 hunks)
• packages/backend/src/repoIndexManager.ts (6 hunks)

🧰 Additional context used

📓 Path-based instructions (1)

**/*

📄 CodeRabbit inference engine (.cursor/rules/style.mdc)

Filenames should always be camelCase. Exception: if there are filenames in the same directory with a format other than camelCase, use that format to keep things consistent.

Files:

• packages/backend/src/constants.ts
• packages/backend/src/repoCompileUtils.ts
• packages/backend/src/index.ts
• packages/backend/src/repoIndexManager.ts
• packages/backend/src/connectionManager.ts

[brendan-kellam]

This PR resolves a couple of issues:

• During termination, we call worker.close() to terminate any in progress jobs. This takes a gracefulTimeout param (defaults 30 seconds). If there are any jobs still in progress after that duration, there is a bug in groupmq where the job's group lock will not be released (see [bug] Group lock is not released after graceful shutdown, resulting in stalling jobs Openpanel-dev/groupmq#8). This resulted in jobs (especially long running connection jobs) being left in PENDING or IN_PROGRESS indefinitely, and new jobs not being queued until the timeout expires. The current workaround is to explicitly free the locks when calling dispose.
• Improved the graceful shutdown handling in index.ts

Summary by CodeRabbit

Release Notes

Bug Fixes

• Implemented timeout handling for background jobs; timed-out jobs are now properly marked as failed with status updates and metrics.
• Added job state validation to prevent execution of jobs in invalid states.

Improvements

• Enhanced graceful shutdown process with comprehensive error handling and error tracking integration.
• Improved lock management during shutdown to prevent deadlocks.
• Extended process signal handling for more reliable termination.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

The pull request introduces graceful shutdown handling for job queues, implements job state validation before execution, enriches job payloads with metadata, adds a reusable signal listener for shutdown orchestration, and updates method signatures to pass AbortSignal instead of AbortController instances.

Changes

Cohort / File(s)	Summary
Graceful timeout and job validation packages/backend/src/connectionManager.ts, packages/backend/src/repoIndexManager.ts	Added graceful-timeout event handlers that mark timed-out jobs as FAILED with status updates and metric emissions. Implemented job state validation in runJob to fetch and verify job status is PENDING or IN_PROGRESS before execution. Added manual Redis lock cleanup during shutdown to release group locks for in-progress jobs. Extended constructors to store Redis instance as private field for lock management.
Shutdown signal handling packages/backend/src/index.ts	Refactored cleanup flow into a new listenToShutdownSignals function that sequentially disposes managers, disconnects services, and handles errors with Sentry. Replaced hard-coded cleanup with signal handler registration loop over SHUTDOWN_SIGNALS and added uncaught exception/rejection handlers.
Shutdown constants packages/backend/src/constants.ts	Added GROUPMQ_WORKER_STOP_GRACEFUL_TIMEOUT_MS (5 seconds) for worker shutdown timeout and SHUTDOWN_SIGNALS array listing Node.js termination signals.
Parameter type refinement packages/backend/src/repoCompileUtils.ts	Changed compileGithubConfig signature to accept signal: AbortSignal instead of abortController: AbortController, forwarding the signal to getGitHubReposFromConfig.

Sequence Diagram(s)

sequenceDiagram
participant Worker
participant JobQueue
participant DB as Prisma/Redis
participant Handler
rect rgb(200, 220, 255)
note over Worker: Job Lifecycle with Graceful Timeout
end
Worker->>Worker: runJob called
Worker->>DB: Fetch current job status
DB-->>Worker: Job status
alt Status is PENDING or IN_PROGRESS
Worker->>Worker: Execute job
alt Normal Completion
Worker->>Handler: Job success
else Graceful Timeout
Handler->>Worker: onJobGracefulTimeout
Worker->>DB: Mark job as FAILED
Worker->>Handler: Update metrics & log
end
else Status is other
Worker->>Handler: Throw error (skip invalid state)
end

Loading

sequenceDiagram
participant System
participant Index as index.ts
participant Managers as ConnectionManager<br/>RepoIndexManager
participant Services as Prisma<br/>Redis<br/>API<br/>Posthog
participant Signals
rect rgb(255, 220, 200)
note over System: Graceful Shutdown Flow
end
Signals->>Index: SIGTERM/SIGINT/etc.
Index->>Index: listenToShutdownSignals
Index->>Managers: dispose() each manager
par Manager Cleanup
Managers->>Managers: Worker.gracefulStop(timeout)
Managers->>Services: Delete Redis lock keys
Managers->>Managers: Log lock release
end
Index->>Services: Disconnect each service
Services-->>Index: Disconnected
Index->>Index: Exit gracefully
Index->>Signals: Remove handlers

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Areas requiring extra attention:

• Lock cleanup logic in connectionManager.ts and repoIndexManager.ts: Manual Redis key deletion during disposal—verify error handling is robust and doesn't prevent shutdown
• Job state validation logic: Ensure the early-exit pattern for invalid job states doesn't suppress legitimate errors or cause jobs to be lost
• Signal handler guard mechanism: Verify the guard against repeated signal processing works correctly and doesn't prevent graceful shutdown if triggered multiple times
• Payload enrichment impact: Confirm job payload expansion (jobId, connectionId, connectionName, orgId) doesn't break existing job consumers or queue serialization
• Parameter type change in repoCompileUtils.ts: Ensure all call sites pass AbortSignal correctly and the change is backward-compatible with consumers

Possibly related PRs

• db performance improvements and job resilience  #200: Modifies connectionManager.ts queue initialization and Redis-backed queue construction with typed Worker<JobPayload>, sharing the same refactoring patterns for queue and worker type safety
• feat(web): Add force resync buttons for repo & connections #610: Updates repoIndexManager.ts job machinery with visibility and return type changes, overlapping with this PR's modifications to the same file's job handling logic

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: implementing graceful shutdown fixes for workers, which is the core objective of the pull request.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[brendan-kellam]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 18fad64 and d43f35b.

📒 Files selected for processing (5)

• packages/backend/src/connectionManager.ts (9 hunks)
• packages/backend/src/constants.ts (1 hunks)
• packages/backend/src/index.ts (3 hunks)
• packages/backend/src/repoCompileUtils.ts (1 hunks)
• packages/backend/src/repoIndexManager.ts (6 hunks)

🧰 Additional context used

📓 Path-based instructions (1)

**/*

📄 CodeRabbit inference engine (.cursor/rules/style.mdc)

Filenames should always be camelCase. Exception: if there are filenames in the same directory with a format other than camelCase, use that format to keep things consistent.

Files:

• packages/backend/src/constants.ts
• packages/backend/src/repoCompileUtils.ts
• packages/backend/src/index.ts
• packages/backend/src/repoIndexManager.ts
• packages/backend/src/connectionManager.ts

[brendan-kellam]

This PR resolves a couple of issues:

• During termination, we call worker.close() to terminate any in progress jobs. This takes a gracefulTimeout param (defaults 30 seconds). If there are any jobs still in progress after that duration, there is a bug in groupmq where the job's group lock will not be released (see [bug] Group lock is not released after graceful shutdown, resulting in stalling jobs Openpanel-dev/groupmq#8). This resulted in jobs (especially long running connection jobs) being left in PENDING or IN_PROGRESS indefinitely, and new jobs not being queued until the timeout expires. The current workaround is to explicitly free the locks when calling dispose.
• Improved the graceful shutdown handling in index.ts

Summary by CodeRabbit

Release Notes

Bug Fixes

• Implemented timeout handling for background jobs; timed-out jobs are now properly marked as failed with status updates and metrics.
• Added job state validation to prevent execution of jobs in invalid states.

Improvements

• Enhanced graceful shutdown process with comprehensive error handling and error tracking integration.
• Improved lock management during shutdown to prevent deadlocks.
• Extended process signal handling for more reliable termination.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

The pull request introduces graceful shutdown handling for job queues, implements job state validation before execution, enriches job payloads with metadata, adds a reusable signal listener for shutdown orchestration, and updates method signatures to pass AbortSignal instead of AbortController instances.

Changes

Cohort / File(s)	Summary
Graceful timeout and job validation packages/backend/src/connectionManager.ts, packages/backend/src/repoIndexManager.ts	Added graceful-timeout event handlers that mark timed-out jobs as FAILED with status updates and metric emissions. Implemented job state validation in runJob to fetch and verify job status is PENDING or IN_PROGRESS before execution. Added manual Redis lock cleanup during shutdown to release group locks for in-progress jobs. Extended constructors to store Redis instance as private field for lock management.
Shutdown signal handling packages/backend/src/index.ts	Refactored cleanup flow into a new listenToShutdownSignals function that sequentially disposes managers, disconnects services, and handles errors with Sentry. Replaced hard-coded cleanup with signal handler registration loop over SHUTDOWN_SIGNALS and added uncaught exception/rejection handlers.
Shutdown constants packages/backend/src/constants.ts	Added GROUPMQ_WORKER_STOP_GRACEFUL_TIMEOUT_MS (5 seconds) for worker shutdown timeout and SHUTDOWN_SIGNALS array listing Node.js termination signals.
Parameter type refinement packages/backend/src/repoCompileUtils.ts	Changed compileGithubConfig signature to accept signal: AbortSignal instead of abortController: AbortController, forwarding the signal to getGitHubReposFromConfig.

Sequence Diagram(s)

sequenceDiagram
participant Worker
participant JobQueue
participant DB as Prisma/Redis
participant Handler
rect rgb(200, 220, 255)
note over Worker: Job Lifecycle with Graceful Timeout
end
Worker->>Worker: runJob called
Worker->>DB: Fetch current job status
DB-->>Worker: Job status
alt Status is PENDING or IN_PROGRESS
Worker->>Worker: Execute job
alt Normal Completion
Worker->>Handler: Job success
else Graceful Timeout
Handler->>Worker: onJobGracefulTimeout
Worker->>DB: Mark job as FAILED
Worker->>Handler: Update metrics & log
end
else Status is other
Worker->>Handler: Throw error (skip invalid state)
end

Loading

sequenceDiagram
participant System
participant Index as index.ts
participant Managers as ConnectionManager<br/>RepoIndexManager
participant Services as Prisma<br/>Redis<br/>API<br/>Posthog
participant Signals
rect rgb(255, 220, 200)
note over System: Graceful Shutdown Flow
end
Signals->>Index: SIGTERM/SIGINT/etc.
Index->>Index: listenToShutdownSignals
Index->>Managers: dispose() each manager
par Manager Cleanup
Managers->>Managers: Worker.gracefulStop(timeout)
Managers->>Services: Delete Redis lock keys
Managers->>Managers: Log lock release
end
Index->>Services: Disconnect each service
Services-->>Index: Disconnected
Index->>Index: Exit gracefully
Index->>Signals: Remove handlers

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Areas requiring extra attention:

• Lock cleanup logic in connectionManager.ts and repoIndexManager.ts: Manual Redis key deletion during disposal—verify error handling is robust and doesn't prevent shutdown
• Job state validation logic: Ensure the early-exit pattern for invalid job states doesn't suppress legitimate errors or cause jobs to be lost
• Signal handler guard mechanism: Verify the guard against repeated signal processing works correctly and doesn't prevent graceful shutdown if triggered multiple times
• Payload enrichment impact: Confirm job payload expansion (jobId, connectionId, connectionName, orgId) doesn't break existing job consumers or queue serialization
• Parameter type change in repoCompileUtils.ts: Ensure all call sites pass AbortSignal correctly and the change is backward-compatible with consumers

Possibly related PRs

• db performance improvements and job resilience  #200: Modifies connectionManager.ts queue initialization and Redis-backed queue construction with typed Worker<JobPayload>, sharing the same refactoring patterns for queue and worker type safety
• feat(web): Add force resync buttons for repo & connections #610: Updates repoIndexManager.ts job machinery with visibility and return type changes, overlapping with this PR's modifications to the same file's job handling logic

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: implementing graceful shutdown fixes for workers, which is the core objective of the pull request.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[brendan-kellam]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 18fad64 and d43f35b.

📒 Files selected for processing (5)

• packages/backend/src/connectionManager.ts (9 hunks)
• packages/backend/src/constants.ts (1 hunks)
• packages/backend/src/index.ts (3 hunks)
• packages/backend/src/repoCompileUtils.ts (1 hunks)
• packages/backend/src/repoIndexManager.ts (6 hunks)

🧰 Additional context used

📓 Path-based instructions (1)

**/*

📄 CodeRabbit inference engine (.cursor/rules/style.mdc)

Filenames should always be camelCase. Exception: if there are filenames in the same directory with a format other than camelCase, use that format to keep things consistent.

Files:

• packages/backend/src/constants.ts
• packages/backend/src/repoCompileUtils.ts
• packages/backend/src/index.ts
• packages/backend/src/repoIndexManager.ts
• packages/backend/src/connectionManager.ts

[brendan-kellam]

This PR resolves a couple of issues:

• During termination, we call worker.close() to terminate any in progress jobs. This takes a gracefulTimeout param (defaults 30 seconds). If there are any jobs still in progress after that duration, there is a bug in groupmq where the job's group lock will not be released (see [bug] Group lock is not released after graceful shutdown, resulting in stalling jobs Openpanel-dev/groupmq#8). This resulted in jobs (especially long running connection jobs) being left in PENDING or IN_PROGRESS indefinitely, and new jobs not being queued until the timeout expires. The current workaround is to explicitly free the locks when calling dispose.
• Improved the graceful shutdown handling in index.ts

Summary by CodeRabbit

Release Notes

Bug Fixes

• Implemented timeout handling for background jobs; timed-out jobs are now properly marked as failed with status updates and metrics.
• Added job state validation to prevent execution of jobs in invalid states.

Improvements

• Enhanced graceful shutdown process with comprehensive error handling and error tracking integration.
• Improved lock management during shutdown to prevent deadlocks.
• Extended process signal handling for more reliable termination.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

The pull request introduces graceful shutdown handling for job queues, implements job state validation before execution, enriches job payloads with metadata, adds a reusable signal listener for shutdown orchestration, and updates method signatures to pass AbortSignal instead of AbortController instances.

Changes

Cohort / File(s)	Summary
Graceful timeout and job validation packages/backend/src/connectionManager.ts, packages/backend/src/repoIndexManager.ts	Added graceful-timeout event handlers that mark timed-out jobs as FAILED with status updates and metric emissions. Implemented job state validation in runJob to fetch and verify job status is PENDING or IN_PROGRESS before execution. Added manual Redis lock cleanup during shutdown to release group locks for in-progress jobs. Extended constructors to store Redis instance as private field for lock management.
Shutdown signal handling packages/backend/src/index.ts	Refactored cleanup flow into a new listenToShutdownSignals function that sequentially disposes managers, disconnects services, and handles errors with Sentry. Replaced hard-coded cleanup with signal handler registration loop over SHUTDOWN_SIGNALS and added uncaught exception/rejection handlers.
Shutdown constants packages/backend/src/constants.ts	Added GROUPMQ_WORKER_STOP_GRACEFUL_TIMEOUT_MS (5 seconds) for worker shutdown timeout and SHUTDOWN_SIGNALS array listing Node.js termination signals.
Parameter type refinement packages/backend/src/repoCompileUtils.ts	Changed compileGithubConfig signature to accept signal: AbortSignal instead of abortController: AbortController, forwarding the signal to getGitHubReposFromConfig.

Sequence Diagram(s)

sequenceDiagram
participant Worker
participant JobQueue
participant DB as Prisma/Redis
participant Handler
rect rgb(200, 220, 255)
note over Worker: Job Lifecycle with Graceful Timeout
end
Worker->>Worker: runJob called
Worker->>DB: Fetch current job status
DB-->>Worker: Job status
alt Status is PENDING or IN_PROGRESS
Worker->>Worker: Execute job
alt Normal Completion
Worker->>Handler: Job success
else Graceful Timeout
Handler->>Worker: onJobGracefulTimeout
Worker->>DB: Mark job as FAILED
Worker->>Handler: Update metrics & log
end
else Status is other
Worker->>Handler: Throw error (skip invalid state)
end

Loading

sequenceDiagram
participant System
participant Index as index.ts
participant Managers as ConnectionManager<br/>RepoIndexManager
participant Services as Prisma<br/>Redis<br/>API<br/>Posthog
participant Signals
rect rgb(255, 220, 200)
note over System: Graceful Shutdown Flow
end
Signals->>Index: SIGTERM/SIGINT/etc.
Index->>Index: listenToShutdownSignals
Index->>Managers: dispose() each manager
par Manager Cleanup
Managers->>Managers: Worker.gracefulStop(timeout)
Managers->>Services: Delete Redis lock keys
Managers->>Managers: Log lock release
end
Index->>Services: Disconnect each service
Services-->>Index: Disconnected
Index->>Index: Exit gracefully
Index->>Signals: Remove handlers

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Areas requiring extra attention:

• Lock cleanup logic in connectionManager.ts and repoIndexManager.ts: Manual Redis key deletion during disposal—verify error handling is robust and doesn't prevent shutdown
• Job state validation logic: Ensure the early-exit pattern for invalid job states doesn't suppress legitimate errors or cause jobs to be lost
• Signal handler guard mechanism: Verify the guard against repeated signal processing works correctly and doesn't prevent graceful shutdown if triggered multiple times
• Payload enrichment impact: Confirm job payload expansion (jobId, connectionId, connectionName, orgId) doesn't break existing job consumers or queue serialization
• Parameter type change in repoCompileUtils.ts: Ensure all call sites pass AbortSignal correctly and the change is backward-compatible with consumers

Possibly related PRs

• db performance improvements and job resilience  #200: Modifies connectionManager.ts queue initialization and Redis-backed queue construction with typed Worker<JobPayload>, sharing the same refactoring patterns for queue and worker type safety
• feat(web): Add force resync buttons for repo & connections #610: Updates repoIndexManager.ts job machinery with visibility and return type changes, overlapping with this PR's modifications to the same file's job handling logic

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: implementing graceful shutdown fixes for workers, which is the core objective of the pull request.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[brendan-kellam]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 18fad64 and d43f35b.

📒 Files selected for processing (5)

• packages/backend/src/connectionManager.ts (9 hunks)
• packages/backend/src/constants.ts (1 hunks)
• packages/backend/src/index.ts (3 hunks)
• packages/backend/src/repoCompileUtils.ts (1 hunks)
• packages/backend/src/repoIndexManager.ts (6 hunks)

🧰 Additional context used

📓 Path-based instructions (1)

**/*

📄 CodeRabbit inference engine (.cursor/rules/style.mdc)

Filenames should always be camelCase. Exception: if there are filenames in the same directory with a format other than camelCase, use that format to keep things consistent.

Files:

• packages/backend/src/constants.ts
• packages/backend/src/repoCompileUtils.ts
• packages/backend/src/index.ts
• packages/backend/src/repoIndexManager.ts
• packages/backend/src/connectionManager.ts