Skip to content

fix(shutdown): handle SIGTERM at tracker boundary - #2133

Merged
josecelano merged 30 commits into
torrust:developfrom
josecelano:2132-add-sigterm-to-main
Sep 3, 2026
Merged

josecelano merged 30 commits into
torrust:developfrom
josecelano:2132-add-sigterm-to-main

Conversation

@josecelano

@josecelano josecelano commented Sep 2, 2026

Copy link
Copy Markdown
Member

Summary

Add Unix SIGTERM handling at the tracker executable boundary. SIGINT and SIGTERM select the shutdown source, emit distinct observable logs, and execute the existing JobManager cancellation and waiting sequence once.

Details

  • Extract wait_for_shutdown_signal() so platform-specific signal registration and source selection are isolated from the executable lifecycle.
  • Retain Ctrl-C support on non-Unix builds and explicitly fail if Tokio cannot install the Ctrl-C handler rather than treating an error as SIGINT.
  • Explicitly reject an unexpectedly closed SIGTERM signal stream rather than reporting it as SIGTERM.
  • Add Unix-only native executable-boundary coverage that launches the Cargo-built tracker binary, waits for health readiness and signal-handler registration, and sends SIGTERM or SIGINT to its exact child PID.
  • Add a reusable lifecycle fixture with isolated port-zero configuration, child-only environment configuration, concurrent output draining, deadline-bounded health polling, graceful shutdown, and force-kill/reap panic cleanup.
  • Refactor the fixture incrementally into workspace, output-capture, and health-check client collaborators; document the decision not to add a separate ReadinessProbe.
  • Record the implementation retrospective and standardize the repository workflow for conditional issue-local retrospectives.
  • Move the SI-1 specification and verification evidence to docs/issues/open/2132-add-sigterm-to-main/.

Closes #2132

Validation

  • cargo build --release
  • cargo test --test lifecycle-signals
  • cargo test
  • linter all
  • Mandatory pre-commit checks on every signed commit
  • Pre-push checks: nightly format/check/docs and full test suite
  • Manual direct release-binary verification for default SIGTERM, explicit SIGTERM, SIGINT, and SIGKILL control behavior

Known transitional behavior

Legacy periodic jobs retain their existing sequential per-job timeout behavior. The documented aggregate shutdown-result and deadline policy remain owned by SI-20. The native fixture's output assertions intentionally require message presence only; stdout and stderr are drained concurrently into one retained buffer and their cross-stream ordering is not asserted.

@codecov

codecov Bot commented Sep 2, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 70.96774% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.26%. Comparing base (c956a44) to head (b0810e3).

Files with missing lines Patch % Lines
src/main.rs 70.96% 7 Missing and 2 partials ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #2133      +/-   ##
===========================================
+ Coverage    84.04%   84.26%   +0.21%     
===========================================
  Files          351      351              
  Lines        27988    28008      +20     
  Branches     27988    28008      +20     
===========================================
+ Hits         23523    23601      +78     
+ Misses        4070     4006      -64     
- Partials       395      401       +6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

src/main.rs currently ignores the Result from tokio::signal::ctrl_c() inside tokio::select!, which can cause an error to be treated as a SIGINT and trigger an unintended shutdown.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds Unix SIGTERM handling at the tracker executable boundary so the existing coordinated shutdown sequence runs for both SIGINT and SIGTERM, with distinct observable logs, and updates the related issue spec/evidence docs accordingly.

Changes:

  • Add Unix-only SIGTERM listener and select SIGINT vs SIGTERM to label shutdown logs in src/main.rs.
  • Update SI-1/#2132 issue spec and verification evidence to reflect completed implementation and manual checks.
  • Update the shutdown EPIC (#1488) to link to the new open issue spec location.
File summaries
File Description
src/main.rs Adds SIGTERM handling (Unix) and shutdown-source logging before running the existing JobManager cancellation/wait sequence.
docs/issues/open/2132-add-sigterm-to-main/verification.md Records post-implementation manual verification evidence and checklist updates.
docs/issues/open/2132-add-sigterm-to-main/ISSUE.md Promotes draft to open issue #2132 and updates acceptance criteria/status narrative.
docs/issues/open/1488-overhaul-tracker-shutdown/ISSUE.md Updates the EPIC table link to reference the new SI-1 issue spec path.
Review details

Suppressed comments (3)

docs/issues/open/2132-add-sigterm-to-main/ISSUE.md:10

  • Frontmatter last-updated-utc is date-only, but the repo template requires UTC timestamp precision (YYYY-MM-DD HH:MM). Update this field to include the time in UTC to match the documented convention.
    docs/issues/open/2132-add-sigterm-to-main/verification.md:161
  • The verification environment captures a personal machine hostname (josecelano-desktop) in the OS line. To avoid leaking identifiable workstation details in repo docs, redact the hostname while keeping kernel/version info.
    docs/issues/open/2132-add-sigterm-to-main/verification.md:156
  • Verification notes say the SIGTERM fix was "uncommitted" and the commit hash line includes a "working tree contains" qualifier, which reduces reproducibility for anyone trying to re-run the evidence later. Prefer re-running verification on a clean working tree and recording the exact commit SHA that contains the change.
  • Files reviewed: 4/4 changed files
  • Comments generated: 1
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/main.rs Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

src/main.rs currently treats sigterm.recv() returning None as a SIGTERM, which can misreport the shutdown source and trigger shutdown unexpectedly if the signal stream closes.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

src/main.rs:59

  • Same as the outer select!: sigterm.recv() yields Option<()>. If it ever returns None, the current arm would still select and report SIGTERM. Handle None explicitly to avoid misreporting the shutdown source.
                _ = sigterm.recv() => "SIGTERM",

docs/issues/open/2132-add-sigterm-to-main/ISSUE.md:10

  • last-updated-utc should include a UTC time (YYYY-MM-DD HH:MM) per the issue spec template; this entry is date-only, which makes updates harder to track consistently.
  • Files reviewed: 12/13 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread src/main.rs Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new tests/lifecycle/native_tracker.rs introduces const fn methods that mutate or use non-const operations and are likely to break compilation under the project MSRV.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (4)

Previously missed (1) — in code that hasn't changed since the last review.

docs/issues/open/AGENTS.md:114

  • The summary table mixes the legacy standalone EPIC example (1978-configuration-overhaul-epic.md above) with a folder-style EPIC example here, but the table label EPIC spec doesn’t clarify which format it refers to. This can mislead readers into thinking EPIC specs are always folder-based even when documenting legacy single-file EPICs.

tests/lifecycle/native_tracker.rs:349

  • child_ref is declared as const fn but calls runtime-only methods (as_ref() + expect(...)). This is very likely to be rejected by const-eval rules and can break compilation; use a normal fn.
    const fn child_ref(&self) -> &Child {

tests/lifecycle/native_tracker.rs:353

  • child_mut is declared as const fn but uses Option::as_mut() + expect(...), which is unlikely to be const-evaluable and can break compilation. Use a normal fn.
    const fn child_mut(&mut self) -> &mut Child {

tests/lifecycle/native_tracker.rs:357

  • output_ref is declared as const fn but uses Option::as_ref() + expect(...), which is unlikely to be const-evaluable and can break compilation. Use a normal fn.
    const fn output_ref(&self) -> &TrackerOutputCapture {
  • Files reviewed: 29/30 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread tests/lifecycle/native_tracker.rs
@josecelano
josecelano marked this pull request as ready for review September 3, 2026 15:39
@josecelano
josecelano requested a review from a team as a code owner September 3, 2026 15:39
@josecelano
josecelano requested a lite review from Copilot September 3, 2026 15:40

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

There are a couple of concrete documentation/maintainability issues in changed files (notably a contradictory naming “Summary Table” and an unnecessary const fn) that should be corrected before merging.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

tests/lifecycle/native_tracker.rs:257

  • take_drop_cleanup_observer is declared as a const fn, but it performs normal runtime state mutation (Option::take()) and returns a non-const type. Marking it const doesn't provide a benefit here and can unnecessarily constrain future changes to this helper.
    /// Returns an observer for the signal that terminated the reaped drop-path child.
    pub const fn take_drop_cleanup_observer(&mut self) -> oneshot::Receiver<Result<i32, String>> {
        self.drop_cleanup_observer
            .take()
            .expect("drop cleanup observer must be taken at most once")
    }
  • Files reviewed: 29/30 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread docs/issues/open/AGENTS.md
@josecelano

Copy link
Copy Markdown
Member Author

ACK b0810e3

@josecelano
josecelano merged commit d54d418 into torrust:develop Sep 3, 2026
17 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add SIGTERM handling at tracker signal boundary

2 participants