fix(collector): fail closed on disk, archiver, and sequence gaps - #1155
proerror77 wants to merge 3 commits into
Conversation
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f9c302f996
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| sequence_gap_delta_json=$sequence_gap_delta | ||
| sequence_gap_baseline=increased | ||
| record_warning "$label: sequence_gap_total increased $prior_total -> $sequence_gap_total (delta=$sequence_gap_delta)" | ||
| record_breach "$label: sequence_gap_total increased $prior_total -> $sequence_gap_total (delta=$sequence_gap_delta)" |
There was a problem hiding this comment.
monday-collector-health.timer runs this stateful script every five minutes, while .github/workflows/monitor-collector-host.yml invokes it only every fifteen minutes. After the local timer observes this newly promoted breach, the same invocation writes the increased counter as the new baseline at line 775; the next local run and the GitHub alerting run therefore see a stable counter, return ok:true, and never open the triage issue. Keep the gap latched (or otherwise expose it to the alerting channel) instead of consuming it on the first non-alerting poll.
AGENTS.md reference: AGENTS.md:L103-L104
Useful? React with 👍 / 👎.
| else | ||
| age=999999 |
There was a problem hiding this comment.
When health.json is valid JSON but updated_at_ns is a string, object, or otherwise non-numeric, this new branch sets an artificial age and continues; the later existing path records only a warning, so the monitor returns ok:true even though LOB health freshness is unverifiable. This also changes the prior arithmetic failure—which the workflow surfaced as a monitor-invocation alert—into a silent non-blocking result. Record a breach in this branch so malformed timestamps fail closed like missing, unparseable, and stale health files.
AGENTS.md reference: AGENTS.md:L103-L104
Useful? React with 👍 / 👎.
4d4e5ee to
72fed78
Compare
Promote df failure, production LOB archiver death, missing or stale health.json, and sequence-gap growth from warnings to hard breaches so the host monitor cannot report ok while those observability gates fail. Co-authored-by: Wild Card <proerror77@users.noreply.github.com>
Keep sequence_gap_total increases latched on the local five-minute timer so the 15-minute monitor-collector-host poll still observes the breach. Treat malformed health.json updated_at_ns as a liveness breach. Co-authored-by: Wild Card <proerror77@users.noreply.github.com>
JSON mode writes breaches in the snapshot object, not as `breach:` stdout lines. Co-authored-by: Wild Card <proerror77@users.noreply.github.com>
72fed78 to
e3ccab8
Compare
Change
Collector host health now fails closed when disk capacity is unobservable, a production LOB archiver is dead,
health.jsonis missing/stale/unparseable/malformed, or LOB sequence gaps grow. Those used to be warnings, sook:truecould still hold while the monitor that exists to catch silent collector death could not see disk, process, or replay liveness.Sequence-gap increases stay latched on the local five-minute timer (
MONDAY_COLLECTOR_HEALTH_LATCH_SEQUENCE_GAPS=1) so the 15-minute GitHubmonitor-collector-hostpoll still observes the breach. The alerting poll then consumes the new baseline. Malformedupdated_at_nsis a breach, not a warning.Non-LOB unit/timer Result, restart-rate deltas, and sequence session/regression rebaselines stay warnings. Failed recovery jobs still require an identity-bearing
resume; this change does not auto-resume them.Issue relationship
None
Validation
git diff --checkdeployment/aliyun/test-monday-collector-health.sh— contract assertions coverdfunavailable, archiver inactive/failed Result, missing/symlink/stale/malformedhealth.json, currentsequence_gaps,sequence_gap_totalincrease, and timer-latch until a non-latch (GitHub) poll consumes the counter.Runtime impact and rollback
Monitor-only. After merge the next collector-health deploy of
monday-collector-health.shandmonday-collector-health.servicewill start paging on these breaches and will latch sequence-gap deltas on the host timer. Rollback is reverting this commit / restoring the previous installed script and unit. No automatic artifact publication beyond ordinary merge-triggered publication.