Skip to content

Fix nightly janitor running dozens of times each midnight - #848

Merged
ajslater merged 1 commit into
developfrom
fix/cron-nightly-midnight-spin
Sep 13, 2026
Merged

ajslater merged 1 commit into
developfrom
fix/cron-nightly-midnight-spin

Conversation

@ajslater

Copy link
Copy Markdown
Owner

The bug

Every night the librarian ran the whole nightly fan-out thirty to fifty
times instead of once: search-index optimize, PRAGMA optimize + VACUUM,
database backup, user-data sidecar snapshot, and the fifteen other nightly
tasks, including the self-update check.

CronThread._get_timeout truncated the wait to whole seconds, so
Condition.wait returned up to a second before local midnight.
_run_expired_jobs(timed_out=True) bumped now up to the slot and queued
the job anyway. _create_task_times then recomputed off the real (still
short) wall clock, get_janitor_time handed back the same midnight, the
timeout came out int(0.3) == 0, and Condition.wait(0) returned at once
as a timeout. The loop span until the clock crossed midnight, queueing a
full JanitorNightlyTask on every pass. Janitor.queue_nightly_tasks fans
each one into all nineteen nightly tasks, and the scribe queue deliberately
never collapses equal tasks.

Introduced in 646fe38, which removed the sleep(2) # fix time rounding problems that had been keeping the recompute on the far side of midnight,
and replaced it with the early-fire bump. The weekly telemeter send shared
the window: its claim stamps updated_at with the same short clock, so the
slot stayed schedulable.

Driving the real CronThread with a fake clock started at 23:59:59.7
produced 28,847 enqueues in 0.3 s. After this change: 1.

The fix

  • Wait out the real remainder: _get_timeout returns a float instead of
    truncating.
  • Run a job only once the clock has reached its slot. _run_expired_jobs
    loses the timed_out bump; an early wake now costs one more trip around
    the loop rather than a night's work.
  • Compute the nightly slot from tomorrow's calendar date rather than
    now + 24 h. On the night daylight saving time ends the local day is 25
    hours long, so get_janitor_time returned the midnight that had just
    fired for the whole hour after it — an hour-long spin.

Tests

tests/test_crond.py gains an early-wake case that freezes the clock 300 ms
short of midnight, runs three loop passes, and requires exactly one nightly
task once the clock crosses; plus a case pinning the sub-second timeout, and
a DST case under America/Los_Angeles. The two CronWakeupTestCase tests
that asserted the old timed_out behavior are replaced.

Verified: 1123 pytest, 500 vitest, make lint and make ty clean.

🤖 Generated with Claude Code

The cron loop asked for a timeout truncated to whole seconds, so every
wait ended up to a second before the slot it waited for. It queued the
job anyway, then recomputed a schedule off the same short clock, which
handed back the slot just run: a timeout of zero, a wait that returns at
once, and the whole nightly fan-out queued again on every pass until the
clock caught up. Roughly thirty to fifty runs of the search-index
optimize, vacuum, database optimize, backup and user-data snapshot every
night. The weekly telemeter send shared the window.

Wait out the real remainder instead, and run a job only once the clock
has reached its slot. An early wake now costs one more trip around the
loop rather than a night's work.

Also compute the nightly slot as tomorrow's calendar date rather than
now plus twenty four hours. On the night daylight saving time ends the
local day is 25 hours long, so the hour after midnight resolved to the
midnight that had just gone by, which would have spun for that whole
hour.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ajslater
ajslater merged commit 11b3210 into develop Sep 13, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant