Fix nightly janitor running dozens of times each midnight - #848
Merged
Merged
Conversation
The cron loop asked for a timeout truncated to whole seconds, so every wait ended up to a second before the slot it waited for. It queued the job anyway, then recomputed a schedule off the same short clock, which handed back the slot just run: a timeout of zero, a wait that returns at once, and the whole nightly fan-out queued again on every pass until the clock caught up. Roughly thirty to fifty runs of the search-index optimize, vacuum, database optimize, backup and user-data snapshot every night. The weekly telemeter send shared the window. Wait out the real remainder instead, and run a job only once the clock has reached its slot. An early wake now costs one more trip around the loop rather than a night's work. Also compute the nightly slot as tomorrow's calendar date rather than now plus twenty four hours. On the night daylight saving time ends the local day is 25 hours long, so the hour after midnight resolved to the midnight that had just gone by, which would have spun for that whole hour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
Every night the librarian ran the whole nightly fan-out thirty to fifty
times instead of once: search-index optimize,
PRAGMA optimize+VACUUM,database backup, user-data sidecar snapshot, and the fifteen other nightly
tasks, including the self-update check.
CronThread._get_timeouttruncated the wait to whole seconds, soCondition.waitreturned up to a second before local midnight._run_expired_jobs(timed_out=True)bumpednowup to the slot and queuedthe job anyway.
_create_task_timesthen recomputed off the real (stillshort) wall clock,
get_janitor_timehanded back the same midnight, thetimeout came out
int(0.3) == 0, andCondition.wait(0)returned at onceas a timeout. The loop span until the clock crossed midnight, queueing a
full
JanitorNightlyTaskon every pass.Janitor.queue_nightly_tasksfanseach one into all nineteen nightly tasks, and the scribe queue deliberately
never collapses equal tasks.
Introduced in 646fe38, which removed the
sleep(2) # fix time rounding problemsthat had been keeping the recompute on the far side of midnight,and replaced it with the early-fire bump. The weekly telemeter send shared
the window: its claim stamps
updated_atwith the same short clock, so theslot stayed schedulable.
Driving the real
CronThreadwith a fake clock started at 23:59:59.7produced 28,847 enqueues in 0.3 s. After this change: 1.
The fix
_get_timeoutreturns a float instead oftruncating.
_run_expired_jobsloses the
timed_outbump; an early wake now costs one more trip aroundthe loop rather than a night's work.
now + 24 h. On the night daylight saving time ends the local day is 25
hours long, so
get_janitor_timereturned the midnight that had justfired for the whole hour after it — an hour-long spin.
Tests
tests/test_crond.pygains an early-wake case that freezes the clock 300 msshort of midnight, runs three loop passes, and requires exactly one nightly
task once the clock crosses; plus a case pinning the sub-second timeout, and
a DST case under
America/Los_Angeles. The twoCronWakeupTestCaseteststhat asserted the old
timed_outbehavior are replaced.Verified: 1123 pytest, 500 vitest,
make lintandmake tyclean.🤖 Generated with Claude Code