Skip to content

fix(trash): work inside the run budget, checkpoint every 50 folders, chain the rest - #7

Merged
jshvn merged 1 commit into
mainfrom
josh/trash-budget
Sep 14, 2026
Merged

jshvn merged 1 commit into
mainfrom
josh/trash-budget

Conversation

@jshvn

@jshvn jshvn commented Sep 14, 2026

Copy link
Copy Markdown
Member

What

Run 34884590321 entered trash with 46,472 files across 6,730 Proton folders, the old paths of a Dropbox reorganization (every content hash is among the files re-uploaded in runs 35 and 36). The phase listed and trashed one folder at a time with no budget check, no progress line and one state push at the very end, so the job timeout killed it, the pushed state kept every mirror row, and each later run would start the same list over.

p50_trash now:

  • reuses p40_batches.should_start against the run budget and stops before a folder that would not fit
  • drops each folder's mirror_objects rows as it lands, so the next run's delta is smaller
  • pushes a checkpoint every 50 folders (<epoch>-trash-<folders>) and a final one when anything is unpushed
  • logs one line per folder
  • sets the run's chain flag when folders remain, so report queues the next run

Test plan

  • task test: 154 passed, two new tests in tests/test_trash.py (budget stop and chain; periodic checkpoints), watched fail before the change
  • task lint, task check
  • After merge: the next dispatched run drains the set over a few chained runs; task status shows mirrored_files falling toward the inventory count and history objects labelled <epoch>-trash-<n>

…chain the rest

A Dropbox reorganization leaves every moved file to trash at its old path, tens of
thousands of files across thousands of folders, and the phase listed and trashed
them one folder at a time with no budget check, no progress line and one state push
at the very end. The job timeout killed it first, the pushed state kept every mirror
row, and each later run started the same list over, converging short of the end.

The phase now reuses the batch phase's should_start rule against the run budget,
drops each folder's mirror rows as it lands, pushes a checkpoint every 50 folders
(label <epoch>-trash-<folders>), logs one line per folder, and sets the run's chain
flag when folders remain so report queues the next run.
@jshvn
jshvn merged commit b205c72 into main Sep 14, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant