fix(trash): work inside the run budget, checkpoint every 50 folders, chain the rest - #7
Merged
Merged
Conversation
…chain the rest A Dropbox reorganization leaves every moved file to trash at its old path, tens of thousands of files across thousands of folders, and the phase listed and trashed them one folder at a time with no budget check, no progress line and one state push at the very end. The job timeout killed it first, the pushed state kept every mirror row, and each later run started the same list over, converging short of the end. The phase now reuses the batch phase's should_start rule against the run budget, drops each folder's mirror rows as it lands, pushes a checkpoint every 50 folders (label <epoch>-trash-<folders>), logs one line per folder, and sets the run's chain flag when folders remain so report queues the next run.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Run 34884590321 entered
trashwith 46,472 files across 6,730 Proton folders, the old paths of a Dropbox reorganization (every content hash is among the files re-uploaded in runs 35 and 36). The phase listed and trashed one folder at a time with no budget check, no progress line and one state push at the very end, so the job timeout killed it, the pushed state kept every mirror row, and each later run would start the same list over.p50_trashnow:p40_batches.should_startagainst the run budget and stops before a folder that would not fitmirror_objectsrows as it lands, so the next run's delta is smaller<epoch>-trash-<folders>) and a final one when anything is unpushedreportqueues the next runTest plan
task test: 154 passed, two new tests intests/test_trash.py(budget stop and chain; periodic checkpoints), watched fail before the changetask lint,task checktask statusshowsmirrored_filesfalling toward the inventory count and history objects labelled<epoch>-trash-<n>