Report what the install is doing, and unwtar in parallel - #89
Open
MatanTiram wants to merge 1 commit into
Open
MatanTiram wants to merge 1 commit into
MatanTiram wants to merge 1 commit into
Conversation
An install announced almost nothing between "started" and "done". Central drove its progress bar by parsing free text, and whole phases - reading the index, scanning the sync cache, verifying, copying - passed with no signal a UI could act on. A run that failed emitted nothing terminal at all, so a broken install and a hung one looked identical from outside. This adds the reporting, and one real speedup on the way past. The state machine now has an entry and an exit. preparing is emitted at the top of the client flow for sync/copy/synccopy, covering the yaml and info-map read, item calculation and the sync-folder scan. failed is emitted from PythonBatchRuntime.log_error, so a failed install is no longer indistinguishable from a hang. ready_to_copy gives a sync-only run a terminal state - before, it stopped on verifying_downloads and never finished. The copy phase reports bytes. Planned and done are now the same unit - source bytes - where before the plan counted wtar items at an expanded estimate while the copy commands reported compressed bytes, so the bar topped out around 77% and then jumped. That leaves WTAR_RATIO with no reader, so it goes. copying is announced after folder creation and the pre-copy actions, which contribute no bytes and used to sit at zero. Unwtar extracts across a process pool. Decompression is CPU-bound and was fully serial. DOWNLOAD_PARALLEL_UNWTAR is a kill switch back to the serial path, and DOWNLOAD_PARALLEL_WORKERS=0 means os.cpu_count(). downloadEvents is the structured channel Central reads instead of parsing text. It does not replace the legacy progress line; that line must keep being emitted and does. Redaction is enforced on every helper: hosts may pass, the rest of a url may not, and local paths are denylisted. DOWNLOAD_TELEMETRY_ENABLED turns the whole channel off without touching the text log. Scope note: the channel defines five event types and this emits four. The fifth, download.retry_decision, needs the retry policy that produces it, so both it and its tests belong with the download engine rather than here. The docstrings in downloadEvents describe the whole contract, including that event. Verified per module, each in its own process, against master: thirteen modules, identical failure sets, zero regressions. 78 tests across the eight modules this adds or touches, all passing, including the client-copy characterization golden that pins the emitted copy script. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An install announces almost nothing between "started" and "done". Central drives its progress bar by parsing free text, and whole phases — reading the index, scanning the sync cache, verifying, copying — pass with no signal a UI can act on. A run that fails emits nothing terminal at all, so a broken install and a hung one look identical from outside.
This adds the reporting, and one real speedup on the way past.
26 files, +3,622 / −66 — of which +1,925 is production code and +1,697 is tests.
The state machine gets an entry and an exit
preparingat the top of the client flow forsync/copy/synccopy— covering the yaml and info-map read, item calculation, and the sync-folder scan. That stretch previously reported nothing at all.failedfromPythonBatchRuntime.log_error, so a failed install is no longer indistinguishable from a hang. This is the one I would most like a second opinion on: it is a new terminal state on the wire.ready_to_copygives a sync-only run a terminal state. Before,instl syncstopped onverifying_downloadsand never finished.The copy phase reports bytes
Planned and done are now the same unit — source bytes. Before, the plan counted wtar items at an expanded estimate while the copy commands reported compressed bytes, so the bar topped out around 77% and then jumped to done. That leaves
WTAR_RATIOwith no reader, so it goes — flagging it explicitly since it is a documented config var.copyingis announced after folder creation and the pre-copy actions, which contribute no bytes and used to leave the bar at zero.Unwtar extracts in parallel
Decompression is CPU-bound and was fully serial.
DOWNLOAD_PARALLEL_UNWTARis a kill switch back to the serial path;DOWNLOAD_PARALLEL_WORKERS: 0meansos.cpu_count().The event channel
downloadEventsis the structured channel Central reads instead of parsing text. It does not replace the legacy progress line — that line must keep being emitted, and does. Redaction is enforced in every helper: hosts may pass, the rest of a URL may not, and local paths are denylisted.DOWNLOAD_TELEMETRY_ENABLEDturns the whole channel off without touching the text log.Scope note: the channel defines five event types and this emits four. The fifth,
download.retry_decision, needs the retry policy that produces it, so both it and its tests belong with the download engine rather than here. The docstrings indownloadEventsdescribe the whole contract including that event, so you will see forward references to adownloadRetrymodule that arrives in the next PR.Verification
repr/evalround-trip forReportDownloadStateunderpybatch/test, per AGENTS.md §8.master3: identical failure sets, zero regressions.instl doitagainst Central's realBuild-index.yamlfrom this branch: exit 0, and the emitted scriptpy_compiles.master3worktree produces a structurally identical script — 316 lines both, differing only in paths, timestamp and__INVOCATION_RANDOM_ID__.eval'd against this branch, then round-tripped throughrepr:Not claimed: a green suite. It is not green here and was not green on
master3.Worth pushing back on
wtarBatchCommandsis the most intrusive change into existing code here. The parallel path is a module-level picklable function that receives only plain data — no command object, noconfig_vars— and all progress reporting stays on the main process. If you would rather it lived elsewhere, say so.src/services/instl/behaviours/shell/progress/.schemaVersionstays at 1 because every change here is additive.WTAR_RATIOremoval, as above.This is independent of the housekeeping PR — either order works. The download engine itself (transfer, resume, retry policy, verify) follows separately and builds on this.