Uh oh!
There was an error while loading. Please reload this page.
deps: lock the runtime dependencies with hashes - #49
Merged
Conversation
requirements.txt carried only floors, so `docker build` today and next month resolved different transitive trees — a bug reported against an image tag couldn't be reliably reproduced, because the tag didn't identify what was in it. A compromised upstream release also landed on the next rebuild with no PR, no diff and no alert. 187 packages are now pinned exactly with hashes, and the Dockerfile installs with --require-hashes. Per the decision on #10, torch is excluded from the lock along with triton and the nvidia-* runtime packages: the Dockerfile installs torch from the cu130 index first, because those wheels bundle the CUDA + cuDNN runtime, and letting PyPI install it again would clobber that with a generic build. torchaudio and torchmetrics are deliberately *not* excluded — they come from PyPI, not from the cu130 step, so they belong in the lock. torch is still pinned in requirements.in even though it never reaches the output, and that turned out to matter: without it the resolver drifts to PyPI's newest torch (2.13.0) and pins the whole tree against a torch the image never installs. With the pin, the resolution reproduces the running container exactly — torchaudio 2.11.0, torchmetrics 1.9.0, numpy 2.4.6, nemo-toolkit 2.7.3. The exclusion list lives in scripts/lock-requirements.sh rather than a comment someone has to retype, and the script fails if a cu130-provided package ever leaks into the output. Dependabot gains the pip ecosystem now that there is something to bump, with torch ignored so it can't move the pin out of lockstep with the Dockerfile. Verified by building and running the image on the GPU box: build succeeds under --require-hashes, torch survives as 2.12.1+cu130 with CUDA available and bf16 supported, and both sample clips transcribe correctly. Also confirmed the hash check actually bites — a forged hash is rejected, the genuine one installs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The second half of #10, taking your call on the fork:
187 packages pinned exactly with hashes;
Dockerfileinstalls with--require-hashes.What's excluded, and what deliberately isn't
Excluded (all installed by the cu130 step that runs first):
torch,triton, and the sixteennvidia-*runtime packages.Not excluded:
torchaudioandtorchmetrics. Worth flagging since they look like they belong with torch — they don't. I checked the running container: the cu130 install doesn't provide them, they come from PyPI as ordinary NeMo dependencies, so leaving them out would mean an unpinned hole in the middle of the tree.The torch pin earns its keep
requirements.inpinstorch==2.12.1even though torch never reaches the output. That isn't decoration — without it the resolver drifts to PyPI's newest torch (2.13.0) and pins everything else against a torch the image never installs.With the pin, the resolution reproduces the running container exactly:
The Dockerfile's
torch==2.12.1and therequirements.inpin must stay in lockstep; both carry a comment saying so, and Dependabot is configured to ignore torch so it can't move one without the other.Verified by building and running it, not by reading it
On the GPU box, separate tag/port/volume so the live daemon was untouched:
--require-hashes2.12.1+cu130,cuda available: True,bf16 supported: True— this was the real risk, since the install upgradescuda-bindings/cuda-pathfinderthat the cu130 step had brought inaudio/clean.wavandaudio/noisy.wavcome back correctExpected sha256 1111… Got 0f17b89f…), genuine ones installAlso here
scripts/lock-requirements.shholds the exclusion list, so it isn't a comment someone has to retype. It fails loudly if a cu130-provided package ever leaks into the lock.pipecosystem. Read its first PR rather than merging on green: both lockfiles areuv-generated, and Dependabot's pip-compile support keys off the pip-tools header, so it may editrequirements.txtdirectly instead of regenerating it. If it does, close the PR and re-run the script.Not included
requirements.inisn't added toserver.yml's path filters — that file is under.github/workflows/, which I can't merge without theworkflowscope. It belongs in #40, which already edits those filters.Closes#10