[CI] Nightly: publish pytorch-nightly multi-arch image (torch + source) - #3353
Conversation
|
Jenkins build for 57c0e74ea7e20ea8d131e540248e589ab15ac756 commit finished as NOT_BUILT |
|
Jenkins build for 52480d283ece84ca1ce2499cd67b5a9b631512c5 commit finished as NOT_BUILT |
|
Jenkins build for 52480d283ece84ca1ce2499cd67b5a9b631512c5 commit finished as FAILURE |
|
Jenkins build for d6d638c8a1e652faaebeaae82eb7bcacfe1170fa commit finished as FAILURE |
jithunnair-amd
left a comment
There was a problem hiding this comment.
@ethanwee1 I was hoping to port @amdfaa 's scripts to ROCm/pytorch and then use them in this workflow, instead of rehashing a lot of the same logic. From your PR description, it seems that should be feasible?
Also left some comments for specific aspects.
|
Jenkins build for 3a30693b2fa228b05f5e01cde56e28c110042af8 commit finished as FAILURE |
3a30693 to
09cea71
Compare
|
Jenkins build for 09cea7192cde2d0b5a1024f7b10330bee9ae4def commit finished as FAILURE |
Build rocm/pytorch-nightly from the pytorch/pytorch nightly branch head and install torch plus ROCm multi-arch wheels from the whl-multi-arch index.
6b283c2 to
9d04265
Compare
The Ubuntu base image's pip is installed by apt and cannot be uninstalled by pip itself. Use it directly with --break-system-packages for the multi-arch wheel install.
The ROCm multi-arch wheel set is large. Avoid retaining pip's download cache during the Docker build and use plain Docker progress for better hosted-runner behavior.
Docker RUN defaults to dash, which rejects printf %q and failed every fork publish since the multi-arch rewrite.
|
@jithunnair-amd addressing the remaining review notes so we can re-review this:
Fork builds of that rewrite were failing on |
ubuntu:24.04 does not ship OpenMP. torch._C failed validation with missing libgomp.so.1 after the image itself built successfully.
|
Fork validation is green: https://github.com/ethanwee1/pytorch/actions/runs/32059342057
@jithunnair-amd this is ready for another look. The |
Use a venv instead of --break-system-packages, publish latest plus date/sha tags, switch to DOCKER_USERNAME/DOCKER_TOKEN, and clone source under /root/pytorch.
Switch the pytorch-nightly image workflow to download.pytorch.org's ROCm 7.14 nightly index. Use the supported ROCm extras available there: libraries and device-all.
download.pytorch.org's ROCm nightly index does not publish rocm-sdk-devel. Locate the installed _rocm_sdk_* root from site-packages instead of calling rocm-sdk path, which requires the devel extra.
Use a Python one-liner to locate the ROCm wheel root so Docker does not parse heredoc body lines as Dockerfile instructions.
The download.pytorch.org ROCm nightly torch wheel requires libatomic.so.1 at import time, so install Ubuntu's libatomic1 runtime package.
Keep workflow_dispatch runs as build-and-validate only. Only scheduled runs log in to Docker Hub and push rocm/pytorch-nightly tags.
| package_index_url: | ||
| description: "Python wheel index for torch + ROCm multi-arch packages" | ||
| type: string | ||
| default: "https://download.pytorch.org/whl/nightly/rocm10.0/" |
There was a problem hiding this comment.
Switch this workflow input to be the ROCm version e.g. 10.0
| IMAGE_NAME: rocm/pytorch-nightly | ||
| PYTORCH_REPO: pytorch/pytorch | ||
| PYTORCH_BRANCH: ${{ inputs.pytorch_branch || 'nightly' }} | ||
| PACKAGE_INDEX_URL: ${{ inputs.package_index_url || 'https://download.pytorch.org/whl/nightly/rocm10.0/' }} |
There was a problem hiding this comment.
| PACKAGE_INDEX_URL: ${{ inputs.package_index_url || 'https://download.pytorch.org/whl/nightly/rocm10.0/' }} | |
| TORCH_PACKAGE_INDEX_URL: ${{ inputs.package_index_url || 'https://download.pytorch.org/whl/nightly/rocm10.0/' }} |
Use a single ROCm version input to derive the PyTorch nightly wheel index and install ROCm devel packages from stable.repo.amd.com/rocm/whl-next with a patch-version wildcard.
After installing rocm[devel], use rocm-sdk path to set ROCM_HOME and ROCM_BIN so the environment points at the devel package location.
Split package indexes so PyTorch wheels come from download.pytorch.org while all rocm packages, including devel, come from stable.repo.amd.com/rocm/whl-next using the configured ROCm version wildcard.
|
@ethanwee1 Thank you for addressing the review comments. Can you please trigger another workflow run on your fork with the latest changes, so we can validate the changes as mentioned in the PR description:
|
What
Adds a nightly/manual workflow that publishes
docker.io/rocm/pytorch-nightlyfrom the actual head ofpytorch/pytorch:nightly.The workflow resolves the nightly branch ref directly and records that exact source SHA in Docker image labels and the workflow summary.
How
The workflow builds a Docker image from
ubuntu:24.04. It uses onerocm_versioninput (default10.0) and two derived package indexes:All ROCm packages now come from
repo.amd.com, including libraries, devel files, and all device packages:PyTorch packages come from
download.pytorch.org:python -m pip install --no-cache-dir --index-url "${TORCH_PACKAGE_INDEX_URL}" torch torchvision torchaudioAfter
rocm[devel]is installed,/etc/rocm_env.shis generated from the devel package usingrocm-sdk path --rootandrocm-sdk path --bin, soROCM_HOME/ROCM_BINpoint at the devel package location.The workflow also clones the source into the image at expanded
~/pytorch(/root/pytorchviaPYTORCH_SRC_DIR):So the published image contains installed nightly PyTorch wheels, ROCm runtime/devel/device wheels from repo.amd.com, and source code from the actual
pytorch/pytorch:nightlybranch head at build time.Tags & secrets
docker.io/rocm/pytorch-nightly.latest.<YYYYMMDD>-<shortsha>.secrets.DOCKER_USERNAMEandsecrets.DOCKER_TOKEN.workflow_dispatchruns build and validate only; they do not log in or push torocm/pytorch-nightly.latestand<YYYYMMDD>-<shortsha>.Triggers & config
schedule: daily at 07:00 UTC.workflow_dispatchinputs (manual runs do not publish):pytorch_branch(defaultnightly)rocm_version(default10.0; derives both torch and ROCm package indexes)push_tag(defaultlatest)Validation
Local validation for the new devel install:
python3 -m pip install --dry-run --no-cache-dir --ignore-installed --index-url https://stable.repo.amd.com/rocm/whl-next/ 'rocm[devel]==10.0.*'The dry-run resolves
rocm-10.0.0,rocm-sdk-core-10.0.0, androcm-sdk-devel-10.0.0.Previous image validation on the Docker Hub
latesttag confirmed the pulled image has/root/pytorch, imports torch, and includes ROCm device packages. A new workflow run should be used to validate the 10.0/devel update end-to-end, includingrocm-sdk pathafter devel installation.Validation now prints
torch.version.hipandtorch.version.rocm.Latest validation run
Manual fork validation for current PR workflow (
1f7db8b96d) passed: https://github.com/ethanwee1/pytorch/actions/runs/35229609533This run used
rocm_version=10.0and manualworkflow_dispatch, so it built and validated only; Docker Hub login/push were skipped as expected.Validated output:
pytorch/pytorch:nightly->f3c0cf56a7577d677c37718c4edc41b8fc7fb51ahttps://stable.repo.amd.com/rocm/whl-next/rocm[libraries,devel,device-all]==10.0.*resolved to 10.0.0 packages, includingrocm-sdk-devel, libraries, and device packages (gfx908,gfx942,gfx950, etc.)https://download.pytorch.org/whl/nightly/rocm10.0/torch 2.15.0.dev20260916+rocm10.0,torch.version.hip 7.15.26333,torch.version.rocm 10.0.0ROCM_HOME/ROCM_BINwere generated fromrocm-sdk pathafter devel installation.