Uh oh!
There was an error while loading. Please reload this page.
Publish Dion to PyPI from a v* tag via Trusted Publishing - #118
Open
John (JohnLangford) wants to merge 2 commits into
Open
Publish Dion to PyPI from a v* tag via Trusted Publishing#118John (JohnLangford) wants to merge 2 commits into
John (JohnLangford) wants to merge 2 commits into
Conversation
added 2 commits
August 24, 2026 13:05
Dion is installable only from git today, so a downstream project that publishes to PyPI cannot declare it as a dependency at all -- PyPI rejects direct-URL requirements in uploaded distributions. vllm-project/speculators#1031 works around this with a lazy import and a manual install message. Requested in microsoft#116. Adds .github/workflows/release.yml: every change to packaging builds and validates the distributions, and a v* tag publishes them. Publishing goes through PyPI Trusted Publishing (OIDC) against a `pypi` environment, so no API token lives in this repo. The tag is checked against setup.py's version before anything is uploaded. RELEASING.md documents the one-time PyPI-side setup and the steps to cut a release. Two things had to be fixed first; both build cleanly and pass `twine check`, which is why neither has been noticed: - author_email was `{kwangjunahn, byronxu}@microsoft.com`, deliberately obfuscated against harvesters. It does not parse as an address (email.utils.parseaddr returns ('', '')) and PyPI validates the field on upload, so the first release would have been rejected at the very last step. Those addresses no longer reach anyone in any case. The author list is paper attribution and is unchanged; only the contact is now live, and project_urls points bug reports at the issue tracker instead of an inbox. - MANIFEST.in did not exist, so the sdist shipped without the requirements_*.txt files that setup.py reads at build time. read_requirements warns and returns [] when they are absent, so building from the sdist *succeeds* and produces a wheel declaring no dependencies at all -- no numpy, no torch. Verified by round-tripping: before, an sdist-built wheel had zero base Requires-Dist; after, it carries numpy and torch>=2.7.1. tests/test_packaging.py covers both. It reads setup.py with ast, so it needs neither torch nor a build, and re-runs against the built artifacts in CI via DION_DIST_DIR -- twine check inspects only the long description and would not catch either bug. Also adds the MIT and Python-version classifiers, which PyPI facets on, and a pyproject.toml [build-system] table so builds stop going through the deprecated `setup.py bdist_wheel` path.
Review follow-ups on the PyPI publishing workflow. Supply chain: pin every action to a commit SHA. The publish job holds an OIDC identity PyPI trusts to upload as `dion`, and `pypa/gh-action-pypi-publish@release/v1` is a mutable *branch* -- a compromise of it runs code inside that job. Adds .github/dependabot.yml so the pins do not rot, and `persist-credentials: false` on the checkout, which does not need to keep a token in .git/config. Tag gate: verify the tag against the version baked into the built distribution filenames rather than re-parsing setup.py with sed. That is the version that will actually be uploaded, it removes the second parser for the same field, and comparing PEP 440 Versions rejects a tag that is not a valid version at all. PR trigger: the paths filter missed README.md -- which is the long_description that `twine check --strict` renders, so a README change could break a release with no pre-tag signal -- along with the other files MANIFEST.in ships and `dion/**`, whose layout find_packages() reads. MANIFEST.in: glob `requirements_*.txt` instead of listing the four files, so a requirements file added later is shipped rather than merely detected as missing. The test now matches names against the include patterns. Tests: - test_wheel_ships_every_package_module -- nothing checked that the wheel contains the code. find_packages() silently drops a subdirectory with no __init__.py, and CI builds the wheel from the sdist, so this covers both. - test_python_classifiers_agree_with_python_requires -- the classifiers advertise 3.9-3.13 with nothing tying them to python_requires. - test_repo_has_requirements_files -- an empty glob made the parametrized manifest test vacuous, which is the silent-pass failure this file exists to prevent. - import packaging directly instead of pytest.importorskip; pytest depends on it, so the skip could only ever hide the PEP 440 check. RELEASING.md: document that Actions must be enabled for repo-file workflows here -- no workflow under .github/ has ever run in this repository, and release.yml did not trigger on its own pull request -- plus why `python -m build` is spelled without flags (only the no-flag form builds the wheel from the sdist, which is the round trip the dependency assertions rely on), the SHA pinning, and that a `v*` tag alone reaches the upload step.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes#116.
Dion is installable only from git today. A project that itself publishes to PyPI therefore cannot declare
dionas a dependency at all — PyPI rejects direct-URL (git+https://) requirements in uploaded distributions. vllm-project/speculators#1031 is a live example, working around it with a lazy import and a manual install message:The
dionname is currently unclaimed on PyPI, so publishing also secures it.What this adds
.github/workflows/release.yml— the repo has no.github/today, so this is the first workflow.twine check --strict, and runs the packaging tests against both the source and the built artifacts.v*tag: rebuilds, verifies the tag matchessetup.py'sversion(sov0.2.0cannot publish0.1.0), and publishes via Trusted Publishing — OIDC against apypienvironment, no API token in the repo.RELEASING.mddocuments the one-time PyPI-side setup and how to cut a release.Two bugs this had to fix first
Both build cleanly and pass
twine check, which is why neither has surfaced.1.
author_emaildoes not parse, and PyPI validates it on upload.email.utils.parseaddrreturns('', '')for this. The first upload would have been rejected at the very last step, after everything else succeeded. Those addresses no longer reach anyone regardless. Theauthorlist is paper attribution and is unchanged — only the contact address is now live, and the newproject_urlssends bug reports to the issue tracker rather than to an inbox.2. The sdist shipped without the
requirements_*.txtfiles — silently.setup.pyreads them at build time. There was noMANIFEST.in, so they were not in the sdist, andread_requirementswarns and returns[]when a file is missing. The build therefore succeeds and produces a wheel that declares no dependencies at all. Round-tripped to confirm:Requires-Distnumpy,torch>=2.7.1The
py3-none-anywheel means most users would never hit it, but--no-binary, downstream packagers, and anyone building from source would have gotten a dependency-less install.Tests
tests/test_packaging.pycovers both. It parsessetup.pywithast, so it needs neither torch nor a build step, and CI re-runs it against the real artifacts viaDION_DIST_DIR.twine checkinspects only the long description and catches neither bug, so this is the only guard.Each test was confirmed to fail against the pre-fix state and pass after.
Also adds the
License :: OSI Approved :: MIT Licenseand Python-version classifiers (PyPI facets on these), and apyproject.toml[build-system]table so builds stop going through the deprecatedsetup.py bdist_wheelpath.Before the first release can happen
Two steps need someone with the accounts; neither can be done from a PR. Both are in
RELEASING.md:dion, ownermicrosoft, repodion, workflowrelease.yml, environmentpypi. A pending publisher is what allows the workflow to create a project that does not exist yet.pypienvironment in Settings → Environments. Worth adding required reviewers so a human approves each upload.Nothing is tagged or published by this PR.
CHANGELOG.md's[Unreleased]section is deliberately left alone — cutting it belongs with the version bump in the release PR. The README still documents thegit+install, which stays accurate until the first release lands.