Build scripts for pip wheels by ksivaman · Pull Request #1036 · NVIDIA/TransformerEngine

ksivaman · 2024-07-23T02:47:25Z

Description

Miscellaneous build fixes and improvements for shipping wheels.

Type of change

Documentation change (change only to the documentation, either a fix or a new content)
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Infra/Build change
Code refractor

Changes

Remove linking to cudnn and nvrtc and instead load them dynamically (NVRTC specifically has SOs for minor cuda versions as well). libtransformer_engine.so needs to define the symbol for cudnn_dlhandle.
Fix wheel builds for JAX as a refactor in [JAX] Splitting csrc/modules.cpp by category #883 broke this.
Resolve edge cases which result in .so conflicts when downloading from pip vs building locally.
Remove residual packages that are generated under dist-packages or site-packages during framework only dists.
Add scripts for generating files for wheel upload.
Use MANIFEST to properly include necessary files for source distributions without polluting global package namespace.
One TE/paddle wheel per python version.
Add sanity tests in CI to test wheel build workflow.
Fix GH workflow for paddle build.
Make python independent wheel for core lib.

Checklist:

I have read and followed the contributing guidelines
The functionality is complete
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

for more information, see https://pre-commit.ci

Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

for more information, see https://pre-commit.ci

Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

for more information, see https://pre-commit.ci

Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

timmoon10

LGTM. One step closer to pip install transformer_engine

ksivaman · 2024-07-24T21:44:41Z

Pipeline 16890187

* Specify python version Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Add classifiers for python Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Add utils to build wheels Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * make wheel scripts Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Add aarch Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Fixes Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Fix paddle wheel Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * PaddlePaddle only builds for x86 Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Add optional fwk deps Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Python3.8; catch install error Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * [wip] cudnn9 compile with paddle support Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * [wip] dont link cudnn Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * dlopen cudnn Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * fixes Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * fix Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Fixes Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * fix Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * dynamically load nvrtc Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Lint Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * remove residual packages; exclude stub from nvrtc .so search Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Exclude builtins from nvrtc .so search Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * properly include files for sdist Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * paddle wheel tie to python version Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Fix paddle build from src [wip] Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Fix workflow paddle build Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix paddle Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Fix paddle Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * fix lint from pr986 Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * fix Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Add sanity wheel test Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Add sanity import to wheel test Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * remove upper limit on paddlepaddle version Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Remove unused imports Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Remove pybind11 dependency Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Fix cpp tests Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Search .sos in cuda home Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fixes Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * CLeanup, remove residual code Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> --------- Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

ksivaman added 20 commits July 10, 2024 07:22

Specify python version
cbe56ce
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

Add classifiers for python
1ad6d8a
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

Add utils to build wheels
7699f30
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

make wheel scripts
2f2439d
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

Add aarch
eecef4c
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

Fixes
2f672d2
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

Merge branch 'NVIDIA:main' into wheels
577c78e

Fix paddle wheel
7106018
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

PaddlePaddle only builds for x86
05b53db
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

Merge branch 'NVIDIA:main' into wheels
da3aac6

Add optional fwk deps
7f6532a
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

Python3.8; catch install error
67fa446
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

[wip] cudnn9 compile with paddle support
b158cf6
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

[wip] dont link cudnn
98452a9
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

dlopen cudnn
1631e6f
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

fixes
0751947
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

fix
0522ee4
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

Fixes
f899e12
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

fix
2a53e20
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

dynamically load nvrtc
d3469ff
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

ksivaman marked this pull request as draft July 23, 2024 02:47

pre-commit-ciBotand others added 6 commits July 23, 2024 02:48

[pre-commit.ci] auto fixes from pre-commit.com hooks
b3d9cb8
for more information, see https://pre-commit.ci

Merge branch 'main' into wheels
14d3e4e

Lint
d950a8b
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

remove residual packages; exclude stub from nvrtc .so search
ceae09c
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

Exclude builtins from nvrtc .so search
63673f4
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

properly include files for sdist
63a2c30
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

ksivaman requested a review from timmoon10 July 23, 2024 20:24

ksivaman self-assigned this Jul 23, 2024

ksivaman added the 1.9.0 label Jul 23, 2024

ksivamanand others added 7 commits July 23, 2024 23:16

Fix paddle build from src [wip]
9370d11
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

Fix workflow paddle build
f2af7f2
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks
a5e5401
for more information, see https://pre-commit.ci

Merge branch 'main' into wheels
f101173

Fix paddle
09758fd
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

Fix paddle
31fcc5a
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

fix lint from pr986
515ecc7
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

ksivaman marked this pull request as ready for review July 24, 2024 00:17

ksivaman added 4 commits July 23, 2024 17:45

fix
4722693
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

Add sanity wheel test
48f3d48
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

Add sanity import to wheel test
25a8019
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

remove upper limit on paddlepaddle version
ec0f486
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

timmoon10 reviewed Jul 24, 2024
View reviewed changes

ksivamanand others added 9 commits July 24, 2024 13:41

Remove unused imports
5af18aa
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

Remove pybind11 dependency
564db35
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

Fix cpp tests
0846065
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

Merge branch 'main' into wheels
960645c
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

Search .sos in cuda home
cb44159
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

[pre-commit.ci] auto fixes from pre-commit.com hooks
e3bfd60
for more information, see https://pre-commit.ci

fixes
cd7760d
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

CLeanup, remove residual code
bdb807a
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>

Merge branch 'main' into wheels
7db2621

timmoon10 approved these changes Jul 24, 2024
View reviewed changes

ksivaman merged commit 0981357 into NVIDIA:mainJul 25, 2024

This was referenced Jul 25, 2024
Fixes for pip wheels #1042
Merged
[PyTorch] Fix linter warnings #1041
Merged

timmoon10 mentioned this pull request Aug 1, 2024
[C/PyTorch] Userbuffers and comm+GEMM overlap algorithms refactored and moved to TE/common #1067
Merged
13 tasks

hungryGeek16 mentioned this pull request May 31, 2026
fix unfused padding causal sdpa #3063
Open

Navigation Menu

Uh oh!

Build scripts for pip wheels - #1036

Build scripts for pip wheels#1036
ksivaman merged 47 commits into
NVIDIA:mainfrom
ksivaman:wheels

ksivaman commented Jul 23, 2024•
edited
Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

timmoon10 left a comment

Uh oh!

ksivaman commented Jul 24, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ksivaman commented Jul 23, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Changes

Checklist:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

timmoon10 left a comment

Choose a reason for hiding this comment

Uh oh!

ksivaman commented Jul 24, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ksivaman commented Jul 23, 2024•
edited
Loading