Skip to content

[PyTorch] Misc fixes for release_v1.6 - #784

Merged
ksivaman merged 5 commits into
NVIDIA:mainfrom
ksivaman:fp8_attention_fixes
Apr 17, 2024
Merged

[PyTorch] Misc fixes for release_v1.6#784
ksivaman merged 5 commits into
NVIDIA:mainfrom
ksivaman:fp8_attention_fixes

Conversation

@ksivaman

@ksivamanksivaman commented Apr 16, 2024•
edited
Loading

Copy link
Copy Markdown
Member
  1. Remove docs for 'is_first_module_in_mha' in Linear API's forward.
  2. Add check for FP8 before calling function to update amaxes.
  3. Only call is_first_fp8_module if any of the inputs or params require grad. We don't want the attribute to be set on a module for which bprop will not be executed.
  4. Fix the type check for grad_output in Linear backward since it is a Tensor (fwd returns 1 output).

@ksivaman
fixes; docs
ffcdc3e
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
@ksivaman
ksivaman requested a review from cyanguwaApril 16, 2024 16:38
@ksivamanksivaman self-assigned this Apr 16, 2024
@ksivaman
Check for FP8
ac70e05
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
@ksivaman
ksivamanforce-pushed the fp8_attention_fixes branch from f682260 to ac70e05CompareApril 16, 2024 23:56
@ksivaman
Merge branch 'NVIDIA:main' into fp8_attention_fixes
4268a80
@ksivaman
Fix LoRa-like use cases
73cbd71
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
@ksivamanksivaman changed the title [PyTorch] Misc fixes from #768[PyTorch] Misc fixes for release_v1.6Apr 17, 2024
@ksivaman
ksivaman requested a review from ptrendxApril 17, 2024 02:20
@ksivaman

ksivaman commented Apr 17, 2024

Copy link
Copy Markdown
MemberAuthor

/te-ci pytorch

timmoon10
timmoon10 reviewed Apr 17, 2024
View reviewed changes
Comment threadtransformer_engine/pytorch/module/layernorm_linear.py Outdated
@ksivaman
Reviews
42dc50b
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
@ksivaman

ksivaman commented Apr 17, 2024

Copy link
Copy Markdown
MemberAuthor

/te-ci pytorch

timmoon10
timmoon10 approved these changes Apr 17, 2024
View reviewed changes

@timmoon10timmoon10 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ksivaman
ksivaman merged commit cd54a8c into NVIDIA:mainApr 17, 2024
ksivaman added a commit that referenced this pull request Apr 17, 2024
@ksivaman
[PyTorch] Misc fixes for release_v1.6 (#784)
770173c
* fixes; docs
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
* Check for FP8
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
* Fix LoRa-like use cases
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
* Reviews
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
---------
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
ksivaman added a commit to ksivaman/TransformerEngine-1 that referenced this pull request Apr 19, 2024
@ksivaman
[PyTorch] Misc fixes for release_v1.6 (NVIDIA#784)
8568967
* fixes; docs
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
* Check for FP8
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
* Fix LoRa-like use cases
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
* Reviews
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
---------
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
ksivaman added a commit that referenced this pull request Apr 19, 2024
@ksivaman
[PyTorch] Misc fixes for release_v1.6 (#784)
09d576d
* fixes; docs
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
* Check for FP8
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
* Fix LoRa-like use cases
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
* Reviews
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
---------
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
pggPL pushed a commit to pggPL/TransformerEngine that referenced this pull request May 23, 2024
@ksivaman@pggPL
[PyTorch] Misc fixes for release_v1.6 (NVIDIA#784)
7e9dbca
* fixes; docs
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
* Check for FP8
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
* Fix LoRa-like use cases
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
* Reviews
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
---------
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
@hungryGeek16hungryGeek16 mentioned this pull request May 31, 2026
Open
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@timmoon10timmoon10timmoon10 approved these changes
@cyanguwacyanguwaAwaiting requested review from cyanguwa
@ptrendxptrendxAwaiting requested review from ptrendx

Assignees

@ksivamanksivaman

Labels

1.6.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@ksivaman@timmoon10