Uh oh!
There was an error while loading. Please reload this page.
Add attention bias and qkv format to context parallelism - #726
Merged
Conversation
Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Signed-off-by: Xiaowei Ren <xren@nvidia.com>
cyanguwa
self-requested a review
March 26, 2024 18:56
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
cyanguwa
commented
Apr 23, 2024
Collaborator
/te-ci pytorch |
Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Signed-off-by: Xiaowei Ren <xren@nvidia.com>
Uh oh!
There was an error while loading. Please reload this page.
cyanguwa
commented
Apr 24, 2024
Collaborator
/te-ci pytorch |
Signed-off-by: Xiaowei Ren <xren@nvidia.com>
cyanguwa
commented
Apr 24, 2024
Collaborator
/te-ci pytorch |
cyanguwa
approved these changes
Apr 26, 2024
phu0ngng referenced
this pull request
in phu0ngng/TransformerEngine
May 3, 2024
* make FusedAttn with CP support bias Signed-off-by: Xiaowei Ren <xren@nvidia.com> * assert Alibi cannot work with CP Signed-off-by: Xiaowei Ren <xren@nvidia.com> * syntax fix Signed-off-by: Xiaowei Ren <xren@nvidia.com> * fix variable name Signed-off-by: Xiaowei Ren <xren@nvidia.com> * fix tensor shapes Signed-off-by: Xiaowei Ren <xren@nvidia.com> * a typo fix Signed-off-by: Xiaowei Ren <xren@nvidia.com> * fix bias indexing for CP Signed-off-by: Xiaowei Ren <xren@nvidia.com> * bug fix Signed-off-by: Xiaowei Ren <xren@nvidia.com> * add attn bias tests Signed-off-by: Xiaowei Ren <xren@nvidia.com> * change dbias update location Signed-off-by: Xiaowei Ren <xren@nvidia.com> * fix CP test model configs Signed-off-by: Xiaowei Ren <xren@nvidia.com> * change CP test sequence length Signed-off-by: Xiaowei Ren <xren@nvidia.com> * make AttnFuncWithCP support qkv format of sbhd Signed-off-by: Xiaowei Ren <xren@nvidia.com> * make sure qkv are contiguous for CP in cuDNN fused attn Signed-off-by: Xiaowei Ren <xren@nvidia.com> * change assert message Signed-off-by: Xiaowei Ren <xren@nvidia.com> * fix code format Signed-off-by: Xiaowei Ren <xren@nvidia.com> --------- Signed-off-by: Xiaowei Ren <xren@nvidia.com> Co-authored-by: cyanguwa <8636796+cyanguwa@users.noreply.github.com>
pggPL pushed a commit
to pggPL/TransformerEngine
that referenced
this pull request
May 23, 2024
* make FusedAttn with CP support bias Signed-off-by: Xiaowei Ren <xren@nvidia.com> * assert Alibi cannot work with CP Signed-off-by: Xiaowei Ren <xren@nvidia.com> * syntax fix Signed-off-by: Xiaowei Ren <xren@nvidia.com> * fix variable name Signed-off-by: Xiaowei Ren <xren@nvidia.com> * fix tensor shapes Signed-off-by: Xiaowei Ren <xren@nvidia.com> * a typo fix Signed-off-by: Xiaowei Ren <xren@nvidia.com> * fix bias indexing for CP Signed-off-by: Xiaowei Ren <xren@nvidia.com> * bug fix Signed-off-by: Xiaowei Ren <xren@nvidia.com> * add attn bias tests Signed-off-by: Xiaowei Ren <xren@nvidia.com> * change dbias update location Signed-off-by: Xiaowei Ren <xren@nvidia.com> * fix CP test model configs Signed-off-by: Xiaowei Ren <xren@nvidia.com> * change CP test sequence length Signed-off-by: Xiaowei Ren <xren@nvidia.com> * make AttnFuncWithCP support qkv format of sbhd Signed-off-by: Xiaowei Ren <xren@nvidia.com> * make sure qkv are contiguous for CP in cuDNN fused attn Signed-off-by: Xiaowei Ren <xren@nvidia.com> * change assert message Signed-off-by: Xiaowei Ren <xren@nvidia.com> * fix code format Signed-off-by: Xiaowei Ren <xren@nvidia.com> --------- Signed-off-by: Xiaowei Ren <xren@nvidia.com> Co-authored-by: cyanguwa <8636796+cyanguwa@users.noreply.github.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add attn_bias and qkv_format to AttnFuncWithCP.