Uh oh!
There was an error while loading. Please reload this page.
remove d2d copies - #64
Merged
Merged
Conversation
Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
ksivaman
commented
Feb 1, 2023
MemberAuthor
@ptrendx Turns out the |
ksivaman
commented
Feb 1, 2023
MemberAuthor
/te-ci |
ptrendx
reviewed
Feb 2, 2023
| A_scale_inverse_arg = A_scale_inverse[A_fp8_tensor]; | ||
| at::Tensor B_scale_inverse_arg = B_scale_inverse.clone(); | ||
| at::Tensor B_scale_inverse_arg = B_scale_inverse; |
Member
There was a problem hiding this comment.
Can't we just use the original value instead? This just makes the code harder to read for no reason.
MemberAuthor
There was a problem hiding this comment.
Sure we could, we still need to check that tensor is not empty though.
ksivaman
commented
Feb 2, 2023
MemberAuthor
/te-ci |
cyanguwa pushed a commit
to cyanguwa/TransformerEngine
that referenced
this pull request
Feb 13, 2023
* remove d2d copies Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * cleanup Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> --------- Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> Signed-off-by: Charlene Yang <charleney@nvidia.com>
ksivaman added a commit
that referenced
this pull request
Feb 22, 2023
* add flash attention to TransformerLayer Signed-off-by: Charlene Yang <charleney@nvidia.com> * Add docs for FP8 calibration (#61) Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> Signed-off-by: Charlene Yang <charleney@nvidia.com> * Fix the integer overflow in fused softmax (#60) Signed-off-by: Przemek Tredak <ptredak@nvidia.com> Signed-off-by: Charlene Yang <charleney@nvidia.com> * prefix flash attn env var with NVTE_ Signed-off-by: Charlene Yang <charleney@nvidia.com> * Address steady memory increase and bloated checkpoints (#63) Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> Signed-off-by: Charlene Yang <charleney@nvidia.com> * fix env var logic Signed-off-by: cyanguwa <cyang.uwa@gmail.com> Signed-off-by: Charlene Yang <charleney@nvidia.com> * fix flash attn env var logic again Signed-off-by: cyanguwa <cyang.uwa@gmail.com> Signed-off-by: Charlene Yang <charleney@nvidia.com> * remove d2d copies (#64) * remove d2d copies Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * cleanup Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> --------- Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> Signed-off-by: Charlene Yang <charleney@nvidia.com> * Increase number of FP8 tensors per GEMM (#22) * Increase number of FP8 tensors per GEMM Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com> * Enable FP8 output tensor for fp8_gemm Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com> * [BERT FP8] Initial TE review comments Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com> * Temporary fix for cuda graph non convergence Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com> * Address review comments-2 Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com> * Review comments-3 Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com> * Cleanup Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com> * Change for New API Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com> * Remove unnecessary clone for D_scale, D_amax Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com> * Avoid Roll for AMAX history size = 1 Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com> * Update onnx_te_gemm API Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com> * Fix Lint errors Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com> --------- Signed-off-by: Vasudevan Rengasamy <vrengasamy@nvidia.com> Signed-off-by: Charlene Yang <charleney@nvidia.com> * Bug fixes from PR 22 (#65) * Bug fixes from PR 22 Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Add FP8 tests to ci Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * bundle unittests for ci Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> --------- Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> Signed-off-by: Charlene Yang <charleney@nvidia.com> * replace rearrange with transpose Signed-off-by: cyanguwa <cyang.uwa@gmail.com> Signed-off-by: Charlene Yang <charleney@nvidia.com> * QKV parameters unfused path fixes and optimization (#66) * Bug fixes from PR 22 Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Add FP8 tests to ci Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Better QKV parameter fusion Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * small fix Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * keep original param for unfused case to retain externally set attrs Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * lint fix Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Fix ONNX exports Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * improve arg naming Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * No need to set data pointers Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * lint Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Assert memory loc in NoopCat Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Handle case of different memory in param and buffer Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * fix assert always true Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Reassign params memory to avoid more concats Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> --------- Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> Signed-off-by: Charlene Yang <charleney@nvidia.com> * Fix gradients when using AMP (#70) retain grad related attrs while casting Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> Signed-off-by: Charlene Yang <charleney@nvidia.com> * fix pylint violations fixed pyline violations such as trailing white spaces and too long lines Signed-off-by: cyanguwa <cyang.uwa@gmail.com> * fix pylint violation on line 264 with R1719 Signed-off-by: cyanguwa <cyang.uwa@gmail.com> * fix two more pylint violations Signed-off-by: cyanguwa <cyang.uwa@gmail.com> * DotProductAttention API Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * Add docs for attention Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * fix assert always true Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * check for correct flash-attn version Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * address review comments Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * lint+build fixes, correct settings for default flash-attn Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * correct version Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * review comments and fixes Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * fix onnx and disable flash-attn export test Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * remove einops dependency Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * cleanup internal API; rm duplication Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * fix Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * only install TE wheel (exclude flash-attn to rm conflicts) Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * forgot to change install wheel path Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * next round review comments Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * fix flash_attn output Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * fix QK layer scaling Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * update docs Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> * review comments and fixes to selective checkpointing Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> --------- Signed-off-by: Charlene Yang <charleney@nvidia.com> Signed-off-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com> Signed-off-by: cyanguwa <cyang.uwa@gmail.com> Co-authored-by: Charlene Yang <charleney@nvidia.com> Co-authored-by: Kirthi Shankar Sivamani <ksivamani@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Kirthi Shankar Sivamani ksivamani@nvidia.com