Uh oh!
There was an error while loading. Please reload this page.
[Paddle] Support recompute - #412
Merged
Merged
Conversation
ContributorAuthor
Add @jeng1220 to take a look. |
Tom-Zheng
marked this pull request as draft
September 2, 2023 06:18
Signed-off-by: Tian Zheng (Engrg-Hardware 1) <tizheng@nvidia.com>
Signed-off-by: Tian Zheng (Engrg-Hardware 1) <tizheng@nvidia.com>
Signed-off-by: Tian Zheng (Engrg-Hardware 1) <tizheng@nvidia.com>
Tom-Zhengforce-pushed
the
gh_add_recompute
branch
from
September 3, 2023 12:37
8c2a786 to
895e0aaCompareTom-Zheng
marked this pull request as ready for review
September 3, 2023 12:38
zlsh80826
commented
Sep 3, 2023
Collaborator
/te-ci |
Signed-off-by: Tian Zheng (Engrg-Hardware 1) <tizheng@nvidia.com>
Tom-Zheng
commented
Sep 3, 2023
ContributorAuthor
@timmoon10 There were some changes since your last approval, would you please take a look? cc: @jeng1220 |
Signed-off-by: Tian Zheng (Engrg-Hardware 1) <tizheng@nvidia.com>
Tom-Zheng
commented
Sep 3, 2023
ContributorAuthor
Changes since last approval:
|
zlsh80826
commented
Sep 4, 2023
Collaborator
/te-ci |
jeng1220
reviewed
Sep 4, 2023
| """ | ||
| # Create 'script.py' file | ||
| with open('script.py', 'w', encoding="utf8") as script_file: | ||
| script_file.write(script_content) |
Contributor
There was a problem hiding this comment.
Just create a file for the code instead of using a string.
Signed-off-by: Tian Zheng (Engrg-Hardware 1) <tizheng@nvidia.com>
Signed-off-by: Tian Zheng (Engrg-Hardware 1) <tizheng@nvidia.com>
jeng1220
commented
Sep 5, 2023
Contributor
LGTM. |
zlsh80826
commented
Sep 5, 2023
Collaborator
/te-ci |
jeng1220
commented
Sep 5, 2023
Contributor
There are some UT failures but they are not related to this PR:
Please review and merge the code if everything looks good. |
timmoon10
approved these changes
Sep 6, 2023
RuiWang1998 pushed a commit
to RuiWang1998/TransformerEngine
that referenced
this pull request
Sep 11, 2023
* Add recompute Signed-off-by: Tian Zheng (Engrg-Hardware 1) <tizheng@nvidia.com> * Support recompute core attention Signed-off-by: Tian Zheng (Engrg-Hardware 1) <tizheng@nvidia.com> * Fix transformer layer recompute Signed-off-by: Tian Zheng (Engrg-Hardware 1) <tizheng@nvidia.com> * Add doc Signed-off-by: Tian Zheng (Engrg-Hardware 1) <tizheng@nvidia.com> * Improve recompute test Signed-off-by: Tian Zheng (Engrg-Hardware 1) <tizheng@nvidia.com> * Improve performance of stack backtrace Signed-off-by: Tian Zheng (Engrg-Hardware 1) <tizheng@nvidia.com> * Improve code stype Signed-off-by: Tian Zheng (Engrg-Hardware 1) <tizheng@nvidia.com> * Fix code style Signed-off-by: Tian Zheng (Engrg-Hardware 1) <tizheng@nvidia.com> --------- Signed-off-by: Tian Zheng (Engrg-Hardware 1) <tizheng@nvidia.com> Signed-off-by: Rui Wang <rui@helixon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR supports recompute for TE Paddle. The user can use
te.recomputeto wrap TE layers, or userecompute_core_attention=TrueinTransformerLayer.forwardto recompute only the core attention part.