Skip to content

Clone tensors to avoid torch.save bloat - #3348

Merged
tjruwase merged 26 commits into
masterfrom
olruwase/issue_3303
May 16, 2023
Merged

Clone tensors to avoid torch.save bloat#3348
tjruwase merged 26 commits into
masterfrom
olruwase/issue_3303

Conversation

@tjruwase

@tjruwase tjruwase commented Apr 22, 2023

Copy link
Copy Markdown
Contributor

Fixes #3303

@tjruwase
tjruwase requested a review from jeffra as a code owner April 22, 2023 00:46
@tjruwase

tjruwase commented Apr 22, 2023

Copy link
Copy Markdown
Contributor Author

TODOs:

  1. Docs
  2. Unit tests (?)

@tjruwase tjruwase mentioned this pull request Apr 22, 2023
Closed
loadams
loadams reviewed May 2, 2023
View reviewed changes
Comment thread csrc/transformer/general_kernels.cu Outdated
@tjruwase

tjruwase commented May 2, 2023

Copy link
Copy Markdown
Contributor Author

@stas00, please see docs https://deepspeed.readthedocs.io/en/rtd-staging/model-checkpointing.html#avoiding-zero-checkpoint-bloat

@stas00

stas00 commented May 2, 2023 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Looking at the rendering - the source formatting appears to be borked. It has :param: and the last section doesn't show up.

And the doc is hard to read as it refers to input, let me try to make a better suggestion

stas00
stas00 reviewed May 2, 2023
View reviewed changes
Comment thread docs/code-docs/source/model-checkpointing.rst Outdated
stas00
stas00 reviewed May 2, 2023
View reviewed changes
Comment thread deepspeed/checkpoint/utils.py Outdated
stas00
stas00 reviewed May 2, 2023
View reviewed changes
Comment thread deepspeed/checkpoint/utils.py Outdated
tjruwase and others added 5 commits May 2, 2023 16:38
@tjruwase @stas00
Update docs/code-docs/source/model-checkpointing.rst
ad77a90 
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
@tjruwase @stas00
Update deepspeed/checkpoint/utils.py
9f029f8 
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
@tjruwase @stas00
Update deepspeed/checkpoint/utils.py
9af1f0e 
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
@tjruwase
Fix url
40140f9
@tjruwase
url fix
30eb070
@tjruwase

tjruwase commented May 2, 2023

Copy link
Copy Markdown
Contributor Author

@stas00, thanks for the feedback. I have applied your suggestions. Please take another look.

@stas00

stas00 commented May 2, 2023

Copy link
Copy Markdown
Collaborator

Looking good now!

@stas00 stas00 mentioned this pull request May 15, 2023
Open
@tjruwase
tjruwase merged commit 5c3ebd7 into master May 16, 2023
@stas00 stas00 mentioned this pull request May 23, 2023
Closed
4 tasks
@mrwyattii
mrwyattii deleted the olruwase/issue_3303 branch July 7, 2023 02:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@stas00 stas00 stas00 left review comments
@loadams loadams loadams left review comments
@mrwyattii mrwyattii Awaiting requested review from mrwyattii
@jomayeri jomayeri Awaiting requested review from jomayeri
@jeffra jeffra Awaiting requested review from jeffra
+1 more reviewer
@ShijieZZZZ ShijieZZZZ ShijieZZZZ approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[BUG] Size of saved model checkpoint becomes much larger after deepspeed.initialize when using ZeRO-2

4 participants

@tjruwase @stas00 @loadams @ShijieZZZZ