Cleaning up tensor/pipe parallel accounting. - #1252

Merged

ShadenSmith merged 4 commits into

deepspeedai:big-sciencefrom

ShadenSmith:shaden/pipe_replicated

Jul 26, 2021

Commits on Jul 26, 2021

removes repeated overflow log
Shaden Smith
committed
f76e59e
pipe_replicated
Shaden Smith
committed
467ebf2
_pipe_replicated -> ds_pipe_replicated
Shaden Smith
committed
d11244a
Adds send/recv fallback to bcast when torch version <= 1.8
Shaden Smith
committed
f66caaf