Sync b/w Eleuther AI's DeeperSpeed and DeepSpeed

[g-karthik]

Can the added changes made in Eleuther AI's DeeperSpeed fork be merged back into DeepSpeed?

I see some critical differences, such as in PipelineModule: former and latter.

When using containerized setups, it's a lot cleaner to just install from one source rather than have multiple things going on.

@tjruwase @jeffra @ShadenSmith

[sdtblck]

I submitted a PR for the critical difference you linked above a while back but I think it went unnoticed. #687

It's been paramount for us to get Pipeline Parallelism working at all, so I'm surprised it's still in the main branch.

[sdtblck]

Aside from that and this EleutherAI@0e95737 (for running inference - majorly hacky and I'm sure there's a better way of doing it), they're mostly the same aside from some minor changes to timers / logging.

[loadams]

Closing as this is stale and not an active issue. If there are other changes from a fork people want merged, please link this when you make a PR.