In multiple places in the deepspeed code there exist hardcoded checks such as:
if self.module.__class__.__name__ == 'GPT2ModelPipe':
or
all('ParallelTransformerLayerPipe' in f.__class__.__name__ for f in funcs)
In order to make the examples work.
This means that it is not possible to integrate deepspeed into an existing codebase by looking at the examples or documentation, one needs to thoroughly read the deepspeed code and then name the modules exactly as stated above to get the desired behavior.
To my knowledge this peculiarity is not documented anywhere.
This has been previously noted for checkpointing here: #688
But it is also necessary in order to send the boolean attention masks across pipeline stages. If one attempts to send the attention masks as described in the tutorial but having the wrong module name, deepspeed will deadlock on the backward pass. I believe this is due to NACL not correctly handing boolean tensor but such workaround should happen in deepspeed, or in the very least there should be warnings.
This should either be clearly documented, especially in the examples, or better yet, fixed.
Thanks
In multiple places in the deepspeed code there exist hardcoded checks such as:
if self.module.__class__.__name__ == 'GPT2ModelPipe':or
all('ParallelTransformerLayerPipe' in f.__class__.__name__ for f in funcs)In order to make the examples work.
This means that it is not possible to integrate deepspeed into an existing codebase by looking at the examples or documentation, one needs to thoroughly read the deepspeed code and then name the modules exactly as stated above to get the desired behavior.
To my knowledge this peculiarity is not documented anywhere.
This has been previously noted for checkpointing here: #688
But it is also necessary in order to send the boolean attention masks across pipeline stages. If one attempts to send the attention masks as described in the tutorial but having the wrong module name, deepspeed will deadlock on the backward pass. I believe this is due to NACL not correctly handing boolean tensor but such workaround should happen in deepspeed, or in the very least there should be warnings.
This should either be clearly documented, especially in the examples, or better yet, fixed.
Thanks