Conversation
…rallelism with DeepSpeed on this HuggingFace model
sgugger
left a comment
There was a problem hiding this comment.
Thanks a lot for your PR!
I made one comment on the first model that can be replicated to all the other ones.
Could you also add a test to make sure the feature works (we might not be able to run it on our 2 GPUs machine, but a 4 GPUs one is coming).
|
|
||
| # Use the class's parameter as the hidden_state's last dimension. | ||
| # This dimension cannot be used in case of enabling tensor-parallelism. |
There was a problem hiding this comment.
I don't think this comment is useful when reading the new code. It creates more confusion than help, only the next one is really important.
There was a problem hiding this comment.
I agree, I can remove this.
| # Use the embed_dim from class rather than hidden_state, this is due to | ||
| # the reason that attn_output can be partitioned across GPUs | ||
| # when using tensor-parallelism, in which case the embed_dimension from | ||
| # the input is not equal to the attention's last dimension after merging | ||
| # heads. |
There was a problem hiding this comment.
We have a 119 char limits so you can use more horizontal space :-)
Also, I suggest the following change, more to the point:
Use the `embed_dim` from the config (stored in the class) rather than `hidden_state` because `attn_output` can be partitioned across GPUs when using tensor-parallelism.
There was a problem hiding this comment.
Sure, I will reformat this :)
We will have a full battery of tests for Deepspeed Inference. I will take care of this, Sylvain. The plan is to have a model zoo style test - identical to Deepspeed ZeRO tests, so to cover as many models as possible. (there will be also Deepspeed ZeRO Inference tests #14253, which is different from Deepspeed Inference) We didn't feel a test was needed for this particular PR since it doesn't change anything for a normal application. |
|
You now need to run |
sgugger
left a comment
There was a problem hiding this comment.
Thanks a lot for all your work on this!
What does this PR do?
This PR adds a minor modification to BartAttention and its copies to support tensor-parallelism with DeepSpeed. This relates to this PR on DeepSpeed side.
Please see the added comments in the code that explain the change.
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@stas00