Uh oh!
There was an error while loading. Please reload this page.
Encoder-decoder Multihead attention cpu optimization - #43
Conversation
Good improvement!
|
Yu Yan (yuyan2do)
left a comment
There was a problem hiding this comment.
Also update readme and number in benchmark script
| key_padding_mask.unsqueeze(1).to( | ||
| torch.bool), float("-inf")) | ||
| else: | ||
| #Not supported |
There was a problem hiding this comment.
add "assert False, reason"
| attn_weights = attn_weights.view(-1, tgt_len, | ||
| *attn_weights.size()[-1:]) | ||
| else: | ||
| q = q.contiguous().view(tgt_len, bsz * self.num_heads, |
There was a problem hiding this comment.
why contiguous is needed here?
There was a problem hiding this comment.
This was present in the earlier implementation, I didn't touch it since my changes are only meant for en-dec attention. I agree this is redundant. I'll remove it.
There was a problem hiding this comment.
There are other places using contiguous. please also check if they can be removed as well.
There was a problem hiding this comment.
I just checked this. In all other places, its present after permute/transpose operations which is essential.
This PR reduces CPU time for encoder-decoder multihead attention by 25-30%. GPU time is reduced by 10%.
Overall generation time reduces from 47.8 to 44.9.
Attaching before/after profile results: