Skip to content
This repository was archived by the owner on Jun 11, 2026. It is now read-only.

Cuda op for ngram repeat blocking - #40

Merged
NickNickGo merged 15 commits into
microsoft:mainfrom
NickNickGo:cuda_op_ngram_block
Nov 13, 2020
Merged

Cuda op for ngram repeat blocking#40
NickNickGo merged 15 commits into
microsoft:mainfrom
NickNickGo:cuda_op_ngram_block

Conversation

@NickNickGo

@NickNickGoNickNickGo commented Sep 24, 2020•
edited
Loading

Copy link
Copy Markdown
Contributor

This PR involves 3 optimizations.

  1. parallelizing ngram blocking across all samples within a batch.
  2. parallelizing ngram blocking across all ngrams within a sample.
  3. Accessing consecutive words from shared mem instead of global mem.

Transformers BART large BS 128 1k samples, throughput change - 9.1 to 11.8 (including model load time)
Fairseq BART large BS 128 1k samples, throughput change - 13.7 to 15.5. Generation time reduces from 48.4 to 39.7

@NickNickGo
NickNickGo requested a review from a teamSeptember 24, 2020 22:59
feihugis
Fei Hu (feihugis) reviewed Sep 25, 2020
View reviewed changes

@feihugisFei Hu (feihugis) left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks NickNickGo for implement this operation! I did not look into the details yet. Leave some general comments:

  1. add the unit tests to make sure the op works as expected for different cases;
  2. add the benchmarking unit test for this op;
  3. check if the inputs are valid either in the Python API or the backend?
  4. add the license header to each file;
  5. Add the docs for both Python and C++ code;

Looking forward to the performance number!

Comment threadfastseq/clib/cuda/ngrb.py Outdated
@NickNickGoNickNickGo mentioned this pull request Sep 25, 2020
Open
@NickNickGo
NickNickGo requested a review from a teamSeptember 25, 2020 07:07
yuyan2do
Yu Yan (yuyan2do) reviewed Sep 25, 2020
View reviewed changes

@yuyan2doYu Yan (yuyan2do) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall it looks good to me. Added some comments for parameter naming.

A tutorial for cuda program, which explain block, thread, shared memory.
https://www.nvidia.com/docs/IO/116711/sc11-cuda-c-basics.pdf

Comment threadfastseq/optimizer/fairseq/beam_search_optimizer_v1.py Outdated
Comment threadfastseq/optimizer/transformers/beam_search_optimizer.py Outdated
Comment threadfastseq/clib/cuda/ngrb_cuda_kernel.cu Outdated
Comment threadfastseq/clib/cuda/ngrb_cuda_kernel.cu Outdated
Comment threadfastseq/clib/cuda/ngrb_cuda_kernel.cu Outdated
Comment threadfastseq/clib/cuda/ngrb_cuda_kernel.cu Outdated
int no_repeat_ngram = 3;
int threads = step - no_repeat_ngram +2;
int shared_mem_size = (step+1) *sizeof(long);
if (threads <=0) return lprobs;

@yuyan2doYu Yan (yuyan2do)Sep 25, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

put return check at beginning of this function.

Comment threadfastseq/clib/cuda/ngrb_cuda_kernel.cu Outdated
Comment threadfastseq/clib/cuda/ngrb_cuda_kernel.cu Outdated
Comment threadfastseq/clib/cuda/ngrb_cuda_kernel.cu Outdated
Comment threadfastseq/clib/cuda/ngrb_cuda_kernel.cu Outdated
JiushengChen
Jiusheng Chen (JiushengChen) reviewed Sep 26, 2020
View reviewed changes
Comment threadfastseq/clib/cuda/ngrb_cuda_kernel.cu Outdated
@JiushengChen

Jiusheng Chen (JiushengChen) commented Sep 26, 2020

Copy link
Copy Markdown
Contributor

Overall it looks good to me. Added some comments for parameter naming.

A tutorial for cuda program, which explain block, thread, shared memory.
https://www.nvidia.com/docs/IO/116711/sc11-cuda-c-basics.pdf

This page explains CUDA various architecture well. https://en.wikipedia.org/wiki/CUDA
Key facts about V100:

  1. V100 has 80 SMs.
  2. Each SM has max 32 blocks, max 2048 threads, max 96k shared mem.

yetingqiaqia
Ting Ye (yetingqiaqia) reviewed Oct 1, 2020
View reviewed changes
Comment threadfastseq/clib/cuda/ngrb.py Outdated
Comment threadfastseq/clib/cuda/ngrb.py Outdated
feihugis
Fei Hu (feihugis) suggested changes Oct 30, 2020
View reviewed changes
Comment threadtests/ops/test_ngram_repeat_block.py Outdated
Comment threadfastseq/clib/cuda/ngram_repeat_block_cuda.cpp Outdated
Comment threadfastseq/clib/cuda/ngram_repeat_block_cuda.cpp Outdated
Comment threadtests/ops/test_ngram_repeat_block.py
Comment threadfastseq/clib/cuda/ngram_repeat_block_cuda.cpp Outdated
Comment threadfastseq/clib/cuda/ngram_repeat_block_cuda_kernel.cu Outdated
Comment on lines +52 to +53
torch::Tensor tokens,
torch::Tensor lprobs,

@feihugisFei Hu (feihugis)Oct 30, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to check the dimensions of tokens and lprobs? For example, the 0th dim of tokens is assumed to be batch dimension?

Could tokens be constant? const torch::Tensor& tokens?

Comment threadfastseq/clib/cuda/ngram_repeat_block_cuda_kernel.cu
Comment threadfastseq/ops/ngram_repeat_block.py
Comment threadtests/ops/test_ngram_repeat_block.py Outdated
feihugis
Fei Hu (feihugis) reviewed Oct 30, 2020
View reviewed changes
Comment threadtests/ops/test_ngram_repeat_block.py Outdated
feihugis
Fei Hu (feihugis) reviewed Oct 30, 2020
View reviewed changes
Comment threadtests/ops/test_ngram_repeat_block.py Outdated
@NickNickGo

NickNickGo commented Nov 11, 2020

Copy link
Copy Markdown
ContributorAuthor

Before :

ModelW/O FastSeq (in samples/s)W/ FastSeq (in samples/s)Speedup
ProphetNet2.710.33.8x
Bart (fs)2.712.54.6x
Bart (hf)3.48.12.4x
DistilBart (hf)4.08.52.1x
T5 (hf)4.87.51.6x
WMT16 En-De (fs)84.0122.01.5x

After:

ModelW/O FastSeq (in samples/s)W/ FastSeq (in samples/s)Speedup
ProphetNet2.710.33.8x
Bart (fs)2.713.35x
Bart (hf)3.49.92.9x
DistilBart (hf)4.011.93x
T5 (hf)4.811.02.3x
WMT16 En-De (fs)84.0124.01.5x

JiushengChen
Jiusheng Chen (JiushengChen) reviewed Nov 11, 2020
View reviewed changes
Comment threadREADME.md Outdated
Comment on lines +23 to +25
- ## How it works?
- We developped a wide range of speedup techniques, including improving beam search efficiency, reducing memory footprint, speeding up calculation for key operations etc, IO speedup etc. To seamlessly connect with community, they were applied to existing models from Fairseq and Huggingface Transformers in the backend, while keeping model interface and usage same as before.
-

@JiushengChenJiusheng Chen (JiushengChen)Nov 11, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why adding "-" here? This is supposed to be an independent section:)

@NickNickGoNickNickGoNov 11, 2020

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed

@NickNickGo
NickNickGo merged commit 6b3c0cb into microsoft:mainNov 13, 2020
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

4 more reviewers
@yetingqiaqiaTing Ye (yetingqiaqia)yetingqiaqia left review comments
@JiushengChenJiusheng Chen (JiushengChen)JiushengChen left review comments
@feihugisFei Hu (feihugis)feihugis approved these changes
@yuyan2doYu Yan (yuyan2do)yuyan2do approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@NickNickGo@JiushengChen@feihugis@yuyan2do@yetingqiaqia