Fairseq v0.10.2 compatible

[Julianne Knott (JulianneKnott)]

Updated generate.py to be compatible with Fairseq v0.10.2.
Speed tbd.

[Jiusheng Chen (JiushengChen)]

Avoid checking in those txt files and old backup file - "generate_old.py".

[Fei Hu (feihugis)]

Thanks Julianne Knott (@JulianneKnott) for this PR! I did a pass and have one question for the files under results: are they used for comparing the outputs from fastseq and fairseq? It will be very helpful if we can add a unit test to automatically check it. We can also integrate it with our CI in the next step.

[Julianne Knott (JulianneKnott)]

Thanks Julianne Knott (@JulianneKnott) for this PR! I did a pass and have one question for the files under results: are they used for comparing the outputs from fastseq and fairseq? It will be very helpful if we can add a unit test to automatically check it. We can also integrate it with our CI in the next step.

Yes. Didn't mean to commit them and will remove them on the next commit. Will look into adding a unit test also.

[Julianne Knott (JulianneKnott)]

Benchmark Info:

Util Model Task Split BatchSize Samples Tokens Bleu Rouge Loss Perplexity Runtime(seconds) Throughput(samples/s) Throughput(tokens/s)
fairseq_v0.10.2 bart.large.cnn cnn_dm/len-1024.bin valid 32 13367 1098036 17.93 NA NA NA 3875 3.4 283.4
fairseq_v0.10.2+fastseq_v0.0.4 bart.large.cnn cnn_dm/len-1024.bin valid 32 13367 1098036 17.93 NA NA NA 3028 4.4 362.6

[Fei Hu (feihugis)]

The current code looks good to me. As this change will break the other parts of the main branch, could we create another branch (e.g., fairseq-0.10.2) for this work? After finishing all the tasks, it can be merged to the main branch.

[Julianne Knott (JulianneKnott)]

Updated beam search optimizer.
Benchmarks:

Util Model Task Split BatchSize Samples Tokens Bleu Rouge Loss Perplexity Runtime(seconds) Throughput(samples/s) Throughput(tokens/s)
fairseq_v0.10.2+fastseq_v0.0.4 bart.large.cnn cnn_dm/len-1024.bin valid 256 13367 1098278 17.90 NA NA NA 718 18.6 1529.6

[Julianne Knott (JulianneKnott)]

Updated el attention optimizer
Benchmarks:

Util Model Task Split BatchSize Samples Tokens Bleu Rouge Loss Perplexity Runtime(seconds) Throughput(samples/s) Throughput(tokens/s)
fairseq_v0.10.2+fastseq_v0.0.4 bart.large.cnn cnn_dm/len-1024.bin valid 320 13367 1098535 17.92 NA NA NA 621 21.5 1769.0

[Fei Hu (feihugis)]

Thanks Julianne Knott (@JulianneKnott) ! LGTM. Just find some very minor issues. After fixing, the PR will be good to merge.

[Fei Hu (feihugis)]

I'm not sure if people will get confused when the child class has the same name with the parent class? What do Julianne Knott (@JulianneKnott) think of it?

[Yu Yan (yuyan2do)]

Looks good. Just added a few suggestions inline.

Another question: Is there any test cover converting a model to its TorchScript version? If not, we need do it manually to verify our change do not break this functionality.

[Yu Yan (yuyan2do)]

Where is this param used?

[Julianne Knott (JulianneKnott)]

fastseq/fastseq/optimizer/fairseq/generate.py

Line 152 in 4a3f17e

ifargs.decode_hypothesis:

also lines: 211, 218, 240 in the same file