[Deepspeed] add many more models to the model zoo test

[stas00]

This PR continues figuring out how to make various models work with Deepspeed (a lot of fixes happen on the Deepspeed side), most models just work out of the box - the main purpose of this PR is to test as many models as possible. so there are no fixes to add.

• update coverage to albert, bart, bert, bigbird_pegasus, big_bird, blenderbot, deberta, deberta_v2, distilbert, electra, flaubert, fsmt, funnel, gpt2, gptj, gpt_neo, layoutlm, led, longformer, marian, mbart, mobilebert, mpnet, pegasus, prophetnet, roberta, squeezebert, t5, t5_v1, vit, xlm_roberta, xlnet

Thanks to @LysandreJik for creating the tiny test models for many of HF models!

Some models I couldn't cover for a variety of reasons unrelated to Deepspeed (missing tokenizers, missing tiny models, missing example scripts to exercise these). But their status is documented in the script. Over time more will be tested.

Blocking events - all resolved:

• [zero3] fix reference counting in backward over multiple forwards deepspeedai/DeepSpeed#1227 (fixes reference counting)
• [zero_to_fp32] fix padding removal deepspeedai/DeepSpeed#1380 (fixes zero_to_fp32 recovery of uneven param shapes)
• [SinusoidalPositionalEmbedding] incorrect dtype when resizing in forward #13665 (fixes positional embeddings: m2m_100 and others)
• Improve z3 trace management deepspeedai/DeepSpeed#1916 (comment) (fixes tracing)
• 0.6.4 Deepspeed release that includes all the merged PRs

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[sgugger]

Very nice work, thanks a lot @stas00 !

[sameeravithana]

nice work @stas00, have you tested Perceiver with DeepSpeed.

[stas00]

Would be glad to do that, @sameeravithana- in order to do that I need is a Trainer-based example script that I can test with.

As you can see from this map:

transformers/tests/deepspeed/test_model_zoo.py

Lines 231 to 270 in 4a419d4

	tasks = dict(
	trans=f"""
	{scripts_dir}/translation/run_translation.py
	--train_file {data_dir_wmt}/train.json
	--source_lang en
	--target_lang ro
	""",
	sum=f"""
	{scripts_dir}/summarization/run_summarization.py
	--train_file {data_dir_xsum}/sample.json
	--max_source_length 12
	--max_target_length 12
	--lang en
	""",
	clm=f"""
	{scripts_dir}/language-modeling/run_clm.py
	--train_file {FIXTURE_DIRECTORY}/sample_text.txt
	--block_size 8
	""",
	mlm=f"""
	{scripts_dir}/language-modeling/run_mlm.py
	--train_file {FIXTURE_DIRECTORY}/sample_text.txt
	""",
	qa=f"""
	{scripts_dir}/question-answering/run_qa.py
	--train_file {data_dir_samples}/SQUAD/sample.json
	""",
	clas=f"""
	{scripts_dir}/text-classification/run_glue.py
	--train_file {data_dir_samples}/MRPC/train.csv
	--max_seq_length 12
	--task_name MRPC
	""",
	img_clas=f"""
	{scripts_dir}/image-classification/run_image_classification.py
	--dataset_name hf-internal-testing/cats_vs_dogs_sample
	--remove_unused_columns False
	--max_steps 10
	--feature_extractor_name {DS_TESTS_DIRECTORY}/vit_feature_extractor.json
	""",

I have each model tested by one of HF Trainer examples. Is there one that can be used with perceiver?