[Latent Consistency Distillation] training stuck at 0%

[rdcoder33]

Describe the bug

When using LCM-LoRA training script, the training just stucks at 0%, tired all different parameters, got same error.

Reproduction

Follow the steps on this page,

[Latent Consistency Distillation Example:]

I have used mutliple datasets from huggingface got same error on all,

my current dataset is in format:

data.tar => shard00001.tar ......... shard00030.tar

each shard has three files, for example shard00001.tar has:

file_1.jpg
file_1.json
file_1.txt

I am not sure if the issue is caused by dataset or something else

Logs

root@55a014677aad:/workspace/diffusers/examples/consistency_distillation# accelerate launch train_lcm_distill_lora_sdxl_wds.py --pretrained_teacher_model=$MODEL_DIR --pretrained_vae_model_name_or_path=madebyollin/sdxl-vae-fp16-fix --mixed_precision=fp16 --resolution=512 --lora_rank=64 --learning_rate=1e-6 --loss_type="huber" --use_fix_crop_and_size --adam_weight_decay=0.0 --max_train_steps=1000 --max_train_samples=4000 --dataloader_num_workers=1 --train_shards_path_or_url='/workspace/output.tar' --validation_steps=200 --checkpointing_steps=200 --checkpoints_total_limit=10 --train_batch_size=1 --gradient_checkpointing --enable_xformers_memory_efficient_attention --gradient_accumulation_steps=1 --use_8bit_adam --resume_from_checkpoint=latest --report_to=wandb --seed=453645634 --push_to_hub
/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py:384: UserWarning: `log_with=wandb` was passed but no supported trackers are currently installed.
  warnings.warn(f"`log_with={log_with}` was passed but no supported trackers are currently installed.")
11/10/2023 12:16:43 - INFO - __main__ - Distributed environment: NO
Num processes: 1
Process index: 0
Local process index: 0
Device: cuda

Mixed precision type: fp16

{'clip_sample_range', 'dynamic_thresholding_ratio', 'variance_type', 'thresholding'} was not found in config. Values will be initialized to default values.
/usr/local/lib/python3.10/dist-packages/transformers/configuration_utils.py:486: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
  warnings.warn(
You are using a model of type clip_text_model to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
You are using a model of type clip_text_model to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
{'attention_type', 'dropout', 'reverse_transformer_layers_per_block'} was not found in config. Values will be initialized to default values.
{'attention_type', 'dropout', 'reverse_transformer_layers_per_block'} was not found in config. Values will be initialized to default values.
11/10/2023 12:16:58 - INFO - __main__ - ***** Running training *****
11/10/2023 12:16:58 - INFO - __main__ -   Num batches each epoch = 4000
11/10/2023 12:16:58 - INFO - __main__ -   Num Epochs = 1
11/10/2023 12:16:58 - INFO - __main__ -   Instantaneous batch size per device = 1
11/10/2023 12:16:58 - INFO - __main__ -   Total train batch size (w. parallel, distributed & accumulation) = 1
11/10/2023 12:16:58 - INFO - __main__ -   Gradient Accumulation steps = 1
11/10/2023 12:16:58 - INFO - __main__ -   Total optimization steps = 1000
Checkpoint 'latest' does not exist. Starting a new training run.
Steps:   0%|                                                                                 | 0/1000 [00:00<?, ?it/s]

System Info

On Rupod Cloud GPU A5000 24GB:

runpod/pytorch:2.1.0-py3.10-cuda11.8.0-devel-ubuntu22.04

root@55a014677aad:/workspace# diffusers-cli env

diffusers-cli env:

- `diffusers` version: 0.23.0.dev0
- Platform: Linux-5.15.0-84-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- PyTorch version (GPU?): 2.1.0+cu118 (True)
- Huggingface_hub version: 0.17.3
- Transformers version: 4.35.0
- Accelerate version: 0.24.1
- xFormers version: 0.0.22.post7+cu118
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

Who can help?

@sayakpaul @patil-suraj @pcuenca @yiyixuxu

[huan085128]

The program is stuck in this line of code, how to solve this problem?

for step, batch in enumerate(train_dataloader):

[rdcoder33]

The program is stuck in this line of code, how to solve this problem?

for step, batch in enumerate(train_dataloader):

Can you run this in your code:

uname -r

To check if kernel is 5.4 or 5.5.0 ?

I just found this in the log, which might be the issue:

Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.

[rdcoder33]

uname -r

Tested on 5.15 kernel it still hangs / stucks at 0% training.

Since, it stops here :

for step, batch in enumerate(train_dataloader):

It must be a dataset issue.

I tired passing dataset in following ways:

 --train_shards_path_or_url='pipe:cat ./output/*.tar' 

--train_shards_path_or_url="pipe:./output/*.tar"

--train_shards_path_or_url="output.tar"

--train_shards_path_or_url="output"

--train_shards_path_or_url="shard00001.tar"

None seems to work. Also the aws dataset in example is not accessible to public i think.

Guys, can you please tell us the right way to pass a web dataset to LCM lora training?

@sayakpaul @patil-suraj @pcuenca @yiyixuxu

[kilimchoi]

Experiencing the same thing. Stuck at 0 when I just put down the url to the tar file. @sayakpaul @patil-suraj

[huan085128]

The program is stuck in this line of code, how to solve this problem?

for step, batch in enumerate(train_dataloader):

Can you run this in your code:

uname -r

To check if kernel is 5.4 or 5.5.0 ?

I just found this in the log, which might be the issue:

Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.

I try to repair it in https://github.com/huan085128/lcm_lora. Welcome to try it.

[rdcoder33]

The program is stuck in this line of code, how to solve this problem?

for step, batch in enumerate(train_dataloader):

Can you run this in your code:
uname -r
To check if kernel is 5.4 or 5.5.0 ?
I just found this in the log, which might be the issue:
Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.

I try to repair it in https://github.com/huan085128/lcm_lora. Welcome to try it.

Thanks for the help. Really appreciated.

Training does get started with your code but it gives this error now:

ValueError("didn't find ['json'] in ['__key__', '__url__', 'jpg']")

I have provided more details in:

huan085128/lcm_lora#1

[Balladie]

In my case I was giving a directory path for --train_shards_path_or_url. Worked by changing it from dataset_dir to dataset_dir/{00000..00063}.tar.

[patrickvonplaten]

@patil-suraj

[ShashwatNigam99]

For me, if I use a single gpu setup, by calling python file.py - it works fine. When using accelerate for a multi-gpu setup, it hangs at the exact step as everyone has been talking about.

[pcuenca]

A reason for this problem could be that the training scripts expect certain keys to be present in the json file inside the webdataset tar files. If your dataset does not include those keys (or a json file for each entry), then all entries will be rejected and the loop will run through completion without selecting any image/text pairs.

Could you please try the scripts in PR #5908 to check if they work for you? If you do, please make sure to adjust these properties according to your dataset characteristics.