During multi-gpus training, each card will execute the cache text embedding operation once?

[pixeli99]

text_encoders = [text_encoder_one, text_encoder_two]
tokenizers = [tokenizer_one, tokenizer_two]
train_dataset = get_train_dataset(args, accelerator)
compute_embeddings_fn = functools.partial(
 compute_embeddings,
 text_encoders=text_encoders,
 tokenizers=tokenizers,
 proportion_empty_prompts=args.proportion_empty_prompts,
)
with accelerator.main_process_first():
 train_dataset = train_dataset.map(compute_embeddings_fn, batched=True)

Each card will execute the above code during training, which will occupy too much disk space. Is this unreasonable? Or is it possible that I have misunderstood because currently, a fill50k training requires 15*8GB of storage space when using 8 cards.

[patrickvonplaten]

cc @williamberman

[williamberman]

Sorry, I'm not sure which training script this code is in? @pixeli99

[pixeli99]

Sorry, I'm not sure which training script this code is in? @pixeli99

hi, @williamberman
Sorry for my late reply. You can take a look here

diffusers/examples/controlnet/train_controlnet_sdxl.py

Line 995 in 3eb498e

text_encoders = [text_encoder_one, text_encoder_two]

My question is whether this code can be selectively executed. If I have to cache these embeddings, it will consume approximately 1TB of my hard drive space, and my dataset size is 500K.

Whether this consumption is normal, and I find that it seems that every GPU will execute this cache, which is very strange. Of course, it may be that I understand it incorrectly

[williamberman]

Ah yes understood, unfortunately I as well am not really familiar with the semantics here of what datasets and accelerate are actually doing. See some of my questions here #4038 (comment)

@sayakpaul probably understands better

[pixeli99]

Yes, it seems that every card will perform cache operations. Did I misunderstand?

[sayakpaul]

There's an answer here by @lhoestq: #4038 (comment)

Also ccing @muellerzr for additional insights in case.

[williamberman]

Ah yes, super helpful @pixeli99 ! So @sayakpaul I believe if you follow up on my comment here #4038 (comment) it should fix this. It looks like it's re doing the same computation on all processes and re-caching the results. Manually telling datasets a hash key will fix this :)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.