batch config issue?

[samyam]

There are a few things in configure train batch size that does not seem correct to me, and there are few things that we do not currently support.

1. The following assertion

train_batch_size == train_micro_batch_size_per_gpu * gradient_accumulation_step * world_size

should always hold but currently it does not in some cases.
For example, when train_micro_batch_size_per_gpu and gradient accumulation steps are None in the ds_cofig its initialized to train_batch_size and 1 respectively which leads to

train_batch_size == train_batch_size * 1 * world_size

2. if train_micro_batch_size_per_gpu > per_device_batch_size, we should throw a config error. Currently, its assigned to be equal to per_device_batch_size.

3. We do not currently support the user providing only the train_micro_batch_size or train_micro_batch_size and gradient _accumulation_steps.

[samyam]

Fixed with pull request Samyamr/batchconfig #33

[rafael-ariascalles]

I am having this same issue using 0.9 but not 0.8 (using a was p4 machine)