[REQUEST] split zero3 checkpoint files into optim states and master weights

[stas00]

Currently, if one wants to pull out fp32 weights from the zero-3 checkpoint they have to load the full checkpoints that contain 3x of data that the user needs. And for huge models this can take hours.

I have already created a PR to drop the optim states here #4025 since otherwise the user needs 3x cpu memory, but the fix can only discard that unneeded data after loading it, so it's still very slow.

One of the solutions I proposed is to split each of the fp32+state1+state2 shard files into 2 files:

1. fp32
2. state1+state2

that way a user not caring about optim states and just wants to retrieve weights will be able to perform the extraction much much faster.

as I mentioned in the PR, the other option is to switch to https://github.com/huggingface/safetensors/ which allows selective loading of tensors, rather than all-in-one loading.

@tjruwase

[xwjiang2010]

Hey I also ran into the same issue. It seems splitting into separate files is a good optimization. I also wonder if saving to fp16 is enough for both resuming training and downstream inference.