fix init weights issue for critic/reward model

[jouw]

Add the following code to disable init weights operation, otherwise it will init model weights and got an error. @hwchen2017

with no_init_weights():

Detailed explanation as belows.

Take Qwen3Model as example, the function call stack is:
Qwen3Model.init() -> Qwen3Model.post_init() -> PreTrainedModel.init_weights()

If we don't add with no_init_weights(): for the code model = model_class.from_config(model_config), the parameter _init_weights will be true, and cause error.

https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py

def init_weights(self):
"""
If needed prunes and maybe initializes weights. If using a custom `PreTrainedModel`, you need to implement any
initialization logic in `_init_weights`.
"""
# Prune heads if needed
if self.config.pruned_heads:
self.prune_heads(self.config.pruned_heads)
if _init_weights:
# Initialize weights
self.initialize_weights()
# Tie weights should be skipped when not initializing all weights
# since from_pretrained(...) calls tie weights anyways
self.tie_weights()

[hwchen2017]

Hi @jouw, can you fix the format and DCO error?

[jouw]

Hi @jouw, can you fix the format and DCO error?

hi @hwchen2017 , I have fixed the error, please help review the change, thanks!

[jouw]

Hi @jouw, can you fix the format and DCO error?

hi @hwchen2017 , I have fixed the errors, can you help merge the change? Thanks!

[tohtana]

Hi @jouw,

It seems this breaks our CI test using DS-Chat. Can you share more about the error you encountered?
Can we revert this fix to unblock PRs on DeepSpeed repo if it is supposed to take time to fix?

[PKUWZP]

Let's revert this PR and fix it.