Redirect the liger loss through the distributed wrapper under DDP

[qgallouedec]

What does this PR do?

Fixes #7245.

compute_loss in DPOTrainer and KTOTrainer entered _forward_redirection only for ZeRO-3 and FSDP, and otherwise ran the loss on the unwrapped model. Under plain DDP that means DistributedDataParallel.forward() never runs, so its reducer is never armed by prepare_for_backward(), the autograd hooks return early, and gradients are never all-reduced. Each rank keeps its own.

Measured on 2 GPUs, one step, different data per rank, comparing every parameter's gradient across ranks:

	params differing	max abs diff
use_liger_kernel=True, before	310 / 310	7.5e-02
use_liger_kernel=True, after	0 / 310	0.0
use_liger_kernel=False	0 / 310	0.0

Introduced in #6372, which unwrapped the model to fix ZeRO-3. Present in v1.10.0, v1.11.0 and v1.12.0.

The condition now matches the one compute_ref_log_probs already uses in both files, and that GRPOTrainer uses in #7077.

is_zero3 is kept rather than folded in: under ZeRO-3 the model is a DeepSpeedEngine, so model is not unwrapped_model should already cover it, but deepspeed is xfailed against the transformers version I have, so I left the existing term alone rather than change a path I could not exercise. Worth simplifying separately.

---

Note

Medium Risk
Changes distributed training gradient synchronization for Liger-enabled DPO/KTO; fixes a silent DDP bug but touches core compute_loss paths used on every training step.

Overview
Fixes incorrect multi-GPU training when DPO or KTO use use_liger_kernel=True. compute_loss now routes the Liger loss through _forward_redirection whenever the passed model is a distributed wrapper (model is not unwrapped_model), in addition to ZeRO-3 and FSDP—so DDP’s forward() runs and gradient all-reduce is armed.

Regression tests were added for DPO and KTO that mock a wrapped model and assert compute_loss calls _forward_redirection, matching the pattern already used for reference log-probs.

^{Reviewed by Cursor Bugbot for commit e02eb31. Bugbot is set up for automated code reviews on this repo. Configure here.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-16T22:16:48.153345Z	e02eb31	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

Superseded. Splitting this in two, because the same fix does not work for both trainers.

KTO is #7247, verified: 310/310 parameters differing across ranks before, 0/310 after, and zero3/zero2/ddp all green.

DPO cannot take this change on main. The fused loss runs torch.func.grad_and_value internally, so running it inside DistributedDataParallel.forward() raises NotImplementedError: Cannot access storage of TensorWrapper on every rank. It turns a silent desync into a crash. It works once #7243 moves DPO onto the chunked path, so it goes there.

Tracked in #7245.