Forward the per-head Muon tag at every muon_update call site - #8612
Open
vineethsaivs wants to merge 1 commit into
Open
vineethsaivs wants to merge 1 commit into
vineethsaivs wants to merge 1 commit into
Conversation
Forward `num_heads` at all three. Signed-off-by: Vineeth Sai <vineethsai4444@gmail.com>
vineethsaivs
requested review from
loadams,
tjruwase and
tohtana
as code owners
September 20, 2026 18:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Per-head Muon (#8384) is silently ignored on three of the six paths that call
muon_update, so the attention projections it tags are orthogonalized whole.deepspeed.initializetags each attention projection withmuon_num_headsand the call sites read it back off the parameter. Three do not: the round-robin branch of ZeRO 1/2get_flat_partition(its unpadded twin does),MuonWithAuxAdam.step, and the new BF16_Optimizer Muon support. Which branch ofget_flat_partitionruns depends onround_robin_gradients, so whether the feature takes effect turns on an unrelated setting, and the update stays finite either way. Forwardnum_headsat all three.Test:
TORCHDYNAMO_DISABLE=1 DS_ACCELERATOR=cpu PYTHONPATH=.:tests python -m pytest tests/unit/runtime/zero/test_per_head_muon.py -q. Two new cases fail before and 78 pass after: one stepsMuonWithAuxAdamon a tagged projection and asserts the applied update is the per-head one rather than the whole-matrix one (0.32 apart here), and one names any call site that drops the tag. yapf 0.43.0 and flake8 pass on the changed files. Executed on Apple M2 Pro CPU; no GPU or distributed run.