Skip to content

Fix PytorchLARS updates when momentum is zero - #2082

Open
mottopanikeiku wants to merge 1 commit into
bitsandbytes-foundation:mainfrom
mottopanikeiku:fix/issue-2079
Open

mottopanikeiku wants to merge 1 commit into
bitsandbytes-foundation:mainfrom
mottopanikeiku:fix/issue-2079

Conversation

@mottopanikeiku

Copy link
Copy Markdown

Fixes #2079.

I use the weight-decayed gradient when PytorchLARS has zero momentum, keeping the existing update-norm limit. This fixes the default step() crash and prevents a zero-momentum group from reusing the previous parameter's momentum update. The native LARS variants are unchanged.

Validation on CPU with Python 3.12 and PyTorch 2.8.0+cpu:

  • OMP_NUM_THREADS=2 python -m pytest tests/test_optim.py -k lars -q: 10 passed, 4 existing bfloat16 skips.
  • Both new regressions fail before the fix.
  • A 16-step training smoke matches PyTorch SGD with mixed momentum groups, weight decay, Nesterov momentum, and closures.
  • pre-commit run --all-files: all hooks passed.

I used OpenAI Codex for investigation, implementation, and testing. This has been manually reviewed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PytorchLARS.step() crashes with UnboundLocalError on its own default momentum=0

1 participant