Skip to content

fix: keep Adam moments in FP32 for BF16 parameters - #210

Open
JYMiracle305 wants to merge 1 commit into
masterfrom
fix/adam-fp32-state
Open

fix: keep Adam moments in FP32 for BF16 parameters#210
JYMiracle305 wants to merge 1 commit into
masterfrom
fix/adam-fp32-state

Conversation

@JYMiracle305

@JYMiracle305JYMiracle305 commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

修改背景

Adam 的一阶动量和二阶动量会在训练过程中持续累积。原实现让 mv 跟随模型参数的数据类型,BF16 参数会因此使用 BF16 optimizer state,容易损失较小的梯度变化。

主要修改

  • Adam 一阶动量 m 和二阶动量 v 统一使用 FP32 保存和计算
  • CUDA Adam kernel 分别处理梯度类型、参数类型和 FP32 optimizer state
  • 同时覆盖普通参数列表和 named-parameter 两种 Adam 构造路径
  • 增加 BF16 参数使用 FP32 Adam state 的创建和更新测试

该 PR 位于 checkpoint stack 中,依赖 #195,后续 PR 为 #196

Base automatically changed from feat/checkpoint-consumed-micro-batches to masterAugust 26, 2026 08:14
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@JYMiracle305