Skip to content

【MIIT program】support matterchat - #305

Open
learncat163 wants to merge 1 commit into
PaddlePaddle:developfrom
learncat163:pr-matterchat
Open

【MIIT program】support matterchat#305
learncat163 wants to merge 1 commit into
PaddlePaddle:developfrom
learncat163:pr-matterchat

Conversation

@learncat163

@learncat163learncat163 commented Jul 7, 2026•
edited
Loading

Copy link
Copy Markdown

MatterChat diff (PyTorch and PaddlePaddle)

AiStudio在线推理案例

MatterChat 由三个模块串联组成: CHGNet (晶体编码器) → Q-Former (跨模态桥接) → Mistral-7B (大语言模型)。本文档分别验证每个模块从 PyTorch 迁移到 PaddlePaddle 后的数值对齐情况。

1. CHGNet Diff

CHGNet 负责将晶体结构 (CIF) 编码为逐原子 embedding。验证用 GaN.cif (4 原子) 作为输入, 对比 Paddle 输出与 PyTorch 参考的 [N_atom, 64] material embedding。

结果

指标值阈值状态
max_diff8.05e-071e-4PASS
mean_diff1.82e-071e-4PASS

代码样例

PyTorch 参考数据生成

importnumpyasnpimporttorchfrompymatgen.coreimportStructurefromModel.chgnet_lib.model.model_embeddingimportCHGNet# MatterChat 原始 PT 代码device=torch.device("cuda")
chgnet=CHGNet.load().to(device).float().eval()
struct=Structure.from_file("fix_inputs/GaN.cif")
withtorch.no_grad():
atom_feas=chgnet.predict_structure_embedding(struct)
ifisinstance(atom_feas, (list, tuple)):
atom_feas=atom_feas[0]
# 保存为参考数据ref=atom_feas.detach().cpu().float().numpy()
np.save("fix_outputs/raw_stage_01_material_embed.npy", ref)
print(f"shape={ref.shape}") # -> (4, 64)

PaddlePaddle 对比

importnumpyasnpimportpaddlefrompymatgen.coreimportStructurefromppmat.models.matterchat.chgnet.model.model_embeddingimportCHGNet# 加载 PyTorch 参考数据ref_embed=np.load("fix_outputs/raw_stage_01_material_embed.npy").astype(np.float64)
# 构建 CHGNet 并加载权重 (CPU)paddle.set_device("cpu")
chgnet=CHGNet()
chgnet_weights=load_sharded_weights(prefix="material_encoder.")
set_state_dict_filtered(chgnet, chgnet_weights, prefix="material_encoder.")
chgnet.eval()
# GPU 推理paddle.set_device("gpu")
gpu_chgnet=CHGNet()
gpu_chgnet.set_state_dict({k: v.cuda() fork, vinchgnet.state_dict().items()})
gpu_chgnet.eval()
struct=Structure.from_file("fix_inputs/GaN.cif")
withpaddle.no_grad():
atom_feas=gpu_chgnet.predict_structure_embedding(struct)
# 对比diff=np.abs(atom_feas.numpy().astype(np.float64) -ref_embed)
print(f"max_diff={diff.max():.2e} mean_diff={diff.mean():.2e}")
# -> max_diff=8.05e-07 mean_diff=1.82e-07 (PASS, thr=1e-4)

2. Q-Former Diff

Q-Former 是 BLIP-2 风格的 BERT 变体, 用 32 个 query token 对 CHGNet 输出做 cross-attention, 产生 [1, 32, 768] 的查询表示。验证 Q-Former 前向输出与基线一致。

结果

指标值阈值状态
max_diff0.00e+001e-4PASS
mean_diff0.00e+001e-4PASS

Q-Former 输出与基线完全一致 (diff=0)。

代码样例

PyTorch 参考数据生成

importnumpyasnpimporttorchfromModel.material_Q_former_baseimportBertConfig, BertLMHeadModel# MatterChat 原始 PT 代码device=torch.device("cuda")
# 加载预训练 Q-Former 权重full_ckpt=torch.load("model_weight/model_weights.pkl", map_location="cpu", weights_only=False)
full_state=full_ckpt["state_dict"]
qformer_config=BertConfig.from_pretrained("bert-base-uncased")
qformer_config.encoder_width=64qformer_config.add_cross_attention=Trueqformer_config.cross_attention_freq=2qformer_config.query_length=32qformer=BertLMHeadModel.from_pretrained("bert-base-uncased", config=qformer_config)
qformer_state= {}
fork, vinfull_state.items():
ifk.startswith("model.Qformer.") ork.startswith("model.query_tokens"):
qformer_state[k.replace("model.", "", 1)] =vqformer.load_state_dict(qformer_state, strict=False)
qformer=qformer.to(device).float().eval()
query_tokens=full_state["model.query_tokens"].to(device).float()
# 输入: 上一步 CHGNet 输出的 material embeddingmaterial_embed=torch.from_numpy(
np.load("fix_outputs/raw_stage_01_material_embed.npy")
).to(device).float()
material_att=torch.ones((1, material_embed.shape[0]), dtype=torch.long, device=device)
withtorch.no_grad():
query_output=qformer.bert(
query_embeds=query_tokens,
encoder_hidden_states=material_embed.unsqueeze(0),
encoder_attention_mask=material_att,
return_dict=True,
)
qf_out=query_output.last_hidden_state[:, :query_tokens.shape[1], :]
# 保存为参考数据ref=qf_out.detach().cpu().float().numpy()
np.save("fix_outputs/raw_stage_02_qformer_out.npy", ref)
print(f"shape={ref.shape}") # -> (1, 32, 768)

PaddlePaddle 对比

importnumpyasnpimportpaddlefromppmat.models.matterchat.q_former.q_former_baseimportBertConfig, BertLMHeadModel# 加载基线数据material_embed=np.load("fix_outputs/raw_stage_01_material_embed.npy")
qt_param=np.load("fix_outputs/paddle_stage_02_query_tokens.npy")
# 构建 Q-Former (CPU)paddle.set_device("cpu")
config=BertConfig()
config.encoder_width=64config.add_cross_attention=Trueconfig.cross_attention_freq=2config.query_length=32qformer=BertLMHeadModel(config)
qformer.cls=Noneqformer.bert.embeddings.word_embeddings=Noneqformer.bert.embeddings.position_embeddings=Noneforlayerinqformer.bert.encoder.layer:
layer.output=Nonelayer.intermediate=Noneqformer_weights=load_sharded_weights(prefix="Qformer.")
set_state_dict_filtered(qformer, qformer_weights, prefix="Qformer.")
qformer.eval()
# GPU 推理paddle.set_device("gpu")
gpu_qformer=BertLMHeadModel(config)
gpu_qformer.set_state_dict({k: v.cuda() fork, vinqformer.state_dict().items()})
gpu_qformer.eval()
qt_gpu=paddle.to_tensor(qt_param).cast(paddle.float32).cuda()
emb_gpu=paddle.to_tensor(material_embed).cuda().unsqueeze(0)
att_gpu=paddle.ones([1, emb_gpu.shape[1]], dtype=paddle.int64).cuda()
withpaddle.no_grad():
query_output=gpu_qformer.bert(
query_embeds=qt_gpu,
encoder_hidden_states=emb_gpu,
encoder_attention_mask=att_gpu,
return_dict=True,
)
qf_out=query_output.last_hidden_state[:, :qt_gpu.shape[1], :]
print(f"max_diff={np.abs(qf_out.numpy().astype(np.float64) -ref).max():.2e}")
# -> max_diff=0.00e+00 (PASS, thr=1e-4)

3. Mistral LLM Diff

Mistral-7B 是 32 层 decoder-only Transformer, 含 RoPE、GQA、SwiGLU。由于模型规模大 (7.3B 参数), float16 跨框架累积误差不可避免, 因此阈值放宽至 1e-3。验证逐层 hidden state、最终 hidden_last 及 Top-20 token 匹配。

结果汇总

测试项通过阈值max_diff状态
32 层 hidden state32/321e-32.97e-04 (layer_31)PASS
hidden_last1/11e-31.12e-04PASS
Top-20 token20/20——PASS

逐层误差

层max_diffmean_diff状态
layer_005.14e-077.92e-09PASS
layer_011.16e-041.06e-07PASS
layer_051.22e-041.70e-07PASS
layer_101.22e-042.25e-07PASS
layer_151.22e-043.15e-07PASS
layer_201.22e-045.78e-07PASS
layer_251.22e-048.60e-07PASS
layer_301.06e-041.51e-06PASS
layer_312.97e-041.99e-06PASS
hidden_last1.12e-041.78e-05PASS

代码样例

PyTorch 参考数据生成

importnumpyasnpimporttorchfromtransformersimportLlamaTokenizerfromtransformers.models.mistral.modeling_mistralimportMistralForCausalLMfromtransformersimportMistralConfigasHFMistralConfigdevice=torch.device("cuda")
MODEL_WEIGHT_DIR="model_weight/Mistral-7B-Instruct-v0.3"# Tokenizetokenizer=LlamaTokenizer.from_pretrained(MODEL_WEIGHT_DIR, use_fast=False)
tokenizer.add_special_tokens({"pad_token": "[PAD]", "bos_token": "<s>",
"eos_token": "</s>", "unk_token": "<unk>"})
prompt="[INST] What is the chemical formula of this material? [/INST]"tokens=tokenizer(prompt, return_tensors="pt", truncation=True, max_length=64)
input_ids=tokens["input_ids"].to(device)
# 加载 Mistral-7Bhf_config=HFMistralConfig.from_pretrained(MODEL_WEIGHT_DIR)
hf_config._attn_implementation="eager"llm=MistralForCausalLM.from_pretrained(MODEL_WEIGHT_DIR, config=hf_config,
torch_dtype=torch.float32).to(device).eval()
llm.resize_token_embeddings(len(tokenizer))
# input embeddingswithtorch.no_grad():
inputs_embeds=llm.model.embed_tokens(input_ids)
np.save("fix_outputs/raw_stage_05_input_embeds.npy",
inputs_embeds.detach().cpu().float().numpy())
# 构建因果掩码seq_len=input_ids.shape[1]
min_dtype=torch.finfo(inputs_embeds.dtype).mincausal_mask=torch.full((seq_len, seq_len), min_dtype, dtype=inputs_embeds.dtype, device=device)
causal_mask=torch.triu(causal_mask, diagonal=1)[None, None, :, :].expand(1, 1, -1, -1)
# 逐层前向, 保存每层 hidden state 作为参考hidden_states=inputs_embedsposition_ids=torch.arange(seq_len, device=device).unsqueeze(0)
forliinrange(32):
withtorch.no_grad():
hidden_states=llm.model.layers[li](
hidden_states, attention_mask=causal_mask,
position_ids=position_ids, use_cache=True,
)[0]
np.save(f"fix_outputs/raw_stage_06_hidden_layer{li:02d}.npy",
hidden_states.detach().cpu().float().numpy())
# 最终 norm + hidden_lastwithtorch.no_grad():
hidden_final=llm.model.norm(hidden_states)
np.save("fix_outputs/raw_stage_06_hidden_last.npy",
hidden_final[0, -1].detach().cpu().float().numpy())

PaddlePaddle 对比

importjsonimportnumpyasnpimportpaddlefromppmat.models.matterchat.mistral.configuration_mistralimportMistralConfigfromppmat.models.matterchat.mistral.modeling_mistralimport (
MistralForCausalLM,
MistralDecoderLayer,
MistralRMSNorm,
)
# 加载 PyTorch 参考数据 (逐层 hidden state)refs= {}
forliinrange(32):
refs[f"stage_06_hidden_layer{li:02d}"] =np.load(
f"fix_outputs/raw_stage_06_hidden_layer{li:02d}.npy"
).astype(np.float64)
refs["stage_05_input_embeds"] =np.load(
"fix_outputs/raw_stage_05_input_embeds.npy"
).astype(np.float64)
emb=refs["stage_05_input_embeds"].astype(np.float32)
seq_len=emb.shape[1]
# 构建因果掩码 + position idsmin_dt=float(paddle.finfo(paddle.float32).min)
causal_mask=paddle.triu(
paddle.full([seq_len, seq_len], min_dt, dtype="float32"), diagonal=1
)[None, None].expand([1, 1, -1, -1])
pos_ids=paddle.arange(seq_len).unsqueeze(0)
cache_pos=paddle.arange(seq_len)
# CPU 加载完整 Mistral-7B, 提取每层权重paddle.set_device("cpu")
llm=MistralForCausalLM(MistralConfig(vocab_size=32769))
load_sharded_weights(llm, weight_dir=WEIGHT_DIR)
llm.eval()
layer_sds= [
{k: vfork, vinllm.model.layers[i].state_dict().items()}
foriinrange(32)
]
# 逐层 GPU 推理 + 对比paddle.set_device("gpu")
hidden=paddle.to_tensor(emb).cuda()
cm=causal_mask.cuda()
pos_ids=pos_ids.cuda()
cache_pos=cache_pos.cuda()
forliinrange(32):
gpu_layer=MistralDecoderLayer(llm.config, li)
gpu_layer.set_state_dict({k: v.cuda() fork, vinlayer_sds[li].items()})
gpu_layer.eval()
withpaddle.no_grad():
hidden=gpu_layer(
hidden, attention_mask=cm,
position_ids=pos_ids, use_cache=False,
cache_position=cache_pos,
)[0]
ref=refs[f"stage_06_hidden_layer{li:02d}"]
diff=np.abs(hidden.cpu().numpy().astype(np.float64) -ref)
status="PASS"ifdiff.max() <1e-3else"FAIL"print(f"layer_{li:02d}: max={diff.max():.2e} mean={diff.mean():.2e} [{status}]")
# -> layer_00: max=5.14e-07 mean=7.92e-09 [PASS]# -> layer_01: max=1.16e-04 mean=1.06e-07 [PASS]# -> ...# -> layer_31: max=2.97e-04 mean=1.99e-06 [PASS]

@learncat163

learncat163 commented Jul 7, 2026

Copy link
Copy Markdown
Author

MatterChat 推理对话示例

1. 单次推理

1.1 一行命令推理

fromppmat.modelsimportbuild_model_from_namefrompymatgen.io.cifimportCifParsermodel, _=build_model_from_name('matterchat_full')
model.to('gpu').eval()
struct=CifParser('Si.cif').get_structures()[0]
print(model.chat(struct, 'what is the chemical formula of this material?'))
# -> The chemical formula of this material is Si.

1.2 指定 config + 本地权重

fromomegaconfimportOmegaConffrompymatgen.io.cifimportCifParserfromppmat.modelsimportbuild_modelfromppmat.utilsimportsave_loadconfig=OmegaConf.to_container(OmegaConf.load('structure_generation/configs/matterchat/matterchat_full.yaml'), resolve=True)
model=build_model(config['Model'])
save_load.load_pretrain(model, './matterchat_full/')
model.to('gpu').eval()
struct=CifParser('Si.cif').get_structures()[0]
print(model.chat(struct, 'what is the chemical formula of this material?'))

2. 多轮对话示例

对同一晶体连续提问多个问题:

fromppmat.modelsimportbuild_model_from_namefrompymatgen.io.cifimportCifParsermodel, _=build_model_from_name("matterchat_full")
model.to("gpu").eval()
struct=CifParser("GaN.cif").get_structures()[0]
# 4 个标准问题prompts= [
"what is the chemical formula of this material?",
"what is the space group of this material?",
"Is this material stable or not?",
"What is the bandgap of this material?",
]
forpromptinprompts:
answer=model.chat(struct, prompt, max_new_tokens=64)
print(f"Q: {prompt}")
print(f"A: {answer}")
print()

输出:

Q: what is the chemical formula of this material?
A: The chemical formula of this material is GaN.
Q: what is the space group of this material?
A: The space group of this material is P6_3mc.
Q: Is this material stable or not?
A: This material is not stable.
Q: What is the bandgap of this material?
A: The bandgap of this material is 1.68300.

3. 批量推理 (多个 CIF)

importosfromppmat.modelsimportbuild_model_from_namefrompymatgen.io.cifimportCifParsermodel, _=build_model_from_name("matterchat_full")
model.to("gpu").eval()
cif_dir="path/to/cif_files"prompt="what is the chemical formula of this material?"forfnameinsorted(os.listdir(cif_dir)):
ifnotfname.endswith(".cif"):
continuestruct=CifParser(os.path.join(cif_dir, fname)).get_structures()[0]
answer=model.chat(struct, prompt, max_new_tokens=64)
print(f"{fname}: {answer}")

@leeleolayleeleolay changed the title support matterchat【MIIT program】support matterchatJul 7, 2026
@paddle-bot

paddle-botBot commented Jul 14, 2026

Copy link
Copy Markdown

Thanks for your contribution!

@paddle-botpaddle-botBot added the contributor External developers label Jul 14, 2026
leeleolay
leeleolay reviewed Jul 16, 2026
View reviewed changes

@leeleolayleeleolay left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

辛苦修改整体的代码规范符合套件风格

Comment threadppmat/datasets/__init__.py
from ppmat.datasets.oc20_s2ef_dataset import OC20S2EFDataset # noqa
from ppmat.datasets.qm9_dataset import QM9Dataset # noqa
from ppmat.datasets.omol25_dataset import OMol25Dataset
from ppmat.models.matterchat.trainer import MTDataset # noqa

@leeleolayleeleolayJul 16, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用已有trainer

Comment threadppmat/datasets/collate_fn.py

@leeleolayleeleolayJul 16, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用默认的collator

Comment threadppmat/models/matterchat/chgnet/graph/__init__.py

@leeleolayleeleolayJul 16, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

套件内已有chgnet，辛苦使用

Comment threadppmat/models/matterchat/chgnet/graph/converter.py

@leeleolayleeleolayJul 16, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用已有的graph_converter

Comment threadppmat/models/matterchat/chgnet/model/dynamics.py

@leeleolayleeleolayJul 16, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ppmatSim已经支持相关的功能，复用已有的

Comment threadppmat/models/matterchat/chgnet/utils/vasp_utils.py

@leeleolayleeleolayJul 16, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vasp在这个模型里的作用是？

Comment threadppmat/models/matterchat/mistral/configuration_mistral.py

@leeleolayleeleolayJul 16, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

不符合已有规范，config不这么处理

@leeleolay

leeleolay commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

@learncat163 重构了部分基础组件，移动了推理器的位置，辛苦基于新的开发和尝试

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@leeleolayleeleolayleeleolay left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

contributorExternal developersMIIT Program

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@learncat163@leeleolay
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
【MIIT program】support matterchat by learncat163 · Pull Request #305 · PaddlePaddle/PaddleMaterials · GitHub
Skip to content
【MIIT program】support matterchat - #305
Open
learncat163 wants to merge 1 commit into
PaddlePaddle:developfrom 
learncat163:pr-matterchat
Open
【MIIT program】support matterchat#305
learncat163 wants to merge 1 commit into
PaddlePaddle:developfrom 
learncat163:pr-matterchat
Conversation
@learncat163
@learncat163learncat163
commented
Jul 7, 2026•

edited

Loading

Copy link


Copy Markdown

MatterChat diff (PyTorch and PaddlePaddle)
AiStudio在线推理案例
MatterChat 由三个模块串联组成: CHGNet (晶体编码器) → Q-Former (跨模态桥接) → Mistral-7B (大语言模型)。本文档分别验证每个模块从 PyTorch 迁移到 PaddlePaddle 后的数值对齐情况。
1. CHGNet Diff
CHGNet 负责将晶体结构 (CIF) 编码为逐原子 embedding。验证用 GaN.cif (4 原子) 作为输入, 对比 Paddle 输出与 PyTorch 参考的 [N_atom, 64] material embedding。
结果
指标值阈值状态
max_diff8.05e-071e-4PASS
mean_diff1.82e-071e-4PASS
代码样例
PyTorch 参考数据生成
importnumpyasnpimporttorchfrompymatgen.coreimportStructurefromModel.chgnet_lib.model.model_embeddingimportCHGNet# MatterChat 原始 PT 代码device=torch.device("cuda")
chgnet=CHGNet.load().to(device).float().eval()
struct=Structure.from_file("fix_inputs/GaN.cif")
withtorch.no_grad():
atom_feas=chgnet.predict_structure_embedding(struct)
ifisinstance(atom_feas, (list, tuple)):
atom_feas=atom_feas[0]
# 保存为参考数据ref=atom_feas.detach().cpu().float().numpy()
np.save("fix_outputs/raw_stage_01_material_embed.npy", ref)
print(f"shape={ref.shape}") # -> (4, 64)
PaddlePaddle 对比
importnumpyasnpimportpaddlefrompymatgen.coreimportStructurefromppmat.models.matterchat.chgnet.model.model_embeddingimportCHGNet# 加载 PyTorch 参考数据ref_embed=np.load("fix_outputs/raw_stage_01_material_embed.npy").astype(np.float64)
# 构建 CHGNet 并加载权重 (CPU)paddle.set_device("cpu")
chgnet=CHGNet()
chgnet_weights=load_sharded_weights(prefix="material_encoder.")
set_state_dict_filtered(chgnet, chgnet_weights, prefix="material_encoder.")
chgnet.eval()
# GPU 推理paddle.set_device("gpu")
gpu_chgnet=CHGNet()
gpu_chgnet.set_state_dict({k: v.cuda() fork, vinchgnet.state_dict().items()})
gpu_chgnet.eval()
struct=Structure.from_file("fix_inputs/GaN.cif")
withpaddle.no_grad():
atom_feas=gpu_chgnet.predict_structure_embedding(struct)
# 对比diff=np.abs(atom_feas.numpy().astype(np.float64) -ref_embed)
print(f"max_diff={diff.max():.2e} mean_diff={diff.mean():.2e}")
# -> max_diff=8.05e-07 mean_diff=1.82e-07 (PASS, thr=1e-4)
2. Q-Former Diff
Q-Former 是 BLIP-2 风格的 BERT 变体, 用 32 个 query token 对 CHGNet 输出做 cross-attention, 产生 [1, 32, 768] 的查询表示。验证 Q-Former 前向输出与基线一致。
结果
指标值阈值状态
max_diff0.00e+001e-4PASS
mean_diff0.00e+001e-4PASS
Q-Former 输出与基线完全一致 (diff=0)。
代码样例
PyTorch 参考数据生成
importnumpyasnpimporttorchfromModel.material_Q_former_baseimportBertConfig, BertLMHeadModel# MatterChat 原始 PT 代码device=torch.device("cuda")
# 加载预训练 Q-Former 权重full_ckpt=torch.load("model_weight/model_weights.pkl", map_location="cpu", weights_only=False)
full_state=full_ckpt["state_dict"]
qformer_config=BertConfig.from_pretrained("bert-base-uncased")
qformer_config.encoder_width=64qformer_config.add_cross_attention=Trueqformer_config.cross_attention_freq=2qformer_config.query_length=32qformer=BertLMHeadModel.from_pretrained("bert-base-uncased", config=qformer_config)
qformer_state= {}
fork, vinfull_state.items():
ifk.startswith("model.Qformer.") ork.startswith("model.query_tokens"):
qformer_state[k.replace("model.", "", 1)] =vqformer.load_state_dict(qformer_state, strict=False)
qformer=qformer.to(device).float().eval()
query_tokens=full_state["model.query_tokens"].to(device).float()
# 输入: 上一步 CHGNet 输出的 material embeddingmaterial_embed=torch.from_numpy(
np.load("fix_outputs/raw_stage_01_material_embed.npy")
).to(device).float()
material_att=torch.ones((1, material_embed.shape[0]), dtype=torch.long, device=device)
withtorch.no_grad():
query_output=qformer.bert(
query_embeds=query_tokens,
encoder_hidden_states=material_embed.unsqueeze(0),
encoder_attention_mask=material_att,
return_dict=True,
)
qf_out=query_output.last_hidden_state[:, :query_tokens.shape[1], :]
# 保存为参考数据ref=qf_out.detach().cpu().float().numpy()
np.save("fix_outputs/raw_stage_02_qformer_out.npy", ref)
print(f"shape={ref.shape}") # -> (1, 32, 768)
PaddlePaddle 对比
importnumpyasnpimportpaddlefromppmat.models.matterchat.q_former.q_former_baseimportBertConfig, BertLMHeadModel# 加载基线数据material_embed=np.load("fix_outputs/raw_stage_01_material_embed.npy")
qt_param=np.load("fix_outputs/paddle_stage_02_query_tokens.npy")
# 构建 Q-Former (CPU)paddle.set_device("cpu")
config=BertConfig()
config.encoder_width=64config.add_cross_attention=Trueconfig.cross_attention_freq=2config.query_length=32qformer=BertLMHeadModel(config)
qformer.cls=Noneqformer.bert.embeddings.word_embeddings=Noneqformer.bert.embeddings.position_embeddings=Noneforlayerinqformer.bert.encoder.layer:
layer.output=Nonelayer.intermediate=Noneqformer_weights=load_sharded_weights(prefix="Qformer.")
set_state_dict_filtered(qformer, qformer_weights, prefix="Qformer.")
qformer.eval()
# GPU 推理paddle.set_device("gpu")
gpu_qformer=BertLMHeadModel(config)
gpu_qformer.set_state_dict({k: v.cuda() fork, vinqformer.state_dict().items()})
gpu_qformer.eval()
qt_gpu=paddle.to_tensor(qt_param).cast(paddle.float32).cuda()
emb_gpu=paddle.to_tensor(material_embed).cuda().unsqueeze(0)
att_gpu=paddle.ones([1, emb_gpu.shape[1]], dtype=paddle.int64).cuda()
withpaddle.no_grad():
query_output=gpu_qformer.bert(
query_embeds=qt_gpu,
encoder_hidden_states=emb_gpu,
encoder_attention_mask=att_gpu,
return_dict=True,
)
qf_out=query_output.last_hidden_state[:, :qt_gpu.shape[1], :]
print(f"max_diff={np.abs(qf_out.numpy().astype(np.float64) -ref).max():.2e}")
# -> max_diff=0.00e+00 (PASS, thr=1e-4)
3. Mistral LLM Diff
Mistral-7B 是 32 层 decoder-only Transformer, 含 RoPE、GQA、SwiGLU。由于模型规模大 (7.3B 参数), float16 跨框架累积误差不可避免, 因此阈值放宽至 1e-3。验证逐层 hidden state、最终 hidden_last 及 Top-20 token 匹配。
结果汇总
测试项通过阈值max_diff状态
32 层 hidden state32/321e-32.97e-04 (layer_31)PASS
hidden_last1/11e-31.12e-04PASS
Top-20 token20/20——PASS
逐层误差
层max_diffmean_diff状态
layer_005.14e-077.92e-09PASS
layer_011.16e-041.06e-07PASS
layer_051.22e-041.70e-07PASS
layer_101.22e-042.25e-07PASS
layer_151.22e-043.15e-07PASS
layer_201.22e-045.78e-07PASS
layer_251.22e-048.60e-07PASS
layer_301.06e-041.51e-06PASS
layer_312.97e-041.99e-06PASS
hidden_last1.12e-041.78e-05PASS
代码样例
PyTorch 参考数据生成
importnumpyasnpimporttorchfromtransformersimportLlamaTokenizerfromtransformers.models.mistral.modeling_mistralimportMistralForCausalLMfromtransformersimportMistralConfigasHFMistralConfigdevice=torch.device("cuda")
MODEL_WEIGHT_DIR="model_weight/Mistral-7B-Instruct-v0.3"# Tokenizetokenizer=LlamaTokenizer.from_pretrained(MODEL_WEIGHT_DIR, use_fast=False)
tokenizer.add_special_tokens({"pad_token": "[PAD]", "bos_token": "<s>",
"eos_token": "</s>", "unk_token": "<unk>"})
prompt="[INST] What is the chemical formula of this material? [/INST]"tokens=tokenizer(prompt, return_tensors="pt", truncation=True, max_length=64)
input_ids=tokens["input_ids"].to(device)
# 加载 Mistral-7Bhf_config=HFMistralConfig.from_pretrained(MODEL_WEIGHT_DIR)
hf_config._attn_implementation="eager"llm=MistralForCausalLM.from_pretrained(MODEL_WEIGHT_DIR, config=hf_config,
torch_dtype=torch.float32).to(device).eval()
llm.resize_token_embeddings(len(tokenizer))
# input embeddingswithtorch.no_grad():
inputs_embeds=llm.model.embed_tokens(input_ids)
np.save("fix_outputs/raw_stage_05_input_embeds.npy",
inputs_embeds.detach().cpu().float().numpy())
# 构建因果掩码seq_len=input_ids.shape[1]
min_dtype=torch.finfo(inputs_embeds.dtype).mincausal_mask=torch.full((seq_len, seq_len), min_dtype, dtype=inputs_embeds.dtype, device=device)
causal_mask=torch.triu(causal_mask, diagonal=1)[None, None, :, :].expand(1, 1, -1, -1)
# 逐层前向, 保存每层 hidden state 作为参考hidden_states=inputs_embedsposition_ids=torch.arange(seq_len, device=device).unsqueeze(0)
forliinrange(32):
withtorch.no_grad():
hidden_states=llm.model.layers[li](
hidden_states, attention_mask=causal_mask,
position_ids=position_ids, use_cache=True,
)[0]
np.save(f"fix_outputs/raw_stage_06_hidden_layer{li:02d}.npy",
hidden_states.detach().cpu().float().numpy())
# 最终 norm + hidden_lastwithtorch.no_grad():
hidden_final=llm.model.norm(hidden_states)
np.save("fix_outputs/raw_stage_06_hidden_last.npy",
hidden_final[0, -1].detach().cpu().float().numpy())
PaddlePaddle 对比
importjsonimportnumpyasnpimportpaddlefromppmat.models.matterchat.mistral.configuration_mistralimportMistralConfigfromppmat.models.matterchat.mistral.modeling_mistralimport (
MistralForCausalLM,
MistralDecoderLayer,
MistralRMSNorm,
)
# 加载 PyTorch 参考数据 (逐层 hidden state)refs= {}
forliinrange(32):
refs[f"stage_06_hidden_layer{li:02d}"] =np.load(
f"fix_outputs/raw_stage_06_hidden_layer{li:02d}.npy"
).astype(np.float64)
refs["stage_05_input_embeds"] =np.load(
"fix_outputs/raw_stage_05_input_embeds.npy"
).astype(np.float64)
emb=refs["stage_05_input_embeds"].astype(np.float32)
seq_len=emb.shape[1]
# 构建因果掩码 + position idsmin_dt=float(paddle.finfo(paddle.float32).min)
causal_mask=paddle.triu(
paddle.full([seq_len, seq_len], min_dt, dtype="float32"), diagonal=1
)[None, None].expand([1, 1, -1, -1])
pos_ids=paddle.arange(seq_len).unsqueeze(0)
cache_pos=paddle.arange(seq_len)
# CPU 加载完整 Mistral-7B, 提取每层权重paddle.set_device("cpu")
llm=MistralForCausalLM(MistralConfig(vocab_size=32769))
load_sharded_weights(llm, weight_dir=WEIGHT_DIR)
llm.eval()
layer_sds= [
{k: vfork, vinllm.model.layers[i].state_dict().items()}
foriinrange(32)
]
# 逐层 GPU 推理 + 对比paddle.set_device("gpu")
hidden=paddle.to_tensor(emb).cuda()
cm=causal_mask.cuda()
pos_ids=pos_ids.cuda()
cache_pos=cache_pos.cuda()
forliinrange(32):
gpu_layer=MistralDecoderLayer(llm.config, li)
gpu_layer.set_state_dict({k: v.cuda() fork, vinlayer_sds[li].items()})
gpu_layer.eval()
withpaddle.no_grad():
hidden=gpu_layer(
hidden, attention_mask=cm,
position_ids=pos_ids, use_cache=False,
cache_position=cache_pos,
)[0]
ref=refs[f"stage_06_hidden_layer{li:02d}"]
diff=np.abs(hidden.cpu().numpy().astype(np.float64) -ref)
status="PASS"ifdiff.max() <1e-3else"FAIL"print(f"layer_{li:02d}: max={diff.max():.2e} mean={diff.mean():.2e} [{status}]")
# -> layer_00: max=5.14e-07 mean=7.92e-09 [PASS]# -> layer_01: max=1.16e-04 mean=1.06e-07 [PASS]# -> ...# -> layer_31: max=2.97e-04 mean=1.99e-06 [PASS]
@learncat163
learncat163
commented
Jul 7, 2026

Copy link


Copy Markdown

Author
MatterChat 推理对话示例
1. 单次推理
1.1 一行命令推理
fromppmat.modelsimportbuild_model_from_namefrompymatgen.io.cifimportCifParsermodel, _=build_model_from_name('matterchat_full')
model.to('gpu').eval()
struct=CifParser('Si.cif').get_structures()[0]
print(model.chat(struct, 'what is the chemical formula of this material?'))
# -> The chemical formula of this material is Si.
1.2 指定 config + 本地权重
fromomegaconfimportOmegaConffrompymatgen.io.cifimportCifParserfromppmat.modelsimportbuild_modelfromppmat.utilsimportsave_loadconfig=OmegaConf.to_container(OmegaConf.load('structure_generation/configs/matterchat/matterchat_full.yaml'), resolve=True)
model=build_model(config['Model'])
save_load.load_pretrain(model, './matterchat_full/')
model.to('gpu').eval()
struct=CifParser('Si.cif').get_structures()[0]
print(model.chat(struct, 'what is the chemical formula of this material?'))
2. 多轮对话示例
对同一晶体连续提问多个问题:
fromppmat.modelsimportbuild_model_from_namefrompymatgen.io.cifimportCifParsermodel, _=build_model_from_name("matterchat_full")
model.to("gpu").eval()
struct=CifParser("GaN.cif").get_structures()[0]
# 4 个标准问题prompts= [
"what is the chemical formula of this material?",
"what is the space group of this material?",
"Is this material stable or not?",
"What is the bandgap of this material?",
]
forpromptinprompts:
answer=model.chat(struct, prompt, max_new_tokens=64)
print(f"Q: {prompt}")
print(f"A: {answer}")
print()
输出:
Q: what is the chemical formula of this material?
A: The chemical formula of this material is GaN.
Q: what is the space group of this material?
A: The space group of this material is P6_3mc.
Q: Is this material stable or not?
A: This material is not stable.
Q: What is the bandgap of this material?
A: The bandgap of this material is 1.68300.

3. 批量推理 (多个 CIF)
importosfromppmat.modelsimportbuild_model_from_namefrompymatgen.io.cifimportCifParsermodel, _=build_model_from_name("matterchat_full")
model.to("gpu").eval()
cif_dir="path/to/cif_files"prompt="what is the chemical formula of this material?"forfnameinsorted(os.listdir(cif_dir)):
ifnotfname.endswith(".cif"):
continuestruct=CifParser(os.path.join(cif_dir, fname)).get_structures()[0]
answer=model.chat(struct, prompt, max_new_tokens=64)
print(f"{fname}: {answer}")
@leeleolayleeleolay
changed the title
support matterchat【MIIT program】support matterchatJul 7, 2026
@paddle-bot
paddle-botBot
commented
Jul 14, 2026

Copy link


Copy Markdown

Thanks for your contribution!
@paddle-botpaddle-botBot
added
the 
contributor
External developers
label
Jul 14, 2026
leeleolay
leeleolay
reviewed
Jul 16, 2026
View reviewed changes
@leeleolayleeleolay
left a comment


Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

辛苦修改整体的代码规范符合套件风格
Comment threadppmat/datasets/__init__.py
from ppmat.datasets.oc20_s2ef_dataset import OC20S2EFDataset # noqa
from ppmat.datasets.qm9_dataset import QM9Dataset # noqa
from ppmat.datasets.omol25_dataset import OMol25Dataset
from ppmat.models.matterchat.trainer import MTDataset # noqa
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用已有trainer
Comment threadppmat/datasets/collate_fn.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用默认的collator
Comment threadppmat/models/matterchat/chgnet/graph/__init__.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

套件内已有chgnet，辛苦使用
Comment threadppmat/models/matterchat/chgnet/graph/converter.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用已有的graph_converter
Comment threadppmat/models/matterchat/chgnet/model/dynamics.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ppmatSim已经支持相关的功能，复用已有的
Comment threadppmat/models/matterchat/chgnet/utils/vasp_utils.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vasp在这个模型里的作用是？
Comment threadppmat/models/matterchat/mistral/configuration_mistral.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

不符合已有规范，config不这么处理
@leeleolay
leeleolay
commented
Aug 11, 2026

Copy link


Copy Markdown

Collaborator
@learncat163 重构了部分基础组件，移动了推理器的位置，辛苦基于新的开发和尝试
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@leeleolayleeleolayleeleolay left review comments

At least 1 approving review is required to merge this pull request.


Assignees


No one assigned


Labels

contributorExternal developersMIIT Program

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@learncat163@leeleolay

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
【MIIT program】support matterchat by learncat163 · Pull Request #305 · PaddlePaddle/PaddleMaterials · GitHub
Skip to content
【MIIT program】support matterchat - #305
Open
learncat163 wants to merge 1 commit into
PaddlePaddle:developfrom 
learncat163:pr-matterchat
Open
【MIIT program】support matterchat#305
learncat163 wants to merge 1 commit into
PaddlePaddle:developfrom 
learncat163:pr-matterchat
Conversation
@learncat163
@learncat163learncat163
commented
Jul 7, 2026•

edited

Loading

Copy link


Copy Markdown

MatterChat diff (PyTorch and PaddlePaddle)
AiStudio在线推理案例
MatterChat 由三个模块串联组成: CHGNet (晶体编码器) → Q-Former (跨模态桥接) → Mistral-7B (大语言模型)。本文档分别验证每个模块从 PyTorch 迁移到 PaddlePaddle 后的数值对齐情况。
1. CHGNet Diff
CHGNet 负责将晶体结构 (CIF) 编码为逐原子 embedding。验证用 GaN.cif (4 原子) 作为输入, 对比 Paddle 输出与 PyTorch 参考的 [N_atom, 64] material embedding。
结果
指标值阈值状态
max_diff8.05e-071e-4PASS
mean_diff1.82e-071e-4PASS
代码样例
PyTorch 参考数据生成
importnumpyasnpimporttorchfrompymatgen.coreimportStructurefromModel.chgnet_lib.model.model_embeddingimportCHGNet# MatterChat 原始 PT 代码device=torch.device("cuda")
chgnet=CHGNet.load().to(device).float().eval()
struct=Structure.from_file("fix_inputs/GaN.cif")
withtorch.no_grad():
atom_feas=chgnet.predict_structure_embedding(struct)
ifisinstance(atom_feas, (list, tuple)):
atom_feas=atom_feas[0]
# 保存为参考数据ref=atom_feas.detach().cpu().float().numpy()
np.save("fix_outputs/raw_stage_01_material_embed.npy", ref)
print(f"shape={ref.shape}") # -> (4, 64)
PaddlePaddle 对比
importnumpyasnpimportpaddlefrompymatgen.coreimportStructurefromppmat.models.matterchat.chgnet.model.model_embeddingimportCHGNet# 加载 PyTorch 参考数据ref_embed=np.load("fix_outputs/raw_stage_01_material_embed.npy").astype(np.float64)
# 构建 CHGNet 并加载权重 (CPU)paddle.set_device("cpu")
chgnet=CHGNet()
chgnet_weights=load_sharded_weights(prefix="material_encoder.")
set_state_dict_filtered(chgnet, chgnet_weights, prefix="material_encoder.")
chgnet.eval()
# GPU 推理paddle.set_device("gpu")
gpu_chgnet=CHGNet()
gpu_chgnet.set_state_dict({k: v.cuda() fork, vinchgnet.state_dict().items()})
gpu_chgnet.eval()
struct=Structure.from_file("fix_inputs/GaN.cif")
withpaddle.no_grad():
atom_feas=gpu_chgnet.predict_structure_embedding(struct)
# 对比diff=np.abs(atom_feas.numpy().astype(np.float64) -ref_embed)
print(f"max_diff={diff.max():.2e} mean_diff={diff.mean():.2e}")
# -> max_diff=8.05e-07 mean_diff=1.82e-07 (PASS, thr=1e-4)
2. Q-Former Diff
Q-Former 是 BLIP-2 风格的 BERT 变体, 用 32 个 query token 对 CHGNet 输出做 cross-attention, 产生 [1, 32, 768] 的查询表示。验证 Q-Former 前向输出与基线一致。
结果
指标值阈值状态
max_diff0.00e+001e-4PASS
mean_diff0.00e+001e-4PASS
Q-Former 输出与基线完全一致 (diff=0)。
代码样例
PyTorch 参考数据生成
importnumpyasnpimporttorchfromModel.material_Q_former_baseimportBertConfig, BertLMHeadModel# MatterChat 原始 PT 代码device=torch.device("cuda")
# 加载预训练 Q-Former 权重full_ckpt=torch.load("model_weight/model_weights.pkl", map_location="cpu", weights_only=False)
full_state=full_ckpt["state_dict"]
qformer_config=BertConfig.from_pretrained("bert-base-uncased")
qformer_config.encoder_width=64qformer_config.add_cross_attention=Trueqformer_config.cross_attention_freq=2qformer_config.query_length=32qformer=BertLMHeadModel.from_pretrained("bert-base-uncased", config=qformer_config)
qformer_state= {}
fork, vinfull_state.items():
ifk.startswith("model.Qformer.") ork.startswith("model.query_tokens"):
qformer_state[k.replace("model.", "", 1)] =vqformer.load_state_dict(qformer_state, strict=False)
qformer=qformer.to(device).float().eval()
query_tokens=full_state["model.query_tokens"].to(device).float()
# 输入: 上一步 CHGNet 输出的 material embeddingmaterial_embed=torch.from_numpy(
np.load("fix_outputs/raw_stage_01_material_embed.npy")
).to(device).float()
material_att=torch.ones((1, material_embed.shape[0]), dtype=torch.long, device=device)
withtorch.no_grad():
query_output=qformer.bert(
query_embeds=query_tokens,
encoder_hidden_states=material_embed.unsqueeze(0),
encoder_attention_mask=material_att,
return_dict=True,
)
qf_out=query_output.last_hidden_state[:, :query_tokens.shape[1], :]
# 保存为参考数据ref=qf_out.detach().cpu().float().numpy()
np.save("fix_outputs/raw_stage_02_qformer_out.npy", ref)
print(f"shape={ref.shape}") # -> (1, 32, 768)
PaddlePaddle 对比
importnumpyasnpimportpaddlefromppmat.models.matterchat.q_former.q_former_baseimportBertConfig, BertLMHeadModel# 加载基线数据material_embed=np.load("fix_outputs/raw_stage_01_material_embed.npy")
qt_param=np.load("fix_outputs/paddle_stage_02_query_tokens.npy")
# 构建 Q-Former (CPU)paddle.set_device("cpu")
config=BertConfig()
config.encoder_width=64config.add_cross_attention=Trueconfig.cross_attention_freq=2config.query_length=32qformer=BertLMHeadModel(config)
qformer.cls=Noneqformer.bert.embeddings.word_embeddings=Noneqformer.bert.embeddings.position_embeddings=Noneforlayerinqformer.bert.encoder.layer:
layer.output=Nonelayer.intermediate=Noneqformer_weights=load_sharded_weights(prefix="Qformer.")
set_state_dict_filtered(qformer, qformer_weights, prefix="Qformer.")
qformer.eval()
# GPU 推理paddle.set_device("gpu")
gpu_qformer=BertLMHeadModel(config)
gpu_qformer.set_state_dict({k: v.cuda() fork, vinqformer.state_dict().items()})
gpu_qformer.eval()
qt_gpu=paddle.to_tensor(qt_param).cast(paddle.float32).cuda()
emb_gpu=paddle.to_tensor(material_embed).cuda().unsqueeze(0)
att_gpu=paddle.ones([1, emb_gpu.shape[1]], dtype=paddle.int64).cuda()
withpaddle.no_grad():
query_output=gpu_qformer.bert(
query_embeds=qt_gpu,
encoder_hidden_states=emb_gpu,
encoder_attention_mask=att_gpu,
return_dict=True,
)
qf_out=query_output.last_hidden_state[:, :qt_gpu.shape[1], :]
print(f"max_diff={np.abs(qf_out.numpy().astype(np.float64) -ref).max():.2e}")
# -> max_diff=0.00e+00 (PASS, thr=1e-4)
3. Mistral LLM Diff
Mistral-7B 是 32 层 decoder-only Transformer, 含 RoPE、GQA、SwiGLU。由于模型规模大 (7.3B 参数), float16 跨框架累积误差不可避免, 因此阈值放宽至 1e-3。验证逐层 hidden state、最终 hidden_last 及 Top-20 token 匹配。
结果汇总
测试项通过阈值max_diff状态
32 层 hidden state32/321e-32.97e-04 (layer_31)PASS
hidden_last1/11e-31.12e-04PASS
Top-20 token20/20——PASS
逐层误差
层max_diffmean_diff状态
layer_005.14e-077.92e-09PASS
layer_011.16e-041.06e-07PASS
layer_051.22e-041.70e-07PASS
layer_101.22e-042.25e-07PASS
layer_151.22e-043.15e-07PASS
layer_201.22e-045.78e-07PASS
layer_251.22e-048.60e-07PASS
layer_301.06e-041.51e-06PASS
layer_312.97e-041.99e-06PASS
hidden_last1.12e-041.78e-05PASS
代码样例
PyTorch 参考数据生成
importnumpyasnpimporttorchfromtransformersimportLlamaTokenizerfromtransformers.models.mistral.modeling_mistralimportMistralForCausalLMfromtransformersimportMistralConfigasHFMistralConfigdevice=torch.device("cuda")
MODEL_WEIGHT_DIR="model_weight/Mistral-7B-Instruct-v0.3"# Tokenizetokenizer=LlamaTokenizer.from_pretrained(MODEL_WEIGHT_DIR, use_fast=False)
tokenizer.add_special_tokens({"pad_token": "[PAD]", "bos_token": "<s>",
"eos_token": "</s>", "unk_token": "<unk>"})
prompt="[INST] What is the chemical formula of this material? [/INST]"tokens=tokenizer(prompt, return_tensors="pt", truncation=True, max_length=64)
input_ids=tokens["input_ids"].to(device)
# 加载 Mistral-7Bhf_config=HFMistralConfig.from_pretrained(MODEL_WEIGHT_DIR)
hf_config._attn_implementation="eager"llm=MistralForCausalLM.from_pretrained(MODEL_WEIGHT_DIR, config=hf_config,
torch_dtype=torch.float32).to(device).eval()
llm.resize_token_embeddings(len(tokenizer))
# input embeddingswithtorch.no_grad():
inputs_embeds=llm.model.embed_tokens(input_ids)
np.save("fix_outputs/raw_stage_05_input_embeds.npy",
inputs_embeds.detach().cpu().float().numpy())
# 构建因果掩码seq_len=input_ids.shape[1]
min_dtype=torch.finfo(inputs_embeds.dtype).mincausal_mask=torch.full((seq_len, seq_len), min_dtype, dtype=inputs_embeds.dtype, device=device)
causal_mask=torch.triu(causal_mask, diagonal=1)[None, None, :, :].expand(1, 1, -1, -1)
# 逐层前向, 保存每层 hidden state 作为参考hidden_states=inputs_embedsposition_ids=torch.arange(seq_len, device=device).unsqueeze(0)
forliinrange(32):
withtorch.no_grad():
hidden_states=llm.model.layers[li](
hidden_states, attention_mask=causal_mask,
position_ids=position_ids, use_cache=True,
)[0]
np.save(f"fix_outputs/raw_stage_06_hidden_layer{li:02d}.npy",
hidden_states.detach().cpu().float().numpy())
# 最终 norm + hidden_lastwithtorch.no_grad():
hidden_final=llm.model.norm(hidden_states)
np.save("fix_outputs/raw_stage_06_hidden_last.npy",
hidden_final[0, -1].detach().cpu().float().numpy())
PaddlePaddle 对比
importjsonimportnumpyasnpimportpaddlefromppmat.models.matterchat.mistral.configuration_mistralimportMistralConfigfromppmat.models.matterchat.mistral.modeling_mistralimport (
MistralForCausalLM,
MistralDecoderLayer,
MistralRMSNorm,
)
# 加载 PyTorch 参考数据 (逐层 hidden state)refs= {}
forliinrange(32):
refs[f"stage_06_hidden_layer{li:02d}"] =np.load(
f"fix_outputs/raw_stage_06_hidden_layer{li:02d}.npy"
).astype(np.float64)
refs["stage_05_input_embeds"] =np.load(
"fix_outputs/raw_stage_05_input_embeds.npy"
).astype(np.float64)
emb=refs["stage_05_input_embeds"].astype(np.float32)
seq_len=emb.shape[1]
# 构建因果掩码 + position idsmin_dt=float(paddle.finfo(paddle.float32).min)
causal_mask=paddle.triu(
paddle.full([seq_len, seq_len], min_dt, dtype="float32"), diagonal=1
)[None, None].expand([1, 1, -1, -1])
pos_ids=paddle.arange(seq_len).unsqueeze(0)
cache_pos=paddle.arange(seq_len)
# CPU 加载完整 Mistral-7B, 提取每层权重paddle.set_device("cpu")
llm=MistralForCausalLM(MistralConfig(vocab_size=32769))
load_sharded_weights(llm, weight_dir=WEIGHT_DIR)
llm.eval()
layer_sds= [
{k: vfork, vinllm.model.layers[i].state_dict().items()}
foriinrange(32)
]
# 逐层 GPU 推理 + 对比paddle.set_device("gpu")
hidden=paddle.to_tensor(emb).cuda()
cm=causal_mask.cuda()
pos_ids=pos_ids.cuda()
cache_pos=cache_pos.cuda()
forliinrange(32):
gpu_layer=MistralDecoderLayer(llm.config, li)
gpu_layer.set_state_dict({k: v.cuda() fork, vinlayer_sds[li].items()})
gpu_layer.eval()
withpaddle.no_grad():
hidden=gpu_layer(
hidden, attention_mask=cm,
position_ids=pos_ids, use_cache=False,
cache_position=cache_pos,
)[0]
ref=refs[f"stage_06_hidden_layer{li:02d}"]
diff=np.abs(hidden.cpu().numpy().astype(np.float64) -ref)
status="PASS"ifdiff.max() <1e-3else"FAIL"print(f"layer_{li:02d}: max={diff.max():.2e} mean={diff.mean():.2e} [{status}]")
# -> layer_00: max=5.14e-07 mean=7.92e-09 [PASS]# -> layer_01: max=1.16e-04 mean=1.06e-07 [PASS]# -> ...# -> layer_31: max=2.97e-04 mean=1.99e-06 [PASS]
@learncat163
learncat163
commented
Jul 7, 2026

Copy link


Copy Markdown

Author
MatterChat 推理对话示例
1. 单次推理
1.1 一行命令推理
fromppmat.modelsimportbuild_model_from_namefrompymatgen.io.cifimportCifParsermodel, _=build_model_from_name('matterchat_full')
model.to('gpu').eval()
struct=CifParser('Si.cif').get_structures()[0]
print(model.chat(struct, 'what is the chemical formula of this material?'))
# -> The chemical formula of this material is Si.
1.2 指定 config + 本地权重
fromomegaconfimportOmegaConffrompymatgen.io.cifimportCifParserfromppmat.modelsimportbuild_modelfromppmat.utilsimportsave_loadconfig=OmegaConf.to_container(OmegaConf.load('structure_generation/configs/matterchat/matterchat_full.yaml'), resolve=True)
model=build_model(config['Model'])
save_load.load_pretrain(model, './matterchat_full/')
model.to('gpu').eval()
struct=CifParser('Si.cif').get_structures()[0]
print(model.chat(struct, 'what is the chemical formula of this material?'))
2. 多轮对话示例
对同一晶体连续提问多个问题:
fromppmat.modelsimportbuild_model_from_namefrompymatgen.io.cifimportCifParsermodel, _=build_model_from_name("matterchat_full")
model.to("gpu").eval()
struct=CifParser("GaN.cif").get_structures()[0]
# 4 个标准问题prompts= [
"what is the chemical formula of this material?",
"what is the space group of this material?",
"Is this material stable or not?",
"What is the bandgap of this material?",
]
forpromptinprompts:
answer=model.chat(struct, prompt, max_new_tokens=64)
print(f"Q: {prompt}")
print(f"A: {answer}")
print()
输出:
Q: what is the chemical formula of this material?
A: The chemical formula of this material is GaN.
Q: what is the space group of this material?
A: The space group of this material is P6_3mc.
Q: Is this material stable or not?
A: This material is not stable.
Q: What is the bandgap of this material?
A: The bandgap of this material is 1.68300.

3. 批量推理 (多个 CIF)
importosfromppmat.modelsimportbuild_model_from_namefrompymatgen.io.cifimportCifParsermodel, _=build_model_from_name("matterchat_full")
model.to("gpu").eval()
cif_dir="path/to/cif_files"prompt="what is the chemical formula of this material?"forfnameinsorted(os.listdir(cif_dir)):
ifnotfname.endswith(".cif"):
continuestruct=CifParser(os.path.join(cif_dir, fname)).get_structures()[0]
answer=model.chat(struct, prompt, max_new_tokens=64)
print(f"{fname}: {answer}")
@leeleolayleeleolay
changed the title
support matterchat【MIIT program】support matterchatJul 7, 2026
@paddle-bot
paddle-botBot
commented
Jul 14, 2026

Copy link


Copy Markdown

Thanks for your contribution!
@paddle-botpaddle-botBot
added
the 
contributor
External developers
label
Jul 14, 2026
leeleolay
leeleolay
reviewed
Jul 16, 2026
View reviewed changes
@leeleolayleeleolay
left a comment


Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

辛苦修改整体的代码规范符合套件风格
Comment threadppmat/datasets/__init__.py
from ppmat.datasets.oc20_s2ef_dataset import OC20S2EFDataset # noqa
from ppmat.datasets.qm9_dataset import QM9Dataset # noqa
from ppmat.datasets.omol25_dataset import OMol25Dataset
from ppmat.models.matterchat.trainer import MTDataset # noqa
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用已有trainer
Comment threadppmat/datasets/collate_fn.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用默认的collator
Comment threadppmat/models/matterchat/chgnet/graph/__init__.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

套件内已有chgnet，辛苦使用
Comment threadppmat/models/matterchat/chgnet/graph/converter.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用已有的graph_converter
Comment threadppmat/models/matterchat/chgnet/model/dynamics.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ppmatSim已经支持相关的功能，复用已有的
Comment threadppmat/models/matterchat/chgnet/utils/vasp_utils.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vasp在这个模型里的作用是？
Comment threadppmat/models/matterchat/mistral/configuration_mistral.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

不符合已有规范，config不这么处理
@leeleolay
leeleolay
commented
Aug 11, 2026

Copy link


Copy Markdown

Collaborator
@learncat163 重构了部分基础组件，移动了推理器的位置，辛苦基于新的开发和尝试
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@leeleolayleeleolayleeleolay left review comments

At least 1 approving review is required to merge this pull request.


Assignees


No one assigned


Labels

contributorExternal developersMIIT Program

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@learncat163@leeleolay

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
【MIIT program】support matterchat by learncat163 · Pull Request #305 · PaddlePaddle/PaddleMaterials · GitHub
Skip to content
【MIIT program】support matterchat - #305
Open
learncat163 wants to merge 1 commit into
PaddlePaddle:developfrom 
learncat163:pr-matterchat
Open
【MIIT program】support matterchat#305
learncat163 wants to merge 1 commit into
PaddlePaddle:developfrom 
learncat163:pr-matterchat
Conversation
@learncat163
@learncat163learncat163
commented
Jul 7, 2026•

edited

Loading

Copy link


Copy Markdown

MatterChat diff (PyTorch and PaddlePaddle)
AiStudio在线推理案例
MatterChat 由三个模块串联组成: CHGNet (晶体编码器) → Q-Former (跨模态桥接) → Mistral-7B (大语言模型)。本文档分别验证每个模块从 PyTorch 迁移到 PaddlePaddle 后的数值对齐情况。
1. CHGNet Diff
CHGNet 负责将晶体结构 (CIF) 编码为逐原子 embedding。验证用 GaN.cif (4 原子) 作为输入, 对比 Paddle 输出与 PyTorch 参考的 [N_atom, 64] material embedding。
结果
指标值阈值状态
max_diff8.05e-071e-4PASS
mean_diff1.82e-071e-4PASS
代码样例
PyTorch 参考数据生成
importnumpyasnpimporttorchfrompymatgen.coreimportStructurefromModel.chgnet_lib.model.model_embeddingimportCHGNet# MatterChat 原始 PT 代码device=torch.device("cuda")
chgnet=CHGNet.load().to(device).float().eval()
struct=Structure.from_file("fix_inputs/GaN.cif")
withtorch.no_grad():
atom_feas=chgnet.predict_structure_embedding(struct)
ifisinstance(atom_feas, (list, tuple)):
atom_feas=atom_feas[0]
# 保存为参考数据ref=atom_feas.detach().cpu().float().numpy()
np.save("fix_outputs/raw_stage_01_material_embed.npy", ref)
print(f"shape={ref.shape}") # -> (4, 64)
PaddlePaddle 对比
importnumpyasnpimportpaddlefrompymatgen.coreimportStructurefromppmat.models.matterchat.chgnet.model.model_embeddingimportCHGNet# 加载 PyTorch 参考数据ref_embed=np.load("fix_outputs/raw_stage_01_material_embed.npy").astype(np.float64)
# 构建 CHGNet 并加载权重 (CPU)paddle.set_device("cpu")
chgnet=CHGNet()
chgnet_weights=load_sharded_weights(prefix="material_encoder.")
set_state_dict_filtered(chgnet, chgnet_weights, prefix="material_encoder.")
chgnet.eval()
# GPU 推理paddle.set_device("gpu")
gpu_chgnet=CHGNet()
gpu_chgnet.set_state_dict({k: v.cuda() fork, vinchgnet.state_dict().items()})
gpu_chgnet.eval()
struct=Structure.from_file("fix_inputs/GaN.cif")
withpaddle.no_grad():
atom_feas=gpu_chgnet.predict_structure_embedding(struct)
# 对比diff=np.abs(atom_feas.numpy().astype(np.float64) -ref_embed)
print(f"max_diff={diff.max():.2e} mean_diff={diff.mean():.2e}")
# -> max_diff=8.05e-07 mean_diff=1.82e-07 (PASS, thr=1e-4)
2. Q-Former Diff
Q-Former 是 BLIP-2 风格的 BERT 变体, 用 32 个 query token 对 CHGNet 输出做 cross-attention, 产生 [1, 32, 768] 的查询表示。验证 Q-Former 前向输出与基线一致。
结果
指标值阈值状态
max_diff0.00e+001e-4PASS
mean_diff0.00e+001e-4PASS
Q-Former 输出与基线完全一致 (diff=0)。
代码样例
PyTorch 参考数据生成
importnumpyasnpimporttorchfromModel.material_Q_former_baseimportBertConfig, BertLMHeadModel# MatterChat 原始 PT 代码device=torch.device("cuda")
# 加载预训练 Q-Former 权重full_ckpt=torch.load("model_weight/model_weights.pkl", map_location="cpu", weights_only=False)
full_state=full_ckpt["state_dict"]
qformer_config=BertConfig.from_pretrained("bert-base-uncased")
qformer_config.encoder_width=64qformer_config.add_cross_attention=Trueqformer_config.cross_attention_freq=2qformer_config.query_length=32qformer=BertLMHeadModel.from_pretrained("bert-base-uncased", config=qformer_config)
qformer_state= {}
fork, vinfull_state.items():
ifk.startswith("model.Qformer.") ork.startswith("model.query_tokens"):
qformer_state[k.replace("model.", "", 1)] =vqformer.load_state_dict(qformer_state, strict=False)
qformer=qformer.to(device).float().eval()
query_tokens=full_state["model.query_tokens"].to(device).float()
# 输入: 上一步 CHGNet 输出的 material embeddingmaterial_embed=torch.from_numpy(
np.load("fix_outputs/raw_stage_01_material_embed.npy")
).to(device).float()
material_att=torch.ones((1, material_embed.shape[0]), dtype=torch.long, device=device)
withtorch.no_grad():
query_output=qformer.bert(
query_embeds=query_tokens,
encoder_hidden_states=material_embed.unsqueeze(0),
encoder_attention_mask=material_att,
return_dict=True,
)
qf_out=query_output.last_hidden_state[:, :query_tokens.shape[1], :]
# 保存为参考数据ref=qf_out.detach().cpu().float().numpy()
np.save("fix_outputs/raw_stage_02_qformer_out.npy", ref)
print(f"shape={ref.shape}") # -> (1, 32, 768)
PaddlePaddle 对比
importnumpyasnpimportpaddlefromppmat.models.matterchat.q_former.q_former_baseimportBertConfig, BertLMHeadModel# 加载基线数据material_embed=np.load("fix_outputs/raw_stage_01_material_embed.npy")
qt_param=np.load("fix_outputs/paddle_stage_02_query_tokens.npy")
# 构建 Q-Former (CPU)paddle.set_device("cpu")
config=BertConfig()
config.encoder_width=64config.add_cross_attention=Trueconfig.cross_attention_freq=2config.query_length=32qformer=BertLMHeadModel(config)
qformer.cls=Noneqformer.bert.embeddings.word_embeddings=Noneqformer.bert.embeddings.position_embeddings=Noneforlayerinqformer.bert.encoder.layer:
layer.output=Nonelayer.intermediate=Noneqformer_weights=load_sharded_weights(prefix="Qformer.")
set_state_dict_filtered(qformer, qformer_weights, prefix="Qformer.")
qformer.eval()
# GPU 推理paddle.set_device("gpu")
gpu_qformer=BertLMHeadModel(config)
gpu_qformer.set_state_dict({k: v.cuda() fork, vinqformer.state_dict().items()})
gpu_qformer.eval()
qt_gpu=paddle.to_tensor(qt_param).cast(paddle.float32).cuda()
emb_gpu=paddle.to_tensor(material_embed).cuda().unsqueeze(0)
att_gpu=paddle.ones([1, emb_gpu.shape[1]], dtype=paddle.int64).cuda()
withpaddle.no_grad():
query_output=gpu_qformer.bert(
query_embeds=qt_gpu,
encoder_hidden_states=emb_gpu,
encoder_attention_mask=att_gpu,
return_dict=True,
)
qf_out=query_output.last_hidden_state[:, :qt_gpu.shape[1], :]
print(f"max_diff={np.abs(qf_out.numpy().astype(np.float64) -ref).max():.2e}")
# -> max_diff=0.00e+00 (PASS, thr=1e-4)
3. Mistral LLM Diff
Mistral-7B 是 32 层 decoder-only Transformer, 含 RoPE、GQA、SwiGLU。由于模型规模大 (7.3B 参数), float16 跨框架累积误差不可避免, 因此阈值放宽至 1e-3。验证逐层 hidden state、最终 hidden_last 及 Top-20 token 匹配。
结果汇总
测试项通过阈值max_diff状态
32 层 hidden state32/321e-32.97e-04 (layer_31)PASS
hidden_last1/11e-31.12e-04PASS
Top-20 token20/20——PASS
逐层误差
层max_diffmean_diff状态
layer_005.14e-077.92e-09PASS
layer_011.16e-041.06e-07PASS
layer_051.22e-041.70e-07PASS
layer_101.22e-042.25e-07PASS
layer_151.22e-043.15e-07PASS
layer_201.22e-045.78e-07PASS
layer_251.22e-048.60e-07PASS
layer_301.06e-041.51e-06PASS
layer_312.97e-041.99e-06PASS
hidden_last1.12e-041.78e-05PASS
代码样例
PyTorch 参考数据生成
importnumpyasnpimporttorchfromtransformersimportLlamaTokenizerfromtransformers.models.mistral.modeling_mistralimportMistralForCausalLMfromtransformersimportMistralConfigasHFMistralConfigdevice=torch.device("cuda")
MODEL_WEIGHT_DIR="model_weight/Mistral-7B-Instruct-v0.3"# Tokenizetokenizer=LlamaTokenizer.from_pretrained(MODEL_WEIGHT_DIR, use_fast=False)
tokenizer.add_special_tokens({"pad_token": "[PAD]", "bos_token": "<s>",
"eos_token": "</s>", "unk_token": "<unk>"})
prompt="[INST] What is the chemical formula of this material? [/INST]"tokens=tokenizer(prompt, return_tensors="pt", truncation=True, max_length=64)
input_ids=tokens["input_ids"].to(device)
# 加载 Mistral-7Bhf_config=HFMistralConfig.from_pretrained(MODEL_WEIGHT_DIR)
hf_config._attn_implementation="eager"llm=MistralForCausalLM.from_pretrained(MODEL_WEIGHT_DIR, config=hf_config,
torch_dtype=torch.float32).to(device).eval()
llm.resize_token_embeddings(len(tokenizer))
# input embeddingswithtorch.no_grad():
inputs_embeds=llm.model.embed_tokens(input_ids)
np.save("fix_outputs/raw_stage_05_input_embeds.npy",
inputs_embeds.detach().cpu().float().numpy())
# 构建因果掩码seq_len=input_ids.shape[1]
min_dtype=torch.finfo(inputs_embeds.dtype).mincausal_mask=torch.full((seq_len, seq_len), min_dtype, dtype=inputs_embeds.dtype, device=device)
causal_mask=torch.triu(causal_mask, diagonal=1)[None, None, :, :].expand(1, 1, -1, -1)
# 逐层前向, 保存每层 hidden state 作为参考hidden_states=inputs_embedsposition_ids=torch.arange(seq_len, device=device).unsqueeze(0)
forliinrange(32):
withtorch.no_grad():
hidden_states=llm.model.layers[li](
hidden_states, attention_mask=causal_mask,
position_ids=position_ids, use_cache=True,
)[0]
np.save(f"fix_outputs/raw_stage_06_hidden_layer{li:02d}.npy",
hidden_states.detach().cpu().float().numpy())
# 最终 norm + hidden_lastwithtorch.no_grad():
hidden_final=llm.model.norm(hidden_states)
np.save("fix_outputs/raw_stage_06_hidden_last.npy",
hidden_final[0, -1].detach().cpu().float().numpy())
PaddlePaddle 对比
importjsonimportnumpyasnpimportpaddlefromppmat.models.matterchat.mistral.configuration_mistralimportMistralConfigfromppmat.models.matterchat.mistral.modeling_mistralimport (
MistralForCausalLM,
MistralDecoderLayer,
MistralRMSNorm,
)
# 加载 PyTorch 参考数据 (逐层 hidden state)refs= {}
forliinrange(32):
refs[f"stage_06_hidden_layer{li:02d}"] =np.load(
f"fix_outputs/raw_stage_06_hidden_layer{li:02d}.npy"
).astype(np.float64)
refs["stage_05_input_embeds"] =np.load(
"fix_outputs/raw_stage_05_input_embeds.npy"
).astype(np.float64)
emb=refs["stage_05_input_embeds"].astype(np.float32)
seq_len=emb.shape[1]
# 构建因果掩码 + position idsmin_dt=float(paddle.finfo(paddle.float32).min)
causal_mask=paddle.triu(
paddle.full([seq_len, seq_len], min_dt, dtype="float32"), diagonal=1
)[None, None].expand([1, 1, -1, -1])
pos_ids=paddle.arange(seq_len).unsqueeze(0)
cache_pos=paddle.arange(seq_len)
# CPU 加载完整 Mistral-7B, 提取每层权重paddle.set_device("cpu")
llm=MistralForCausalLM(MistralConfig(vocab_size=32769))
load_sharded_weights(llm, weight_dir=WEIGHT_DIR)
llm.eval()
layer_sds= [
{k: vfork, vinllm.model.layers[i].state_dict().items()}
foriinrange(32)
]
# 逐层 GPU 推理 + 对比paddle.set_device("gpu")
hidden=paddle.to_tensor(emb).cuda()
cm=causal_mask.cuda()
pos_ids=pos_ids.cuda()
cache_pos=cache_pos.cuda()
forliinrange(32):
gpu_layer=MistralDecoderLayer(llm.config, li)
gpu_layer.set_state_dict({k: v.cuda() fork, vinlayer_sds[li].items()})
gpu_layer.eval()
withpaddle.no_grad():
hidden=gpu_layer(
hidden, attention_mask=cm,
position_ids=pos_ids, use_cache=False,
cache_position=cache_pos,
)[0]
ref=refs[f"stage_06_hidden_layer{li:02d}"]
diff=np.abs(hidden.cpu().numpy().astype(np.float64) -ref)
status="PASS"ifdiff.max() <1e-3else"FAIL"print(f"layer_{li:02d}: max={diff.max():.2e} mean={diff.mean():.2e} [{status}]")
# -> layer_00: max=5.14e-07 mean=7.92e-09 [PASS]# -> layer_01: max=1.16e-04 mean=1.06e-07 [PASS]# -> ...# -> layer_31: max=2.97e-04 mean=1.99e-06 [PASS]
@learncat163
learncat163
commented
Jul 7, 2026

Copy link


Copy Markdown

Author
MatterChat 推理对话示例
1. 单次推理
1.1 一行命令推理
fromppmat.modelsimportbuild_model_from_namefrompymatgen.io.cifimportCifParsermodel, _=build_model_from_name('matterchat_full')
model.to('gpu').eval()
struct=CifParser('Si.cif').get_structures()[0]
print(model.chat(struct, 'what is the chemical formula of this material?'))
# -> The chemical formula of this material is Si.
1.2 指定 config + 本地权重
fromomegaconfimportOmegaConffrompymatgen.io.cifimportCifParserfromppmat.modelsimportbuild_modelfromppmat.utilsimportsave_loadconfig=OmegaConf.to_container(OmegaConf.load('structure_generation/configs/matterchat/matterchat_full.yaml'), resolve=True)
model=build_model(config['Model'])
save_load.load_pretrain(model, './matterchat_full/')
model.to('gpu').eval()
struct=CifParser('Si.cif').get_structures()[0]
print(model.chat(struct, 'what is the chemical formula of this material?'))
2. 多轮对话示例
对同一晶体连续提问多个问题:
fromppmat.modelsimportbuild_model_from_namefrompymatgen.io.cifimportCifParsermodel, _=build_model_from_name("matterchat_full")
model.to("gpu").eval()
struct=CifParser("GaN.cif").get_structures()[0]
# 4 个标准问题prompts= [
"what is the chemical formula of this material?",
"what is the space group of this material?",
"Is this material stable or not?",
"What is the bandgap of this material?",
]
forpromptinprompts:
answer=model.chat(struct, prompt, max_new_tokens=64)
print(f"Q: {prompt}")
print(f"A: {answer}")
print()
输出:
Q: what is the chemical formula of this material?
A: The chemical formula of this material is GaN.
Q: what is the space group of this material?
A: The space group of this material is P6_3mc.
Q: Is this material stable or not?
A: This material is not stable.
Q: What is the bandgap of this material?
A: The bandgap of this material is 1.68300.

3. 批量推理 (多个 CIF)
importosfromppmat.modelsimportbuild_model_from_namefrompymatgen.io.cifimportCifParsermodel, _=build_model_from_name("matterchat_full")
model.to("gpu").eval()
cif_dir="path/to/cif_files"prompt="what is the chemical formula of this material?"forfnameinsorted(os.listdir(cif_dir)):
ifnotfname.endswith(".cif"):
continuestruct=CifParser(os.path.join(cif_dir, fname)).get_structures()[0]
answer=model.chat(struct, prompt, max_new_tokens=64)
print(f"{fname}: {answer}")
@leeleolayleeleolay
changed the title
support matterchat【MIIT program】support matterchatJul 7, 2026
@paddle-bot
paddle-botBot
commented
Jul 14, 2026

Copy link


Copy Markdown

Thanks for your contribution!
@paddle-botpaddle-botBot
added
the 
contributor
External developers
label
Jul 14, 2026
leeleolay
leeleolay
reviewed
Jul 16, 2026
View reviewed changes
@leeleolayleeleolay
left a comment


Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

辛苦修改整体的代码规范符合套件风格
Comment threadppmat/datasets/__init__.py
from ppmat.datasets.oc20_s2ef_dataset import OC20S2EFDataset # noqa
from ppmat.datasets.qm9_dataset import QM9Dataset # noqa
from ppmat.datasets.omol25_dataset import OMol25Dataset
from ppmat.models.matterchat.trainer import MTDataset # noqa
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用已有trainer
Comment threadppmat/datasets/collate_fn.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用默认的collator
Comment threadppmat/models/matterchat/chgnet/graph/__init__.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

套件内已有chgnet，辛苦使用
Comment threadppmat/models/matterchat/chgnet/graph/converter.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用已有的graph_converter
Comment threadppmat/models/matterchat/chgnet/model/dynamics.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ppmatSim已经支持相关的功能，复用已有的
Comment threadppmat/models/matterchat/chgnet/utils/vasp_utils.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vasp在这个模型里的作用是？
Comment threadppmat/models/matterchat/mistral/configuration_mistral.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

不符合已有规范，config不这么处理
@leeleolay
leeleolay
commented
Aug 11, 2026

Copy link


Copy Markdown

Collaborator
@learncat163 重构了部分基础组件，移动了推理器的位置，辛苦基于新的开发和尝试
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@leeleolayleeleolayleeleolay left review comments

At least 1 approving review is required to merge this pull request.


Assignees


No one assigned


Labels

contributorExternal developersMIIT Program

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@learncat163@leeleolay

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
【MIIT program】support matterchat by learncat163 · Pull Request #305 · PaddlePaddle/PaddleMaterials · GitHub
Skip to content
【MIIT program】support matterchat - #305
Open
learncat163 wants to merge 1 commit into
PaddlePaddle:developfrom 
learncat163:pr-matterchat
Open
【MIIT program】support matterchat#305
learncat163 wants to merge 1 commit into
PaddlePaddle:developfrom 
learncat163:pr-matterchat
Conversation
@learncat163
@learncat163learncat163
commented
Jul 7, 2026•

edited

Loading

Copy link


Copy Markdown

MatterChat diff (PyTorch and PaddlePaddle)
AiStudio在线推理案例
MatterChat 由三个模块串联组成: CHGNet (晶体编码器) → Q-Former (跨模态桥接) → Mistral-7B (大语言模型)。本文档分别验证每个模块从 PyTorch 迁移到 PaddlePaddle 后的数值对齐情况。
1. CHGNet Diff
CHGNet 负责将晶体结构 (CIF) 编码为逐原子 embedding。验证用 GaN.cif (4 原子) 作为输入, 对比 Paddle 输出与 PyTorch 参考的 [N_atom, 64] material embedding。
结果
指标值阈值状态
max_diff8.05e-071e-4PASS
mean_diff1.82e-071e-4PASS
代码样例
PyTorch 参考数据生成
importnumpyasnpimporttorchfrompymatgen.coreimportStructurefromModel.chgnet_lib.model.model_embeddingimportCHGNet# MatterChat 原始 PT 代码device=torch.device("cuda")
chgnet=CHGNet.load().to(device).float().eval()
struct=Structure.from_file("fix_inputs/GaN.cif")
withtorch.no_grad():
atom_feas=chgnet.predict_structure_embedding(struct)
ifisinstance(atom_feas, (list, tuple)):
atom_feas=atom_feas[0]
# 保存为参考数据ref=atom_feas.detach().cpu().float().numpy()
np.save("fix_outputs/raw_stage_01_material_embed.npy", ref)
print(f"shape={ref.shape}") # -> (4, 64)
PaddlePaddle 对比
importnumpyasnpimportpaddlefrompymatgen.coreimportStructurefromppmat.models.matterchat.chgnet.model.model_embeddingimportCHGNet# 加载 PyTorch 参考数据ref_embed=np.load("fix_outputs/raw_stage_01_material_embed.npy").astype(np.float64)
# 构建 CHGNet 并加载权重 (CPU)paddle.set_device("cpu")
chgnet=CHGNet()
chgnet_weights=load_sharded_weights(prefix="material_encoder.")
set_state_dict_filtered(chgnet, chgnet_weights, prefix="material_encoder.")
chgnet.eval()
# GPU 推理paddle.set_device("gpu")
gpu_chgnet=CHGNet()
gpu_chgnet.set_state_dict({k: v.cuda() fork, vinchgnet.state_dict().items()})
gpu_chgnet.eval()
struct=Structure.from_file("fix_inputs/GaN.cif")
withpaddle.no_grad():
atom_feas=gpu_chgnet.predict_structure_embedding(struct)
# 对比diff=np.abs(atom_feas.numpy().astype(np.float64) -ref_embed)
print(f"max_diff={diff.max():.2e} mean_diff={diff.mean():.2e}")
# -> max_diff=8.05e-07 mean_diff=1.82e-07 (PASS, thr=1e-4)
2. Q-Former Diff
Q-Former 是 BLIP-2 风格的 BERT 变体, 用 32 个 query token 对 CHGNet 输出做 cross-attention, 产生 [1, 32, 768] 的查询表示。验证 Q-Former 前向输出与基线一致。
结果
指标值阈值状态
max_diff0.00e+001e-4PASS
mean_diff0.00e+001e-4PASS
Q-Former 输出与基线完全一致 (diff=0)。
代码样例
PyTorch 参考数据生成
importnumpyasnpimporttorchfromModel.material_Q_former_baseimportBertConfig, BertLMHeadModel# MatterChat 原始 PT 代码device=torch.device("cuda")
# 加载预训练 Q-Former 权重full_ckpt=torch.load("model_weight/model_weights.pkl", map_location="cpu", weights_only=False)
full_state=full_ckpt["state_dict"]
qformer_config=BertConfig.from_pretrained("bert-base-uncased")
qformer_config.encoder_width=64qformer_config.add_cross_attention=Trueqformer_config.cross_attention_freq=2qformer_config.query_length=32qformer=BertLMHeadModel.from_pretrained("bert-base-uncased", config=qformer_config)
qformer_state= {}
fork, vinfull_state.items():
ifk.startswith("model.Qformer.") ork.startswith("model.query_tokens"):
qformer_state[k.replace("model.", "", 1)] =vqformer.load_state_dict(qformer_state, strict=False)
qformer=qformer.to(device).float().eval()
query_tokens=full_state["model.query_tokens"].to(device).float()
# 输入: 上一步 CHGNet 输出的 material embeddingmaterial_embed=torch.from_numpy(
np.load("fix_outputs/raw_stage_01_material_embed.npy")
).to(device).float()
material_att=torch.ones((1, material_embed.shape[0]), dtype=torch.long, device=device)
withtorch.no_grad():
query_output=qformer.bert(
query_embeds=query_tokens,
encoder_hidden_states=material_embed.unsqueeze(0),
encoder_attention_mask=material_att,
return_dict=True,
)
qf_out=query_output.last_hidden_state[:, :query_tokens.shape[1], :]
# 保存为参考数据ref=qf_out.detach().cpu().float().numpy()
np.save("fix_outputs/raw_stage_02_qformer_out.npy", ref)
print(f"shape={ref.shape}") # -> (1, 32, 768)
PaddlePaddle 对比
importnumpyasnpimportpaddlefromppmat.models.matterchat.q_former.q_former_baseimportBertConfig, BertLMHeadModel# 加载基线数据material_embed=np.load("fix_outputs/raw_stage_01_material_embed.npy")
qt_param=np.load("fix_outputs/paddle_stage_02_query_tokens.npy")
# 构建 Q-Former (CPU)paddle.set_device("cpu")
config=BertConfig()
config.encoder_width=64config.add_cross_attention=Trueconfig.cross_attention_freq=2config.query_length=32qformer=BertLMHeadModel(config)
qformer.cls=Noneqformer.bert.embeddings.word_embeddings=Noneqformer.bert.embeddings.position_embeddings=Noneforlayerinqformer.bert.encoder.layer:
layer.output=Nonelayer.intermediate=Noneqformer_weights=load_sharded_weights(prefix="Qformer.")
set_state_dict_filtered(qformer, qformer_weights, prefix="Qformer.")
qformer.eval()
# GPU 推理paddle.set_device("gpu")
gpu_qformer=BertLMHeadModel(config)
gpu_qformer.set_state_dict({k: v.cuda() fork, vinqformer.state_dict().items()})
gpu_qformer.eval()
qt_gpu=paddle.to_tensor(qt_param).cast(paddle.float32).cuda()
emb_gpu=paddle.to_tensor(material_embed).cuda().unsqueeze(0)
att_gpu=paddle.ones([1, emb_gpu.shape[1]], dtype=paddle.int64).cuda()
withpaddle.no_grad():
query_output=gpu_qformer.bert(
query_embeds=qt_gpu,
encoder_hidden_states=emb_gpu,
encoder_attention_mask=att_gpu,
return_dict=True,
)
qf_out=query_output.last_hidden_state[:, :qt_gpu.shape[1], :]
print(f"max_diff={np.abs(qf_out.numpy().astype(np.float64) -ref).max():.2e}")
# -> max_diff=0.00e+00 (PASS, thr=1e-4)
3. Mistral LLM Diff
Mistral-7B 是 32 层 decoder-only Transformer, 含 RoPE、GQA、SwiGLU。由于模型规模大 (7.3B 参数), float16 跨框架累积误差不可避免, 因此阈值放宽至 1e-3。验证逐层 hidden state、最终 hidden_last 及 Top-20 token 匹配。
结果汇总
测试项通过阈值max_diff状态
32 层 hidden state32/321e-32.97e-04 (layer_31)PASS
hidden_last1/11e-31.12e-04PASS
Top-20 token20/20——PASS
逐层误差
层max_diffmean_diff状态
layer_005.14e-077.92e-09PASS
layer_011.16e-041.06e-07PASS
layer_051.22e-041.70e-07PASS
layer_101.22e-042.25e-07PASS
layer_151.22e-043.15e-07PASS
layer_201.22e-045.78e-07PASS
layer_251.22e-048.60e-07PASS
layer_301.06e-041.51e-06PASS
layer_312.97e-041.99e-06PASS
hidden_last1.12e-041.78e-05PASS
代码样例
PyTorch 参考数据生成
importnumpyasnpimporttorchfromtransformersimportLlamaTokenizerfromtransformers.models.mistral.modeling_mistralimportMistralForCausalLMfromtransformersimportMistralConfigasHFMistralConfigdevice=torch.device("cuda")
MODEL_WEIGHT_DIR="model_weight/Mistral-7B-Instruct-v0.3"# Tokenizetokenizer=LlamaTokenizer.from_pretrained(MODEL_WEIGHT_DIR, use_fast=False)
tokenizer.add_special_tokens({"pad_token": "[PAD]", "bos_token": "<s>",
"eos_token": "</s>", "unk_token": "<unk>"})
prompt="[INST] What is the chemical formula of this material? [/INST]"tokens=tokenizer(prompt, return_tensors="pt", truncation=True, max_length=64)
input_ids=tokens["input_ids"].to(device)
# 加载 Mistral-7Bhf_config=HFMistralConfig.from_pretrained(MODEL_WEIGHT_DIR)
hf_config._attn_implementation="eager"llm=MistralForCausalLM.from_pretrained(MODEL_WEIGHT_DIR, config=hf_config,
torch_dtype=torch.float32).to(device).eval()
llm.resize_token_embeddings(len(tokenizer))
# input embeddingswithtorch.no_grad():
inputs_embeds=llm.model.embed_tokens(input_ids)
np.save("fix_outputs/raw_stage_05_input_embeds.npy",
inputs_embeds.detach().cpu().float().numpy())
# 构建因果掩码seq_len=input_ids.shape[1]
min_dtype=torch.finfo(inputs_embeds.dtype).mincausal_mask=torch.full((seq_len, seq_len), min_dtype, dtype=inputs_embeds.dtype, device=device)
causal_mask=torch.triu(causal_mask, diagonal=1)[None, None, :, :].expand(1, 1, -1, -1)
# 逐层前向, 保存每层 hidden state 作为参考hidden_states=inputs_embedsposition_ids=torch.arange(seq_len, device=device).unsqueeze(0)
forliinrange(32):
withtorch.no_grad():
hidden_states=llm.model.layers[li](
hidden_states, attention_mask=causal_mask,
position_ids=position_ids, use_cache=True,
)[0]
np.save(f"fix_outputs/raw_stage_06_hidden_layer{li:02d}.npy",
hidden_states.detach().cpu().float().numpy())
# 最终 norm + hidden_lastwithtorch.no_grad():
hidden_final=llm.model.norm(hidden_states)
np.save("fix_outputs/raw_stage_06_hidden_last.npy",
hidden_final[0, -1].detach().cpu().float().numpy())
PaddlePaddle 对比
importjsonimportnumpyasnpimportpaddlefromppmat.models.matterchat.mistral.configuration_mistralimportMistralConfigfromppmat.models.matterchat.mistral.modeling_mistralimport (
MistralForCausalLM,
MistralDecoderLayer,
MistralRMSNorm,
)
# 加载 PyTorch 参考数据 (逐层 hidden state)refs= {}
forliinrange(32):
refs[f"stage_06_hidden_layer{li:02d}"] =np.load(
f"fix_outputs/raw_stage_06_hidden_layer{li:02d}.npy"
).astype(np.float64)
refs["stage_05_input_embeds"] =np.load(
"fix_outputs/raw_stage_05_input_embeds.npy"
).astype(np.float64)
emb=refs["stage_05_input_embeds"].astype(np.float32)
seq_len=emb.shape[1]
# 构建因果掩码 + position idsmin_dt=float(paddle.finfo(paddle.float32).min)
causal_mask=paddle.triu(
paddle.full([seq_len, seq_len], min_dt, dtype="float32"), diagonal=1
)[None, None].expand([1, 1, -1, -1])
pos_ids=paddle.arange(seq_len).unsqueeze(0)
cache_pos=paddle.arange(seq_len)
# CPU 加载完整 Mistral-7B, 提取每层权重paddle.set_device("cpu")
llm=MistralForCausalLM(MistralConfig(vocab_size=32769))
load_sharded_weights(llm, weight_dir=WEIGHT_DIR)
llm.eval()
layer_sds= [
{k: vfork, vinllm.model.layers[i].state_dict().items()}
foriinrange(32)
]
# 逐层 GPU 推理 + 对比paddle.set_device("gpu")
hidden=paddle.to_tensor(emb).cuda()
cm=causal_mask.cuda()
pos_ids=pos_ids.cuda()
cache_pos=cache_pos.cuda()
forliinrange(32):
gpu_layer=MistralDecoderLayer(llm.config, li)
gpu_layer.set_state_dict({k: v.cuda() fork, vinlayer_sds[li].items()})
gpu_layer.eval()
withpaddle.no_grad():
hidden=gpu_layer(
hidden, attention_mask=cm,
position_ids=pos_ids, use_cache=False,
cache_position=cache_pos,
)[0]
ref=refs[f"stage_06_hidden_layer{li:02d}"]
diff=np.abs(hidden.cpu().numpy().astype(np.float64) -ref)
status="PASS"ifdiff.max() <1e-3else"FAIL"print(f"layer_{li:02d}: max={diff.max():.2e} mean={diff.mean():.2e} [{status}]")
# -> layer_00: max=5.14e-07 mean=7.92e-09 [PASS]# -> layer_01: max=1.16e-04 mean=1.06e-07 [PASS]# -> ...# -> layer_31: max=2.97e-04 mean=1.99e-06 [PASS]
@learncat163
learncat163
commented
Jul 7, 2026

Copy link


Copy Markdown

Author
MatterChat 推理对话示例
1. 单次推理
1.1 一行命令推理
fromppmat.modelsimportbuild_model_from_namefrompymatgen.io.cifimportCifParsermodel, _=build_model_from_name('matterchat_full')
model.to('gpu').eval()
struct=CifParser('Si.cif').get_structures()[0]
print(model.chat(struct, 'what is the chemical formula of this material?'))
# -> The chemical formula of this material is Si.
1.2 指定 config + 本地权重
fromomegaconfimportOmegaConffrompymatgen.io.cifimportCifParserfromppmat.modelsimportbuild_modelfromppmat.utilsimportsave_loadconfig=OmegaConf.to_container(OmegaConf.load('structure_generation/configs/matterchat/matterchat_full.yaml'), resolve=True)
model=build_model(config['Model'])
save_load.load_pretrain(model, './matterchat_full/')
model.to('gpu').eval()
struct=CifParser('Si.cif').get_structures()[0]
print(model.chat(struct, 'what is the chemical formula of this material?'))
2. 多轮对话示例
对同一晶体连续提问多个问题:
fromppmat.modelsimportbuild_model_from_namefrompymatgen.io.cifimportCifParsermodel, _=build_model_from_name("matterchat_full")
model.to("gpu").eval()
struct=CifParser("GaN.cif").get_structures()[0]
# 4 个标准问题prompts= [
"what is the chemical formula of this material?",
"what is the space group of this material?",
"Is this material stable or not?",
"What is the bandgap of this material?",
]
forpromptinprompts:
answer=model.chat(struct, prompt, max_new_tokens=64)
print(f"Q: {prompt}")
print(f"A: {answer}")
print()
输出:
Q: what is the chemical formula of this material?
A: The chemical formula of this material is GaN.
Q: what is the space group of this material?
A: The space group of this material is P6_3mc.
Q: Is this material stable or not?
A: This material is not stable.
Q: What is the bandgap of this material?
A: The bandgap of this material is 1.68300.

3. 批量推理 (多个 CIF)
importosfromppmat.modelsimportbuild_model_from_namefrompymatgen.io.cifimportCifParsermodel, _=build_model_from_name("matterchat_full")
model.to("gpu").eval()
cif_dir="path/to/cif_files"prompt="what is the chemical formula of this material?"forfnameinsorted(os.listdir(cif_dir)):
ifnotfname.endswith(".cif"):
continuestruct=CifParser(os.path.join(cif_dir, fname)).get_structures()[0]
answer=model.chat(struct, prompt, max_new_tokens=64)
print(f"{fname}: {answer}")
@leeleolayleeleolay
changed the title
support matterchat【MIIT program】support matterchatJul 7, 2026
@paddle-bot
paddle-botBot
commented
Jul 14, 2026

Copy link


Copy Markdown

Thanks for your contribution!
@paddle-botpaddle-botBot
added
the 
contributor
External developers
label
Jul 14, 2026
leeleolay
leeleolay
reviewed
Jul 16, 2026
View reviewed changes
@leeleolayleeleolay
left a comment


Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

辛苦修改整体的代码规范符合套件风格
Comment threadppmat/datasets/__init__.py
from ppmat.datasets.oc20_s2ef_dataset import OC20S2EFDataset # noqa
from ppmat.datasets.qm9_dataset import QM9Dataset # noqa
from ppmat.datasets.omol25_dataset import OMol25Dataset
from ppmat.models.matterchat.trainer import MTDataset # noqa
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用已有trainer
Comment threadppmat/datasets/collate_fn.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用默认的collator
Comment threadppmat/models/matterchat/chgnet/graph/__init__.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

套件内已有chgnet，辛苦使用
Comment threadppmat/models/matterchat/chgnet/graph/converter.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用已有的graph_converter
Comment threadppmat/models/matterchat/chgnet/model/dynamics.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ppmatSim已经支持相关的功能，复用已有的
Comment threadppmat/models/matterchat/chgnet/utils/vasp_utils.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vasp在这个模型里的作用是？
Comment threadppmat/models/matterchat/mistral/configuration_mistral.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

不符合已有规范，config不这么处理
@leeleolay
leeleolay
commented
Aug 11, 2026

Copy link


Copy Markdown

Collaborator
@learncat163 重构了部分基础组件，移动了推理器的位置，辛苦基于新的开发和尝试
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@leeleolayleeleolayleeleolay left review comments

At least 1 approving review is required to merge this pull request.


Assignees


No one assigned


Labels

contributorExternal developersMIIT Program

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@learncat163@leeleolay

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
【MIIT program】support matterchat by learncat163 · Pull Request #305 · PaddlePaddle/PaddleMaterials · GitHub
Skip to content
【MIIT program】support matterchat - #305
Open
learncat163 wants to merge 1 commit into
PaddlePaddle:developfrom 
learncat163:pr-matterchat
Open
【MIIT program】support matterchat#305
learncat163 wants to merge 1 commit into
PaddlePaddle:developfrom 
learncat163:pr-matterchat
Conversation
@learncat163
@learncat163learncat163
commented
Jul 7, 2026•

edited

Loading

Copy link


Copy Markdown

MatterChat diff (PyTorch and PaddlePaddle)
AiStudio在线推理案例
MatterChat 由三个模块串联组成: CHGNet (晶体编码器) → Q-Former (跨模态桥接) → Mistral-7B (大语言模型)。本文档分别验证每个模块从 PyTorch 迁移到 PaddlePaddle 后的数值对齐情况。
1. CHGNet Diff
CHGNet 负责将晶体结构 (CIF) 编码为逐原子 embedding。验证用 GaN.cif (4 原子) 作为输入, 对比 Paddle 输出与 PyTorch 参考的 [N_atom, 64] material embedding。
结果
指标值阈值状态
max_diff8.05e-071e-4PASS
mean_diff1.82e-071e-4PASS
代码样例
PyTorch 参考数据生成
importnumpyasnpimporttorchfrompymatgen.coreimportStructurefromModel.chgnet_lib.model.model_embeddingimportCHGNet# MatterChat 原始 PT 代码device=torch.device("cuda")
chgnet=CHGNet.load().to(device).float().eval()
struct=Structure.from_file("fix_inputs/GaN.cif")
withtorch.no_grad():
atom_feas=chgnet.predict_structure_embedding(struct)
ifisinstance(atom_feas, (list, tuple)):
atom_feas=atom_feas[0]
# 保存为参考数据ref=atom_feas.detach().cpu().float().numpy()
np.save("fix_outputs/raw_stage_01_material_embed.npy", ref)
print(f"shape={ref.shape}") # -> (4, 64)
PaddlePaddle 对比
importnumpyasnpimportpaddlefrompymatgen.coreimportStructurefromppmat.models.matterchat.chgnet.model.model_embeddingimportCHGNet# 加载 PyTorch 参考数据ref_embed=np.load("fix_outputs/raw_stage_01_material_embed.npy").astype(np.float64)
# 构建 CHGNet 并加载权重 (CPU)paddle.set_device("cpu")
chgnet=CHGNet()
chgnet_weights=load_sharded_weights(prefix="material_encoder.")
set_state_dict_filtered(chgnet, chgnet_weights, prefix="material_encoder.")
chgnet.eval()
# GPU 推理paddle.set_device("gpu")
gpu_chgnet=CHGNet()
gpu_chgnet.set_state_dict({k: v.cuda() fork, vinchgnet.state_dict().items()})
gpu_chgnet.eval()
struct=Structure.from_file("fix_inputs/GaN.cif")
withpaddle.no_grad():
atom_feas=gpu_chgnet.predict_structure_embedding(struct)
# 对比diff=np.abs(atom_feas.numpy().astype(np.float64) -ref_embed)
print(f"max_diff={diff.max():.2e} mean_diff={diff.mean():.2e}")
# -> max_diff=8.05e-07 mean_diff=1.82e-07 (PASS, thr=1e-4)
2. Q-Former Diff
Q-Former 是 BLIP-2 风格的 BERT 变体, 用 32 个 query token 对 CHGNet 输出做 cross-attention, 产生 [1, 32, 768] 的查询表示。验证 Q-Former 前向输出与基线一致。
结果
指标值阈值状态
max_diff0.00e+001e-4PASS
mean_diff0.00e+001e-4PASS
Q-Former 输出与基线完全一致 (diff=0)。
代码样例
PyTorch 参考数据生成
importnumpyasnpimporttorchfromModel.material_Q_former_baseimportBertConfig, BertLMHeadModel# MatterChat 原始 PT 代码device=torch.device("cuda")
# 加载预训练 Q-Former 权重full_ckpt=torch.load("model_weight/model_weights.pkl", map_location="cpu", weights_only=False)
full_state=full_ckpt["state_dict"]
qformer_config=BertConfig.from_pretrained("bert-base-uncased")
qformer_config.encoder_width=64qformer_config.add_cross_attention=Trueqformer_config.cross_attention_freq=2qformer_config.query_length=32qformer=BertLMHeadModel.from_pretrained("bert-base-uncased", config=qformer_config)
qformer_state= {}
fork, vinfull_state.items():
ifk.startswith("model.Qformer.") ork.startswith("model.query_tokens"):
qformer_state[k.replace("model.", "", 1)] =vqformer.load_state_dict(qformer_state, strict=False)
qformer=qformer.to(device).float().eval()
query_tokens=full_state["model.query_tokens"].to(device).float()
# 输入: 上一步 CHGNet 输出的 material embeddingmaterial_embed=torch.from_numpy(
np.load("fix_outputs/raw_stage_01_material_embed.npy")
).to(device).float()
material_att=torch.ones((1, material_embed.shape[0]), dtype=torch.long, device=device)
withtorch.no_grad():
query_output=qformer.bert(
query_embeds=query_tokens,
encoder_hidden_states=material_embed.unsqueeze(0),
encoder_attention_mask=material_att,
return_dict=True,
)
qf_out=query_output.last_hidden_state[:, :query_tokens.shape[1], :]
# 保存为参考数据ref=qf_out.detach().cpu().float().numpy()
np.save("fix_outputs/raw_stage_02_qformer_out.npy", ref)
print(f"shape={ref.shape}") # -> (1, 32, 768)
PaddlePaddle 对比
importnumpyasnpimportpaddlefromppmat.models.matterchat.q_former.q_former_baseimportBertConfig, BertLMHeadModel# 加载基线数据material_embed=np.load("fix_outputs/raw_stage_01_material_embed.npy")
qt_param=np.load("fix_outputs/paddle_stage_02_query_tokens.npy")
# 构建 Q-Former (CPU)paddle.set_device("cpu")
config=BertConfig()
config.encoder_width=64config.add_cross_attention=Trueconfig.cross_attention_freq=2config.query_length=32qformer=BertLMHeadModel(config)
qformer.cls=Noneqformer.bert.embeddings.word_embeddings=Noneqformer.bert.embeddings.position_embeddings=Noneforlayerinqformer.bert.encoder.layer:
layer.output=Nonelayer.intermediate=Noneqformer_weights=load_sharded_weights(prefix="Qformer.")
set_state_dict_filtered(qformer, qformer_weights, prefix="Qformer.")
qformer.eval()
# GPU 推理paddle.set_device("gpu")
gpu_qformer=BertLMHeadModel(config)
gpu_qformer.set_state_dict({k: v.cuda() fork, vinqformer.state_dict().items()})
gpu_qformer.eval()
qt_gpu=paddle.to_tensor(qt_param).cast(paddle.float32).cuda()
emb_gpu=paddle.to_tensor(material_embed).cuda().unsqueeze(0)
att_gpu=paddle.ones([1, emb_gpu.shape[1]], dtype=paddle.int64).cuda()
withpaddle.no_grad():
query_output=gpu_qformer.bert(
query_embeds=qt_gpu,
encoder_hidden_states=emb_gpu,
encoder_attention_mask=att_gpu,
return_dict=True,
)
qf_out=query_output.last_hidden_state[:, :qt_gpu.shape[1], :]
print(f"max_diff={np.abs(qf_out.numpy().astype(np.float64) -ref).max():.2e}")
# -> max_diff=0.00e+00 (PASS, thr=1e-4)
3. Mistral LLM Diff
Mistral-7B 是 32 层 decoder-only Transformer, 含 RoPE、GQA、SwiGLU。由于模型规模大 (7.3B 参数), float16 跨框架累积误差不可避免, 因此阈值放宽至 1e-3。验证逐层 hidden state、最终 hidden_last 及 Top-20 token 匹配。
结果汇总
测试项通过阈值max_diff状态
32 层 hidden state32/321e-32.97e-04 (layer_31)PASS
hidden_last1/11e-31.12e-04PASS
Top-20 token20/20——PASS
逐层误差
层max_diffmean_diff状态
layer_005.14e-077.92e-09PASS
layer_011.16e-041.06e-07PASS
layer_051.22e-041.70e-07PASS
layer_101.22e-042.25e-07PASS
layer_151.22e-043.15e-07PASS
layer_201.22e-045.78e-07PASS
layer_251.22e-048.60e-07PASS
layer_301.06e-041.51e-06PASS
layer_312.97e-041.99e-06PASS
hidden_last1.12e-041.78e-05PASS
代码样例
PyTorch 参考数据生成
importnumpyasnpimporttorchfromtransformersimportLlamaTokenizerfromtransformers.models.mistral.modeling_mistralimportMistralForCausalLMfromtransformersimportMistralConfigasHFMistralConfigdevice=torch.device("cuda")
MODEL_WEIGHT_DIR="model_weight/Mistral-7B-Instruct-v0.3"# Tokenizetokenizer=LlamaTokenizer.from_pretrained(MODEL_WEIGHT_DIR, use_fast=False)
tokenizer.add_special_tokens({"pad_token": "[PAD]", "bos_token": "<s>",
"eos_token": "</s>", "unk_token": "<unk>"})
prompt="[INST] What is the chemical formula of this material? [/INST]"tokens=tokenizer(prompt, return_tensors="pt", truncation=True, max_length=64)
input_ids=tokens["input_ids"].to(device)
# 加载 Mistral-7Bhf_config=HFMistralConfig.from_pretrained(MODEL_WEIGHT_DIR)
hf_config._attn_implementation="eager"llm=MistralForCausalLM.from_pretrained(MODEL_WEIGHT_DIR, config=hf_config,
torch_dtype=torch.float32).to(device).eval()
llm.resize_token_embeddings(len(tokenizer))
# input embeddingswithtorch.no_grad():
inputs_embeds=llm.model.embed_tokens(input_ids)
np.save("fix_outputs/raw_stage_05_input_embeds.npy",
inputs_embeds.detach().cpu().float().numpy())
# 构建因果掩码seq_len=input_ids.shape[1]
min_dtype=torch.finfo(inputs_embeds.dtype).mincausal_mask=torch.full((seq_len, seq_len), min_dtype, dtype=inputs_embeds.dtype, device=device)
causal_mask=torch.triu(causal_mask, diagonal=1)[None, None, :, :].expand(1, 1, -1, -1)
# 逐层前向, 保存每层 hidden state 作为参考hidden_states=inputs_embedsposition_ids=torch.arange(seq_len, device=device).unsqueeze(0)
forliinrange(32):
withtorch.no_grad():
hidden_states=llm.model.layers[li](
hidden_states, attention_mask=causal_mask,
position_ids=position_ids, use_cache=True,
)[0]
np.save(f"fix_outputs/raw_stage_06_hidden_layer{li:02d}.npy",
hidden_states.detach().cpu().float().numpy())
# 最终 norm + hidden_lastwithtorch.no_grad():
hidden_final=llm.model.norm(hidden_states)
np.save("fix_outputs/raw_stage_06_hidden_last.npy",
hidden_final[0, -1].detach().cpu().float().numpy())
PaddlePaddle 对比
importjsonimportnumpyasnpimportpaddlefromppmat.models.matterchat.mistral.configuration_mistralimportMistralConfigfromppmat.models.matterchat.mistral.modeling_mistralimport (
MistralForCausalLM,
MistralDecoderLayer,
MistralRMSNorm,
)
# 加载 PyTorch 参考数据 (逐层 hidden state)refs= {}
forliinrange(32):
refs[f"stage_06_hidden_layer{li:02d}"] =np.load(
f"fix_outputs/raw_stage_06_hidden_layer{li:02d}.npy"
).astype(np.float64)
refs["stage_05_input_embeds"] =np.load(
"fix_outputs/raw_stage_05_input_embeds.npy"
).astype(np.float64)
emb=refs["stage_05_input_embeds"].astype(np.float32)
seq_len=emb.shape[1]
# 构建因果掩码 + position idsmin_dt=float(paddle.finfo(paddle.float32).min)
causal_mask=paddle.triu(
paddle.full([seq_len, seq_len], min_dt, dtype="float32"), diagonal=1
)[None, None].expand([1, 1, -1, -1])
pos_ids=paddle.arange(seq_len).unsqueeze(0)
cache_pos=paddle.arange(seq_len)
# CPU 加载完整 Mistral-7B, 提取每层权重paddle.set_device("cpu")
llm=MistralForCausalLM(MistralConfig(vocab_size=32769))
load_sharded_weights(llm, weight_dir=WEIGHT_DIR)
llm.eval()
layer_sds= [
{k: vfork, vinllm.model.layers[i].state_dict().items()}
foriinrange(32)
]
# 逐层 GPU 推理 + 对比paddle.set_device("gpu")
hidden=paddle.to_tensor(emb).cuda()
cm=causal_mask.cuda()
pos_ids=pos_ids.cuda()
cache_pos=cache_pos.cuda()
forliinrange(32):
gpu_layer=MistralDecoderLayer(llm.config, li)
gpu_layer.set_state_dict({k: v.cuda() fork, vinlayer_sds[li].items()})
gpu_layer.eval()
withpaddle.no_grad():
hidden=gpu_layer(
hidden, attention_mask=cm,
position_ids=pos_ids, use_cache=False,
cache_position=cache_pos,
)[0]
ref=refs[f"stage_06_hidden_layer{li:02d}"]
diff=np.abs(hidden.cpu().numpy().astype(np.float64) -ref)
status="PASS"ifdiff.max() <1e-3else"FAIL"print(f"layer_{li:02d}: max={diff.max():.2e} mean={diff.mean():.2e} [{status}]")
# -> layer_00: max=5.14e-07 mean=7.92e-09 [PASS]# -> layer_01: max=1.16e-04 mean=1.06e-07 [PASS]# -> ...# -> layer_31: max=2.97e-04 mean=1.99e-06 [PASS]
@learncat163
learncat163
commented
Jul 7, 2026

Copy link


Copy Markdown

Author
MatterChat 推理对话示例
1. 单次推理
1.1 一行命令推理
fromppmat.modelsimportbuild_model_from_namefrompymatgen.io.cifimportCifParsermodel, _=build_model_from_name('matterchat_full')
model.to('gpu').eval()
struct=CifParser('Si.cif').get_structures()[0]
print(model.chat(struct, 'what is the chemical formula of this material?'))
# -> The chemical formula of this material is Si.
1.2 指定 config + 本地权重
fromomegaconfimportOmegaConffrompymatgen.io.cifimportCifParserfromppmat.modelsimportbuild_modelfromppmat.utilsimportsave_loadconfig=OmegaConf.to_container(OmegaConf.load('structure_generation/configs/matterchat/matterchat_full.yaml'), resolve=True)
model=build_model(config['Model'])
save_load.load_pretrain(model, './matterchat_full/')
model.to('gpu').eval()
struct=CifParser('Si.cif').get_structures()[0]
print(model.chat(struct, 'what is the chemical formula of this material?'))
2. 多轮对话示例
对同一晶体连续提问多个问题:
fromppmat.modelsimportbuild_model_from_namefrompymatgen.io.cifimportCifParsermodel, _=build_model_from_name("matterchat_full")
model.to("gpu").eval()
struct=CifParser("GaN.cif").get_structures()[0]
# 4 个标准问题prompts= [
"what is the chemical formula of this material?",
"what is the space group of this material?",
"Is this material stable or not?",
"What is the bandgap of this material?",
]
forpromptinprompts:
answer=model.chat(struct, prompt, max_new_tokens=64)
print(f"Q: {prompt}")
print(f"A: {answer}")
print()
输出:
Q: what is the chemical formula of this material?
A: The chemical formula of this material is GaN.
Q: what is the space group of this material?
A: The space group of this material is P6_3mc.
Q: Is this material stable or not?
A: This material is not stable.
Q: What is the bandgap of this material?
A: The bandgap of this material is 1.68300.

3. 批量推理 (多个 CIF)
importosfromppmat.modelsimportbuild_model_from_namefrompymatgen.io.cifimportCifParsermodel, _=build_model_from_name("matterchat_full")
model.to("gpu").eval()
cif_dir="path/to/cif_files"prompt="what is the chemical formula of this material?"forfnameinsorted(os.listdir(cif_dir)):
ifnotfname.endswith(".cif"):
continuestruct=CifParser(os.path.join(cif_dir, fname)).get_structures()[0]
answer=model.chat(struct, prompt, max_new_tokens=64)
print(f"{fname}: {answer}")
@leeleolayleeleolay
changed the title
support matterchat【MIIT program】support matterchatJul 7, 2026
@paddle-bot
paddle-botBot
commented
Jul 14, 2026

Copy link


Copy Markdown

Thanks for your contribution!
@paddle-botpaddle-botBot
added
the 
contributor
External developers
label
Jul 14, 2026
leeleolay
leeleolay
reviewed
Jul 16, 2026
View reviewed changes
@leeleolayleeleolay
left a comment


Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

辛苦修改整体的代码规范符合套件风格
Comment threadppmat/datasets/__init__.py
from ppmat.datasets.oc20_s2ef_dataset import OC20S2EFDataset # noqa
from ppmat.datasets.qm9_dataset import QM9Dataset # noqa
from ppmat.datasets.omol25_dataset import OMol25Dataset
from ppmat.models.matterchat.trainer import MTDataset # noqa
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用已有trainer
Comment threadppmat/datasets/collate_fn.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用默认的collator
Comment threadppmat/models/matterchat/chgnet/graph/__init__.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

套件内已有chgnet，辛苦使用
Comment threadppmat/models/matterchat/chgnet/graph/converter.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用已有的graph_converter
Comment threadppmat/models/matterchat/chgnet/model/dynamics.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ppmatSim已经支持相关的功能，复用已有的
Comment threadppmat/models/matterchat/chgnet/utils/vasp_utils.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vasp在这个模型里的作用是？
Comment threadppmat/models/matterchat/mistral/configuration_mistral.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

不符合已有规范，config不这么处理
@leeleolay
leeleolay
commented
Aug 11, 2026

Copy link


Copy Markdown

Collaborator
@learncat163 重构了部分基础组件，移动了推理器的位置，辛苦基于新的开发和尝试
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@leeleolayleeleolayleeleolay left review comments

At least 1 approving review is required to merge this pull request.


Assignees


No one assigned


Labels

contributorExternal developersMIIT Program

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@learncat163@leeleolay

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
【MIIT program】support matterchat by learncat163 · Pull Request #305 · PaddlePaddle/PaddleMaterials · GitHub
Skip to content
【MIIT program】support matterchat - #305
Open
learncat163 wants to merge 1 commit into
PaddlePaddle:developfrom 
learncat163:pr-matterchat
Open
【MIIT program】support matterchat#305
learncat163 wants to merge 1 commit into
PaddlePaddle:developfrom 
learncat163:pr-matterchat
Conversation
@learncat163
@learncat163learncat163
commented
Jul 7, 2026•

edited

Loading

Copy link


Copy Markdown

MatterChat diff (PyTorch and PaddlePaddle)
AiStudio在线推理案例
MatterChat 由三个模块串联组成: CHGNet (晶体编码器) → Q-Former (跨模态桥接) → Mistral-7B (大语言模型)。本文档分别验证每个模块从 PyTorch 迁移到 PaddlePaddle 后的数值对齐情况。
1. CHGNet Diff
CHGNet 负责将晶体结构 (CIF) 编码为逐原子 embedding。验证用 GaN.cif (4 原子) 作为输入, 对比 Paddle 输出与 PyTorch 参考的 [N_atom, 64] material embedding。
结果
指标值阈值状态
max_diff8.05e-071e-4PASS
mean_diff1.82e-071e-4PASS
代码样例
PyTorch 参考数据生成
importnumpyasnpimporttorchfrompymatgen.coreimportStructurefromModel.chgnet_lib.model.model_embeddingimportCHGNet# MatterChat 原始 PT 代码device=torch.device("cuda")
chgnet=CHGNet.load().to(device).float().eval()
struct=Structure.from_file("fix_inputs/GaN.cif")
withtorch.no_grad():
atom_feas=chgnet.predict_structure_embedding(struct)
ifisinstance(atom_feas, (list, tuple)):
atom_feas=atom_feas[0]
# 保存为参考数据ref=atom_feas.detach().cpu().float().numpy()
np.save("fix_outputs/raw_stage_01_material_embed.npy", ref)
print(f"shape={ref.shape}") # -> (4, 64)
PaddlePaddle 对比
importnumpyasnpimportpaddlefrompymatgen.coreimportStructurefromppmat.models.matterchat.chgnet.model.model_embeddingimportCHGNet# 加载 PyTorch 参考数据ref_embed=np.load("fix_outputs/raw_stage_01_material_embed.npy").astype(np.float64)
# 构建 CHGNet 并加载权重 (CPU)paddle.set_device("cpu")
chgnet=CHGNet()
chgnet_weights=load_sharded_weights(prefix="material_encoder.")
set_state_dict_filtered(chgnet, chgnet_weights, prefix="material_encoder.")
chgnet.eval()
# GPU 推理paddle.set_device("gpu")
gpu_chgnet=CHGNet()
gpu_chgnet.set_state_dict({k: v.cuda() fork, vinchgnet.state_dict().items()})
gpu_chgnet.eval()
struct=Structure.from_file("fix_inputs/GaN.cif")
withpaddle.no_grad():
atom_feas=gpu_chgnet.predict_structure_embedding(struct)
# 对比diff=np.abs(atom_feas.numpy().astype(np.float64) -ref_embed)
print(f"max_diff={diff.max():.2e} mean_diff={diff.mean():.2e}")
# -> max_diff=8.05e-07 mean_diff=1.82e-07 (PASS, thr=1e-4)
2. Q-Former Diff
Q-Former 是 BLIP-2 风格的 BERT 变体, 用 32 个 query token 对 CHGNet 输出做 cross-attention, 产生 [1, 32, 768] 的查询表示。验证 Q-Former 前向输出与基线一致。
结果
指标值阈值状态
max_diff0.00e+001e-4PASS
mean_diff0.00e+001e-4PASS
Q-Former 输出与基线完全一致 (diff=0)。
代码样例
PyTorch 参考数据生成
importnumpyasnpimporttorchfromModel.material_Q_former_baseimportBertConfig, BertLMHeadModel# MatterChat 原始 PT 代码device=torch.device("cuda")
# 加载预训练 Q-Former 权重full_ckpt=torch.load("model_weight/model_weights.pkl", map_location="cpu", weights_only=False)
full_state=full_ckpt["state_dict"]
qformer_config=BertConfig.from_pretrained("bert-base-uncased")
qformer_config.encoder_width=64qformer_config.add_cross_attention=Trueqformer_config.cross_attention_freq=2qformer_config.query_length=32qformer=BertLMHeadModel.from_pretrained("bert-base-uncased", config=qformer_config)
qformer_state= {}
fork, vinfull_state.items():
ifk.startswith("model.Qformer.") ork.startswith("model.query_tokens"):
qformer_state[k.replace("model.", "", 1)] =vqformer.load_state_dict(qformer_state, strict=False)
qformer=qformer.to(device).float().eval()
query_tokens=full_state["model.query_tokens"].to(device).float()
# 输入: 上一步 CHGNet 输出的 material embeddingmaterial_embed=torch.from_numpy(
np.load("fix_outputs/raw_stage_01_material_embed.npy")
).to(device).float()
material_att=torch.ones((1, material_embed.shape[0]), dtype=torch.long, device=device)
withtorch.no_grad():
query_output=qformer.bert(
query_embeds=query_tokens,
encoder_hidden_states=material_embed.unsqueeze(0),
encoder_attention_mask=material_att,
return_dict=True,
)
qf_out=query_output.last_hidden_state[:, :query_tokens.shape[1], :]
# 保存为参考数据ref=qf_out.detach().cpu().float().numpy()
np.save("fix_outputs/raw_stage_02_qformer_out.npy", ref)
print(f"shape={ref.shape}") # -> (1, 32, 768)
PaddlePaddle 对比
importnumpyasnpimportpaddlefromppmat.models.matterchat.q_former.q_former_baseimportBertConfig, BertLMHeadModel# 加载基线数据material_embed=np.load("fix_outputs/raw_stage_01_material_embed.npy")
qt_param=np.load("fix_outputs/paddle_stage_02_query_tokens.npy")
# 构建 Q-Former (CPU)paddle.set_device("cpu")
config=BertConfig()
config.encoder_width=64config.add_cross_attention=Trueconfig.cross_attention_freq=2config.query_length=32qformer=BertLMHeadModel(config)
qformer.cls=Noneqformer.bert.embeddings.word_embeddings=Noneqformer.bert.embeddings.position_embeddings=Noneforlayerinqformer.bert.encoder.layer:
layer.output=Nonelayer.intermediate=Noneqformer_weights=load_sharded_weights(prefix="Qformer.")
set_state_dict_filtered(qformer, qformer_weights, prefix="Qformer.")
qformer.eval()
# GPU 推理paddle.set_device("gpu")
gpu_qformer=BertLMHeadModel(config)
gpu_qformer.set_state_dict({k: v.cuda() fork, vinqformer.state_dict().items()})
gpu_qformer.eval()
qt_gpu=paddle.to_tensor(qt_param).cast(paddle.float32).cuda()
emb_gpu=paddle.to_tensor(material_embed).cuda().unsqueeze(0)
att_gpu=paddle.ones([1, emb_gpu.shape[1]], dtype=paddle.int64).cuda()
withpaddle.no_grad():
query_output=gpu_qformer.bert(
query_embeds=qt_gpu,
encoder_hidden_states=emb_gpu,
encoder_attention_mask=att_gpu,
return_dict=True,
)
qf_out=query_output.last_hidden_state[:, :qt_gpu.shape[1], :]
print(f"max_diff={np.abs(qf_out.numpy().astype(np.float64) -ref).max():.2e}")
# -> max_diff=0.00e+00 (PASS, thr=1e-4)
3. Mistral LLM Diff
Mistral-7B 是 32 层 decoder-only Transformer, 含 RoPE、GQA、SwiGLU。由于模型规模大 (7.3B 参数), float16 跨框架累积误差不可避免, 因此阈值放宽至 1e-3。验证逐层 hidden state、最终 hidden_last 及 Top-20 token 匹配。
结果汇总
测试项通过阈值max_diff状态
32 层 hidden state32/321e-32.97e-04 (layer_31)PASS
hidden_last1/11e-31.12e-04PASS
Top-20 token20/20——PASS
逐层误差
层max_diffmean_diff状态
layer_005.14e-077.92e-09PASS
layer_011.16e-041.06e-07PASS
layer_051.22e-041.70e-07PASS
layer_101.22e-042.25e-07PASS
layer_151.22e-043.15e-07PASS
layer_201.22e-045.78e-07PASS
layer_251.22e-048.60e-07PASS
layer_301.06e-041.51e-06PASS
layer_312.97e-041.99e-06PASS
hidden_last1.12e-041.78e-05PASS
代码样例
PyTorch 参考数据生成
importnumpyasnpimporttorchfromtransformersimportLlamaTokenizerfromtransformers.models.mistral.modeling_mistralimportMistralForCausalLMfromtransformersimportMistralConfigasHFMistralConfigdevice=torch.device("cuda")
MODEL_WEIGHT_DIR="model_weight/Mistral-7B-Instruct-v0.3"# Tokenizetokenizer=LlamaTokenizer.from_pretrained(MODEL_WEIGHT_DIR, use_fast=False)
tokenizer.add_special_tokens({"pad_token": "[PAD]", "bos_token": "<s>",
"eos_token": "</s>", "unk_token": "<unk>"})
prompt="[INST] What is the chemical formula of this material? [/INST]"tokens=tokenizer(prompt, return_tensors="pt", truncation=True, max_length=64)
input_ids=tokens["input_ids"].to(device)
# 加载 Mistral-7Bhf_config=HFMistralConfig.from_pretrained(MODEL_WEIGHT_DIR)
hf_config._attn_implementation="eager"llm=MistralForCausalLM.from_pretrained(MODEL_WEIGHT_DIR, config=hf_config,
torch_dtype=torch.float32).to(device).eval()
llm.resize_token_embeddings(len(tokenizer))
# input embeddingswithtorch.no_grad():
inputs_embeds=llm.model.embed_tokens(input_ids)
np.save("fix_outputs/raw_stage_05_input_embeds.npy",
inputs_embeds.detach().cpu().float().numpy())
# 构建因果掩码seq_len=input_ids.shape[1]
min_dtype=torch.finfo(inputs_embeds.dtype).mincausal_mask=torch.full((seq_len, seq_len), min_dtype, dtype=inputs_embeds.dtype, device=device)
causal_mask=torch.triu(causal_mask, diagonal=1)[None, None, :, :].expand(1, 1, -1, -1)
# 逐层前向, 保存每层 hidden state 作为参考hidden_states=inputs_embedsposition_ids=torch.arange(seq_len, device=device).unsqueeze(0)
forliinrange(32):
withtorch.no_grad():
hidden_states=llm.model.layers[li](
hidden_states, attention_mask=causal_mask,
position_ids=position_ids, use_cache=True,
)[0]
np.save(f"fix_outputs/raw_stage_06_hidden_layer{li:02d}.npy",
hidden_states.detach().cpu().float().numpy())
# 最终 norm + hidden_lastwithtorch.no_grad():
hidden_final=llm.model.norm(hidden_states)
np.save("fix_outputs/raw_stage_06_hidden_last.npy",
hidden_final[0, -1].detach().cpu().float().numpy())
PaddlePaddle 对比
importjsonimportnumpyasnpimportpaddlefromppmat.models.matterchat.mistral.configuration_mistralimportMistralConfigfromppmat.models.matterchat.mistral.modeling_mistralimport (
MistralForCausalLM,
MistralDecoderLayer,
MistralRMSNorm,
)
# 加载 PyTorch 参考数据 (逐层 hidden state)refs= {}
forliinrange(32):
refs[f"stage_06_hidden_layer{li:02d}"] =np.load(
f"fix_outputs/raw_stage_06_hidden_layer{li:02d}.npy"
).astype(np.float64)
refs["stage_05_input_embeds"] =np.load(
"fix_outputs/raw_stage_05_input_embeds.npy"
).astype(np.float64)
emb=refs["stage_05_input_embeds"].astype(np.float32)
seq_len=emb.shape[1]
# 构建因果掩码 + position idsmin_dt=float(paddle.finfo(paddle.float32).min)
causal_mask=paddle.triu(
paddle.full([seq_len, seq_len], min_dt, dtype="float32"), diagonal=1
)[None, None].expand([1, 1, -1, -1])
pos_ids=paddle.arange(seq_len).unsqueeze(0)
cache_pos=paddle.arange(seq_len)
# CPU 加载完整 Mistral-7B, 提取每层权重paddle.set_device("cpu")
llm=MistralForCausalLM(MistralConfig(vocab_size=32769))
load_sharded_weights(llm, weight_dir=WEIGHT_DIR)
llm.eval()
layer_sds= [
{k: vfork, vinllm.model.layers[i].state_dict().items()}
foriinrange(32)
]
# 逐层 GPU 推理 + 对比paddle.set_device("gpu")
hidden=paddle.to_tensor(emb).cuda()
cm=causal_mask.cuda()
pos_ids=pos_ids.cuda()
cache_pos=cache_pos.cuda()
forliinrange(32):
gpu_layer=MistralDecoderLayer(llm.config, li)
gpu_layer.set_state_dict({k: v.cuda() fork, vinlayer_sds[li].items()})
gpu_layer.eval()
withpaddle.no_grad():
hidden=gpu_layer(
hidden, attention_mask=cm,
position_ids=pos_ids, use_cache=False,
cache_position=cache_pos,
)[0]
ref=refs[f"stage_06_hidden_layer{li:02d}"]
diff=np.abs(hidden.cpu().numpy().astype(np.float64) -ref)
status="PASS"ifdiff.max() <1e-3else"FAIL"print(f"layer_{li:02d}: max={diff.max():.2e} mean={diff.mean():.2e} [{status}]")
# -> layer_00: max=5.14e-07 mean=7.92e-09 [PASS]# -> layer_01: max=1.16e-04 mean=1.06e-07 [PASS]# -> ...# -> layer_31: max=2.97e-04 mean=1.99e-06 [PASS]
@learncat163
learncat163
commented
Jul 7, 2026

Copy link


Copy Markdown

Author
MatterChat 推理对话示例
1. 单次推理
1.1 一行命令推理
fromppmat.modelsimportbuild_model_from_namefrompymatgen.io.cifimportCifParsermodel, _=build_model_from_name('matterchat_full')
model.to('gpu').eval()
struct=CifParser('Si.cif').get_structures()[0]
print(model.chat(struct, 'what is the chemical formula of this material?'))
# -> The chemical formula of this material is Si.
1.2 指定 config + 本地权重
fromomegaconfimportOmegaConffrompymatgen.io.cifimportCifParserfromppmat.modelsimportbuild_modelfromppmat.utilsimportsave_loadconfig=OmegaConf.to_container(OmegaConf.load('structure_generation/configs/matterchat/matterchat_full.yaml'), resolve=True)
model=build_model(config['Model'])
save_load.load_pretrain(model, './matterchat_full/')
model.to('gpu').eval()
struct=CifParser('Si.cif').get_structures()[0]
print(model.chat(struct, 'what is the chemical formula of this material?'))
2. 多轮对话示例
对同一晶体连续提问多个问题:
fromppmat.modelsimportbuild_model_from_namefrompymatgen.io.cifimportCifParsermodel, _=build_model_from_name("matterchat_full")
model.to("gpu").eval()
struct=CifParser("GaN.cif").get_structures()[0]
# 4 个标准问题prompts= [
"what is the chemical formula of this material?",
"what is the space group of this material?",
"Is this material stable or not?",
"What is the bandgap of this material?",
]
forpromptinprompts:
answer=model.chat(struct, prompt, max_new_tokens=64)
print(f"Q: {prompt}")
print(f"A: {answer}")
print()
输出:
Q: what is the chemical formula of this material?
A: The chemical formula of this material is GaN.
Q: what is the space group of this material?
A: The space group of this material is P6_3mc.
Q: Is this material stable or not?
A: This material is not stable.
Q: What is the bandgap of this material?
A: The bandgap of this material is 1.68300.

3. 批量推理 (多个 CIF)
importosfromppmat.modelsimportbuild_model_from_namefrompymatgen.io.cifimportCifParsermodel, _=build_model_from_name("matterchat_full")
model.to("gpu").eval()
cif_dir="path/to/cif_files"prompt="what is the chemical formula of this material?"forfnameinsorted(os.listdir(cif_dir)):
ifnotfname.endswith(".cif"):
continuestruct=CifParser(os.path.join(cif_dir, fname)).get_structures()[0]
answer=model.chat(struct, prompt, max_new_tokens=64)
print(f"{fname}: {answer}")
@leeleolayleeleolay
changed the title
support matterchat【MIIT program】support matterchatJul 7, 2026
@paddle-bot
paddle-botBot
commented
Jul 14, 2026

Copy link


Copy Markdown

Thanks for your contribution!
@paddle-botpaddle-botBot
added
the 
contributor
External developers
label
Jul 14, 2026
leeleolay
leeleolay
reviewed
Jul 16, 2026
View reviewed changes
@leeleolayleeleolay
left a comment


Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

辛苦修改整体的代码规范符合套件风格
Comment threadppmat/datasets/__init__.py
from ppmat.datasets.oc20_s2ef_dataset import OC20S2EFDataset # noqa
from ppmat.datasets.qm9_dataset import QM9Dataset # noqa
from ppmat.datasets.omol25_dataset import OMol25Dataset
from ppmat.models.matterchat.trainer import MTDataset # noqa
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用已有trainer
Comment threadppmat/datasets/collate_fn.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用默认的collator
Comment threadppmat/models/matterchat/chgnet/graph/__init__.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

套件内已有chgnet，辛苦使用
Comment threadppmat/models/matterchat/chgnet/graph/converter.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用已有的graph_converter
Comment threadppmat/models/matterchat/chgnet/model/dynamics.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ppmatSim已经支持相关的功能，复用已有的
Comment threadppmat/models/matterchat/chgnet/utils/vasp_utils.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vasp在这个模型里的作用是？
Comment threadppmat/models/matterchat/mistral/configuration_mistral.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

不符合已有规范，config不这么处理
@leeleolay
leeleolay
commented
Aug 11, 2026

Copy link


Copy Markdown

Collaborator
@learncat163 重构了部分基础组件，移动了推理器的位置，辛苦基于新的开发和尝试
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@leeleolayleeleolayleeleolay left review comments

At least 1 approving review is required to merge this pull request.


Assignees


No one assigned


Labels

contributorExternal developersMIIT Program

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@learncat163@leeleolay

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
【MIIT program】support matterchat by learncat163 · Pull Request #305 · PaddlePaddle/PaddleMaterials · GitHub
Skip to content
【MIIT program】support matterchat - #305
Open
learncat163 wants to merge 1 commit into
PaddlePaddle:developfrom 
learncat163:pr-matterchat
Open
【MIIT program】support matterchat#305
learncat163 wants to merge 1 commit into
PaddlePaddle:developfrom 
learncat163:pr-matterchat
Conversation
@learncat163
@learncat163learncat163
commented
Jul 7, 2026•

edited

Loading

Copy link


Copy Markdown

MatterChat diff (PyTorch and PaddlePaddle)
AiStudio在线推理案例
MatterChat 由三个模块串联组成: CHGNet (晶体编码器) → Q-Former (跨模态桥接) → Mistral-7B (大语言模型)。本文档分别验证每个模块从 PyTorch 迁移到 PaddlePaddle 后的数值对齐情况。
1. CHGNet Diff
CHGNet 负责将晶体结构 (CIF) 编码为逐原子 embedding。验证用 GaN.cif (4 原子) 作为输入, 对比 Paddle 输出与 PyTorch 参考的 [N_atom, 64] material embedding。
结果
指标值阈值状态
max_diff8.05e-071e-4PASS
mean_diff1.82e-071e-4PASS
代码样例
PyTorch 参考数据生成
importnumpyasnpimporttorchfrompymatgen.coreimportStructurefromModel.chgnet_lib.model.model_embeddingimportCHGNet# MatterChat 原始 PT 代码device=torch.device("cuda")
chgnet=CHGNet.load().to(device).float().eval()
struct=Structure.from_file("fix_inputs/GaN.cif")
withtorch.no_grad():
atom_feas=chgnet.predict_structure_embedding(struct)
ifisinstance(atom_feas, (list, tuple)):
atom_feas=atom_feas[0]
# 保存为参考数据ref=atom_feas.detach().cpu().float().numpy()
np.save("fix_outputs/raw_stage_01_material_embed.npy", ref)
print(f"shape={ref.shape}") # -> (4, 64)
PaddlePaddle 对比
importnumpyasnpimportpaddlefrompymatgen.coreimportStructurefromppmat.models.matterchat.chgnet.model.model_embeddingimportCHGNet# 加载 PyTorch 参考数据ref_embed=np.load("fix_outputs/raw_stage_01_material_embed.npy").astype(np.float64)
# 构建 CHGNet 并加载权重 (CPU)paddle.set_device("cpu")
chgnet=CHGNet()
chgnet_weights=load_sharded_weights(prefix="material_encoder.")
set_state_dict_filtered(chgnet, chgnet_weights, prefix="material_encoder.")
chgnet.eval()
# GPU 推理paddle.set_device("gpu")
gpu_chgnet=CHGNet()
gpu_chgnet.set_state_dict({k: v.cuda() fork, vinchgnet.state_dict().items()})
gpu_chgnet.eval()
struct=Structure.from_file("fix_inputs/GaN.cif")
withpaddle.no_grad():
atom_feas=gpu_chgnet.predict_structure_embedding(struct)
# 对比diff=np.abs(atom_feas.numpy().astype(np.float64) -ref_embed)
print(f"max_diff={diff.max():.2e} mean_diff={diff.mean():.2e}")
# -> max_diff=8.05e-07 mean_diff=1.82e-07 (PASS, thr=1e-4)
2. Q-Former Diff
Q-Former 是 BLIP-2 风格的 BERT 变体, 用 32 个 query token 对 CHGNet 输出做 cross-attention, 产生 [1, 32, 768] 的查询表示。验证 Q-Former 前向输出与基线一致。
结果
指标值阈值状态
max_diff0.00e+001e-4PASS
mean_diff0.00e+001e-4PASS
Q-Former 输出与基线完全一致 (diff=0)。
代码样例
PyTorch 参考数据生成
importnumpyasnpimporttorchfromModel.material_Q_former_baseimportBertConfig, BertLMHeadModel# MatterChat 原始 PT 代码device=torch.device("cuda")
# 加载预训练 Q-Former 权重full_ckpt=torch.load("model_weight/model_weights.pkl", map_location="cpu", weights_only=False)
full_state=full_ckpt["state_dict"]
qformer_config=BertConfig.from_pretrained("bert-base-uncased")
qformer_config.encoder_width=64qformer_config.add_cross_attention=Trueqformer_config.cross_attention_freq=2qformer_config.query_length=32qformer=BertLMHeadModel.from_pretrained("bert-base-uncased", config=qformer_config)
qformer_state= {}
fork, vinfull_state.items():
ifk.startswith("model.Qformer.") ork.startswith("model.query_tokens"):
qformer_state[k.replace("model.", "", 1)] =vqformer.load_state_dict(qformer_state, strict=False)
qformer=qformer.to(device).float().eval()
query_tokens=full_state["model.query_tokens"].to(device).float()
# 输入: 上一步 CHGNet 输出的 material embeddingmaterial_embed=torch.from_numpy(
np.load("fix_outputs/raw_stage_01_material_embed.npy")
).to(device).float()
material_att=torch.ones((1, material_embed.shape[0]), dtype=torch.long, device=device)
withtorch.no_grad():
query_output=qformer.bert(
query_embeds=query_tokens,
encoder_hidden_states=material_embed.unsqueeze(0),
encoder_attention_mask=material_att,
return_dict=True,
)
qf_out=query_output.last_hidden_state[:, :query_tokens.shape[1], :]
# 保存为参考数据ref=qf_out.detach().cpu().float().numpy()
np.save("fix_outputs/raw_stage_02_qformer_out.npy", ref)
print(f"shape={ref.shape}") # -> (1, 32, 768)
PaddlePaddle 对比
importnumpyasnpimportpaddlefromppmat.models.matterchat.q_former.q_former_baseimportBertConfig, BertLMHeadModel# 加载基线数据material_embed=np.load("fix_outputs/raw_stage_01_material_embed.npy")
qt_param=np.load("fix_outputs/paddle_stage_02_query_tokens.npy")
# 构建 Q-Former (CPU)paddle.set_device("cpu")
config=BertConfig()
config.encoder_width=64config.add_cross_attention=Trueconfig.cross_attention_freq=2config.query_length=32qformer=BertLMHeadModel(config)
qformer.cls=Noneqformer.bert.embeddings.word_embeddings=Noneqformer.bert.embeddings.position_embeddings=Noneforlayerinqformer.bert.encoder.layer:
layer.output=Nonelayer.intermediate=Noneqformer_weights=load_sharded_weights(prefix="Qformer.")
set_state_dict_filtered(qformer, qformer_weights, prefix="Qformer.")
qformer.eval()
# GPU 推理paddle.set_device("gpu")
gpu_qformer=BertLMHeadModel(config)
gpu_qformer.set_state_dict({k: v.cuda() fork, vinqformer.state_dict().items()})
gpu_qformer.eval()
qt_gpu=paddle.to_tensor(qt_param).cast(paddle.float32).cuda()
emb_gpu=paddle.to_tensor(material_embed).cuda().unsqueeze(0)
att_gpu=paddle.ones([1, emb_gpu.shape[1]], dtype=paddle.int64).cuda()
withpaddle.no_grad():
query_output=gpu_qformer.bert(
query_embeds=qt_gpu,
encoder_hidden_states=emb_gpu,
encoder_attention_mask=att_gpu,
return_dict=True,
)
qf_out=query_output.last_hidden_state[:, :qt_gpu.shape[1], :]
print(f"max_diff={np.abs(qf_out.numpy().astype(np.float64) -ref).max():.2e}")
# -> max_diff=0.00e+00 (PASS, thr=1e-4)
3. Mistral LLM Diff
Mistral-7B 是 32 层 decoder-only Transformer, 含 RoPE、GQA、SwiGLU。由于模型规模大 (7.3B 参数), float16 跨框架累积误差不可避免, 因此阈值放宽至 1e-3。验证逐层 hidden state、最终 hidden_last 及 Top-20 token 匹配。
结果汇总
测试项通过阈值max_diff状态
32 层 hidden state32/321e-32.97e-04 (layer_31)PASS
hidden_last1/11e-31.12e-04PASS
Top-20 token20/20——PASS
逐层误差
层max_diffmean_diff状态
layer_005.14e-077.92e-09PASS
layer_011.16e-041.06e-07PASS
layer_051.22e-041.70e-07PASS
layer_101.22e-042.25e-07PASS
layer_151.22e-043.15e-07PASS
layer_201.22e-045.78e-07PASS
layer_251.22e-048.60e-07PASS
layer_301.06e-041.51e-06PASS
layer_312.97e-041.99e-06PASS
hidden_last1.12e-041.78e-05PASS
代码样例
PyTorch 参考数据生成
importnumpyasnpimporttorchfromtransformersimportLlamaTokenizerfromtransformers.models.mistral.modeling_mistralimportMistralForCausalLMfromtransformersimportMistralConfigasHFMistralConfigdevice=torch.device("cuda")
MODEL_WEIGHT_DIR="model_weight/Mistral-7B-Instruct-v0.3"# Tokenizetokenizer=LlamaTokenizer.from_pretrained(MODEL_WEIGHT_DIR, use_fast=False)
tokenizer.add_special_tokens({"pad_token": "[PAD]", "bos_token": "<s>",
"eos_token": "</s>", "unk_token": "<unk>"})
prompt="[INST] What is the chemical formula of this material? [/INST]"tokens=tokenizer(prompt, return_tensors="pt", truncation=True, max_length=64)
input_ids=tokens["input_ids"].to(device)
# 加载 Mistral-7Bhf_config=HFMistralConfig.from_pretrained(MODEL_WEIGHT_DIR)
hf_config._attn_implementation="eager"llm=MistralForCausalLM.from_pretrained(MODEL_WEIGHT_DIR, config=hf_config,
torch_dtype=torch.float32).to(device).eval()
llm.resize_token_embeddings(len(tokenizer))
# input embeddingswithtorch.no_grad():
inputs_embeds=llm.model.embed_tokens(input_ids)
np.save("fix_outputs/raw_stage_05_input_embeds.npy",
inputs_embeds.detach().cpu().float().numpy())
# 构建因果掩码seq_len=input_ids.shape[1]
min_dtype=torch.finfo(inputs_embeds.dtype).mincausal_mask=torch.full((seq_len, seq_len), min_dtype, dtype=inputs_embeds.dtype, device=device)
causal_mask=torch.triu(causal_mask, diagonal=1)[None, None, :, :].expand(1, 1, -1, -1)
# 逐层前向, 保存每层 hidden state 作为参考hidden_states=inputs_embedsposition_ids=torch.arange(seq_len, device=device).unsqueeze(0)
forliinrange(32):
withtorch.no_grad():
hidden_states=llm.model.layers[li](
hidden_states, attention_mask=causal_mask,
position_ids=position_ids, use_cache=True,
)[0]
np.save(f"fix_outputs/raw_stage_06_hidden_layer{li:02d}.npy",
hidden_states.detach().cpu().float().numpy())
# 最终 norm + hidden_lastwithtorch.no_grad():
hidden_final=llm.model.norm(hidden_states)
np.save("fix_outputs/raw_stage_06_hidden_last.npy",
hidden_final[0, -1].detach().cpu().float().numpy())
PaddlePaddle 对比
importjsonimportnumpyasnpimportpaddlefromppmat.models.matterchat.mistral.configuration_mistralimportMistralConfigfromppmat.models.matterchat.mistral.modeling_mistralimport (
MistralForCausalLM,
MistralDecoderLayer,
MistralRMSNorm,
)
# 加载 PyTorch 参考数据 (逐层 hidden state)refs= {}
forliinrange(32):
refs[f"stage_06_hidden_layer{li:02d}"] =np.load(
f"fix_outputs/raw_stage_06_hidden_layer{li:02d}.npy"
).astype(np.float64)
refs["stage_05_input_embeds"] =np.load(
"fix_outputs/raw_stage_05_input_embeds.npy"
).astype(np.float64)
emb=refs["stage_05_input_embeds"].astype(np.float32)
seq_len=emb.shape[1]
# 构建因果掩码 + position idsmin_dt=float(paddle.finfo(paddle.float32).min)
causal_mask=paddle.triu(
paddle.full([seq_len, seq_len], min_dt, dtype="float32"), diagonal=1
)[None, None].expand([1, 1, -1, -1])
pos_ids=paddle.arange(seq_len).unsqueeze(0)
cache_pos=paddle.arange(seq_len)
# CPU 加载完整 Mistral-7B, 提取每层权重paddle.set_device("cpu")
llm=MistralForCausalLM(MistralConfig(vocab_size=32769))
load_sharded_weights(llm, weight_dir=WEIGHT_DIR)
llm.eval()
layer_sds= [
{k: vfork, vinllm.model.layers[i].state_dict().items()}
foriinrange(32)
]
# 逐层 GPU 推理 + 对比paddle.set_device("gpu")
hidden=paddle.to_tensor(emb).cuda()
cm=causal_mask.cuda()
pos_ids=pos_ids.cuda()
cache_pos=cache_pos.cuda()
forliinrange(32):
gpu_layer=MistralDecoderLayer(llm.config, li)
gpu_layer.set_state_dict({k: v.cuda() fork, vinlayer_sds[li].items()})
gpu_layer.eval()
withpaddle.no_grad():
hidden=gpu_layer(
hidden, attention_mask=cm,
position_ids=pos_ids, use_cache=False,
cache_position=cache_pos,
)[0]
ref=refs[f"stage_06_hidden_layer{li:02d}"]
diff=np.abs(hidden.cpu().numpy().astype(np.float64) -ref)
status="PASS"ifdiff.max() <1e-3else"FAIL"print(f"layer_{li:02d}: max={diff.max():.2e} mean={diff.mean():.2e} [{status}]")
# -> layer_00: max=5.14e-07 mean=7.92e-09 [PASS]# -> layer_01: max=1.16e-04 mean=1.06e-07 [PASS]# -> ...# -> layer_31: max=2.97e-04 mean=1.99e-06 [PASS]
@learncat163
learncat163
commented
Jul 7, 2026

Copy link


Copy Markdown

Author
MatterChat 推理对话示例
1. 单次推理
1.1 一行命令推理
fromppmat.modelsimportbuild_model_from_namefrompymatgen.io.cifimportCifParsermodel, _=build_model_from_name('matterchat_full')
model.to('gpu').eval()
struct=CifParser('Si.cif').get_structures()[0]
print(model.chat(struct, 'what is the chemical formula of this material?'))
# -> The chemical formula of this material is Si.
1.2 指定 config + 本地权重
fromomegaconfimportOmegaConffrompymatgen.io.cifimportCifParserfromppmat.modelsimportbuild_modelfromppmat.utilsimportsave_loadconfig=OmegaConf.to_container(OmegaConf.load('structure_generation/configs/matterchat/matterchat_full.yaml'), resolve=True)
model=build_model(config['Model'])
save_load.load_pretrain(model, './matterchat_full/')
model.to('gpu').eval()
struct=CifParser('Si.cif').get_structures()[0]
print(model.chat(struct, 'what is the chemical formula of this material?'))
2. 多轮对话示例
对同一晶体连续提问多个问题:
fromppmat.modelsimportbuild_model_from_namefrompymatgen.io.cifimportCifParsermodel, _=build_model_from_name("matterchat_full")
model.to("gpu").eval()
struct=CifParser("GaN.cif").get_structures()[0]
# 4 个标准问题prompts= [
"what is the chemical formula of this material?",
"what is the space group of this material?",
"Is this material stable or not?",
"What is the bandgap of this material?",
]
forpromptinprompts:
answer=model.chat(struct, prompt, max_new_tokens=64)
print(f"Q: {prompt}")
print(f"A: {answer}")
print()
输出:
Q: what is the chemical formula of this material?
A: The chemical formula of this material is GaN.
Q: what is the space group of this material?
A: The space group of this material is P6_3mc.
Q: Is this material stable or not?
A: This material is not stable.
Q: What is the bandgap of this material?
A: The bandgap of this material is 1.68300.

3. 批量推理 (多个 CIF)
importosfromppmat.modelsimportbuild_model_from_namefrompymatgen.io.cifimportCifParsermodel, _=build_model_from_name("matterchat_full")
model.to("gpu").eval()
cif_dir="path/to/cif_files"prompt="what is the chemical formula of this material?"forfnameinsorted(os.listdir(cif_dir)):
ifnotfname.endswith(".cif"):
continuestruct=CifParser(os.path.join(cif_dir, fname)).get_structures()[0]
answer=model.chat(struct, prompt, max_new_tokens=64)
print(f"{fname}: {answer}")
@leeleolayleeleolay
changed the title
support matterchat【MIIT program】support matterchatJul 7, 2026
@paddle-bot
paddle-botBot
commented
Jul 14, 2026

Copy link


Copy Markdown

Thanks for your contribution!
@paddle-botpaddle-botBot
added
the 
contributor
External developers
label
Jul 14, 2026
leeleolay
leeleolay
reviewed
Jul 16, 2026
View reviewed changes
@leeleolayleeleolay
left a comment


Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

辛苦修改整体的代码规范符合套件风格
Comment threadppmat/datasets/__init__.py
from ppmat.datasets.oc20_s2ef_dataset import OC20S2EFDataset # noqa
from ppmat.datasets.qm9_dataset import QM9Dataset # noqa
from ppmat.datasets.omol25_dataset import OMol25Dataset
from ppmat.models.matterchat.trainer import MTDataset # noqa
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用已有trainer
Comment threadppmat/datasets/collate_fn.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用默认的collator
Comment threadppmat/models/matterchat/chgnet/graph/__init__.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

套件内已有chgnet，辛苦使用
Comment threadppmat/models/matterchat/chgnet/graph/converter.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

使用已有的graph_converter
Comment threadppmat/models/matterchat/chgnet/model/dynamics.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ppmatSim已经支持相关的功能，复用已有的
Comment threadppmat/models/matterchat/chgnet/utils/vasp_utils.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vasp在这个模型里的作用是？
Comment threadppmat/models/matterchat/mistral/configuration_mistral.py
@leeleolayleeleolayJul 16, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

不符合已有规范，config不这么处理
@leeleolay
leeleolay
commented
Aug 11, 2026

Copy link


Copy Markdown

Collaborator
@learncat163 重构了部分基础组件，移动了推理器的位置，辛苦基于新的开发和尝试
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@leeleolayleeleolayleeleolay left review comments

At least 1 approving review is required to merge this pull request.


Assignees


No one assigned


Labels

contributorExternal developersMIIT Program

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@learncat163@leeleolay