Name		Name	Last commit message	Last commit date
example		example
go.mod		go.mod
readme.md		readme.md
tokenizer.go		tokenizer.go

Repository files navigation

Go语言实现大模型之tokenizer组件

tokenizer根据训练数据提取词组信息表，计算机可以通过词表进行输入文本的组词

About

No description, website, or topics provided.

1 star

1 watching

Report repository

Releases

Packages

Used by

Contributors

Languages