Use external hf_tokenizer in llama runner - #9112

Merged

jackzhxng merged 23 commits into

mainfrom

jz/cpp-no-op-tokenizer

Apr 30, 2025

Commits on Apr 9, 2025

Bump external tokenizer submodule version
jackzhxng
committed
4aca3b1
Add stub impl of json tokenizer to llama runner
jackzhxng
committed
13408fb
Cmake
jackzhxng
committed
d790908
Remove stub
jackzhxng
committed
97110ac
Scott pr review
jackzhxng
committed
173308e

Commits on Apr 15, 2025

Target link tokenizers library
jackzhxng
committed
dc8a31a

Commits on Apr 18, 2025

Merge branch 'main' into jz/cpp-no-op-tokenizer
jackzhxng
committed
c8eacc8
Remove unused err
jackzhxng
committed
fda4ea5

Commits on Apr 25, 2025

Merge branch 'main' into jz/cpp-no-op-tokenizer
jackzhxng
committed
6cc6783
Fix merge error
jackzhxng
committed
aac4832

Commits on Apr 28, 2025

Pin bump tokenizers
jackzhxng
committed
f0406e2
Fix qnn build
jackzhxng
committed
7fdb8ae
Nevermind
jackzhxng
committed
cb12e06
Try to fix qnn
jackzhxng
committed
8f0c532
Fix unicode in qnn
jackzhxng
committed
846951e
Fix qaihub
jackzhxng
committed
ac83e12
Fix local shadowing
jackzhxng
committed
c908968

Commits on Apr 29, 2025

Fix duplicate symbol
jackzhxng
committed
30d770b
Bump tokenizers
jackzhxng
committed
6cc40bb
Merge branch 'main' into jz/cpp-no-op-tokenizer
jackzhxng
committed
8be6a6b
Lint
jackzhxng
committed
194c829

Commits on Apr 30, 2025

Merge branch 'main' into jz/cpp-no-op-tokenizer
jackzhxng
committed
73aeb21
Bump tokenizer dep one last time
jackzhxng
committed
4ff5d8b