Korean text normalization and language preparation package for LM in Kaldi-based ASR system
-
Updated
Apr 23, 2020 - Python
#
subword
Here are 8 public repositories matching this topic...
Simple-to-use scoring function for arbitrarily tokenized texts.
-
Updated
Feb 19, 2025 - Python
Subword-augmented Embedding for Cloze Reading Comprehension (COLING 2018)
-
Updated
Nov 6, 2018 - Python
johnny - a neural network graph based DEPendency Parser
-
Updated
Mar 25, 2021 - Python
A framework for generating subword vocabulary from a tensorflow dataset and building custom BERT tokenizer models.
machine-learning deep-learning tensorflow machine-translation vocabulary-builder bert subword wordpiece berttokenizer tensorflow-text
-
Updated
Jul 6, 2021 - Python
200k-vocab SentencePiece (Unigram) tokenizer for German-primary LLMs — German/English/code, low fertility, byte-fallback, chat-template tokens. From the Auralis/Helix project.
-
Updated
Jun 19, 2026 - Python
Com la tokenització fractura la morfologia catalana i si una segmentació conscient dels morfemes recupera la geometria. Provat en 3 llengües indoeuropees (català, castellà, anglès): el català es fragmenta ~1,7× més que l'anglès; forçar el tall morfèmic recupera la composicionalitat (robust a portadora i replicat en castellà).
nlp morphology catalan gemma tokenization subword salamandra llm qwen low-resource-language representation-geometry open-weight-models
-
Updated
May 28, 2026 - Python
Subword Neural Machine Translation
-
Updated
Jun 20, 2017 - Python
Add this topic to your repo
To associate your repository with the subword topic, visit your repo's landing page and select "manage topics."