Bundle of more than 30 new text objects for Neovim.
-
Updated
Aug 3, 2026 - Lua
#
subword
Here are 22 public repositories matching this topic...
Korean text normalization and language preparation package for LM in Kaldi-based ASR system
-
Updated
Apr 23, 2020 - Python
Simple-to-use scoring function for arbitrarily tokenized texts.
-
Updated
Feb 19, 2025 - Python
Keyword Search Recipe for Subword ASR
-
Updated
Jul 12, 2019 - Shell
Subword-augmented Embedding for Cloze Reading Comprehension (COLING 2018)
-
Updated
Nov 6, 2018 - Python
johnny - a neural network graph based DEPendency Parser
-
Updated
Mar 25, 2021 - Python
Effective Subword Segmentation for Text Comprehension (TASLP 2019)
-
Updated
Nov 6, 2018 - C++
Unsupervised Word Segmentation using Minimum Description Length for Neural Machine Translation (NMT)
-
Updated
Dec 21, 2018 - C++
A causal intervention framework to learn robust and interpretable character representations inside subword-based language models
-
Updated
Jul 10, 2023 - Jupyter Notebook
A subword-level GRU language model inference engine.
language cpp fortran engine language-learning gru rnn language-model rnn-model subword subword-tokenization
-
Updated
Aug 30, 2026 - C++
An implementation of subword division algorithm proposed in T. Mikolov (2012).
-
Updated
Sep 25, 2019 - HTML
-
Updated
Jan 8, 2023 - Jupyter Notebook
A framework for generating subword vocabulary from a tensorflow dataset and building custom BERT tokenizer models.
machine-learning deep-learning tensorflow machine-translation vocabulary-builder bert subword wordpiece berttokenizer tensorflow-text
-
Updated
Jul 6, 2021 - Python
This repository contains source code implementation of assignments for NTU's MSAI course AI6127 on Deep Neural Networks for Natural Language Processing (2019 Sem 2).
-
Updated
Dec 11, 2020 - Jupyter Notebook
🧠 Build and explore a minimal Byte Pair Encoding tokenizer in TypeScript, training and encoding text using raw UTF-8 bytes without external libraries.
nlp open-source education encoding machine-learning typescript ai compiler decoding developer-tools text-processing language-models utf8 subword bun bpe byte-pair-encoding llm
-
Updated
Sep 15, 2026 - TypeScript
Tokenization is a way of separating a piece of text into smaller units called tokens. Here, tokens can be either words, characters, or subwords. Hence, tokenization can be broadly classified into 3 types – word, character, and subword (n-gram characters) tokenization.
cat nlp count tensorflow tokenizer natural-language character sentence keras-classification-models subword nerual-network imdb-dataset deep-learning-architectures rnn-keras smaller-units tokenizer-nlp
-
Updated
Jun 30, 2021 - Jupyter Notebook
200k-vocab SentencePiece (Unigram) tokenizer for German-primary LLMs — German/English/code, low fertility, byte-fallback, chat-template tokens. From the Auralis/Helix project.
-
Updated
Jun 19, 2026 - Python
Com la tokenització fractura la morfologia catalana i si una segmentació conscient dels morfemes recupera la geometria. Provat en 3 llengües indoeuropees (català, castellà, anglès): el català es fragmenta ~1,7× més que l'anglès; forçar el tall morfèmic recupera la composicionalitat (robust a portadora i replicat en castellà).
nlp morphology catalan gemma tokenization subword salamandra llm qwen low-resource-language representation-geometry open-weight-models
-
Updated
May 28, 2026 - Python
Subword Neural Machine Translation
-
Updated
Jun 20, 2017 - Python
Byte-pair encoding (BPE) subword tokenizer in pure Standard ML: train, encode, decode. Deterministic, MLton + Poly/ML.
nlp library tokenizer pure sml standard-ml mlton deterministic subword polyml bpe byte-pair-encoding sjqtentacles-web
-
Updated
Jul 11, 2026 - Standard ML
Add this topic to your repo
To associate your repository with the subword topic, visit your repo's landing page and select "manage topics."