TokenizersNet

A full-featured .NET tokenization library with complete tokenizer.json compatibility. Implements the composable pipeline architecture — normalizer, pre-tokenizer, model, post-processor, decoder — with full support for BPE, WordPiece, WordLevel, and Unigram models.

Quick start

Load from the hub

usingTokenizersNet;Tokenizertokenizer=awaitTokenizer.FromPretrainedAsync("gpt2");Encodingencoding=tokenizer.Encode("Hello, world!",addSpecialTokens:false);Console.WriteLine(string.Join(", ",encoding.Tokens));// Hello , world , !

Load from a local file

Tokenizertokenizer=Tokenizer.Load("path/to/tokenizer.json");

Encode and decode

// Single sequenceEncodingencoding=tokenizer.Encode("Hey there!",addSpecialTokens:true);Console.WriteLine(encoding.Ids);// token idsConsole.WriteLine(encoding.Tokens);// token stringsConsole.WriteLine(encoding.Offsets);// (start, end) into the original stringConsole.WriteLine(encoding.Words);// word index per token// Sequence pairEncodingpair=tokenizer.EncodePair("Hello","world",addSpecialTokens:true);Console.WriteLine(pair.TypeIds);// 0 for sequence A, 1 for sequence B// BatchIReadOnlyList<Encoding>batch=tokenizer.EncodeBatch(new[]{"First sentence.","Second sentence."},addSpecialTokens:false);// Decodestringtext=tokenizer.Decode(encoding.Ids,skipSpecialTokens:true);

Truncation and padding

tokenizer.WithTruncation(newTruncationParams{MaxLength=512,Strategy=TruncationStrategy.LongestFirst,Direction=TruncationDirection.Right,});tokenizer.WithPadding(newPaddingParams{Strategy=PaddingStrategy.BatchLongest,Direction=PaddingDirection.Right,PadToken="[PAD]",});IReadOnlyList<Encoding>padded=tokenizer.EncodeBatch(sentences,addSpecialTokens:true);

Streaming decode

DecodeStream<IModel,INormalizer,IPreTokenizer,IPostProcessor,IDecoder>stream=tokenizer.DecodeStream(skipSpecialTokens:false);foreach(intidinids){string?chunk=stream.Step(id);if(chunk!=null){Console.Write(chunk);}}

Pipeline architecture

A tokenizer is a composable pipeline of five optional components:

Component	Role	Examples
INormalizer	Text normalization	BertNormalizer, NfcNormalizer, LowercaseNormalizer, SequenceNormalizer
IPreTokenizer	Initial splitting	ByteLevelPreTokenizer, BertPreTokenizer, MetaspacePreTokenizer, SplitPreTokenizer
IModel	Tokenization model	BpeModel, WordPieceModel, WordLevelModel, UnigramModel
IPostProcessor	Special-token insertion	BertPostProcessor, RobertaPostProcessor, TemplatePostProcessor
IDecoder	Token-to-string reconstruction	ByteLevelDecoder, WordPieceDecoder, MetaspaceDecoder, BpeDecoder

Building a tokenizer from scratch

usingTokenizersNet;usingTokenizersNet.Models;usingTokenizersNet.Normalizers;usingTokenizersNet.PreTokenizers;usingTokenizersNet.PostProcessors;usingTokenizersNet.Decoders;// WordPiece (BERT-style)varmodel=newWordPieceModel(vocabulary,newWordPieceModelOptions{UnknownToken="[UNK]",ContinuingSubwordPrefix="##",MaxInputCharsPerWord=100,});Tokenizertokenizer=newTokenizer(model);tokenizer.WithNormalizer(newBertNormalizer());tokenizer.WithPreTokenizer(newBertPreTokenizer());tokenizer.WithPostProcessor(newBertPostProcessor("[CLS]",101,"[SEP]",102));tokenizer.WithDecoder(newWordPieceDecoder());tokenizer.Save("tokenizer.json");

Training

usingTokenizersNet;usingTokenizersNet.Models;usingTokenizersNet.Trainers;usingTokenizersNet.PreTokenizers;usingTokenizersNet.Decoders;Tokenizertokenizer=newTokenizer(newBpeModel());tokenizer.WithPreTokenizer(newByteLevelPreTokenizer());tokenizer.WithDecoder(newByteLevelDecoder());BpeTrainertrainer=newBpeTrainer{VocabSize=30_000,MinFrequency=2,SpecialTokens=new[]{AddedToken.From("<unk>",special:true)},};tokenizer.Train(trainer,new[]{"path/to/corpus.txt"});tokenizer.Save("tokenizer.json");

Added tokens

// Special tokens (bypass pre-tokenizer and model)tokenizer.AddSpecialTokens(new[]{AddedToken.From("[CLS]",special:true),AddedToken.From("[SEP]",special:true),AddedToken.From("[MASK]",special:true),});// Non-special added tokenstokenizer.AddTokens(new[]{AddedToken.From("url"),AddedToken.From("email"),});

Encoding properties

Property	Type	Description
Ids	IReadOnlyList<int>	Token ids
Tokens	IReadOnlyList<string>	Token strings
TypeIds	IReadOnlyList<int>	Sequence index (0 = A, 1 = B)
Words	IReadOnlyList<int?>	Word index per token
Offsets	IReadOnlyList<Offsets>	(Start, End) into the original string
AttentionMask	IReadOnlyList<int>	1 for real tokens, 0 for padding
SpecialTokensMask	IReadOnlyList<int>	1 for special tokens
Overflowing	IReadOnlyList<Encoding>	Overflow encodings when truncating
Length	int	Number of tokens

Running the sample app

dotnet run --project samples/TokenizersNet.Sample.csproj -- serialization
dotnet run --project samples/TokenizersNet.Sample.csproj -- encode-batch
dotnet run --project samples/TokenizersNet.Sample.csproj -- encode-batch path/to/file.txt

---

Acknowledgements

TokenizersNet is a clean-room .NET implementation of the tokenization pipeline originally designed and published by the Hugging Face tokenizers team. The architecture, pipeline model, tokenizer.json format, and algorithmic design all originate from their work. This library exists to bring that same capability natively to the .NET ecosystem.