go-token-boundary

A Go-based CLI tool for analyzing and visualizing token boundaries in natural language text using different tokenization strategies.

Features

• Multiple Tokenization Strategies:

  • Whitespace: Splits text on whitespace boundaries (traditional word tokenization)
  • Unicode Grapheme: Splits text into Unicode grapheme clusters (handles combining characters, emoji, etc.)
  • BPE-like Heuristic: Subword tokenization using deterministic heuristics (handles camelCase, suffixes, punctuation)

• Comprehensive Analysis:

  • Token boundary maps showing exact positions
  • Detailed statistics (token count, lengths, character types, Unicode handling)
  • Ambiguity zone detection (positions where strategies disagree)

• Output Formats:

  • Human-readable text output
  • JSON output (compact or pretty-printed)

• Unicode Support:

  • Proper handling of non-ASCII characters
  • Combining marks and diacritics
  • Multi-byte characters (Chinese, Japanese, emoji, etc.)

Installation

From Source

git clone https://github.com/BaseMax/go-token-boundary.git
cd go-token-boundary
go build -o go-token-boundary ./cmd/go-token-boundary

Using Go Install

go install github.com/BaseMax/go-token-boundary/cmd/go-token-boundary@latest

Usage

Basic Usage

# Analyze a simple sentence
go-token-boundary "Hello, world!"

# Analyze with JSON output
go-token-boundary --format json "Hello, world!"

# Pretty-printed JSON
go-token-boundary --format json --pretty "Hello, world!"

Reading from stdin

# Pipe text to the tool
echo "tokenization test" | go-token-boundary --stdin

# Read from a file
cat myfile.txt | go-token-boundary --stdin

Examples

Example 1: Simple Text

$ go-token-boundary "Hello, world!"

Output shows:

• Whitespace tokenizer: [Hello,] | [world!]
• Grapheme tokenizer: Each character as a separate token
• BPE tokenizer: [Hello] | [,] | [ ] | [world] | [!]

Example 2: CamelCase Handling

$ go-token-boundary "parseHTMLDocument"

The BPE tokenizer intelligently splits:

• [parse] | [HTML] | [Docu] | [ment]

Example 3: Unicode Text

$ go-token-boundary "Héllo wörld! 你好"

Properly handles:

• Accented characters (é, ö)
• Multi-byte Unicode (你好)
• Provides Unicode character count statistics

Example 4: Suffix Detection

$ go-token-boundary "testing running"

BPE tokenizer splits common suffixes:

• [test] | [ing] | [ ] | [run] | [ning]

Command-Line Options

Option	Description
--format <format>	Output format: text (default) or json
--pretty	Pretty print JSON output (only with --format json)
--stdin	Read input from stdin instead of command arguments
--version	Print version and exit
--help	Show help message

Output Format

Text Output

The text output includes:

1. Boundary Maps: For each strategy, shows tokens and boundary positions
2. Statistics: Comprehensive statistics including:
   • Token count and unique token count
   • Average, minimum, and maximum token lengths
   • Character type counts (alphabetic, numeric, punctuation, whitespace)
   • Unicode (non-ASCII) character count
3. Ambiguity Zones: Positions where different strategies disagree, with context and reasoning

JSON Output

The JSON output provides a structured representation of the analysis:

{
  "Input": "Hello world",
  "BoundaryMaps": {
    "whitespace": { ... },
    "grapheme": { ... },
    "bpe": { ... }
  },
  "Statistics": {
    "whitespace": { ... },
    "grapheme": { ... },
    "bpe": { ... }
  },
  "AmbiguityZones": [ ... ]
}

Tokenization Strategies

Whitespace Tokenizer

Splits text on Unicode whitespace characters. This is the traditional approach used in many NLP applications.

Example:

• Input: "Hello, world!"
• Tokens: ["Hello,", "world!"]

Grapheme Tokenizer

Splits text into Unicode grapheme clusters - the smallest units that represent a single visual character. Properly handles:

• Combining diacritical marks (é = e + ́)
• Multi-codepoint emoji
• Special sequences like CRLF

Example:

• Input: "Héllo"
• Tokens: ["H", "é", "l", "l", "o"] (é is treated as one grapheme)

BPE-like Tokenizer

Uses deterministic heuristics to perform subword tokenization similar to Byte Pair Encoding, but without requiring trained merge rules:

• Splits on punctuation
• Splits on case changes (camelCase, PascalCase)
• Splits common suffixes (-ing, -ed, -tion, etc.)
• Handles number sequences
• Processes whitespace separately

Example:

• Input: "parseHTMLDocument"
• Tokens: ["parse", "HTML", "Docu", "ment"]

Ambiguity Zones

Ambiguity zones identify positions where different tokenization strategies disagree. This helps understand:

• Where tokenization decisions are non-trivial
• Potential issues in multi-strategy NLP pipelines
• Linguistic complexity in the input text

Each ambiguity zone includes:

• Position in the text
• Length of the ambiguous region
• Context (surrounding text)
• Reason for the ambiguity
• Which strategies have a boundary at that position

Use Cases

• NLP Development: Compare tokenization approaches for your text data
• Linguistic Analysis: Understand how different strategies handle complex text
• Unicode Testing: Verify proper handling of international text
• Education: Learn about tokenization and its challenges
• Debugging: Diagnose tokenization issues in NLP pipelines

Development

Running Tests

go test -v ./tokenizer

Building

go build -o go-token-boundary ./cmd/go-token-boundary

Project Structure

.
├── cmd/
│   └── go-token-boundary/
│       └── main.go          # CLI entry point
├── tokenizer/
│   ├── types.go             # Core types and interfaces
│   ├── whitespace.go        # Whitespace tokenizer
│   ├── grapheme.go          # Grapheme tokenizer
│   ├── bpe.go               # BPE-like tokenizer
│   ├── analyzer.go          # Analysis and ambiguity detection
│   ├── json.go              # JSON formatting
│   └── tokenizer_test.go    # Tests
├── go.mod
└── README.md

Technical Details

Linguistic Correctness

• Proper Unicode grapheme cluster segmentation
• Handles combining marks and diacritics correctly
• Recognizes Unicode character categories
• Supports all Unicode scripts

Performance

• Pure Go implementation with no external dependencies
• Efficient rune-based processing
• Suitable for interactive CLI use

No ML Required

This is a purely deterministic analysis tool. It doesn't require:

• Pre-trained models
• Training data
• External ML libraries

All tokenization strategies are rule-based and deterministic.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Author

Max Base (@BaseMax)

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.