🪙 Token::Resolver

if ci_badges.map(&:color).detect { it != "green"} ☝️ let me know on Discord or RubyForum, as I may have missed the notification.

---

if ci_badges.map(&:color).all? { it == "green"} 👇️ send money so I can do more of this. FLOSS maintenance is now my full-time job.

👣 How will this project approach the September 2025 hostile takeover of RubyGems? 🚑️

I've summarized my thoughts in this blog post.

🌻 Synopsis

Token::Resolver is a configurable PEG-based token parser and resolver for structured token detection and replacement in arbitrary text.

Detects structured tokens like {KJ|GEM_NAME} in any file format and resolves them against a replacement map. The token structure (delimiters, separators, segment count) is fully configurable.

# One-liner: parse and resolveresult=Token::Resolver.resolve("Hello {KJ|NAME}, welcome to {KJ|PROJECT}!",{"KJ|NAME"=>"World","KJ|PROJECT"=>"my-app"})# => "Hello World, welcome to my-app!"

💡 Info you can shake a stick at

Tokens to Remember
Works with JRuby
Works with Truffle Ruby
Works with MRI Ruby 4
Works with MRI Ruby 3
Support & Community
Source
Documentation
Compliance
Style
Maintainer 🎖️
... 💖	🧊🐙🛖🧪

Compatibility

Compatible with MRI Ruby 3.2.0+, and concordant releases of JRuby, and TruffleRuby. CI workflows and Appraisals are generated for MRI Ruby 3.2.0+. This test floor is configured by ruby.test_minimum in .kettle-jem.yml and may be higher than the gem's runtime compatibility floor when legacy Rubies are not practical for the current toolchain.

The amazing test matrix is powered by the kettle-dev stack.

How kettle-dev manages complexity in tests

Gem	Source	Role	Total downloads
appraisal2	GitHub	multi-dependency Appraisal matrix generation
appraisal2-rubocop	GitHub	RuboCop Appraisal generator integration
kettle-dev	GitHub	development, release, and CI workflow tooling
kettle-jem	GitHub	Appraisals & CI workflow templates
kettle-soup-cover	GitHub	SimpleCov coverage policy and reporting
kettle-test	GitHub	standard test runner and coverage harness
rubocop-lts	GitHub	Ruby-version-aware linting
turbo_tests2	GitHub	parallel test execution

Federated DVCS

Find this repo on federated forges (Coming soon!)

Federated DVCS Repository	Status	Issues	PRs	Wiki	CI
🧪 kettle-dev/token-resolver on GitLab	The Truth	💚	💚	💚	🐭 Tiny Matrix
🧊 kettle-dev/token-resolver on CodeBerg	An Ethical Mirror (Donate)	💚	💚	➖	⭕️ No Matrix
🐙 kettle-dev/token-resolver on GitHub	Another Mirror	💚	💚	💚	💯 Full Matrix

Enterprise Support

Available as part of the Tidelift Subscription.

Need enterprise-level guarantees?

The maintainers of this and thousands of other packages are working with Tidelift to deliver commercial support and maintenance for the open source packages you use to build your applications. Save time, reduce risk, and improve code health, while paying the maintainers of the exact packages you use.

• 💡Subscribe for support guarantees covering all your FLOSS dependencies
• 💡Tidelift is part of Sonar
• 💡Tidelift pays maintainers to maintain the software you depend on!
  📊@Pointy Haired Boss: An enterprise support subscription is "never gonna let you down", and supports open source maintainers

Alternatively:

• 
• 
• 
• 

✨ Installation

Install the gem and add to the application's Gemfile by executing:

bundle add token-resolver

If bundler is not being used to manage dependencies, install the gem by executing:

gem install token-resolver

⚙️ Configuration

Token Config Options

Option	Default	Description
pre	"{"	Opening delimiter
post	"}"	Closing delimiter
separators	["|"] (pipe)	Segment separators (sequential; last repeats)
min_segments	2	Minimum segments for a valid token
max_segments	nil	Maximum segments (nil = unlimited)
segment_pattern	"[A-Za-z0-9_]"	Parslet character class for valid segment content

Segment Character Constraints

Token segments (the parts between delimiters and separators) only match characters that conform to the segment_pattern. By default, this is word characters: uppercase and lowercase letters, digits, and underscores.

This prevents false positives with syntax that structurally resembles tokens but isn't:

# These are NOT parsed as tokens (spaces, punctuation disqualify them):"items.map { |x| x.to_s }"# Ruby block parameters"${CLASSPATH:+:$CLASSPATH}"# Shell variable expansion"cert_chain.select! { |fp| File.exist? }"# Ruby block with expressions

If you need different characters in your token segments, provide a custom pattern:

# Allow hyphens in segments: {NS|my-key}config=Token::Resolver::Config.new(segment_pattern: "[A-Za-z0-9_-]")

🔧 Basic Usage

Basic Token Resolution

require"token/resolver"# Parse a document to inspect tokensdoc=Token::Resolver.parse("Deploy {KJ|GEM_NAME} to {KJ|GH_ORG}")doc.token_keys# => ["KJ|GEM_NAME", "KJ|GH_ORG"]doc.text_only?# => false# Resolve tokensresult=Token::Resolver.resolve("Deploy {KJ|GEM_NAME} to {KJ|GH_ORG}",{"KJ|GEM_NAME"=>"my-gem","KJ|GH_ORG"=>"my-org"})# => "Deploy my-gem to my-org"

Handling Missing Tokens

# Default: raise on unresolved tokensToken::Resolver.resolve("{KJ|MISSING}",{})# => raises Token::Resolver::UnresolvedTokenError# Keep unresolved tokens as-isToken::Resolver.resolve("{KJ|MISSING}",{},on_missing: :keep)# => "{KJ|MISSING}"# Remove unresolved tokensToken::Resolver.resolve("{KJ|MISSING}",{},on_missing: :remove)# => ""

Custom Token Structure

# Tokens like <<SECTION:NAME>>config=Token::Resolver::Config.new(pre: "<<",post: ">>",separators: [":"])Token::Resolver.resolve("Hello <<NS:NAME>>!",{"NS:NAME"=>"World"},config: config)# => "Hello World!"

Multi-Segment Tokens with Sequential Separators

# Tokens like {KJ|SECTION:SUBSECTION}config=Token::Resolver::Config.new(separators: ["|",":"]# First boundary uses |, second uses :, rest repeat :)doc=Token::Resolver.parse("{KJ|META:AUTHOR}",config: config)doc.tokens.first.key# => "KJ|META:AUTHOR"doc.tokens.first.prefix# => "KJ"doc.tokens.first.segments# => ["KJ", "META", "AUTHOR"]

Step-by-Step API

# Parsedoc=Token::Resolver::Document.new("Hello {KJ|NAME}!")# Inspectdoc.nodes# => [Text("Hello "), Token(["KJ", "NAME"]), Text("!")]doc.tokens# => [Token(["KJ", "NAME"])]doc.token_keys# => ["KJ|NAME"]doc.to_s# => "Hello {KJ|NAME}!" (roundtrip fidelity)# Resolveresolver=Token::Resolver::Resolve.new(on_missing: :raise)result=resolver.resolve(doc,{"KJ|NAME"=>"World"})# => "Hello World!"

🔐 Security

See SECURITY.md.

🤝 Contributing

If you need some ideas of where to help, you could work on adding more code coverage, or if it is already 💯 (see below) check issues or PRs, or use the gem and think about how it could be better.

We so if you make changes, remember to update it.

See CONTRIBUTING.md for more detailed instructions.

🚀 Release Instructions

See CONTRIBUTING.md.

Code Coverage

Coverage service badges

🪇 Code of Conduct

Everyone interacting with this project's codebases, issue trackers, chat rooms and mailing lists agrees to follow the .

🌈 Contributors

Made with contributors-img.

Also see GitLab Contributors: https://gitlab.com/kettle-dev/token-resolver/-/graphs/main

⭐️ Star History

📌 Versioning

This library follows for its public API where practical. For most applications, prefer the Pessimistic Version Constraint with two digits of precision.

For example:

spec.add_dependency("token-resolver","~> 2.0")

📌 Is "Platform Support" part of the public API? More details inside.

Dropping support for a platform can be a breaking change for affected users. If a release changes supported platforms, it should be called out clearly in the changelog and versioned with that impact in mind.

To get a better understanding of how SemVer is intended to work over a project's lifetime, read this article from the creator of SemVer:

• "Major Version Numbers are Not Sacred"

See CHANGELOG.md for a list of releases.

📄 License

The gem is available under the following licenses: AGPL-3.0-only, PolyForm-Small-Business-1.0.0. See LICENSE.md for details.

If none of the available licenses suit your use case, please contact us to discuss a custom commercial license.

© Copyright

See LICENSE.md for the official copyright notice.

Copyright holders

• Required Notice: Copyright (c) 2026 Peter H. Boling

🤑 A request for help

Maintainers have teeth and need to pay their dentists. After getting laid off in an RIF in March, and encountering difficulty finding a new one, I began spending most of my time building open source tools. I'm hoping to be able to pay for my kids' health insurance this month, so if you value the work I am doing, I need your support. Please consider sponsoring me or the project.

To join the community or get help, use the RubyForum or Discord.

To say "thanks!" ☝️ Join the community or 👇️ send money.

💌 💌 💌

Please give the project a star ⭐ ♥.

Many parts of this project are actively managed by a kettle-jem smart template utilizing StructuredMerge.org merge contracts.

Thanks for RTFM. ☺️

Field	Value
Package	token-resolver
Description	🪙 Token::Resolver provides configurable PEG-based (parslet) parsing and resolution of structured tokens (e.g., {KJ|GEM_NAME}) in arbitrary text. Useful for template ETL pipelines where tokens in template files must be resolved before format-specific merging.
Homepage	https://github.com/kettle-dev/token-resolver
Source	https://github.com/kettle-dev/token-resolver
License	AGPL-3.0-only OR PolyForm-Small-Business-1.0.0
Funding	https://github.com/sponsors/pboling, https://ko-fi.com/pboling, https://liberapay.com/pboling/donate, https://opencollective.com/kettle-dev, https://thanks.dev/u/gh/pboling, https://tidelift.com/funding/github/rubygems/token-resolver, https://www.buymeacoffee.com/pboling