chunk-smart

Structure-aware text chunker for RAG pipelines.

chunk-smart detects the content type of input text -- markdown, code, JSON, HTML, YAML, or plain text -- and splits it at natural structural boundaries rather than blindly by character count. Markdown is split at heading boundaries, code at function and class boundaries, JSON at top-level keys or array element groups, and plain text at paragraph and sentence boundaries. Every chunk carries rich metadata including positional offsets, token counts, heading context, and overlap indicators.

The package has zero runtime dependencies. All detection and splitting logic uses hand-written scanners and regex patterns. Token counting uses an approximate heuristic (1 token per 4 characters) suitable for most LLM tokenizers. The same input with the same options always produces the same output -- no LLM calls, no network access, fully deterministic.

Installation

npm install chunk-smart

Quick Start

import{chunk}from'chunk-smart';consttext=`# IntroductionThis is the first section with some content.## DetailsHere are the details of the implementation.## ConclusionFinal thoughts on the topic.`;constchunks=chunk(text);for(constcofchunks){console.log(c.metadata.index,c.metadata.contentType,c.metadata.tokenCount,c.content.slice(0,60));}

Output:

0 markdown 15 # Introduction\n\nThis is the first section with some c
1 markdown 14 ## Details\n\nHere are the details of the implementatio
2 markdown 10 ## Conclusion\n\nFinal thoughts on the topic.

Features

• Auto-detection -- Identifies content type (markdown, code, JSON, HTML, YAML, plain text) from structural markers with a confidence score.
• Structure-aware splitting -- Splits at headings, function/class boundaries, JSON keys, paragraph breaks, and sentence endings instead of arbitrary character positions.
• Rich metadata -- Every chunk includes its sequential index, character offsets in the original text, token count, character count, content type, heading context (markdown), detected code language, and overlap indicators.
• Configurable chunk sizing -- Control maximum and minimum token counts per chunk, with an approximate tokenizer (1 token = 4 characters).
• Overlap support -- Configurable token-based overlap between adjacent chunks for continuity in retrieval pipelines.
• Factory pattern -- createChunker produces a reusable chunker instance with preset defaults, avoiding repeated option parsing.
• Type-specific methods -- Dedicated chunkMarkdown, chunkCode, and chunkJSON methods bypass auto-detection when you know the input format.
• Zero dependencies -- No runtime dependencies. Ships as compiled CommonJS with TypeScript declarations.
• Deterministic -- Same input and options always produce the same output. No LLM calls, no network access.

API Reference

chunk(text, options?)

Splits a text string into an array of Chunk objects. Auto-detects content type unless contentType is specified in options.

Signature:

functionchunk(text: string,options?: ChunkOptions): Chunk[]

Parameters:

Parameter	Type	Description
text	string	The text to split into chunks.
options	ChunkOptions	Optional configuration for chunking behavior.

Returns:Chunk[] -- An array of chunk objects, each containing content and metadata.

Example:

import{chunk}from'chunk-smart';// Auto-detect content typeconstchunks=chunk('# Hello\n\nWorld.\n\n## Section\n\nDetails.');// Force content type and limit chunk sizeconstsmallChunks=chunk(longText,{maxTokens: 256,contentType: 'markdown',});// Enable overlap between chunksconstoverlapping=chunk(document,{maxTokens: 512,overlap: 50,});

---

createChunker(defaultOptions?)

Creates a reusable chunker instance with preset default options. All methods on the returned Chunker accept optional overrides that are merged with the defaults.

Signature:

functioncreateChunker(defaultOptions?: ChunkOptions): Chunker

Parameters:

Parameter	Type	Description
defaultOptions	ChunkOptions	Default options applied to every call on the returned chunker.

Returns:Chunker -- An object with chunk, chunkMarkdown, chunkCode, chunkJSON, and detectContentType methods.

Example:

import{createChunker}from'chunk-smart';constchunker=createChunker({maxTokens: 256,overlap: 20});constmdChunks=chunker.chunkMarkdown(markdownText);constcodeChunks=chunker.chunkCode(sourceCode);constjsonChunks=chunker.chunkJSON(jsonString);constautoChunks=chunker.chunk(unknownText);

---

Chunker Interface

The object returned by createChunker.

interfaceChunker{chunk(text: string,overrides?: Partial<ChunkOptions>): Chunk[];chunkMarkdown(text: string,options?: Partial<ChunkOptions>): Chunk[];chunkCode(text: string,options?: Partial<ChunkOptions>): Chunk[];chunkJSON(text: string,options?: Partial<ChunkOptions>): Chunk[];detectContentType(text: string): DetectResult;}

Method	Description
chunk	Auto-detects content type and splits accordingly.
chunkMarkdown	Forces contentType: 'markdown' and splits at heading boundaries, then paragraphs, then sentences.
chunkCode	Forces contentType: 'code' and splits at function, class, def, const, let, var boundaries, then blank lines.
chunkJSON	Forces contentType: 'json' and splits at top-level object keys or array element groups. Falls back to token-based splitting for invalid JSON.
detectContentType	Returns the detected content type and confidence score without chunking.

---

detectContentType(text)

Analyzes text and returns the detected content type with a confidence score.

Signature:

functiondetectContentType(text: string): DetectResult

Parameters:

Parameter	Type	Description
text	string	The text to analyze.

Returns:DetectResult -- An object with type and confidence.

Detection heuristics:

Content Type	Signals	Confidence
json	Starts with { or [; valid JSON.parse	0.95
json	Starts with { or [; invalid parse	0.70
html	<!DOCTYPE html>, <html>, or common block tags (div, p, span, body, etc.)	0.90
yaml	--- marker or 2+ key: value lines; no < characters	0.80
markdown	# headings, triple backtick fences, **bold**, list items, [link](url) (score >= 2)	0.80
code	function, class, def, import, const/let/var assignments, indented {}; lines (score >= 2)	0.70
text	No structural markers detected	0.50

Example:

import{detectContentType}from'chunk-smart';detectContentType('{"key": "value"}');// { type: 'json', confidence: 0.95 }detectContentType('# Title\n\nParagraph text.');// { type: 'markdown', confidence: 0.8 }detectContentType('function greet() { return "hi"; }');// { type: 'code', confidence: 0.7 }detectContentType('Just plain text with no markers.');// { type: 'text', confidence: 0.5 }

Configuration

ChunkOptions

Option	Type	Default	Description
maxTokens	number	512	Maximum tokens per chunk. 1 token is approximately 4 characters.
minTokens	number	50	Minimum tokens per chunk (informational).
overlap	number	0	Number of tokens of overlap between adjacent chunks.
contentType	ContentType	auto-detected	Force a specific content type instead of auto-detecting. One of 'markdown', 'code', 'html', 'json', 'yaml', 'text'.
preserveStructure	boolean	true	When true, uses boundary-aware splitting (paragraphs, sentences). When false, falls back to fixed-size token splitting for html, yaml, and text types.

ContentType

typeContentType='markdown'|'code'|'html'|'json'|'yaml'|'text'

Content Type Splitting Strategies

Type	Primary Boundary	Secondary Boundary	Tertiary Boundary
markdown	# headings	Double newlines (paragraphs)	Sentence endings
code	function/class/def/const/let/var declarations	Blank lines	Character-level hard split
json	Top-level object keys or array element groups	Token-based fallback (invalid JSON)	--
html	Paragraphs	Sentences	Word boundaries
yaml	Paragraphs	Sentences	Word boundaries
text	Double newlines (paragraphs)	Sentence endings (.!?)	Word boundaries

Types

Chunk

interfaceChunk{content: string;metadata: ChunkMetadata;}

ChunkMetadata

interfaceChunkMetadata{index: number;// Sequential position in the result array (0-based)startOffset: number;// Character offset of chunk start in the original textendOffset: number;// Character offset of chunk end in the original texttokenCount: number;// Approximate token count: Math.ceil(content.length / 4)charCount: number;// Character count: content.lengthcontentType: ContentType;// Detected or specified content typeheadings: string[];// Headings found within the chunk (markdown only)codeLanguage?: string;// Language detected from ``` fence or #! shebang (code only)overlapBefore: number;// Characters of overlap with the previous chunkoverlapAfter: number;// Characters of overlap with the next chunk}

DetectResult

interfaceDetectResult{type: ContentType;confidence: number;// 0.0 to 1.0}

Error Handling

chunk-smart handles edge cases gracefully without throwing exceptions:

• Empty string -- Returns an empty array [].
• Whitespace-only input -- Returns an empty array [].
• Invalid JSON with contentType: 'json' -- Falls back to token-based splitting.
• Single word exceeding maxTokens -- Hard-splits at the character limit to guarantee every chunk respects the size constraint.
• No structural boundaries found -- Falls back to the next finer boundary level (paragraphs to sentences to words to characters).

Advanced Usage

Chunking a Codebase for Retrieval

import{createChunker}from'chunk-smart';import{readFileSync}from'node:fs';constchunker=createChunker({maxTokens: 512,overlap: 30});constsource=readFileSync('src/parser.ts','utf-8');constchunks=chunker.chunkCode(source);for(constcofchunks){console.log(`Chunk ${c.metadata.index}: ${c.metadata.tokenCount} tokens`);if(c.metadata.codeLanguage){console.log(` Language: ${c.metadata.codeLanguage}`);}}

Processing Large JSON API Responses

import{chunk}from'chunk-smart';constapiResponse=JSON.stringify(largeDataset,null,2);constchunks=chunk(apiResponse,{maxTokens: 1024,contentType: 'json',});// Each chunk is valid JSON (subset of top-level keys or array elements)for(constcofchunks){constparsed=JSON.parse(c.content);console.log(`Chunk ${c.metadata.index}: ${Object.keys(parsed).length} keys`);}

Markdown Documentation with Heading Context

import{chunk}from'chunk-smart';constdocs=readFileSync('API.md','utf-8');constchunks=chunk(docs,{maxTokens: 256,contentType: 'markdown'});for(constcofchunks){if(c.metadata.headings.length>0){console.log(`Section: ${c.metadata.headings.join(' > ')}`);}console.log(` Offset: ${c.metadata.startOffset}-${c.metadata.endOffset}`);console.log(` Tokens: ${c.metadata.tokenCount}`);}

Disabling Structure-Aware Splitting

import{chunk}from'chunk-smart';// Fixed-size token splitting without boundary awarenessconstchunks=chunk(text,{maxTokens: 128,overlap: 20,preserveStructure: false,contentType: 'text',});

Using Per-Call Overrides with a Chunker Instance

import{createChunker}from'chunk-smart';constchunker=createChunker({maxTokens: 512});// Override maxTokens for a single callconstsmall=chunker.chunk(text,{maxTokens: 128});// Detect content type without chunkingconstdetected=chunker.detectContentType(unknownInput);console.log(detected.type,detected.confidence);

TypeScript

chunk-smart is written in TypeScript with strict mode enabled. Type declarations are shipped alongside the compiled JavaScript in the dist/ directory.

All public types are exported from the package entry point:

import{chunk,createChunker,detectContentType,}from'chunk-smart';importtype{ContentType,DetectResult,ChunkMetadata,Chunk,ChunkOptions,Chunker,}from'chunk-smart';

License

MIT