retriv

Hybrid search for TypeScript/JavaScript projects. AST-aware chunking, camelCase tokenization, local-first with optional cloud backends.

Why?

Building search for TS/JS codebases is harder than it looks:

• Keyword search misses semantic matches — "authentication" won't find verifyCredentials()
• Vector search misses exact identifiers — getUserName returns fuzzy matches instead of the function
• Generic chunkers break code mid-function — you need AST-aware splitting that understands TS/JS syntax

retriv is purpose-built for the JS ecosystem: TypeScript compiler API for AST parsing (zero native deps), automatic camelCase/snake_case tokenization, hybrid BM25+vector search with RRF fusion. Start with SQLite, scale to Turso/Upstash/Cloudflare when needed.

Made possible by my Sponsor Program 💖 Follow me @harlan_zw 🐦 • Join Discord for help

Features

• 🎯 Hybrid search — BM25 keywords + vector semantic, merged via RRF
• 🌳 TS/JS AST chunking — TypeScript compiler API splits on function/class boundaries, extracts signatures and scope (zero native deps)
• 🔤 Code-aware tokenization — getUserName → get User Name getUserName for better BM25 recall
• 📦 Local to cloud — start with SQLite, scale to Turso/Upstash/Cloudflare
• 🔍 Metadata filtering — narrow by file type, path prefix, or custom fields

Installation

pnpm add retriv

Tip

Generate an Agent Skill for this package using skilld:

npx skilld add retriv

Quick Start - Local Hybrid Search

1. Install extra dependencies:

# typescript for AST chunking, sqlite-vec for vector storage, transformers for local embeddings
pnpm add typescript sqlite-vec @huggingface/transformers

2. Create your retriv search instance:

import{createRetriv}from'retriv'import{autoChunker}from'retriv/chunkers/auto'importsqlitefrom'retriv/db/sqlite'import{transformersJs}from'retriv/embeddings/transformers-js'constsearch=awaitcreateRetriv({driver: sqlite({path: './search.db',embeddings: transformersJs(),}),chunking: autoChunker(),// TS/JS AST + markdown splitting})

3. Index documents:

awaitsearch.index([{id: 'src/auth.ts',content: authFileContents,metadata: {type: 'code',lang: 'typescript'}},{id: 'docs/guide.md',content: guideContents,metadata: {type: 'docs'}},])

4. Search!

// hybrid search finds both code and docsconstresults=awaitsearch.search('password hashing',{returnContent: true})// [// {// id: 'src/auth.ts#chunk-2', score: 0.82,// content: 'async function hashPassword(raw: string) {\n ...',// _chunk: {// parentId: 'src/auth.ts', index: 2,// range: [140, 312], lineRange: [12, 28],// entities: [{ name: 'hashPassword', type: 'function' }],// scope: [{ name: 'AuthService', type: 'class' }],// },// },// {// id: 'docs/guide.md#chunk-0', score: 0.71,// content: '## Password Hashing\n\nUse bcrypt with...',// _chunk: { parentId: 'docs/guide.md', index: 0, range: [0, 487] },// },// ]// filter to just code filesawaitsearch.search('getUserName',{filter: {type: 'code'}})// [// { id: 'src/auth.ts#chunk-0', score: 0.91, _chunk: { parentId: 'src/auth.ts', index: 0, range: [0, 139] }},// ]

Split-Category Search

When one category of documents (e.g. prose docs) outnumbers another (e.g. code definitions), the majority can drown out minority results. Split-category search fixes this by running parallel filtered searches per category and fusing with RRF — each category gets equal representation regardless of volume.

constsearch=awaitcreateRetriv({driver: sqlite({path: './search.db',embeddings: transformersJs()}),categories: doc=>doc.metadata?.type||'other',})awaitsearch.index([{id: 'src/auth.ts',content: authCode,metadata: {type: 'code'}},{id: 'docs/guide.md',content: guide,metadata: {type: 'docs'}},{id: 'docs/api.md',content: apiRef,metadata: {type: 'docs'}},])// Code results won't be buried by docs, even when outnumberedawaitsearch.search('authentication')

The categories function receives each document at index time and returns a category string. The category is stored in metadata.category automatically. At search time, one query runs per seen category and results are fused with RRF.

Categories can be derived from any document property:

// By file extension
categories: doc=>/\.(?:ts|js)$/.test(doc.id) ? 'code' : 'docs'// By explicit metadata
categories: doc=>doc.metadata?.category

Cloud Embeddings

pnpm add @ai-sdk/openai ai sqlite-vec

import{openai}from'retriv/embeddings/openai'constsearch=awaitcreateRetriv({driver: sqlite({path: './search.db',embeddings: openai(),// uses OPENAI_API_KEY env}),})

Cloud Vector DB

For serverless or edge deployments, compose separate vector and keyword drivers:

import{createRetriv}from'retriv'importlibsqlfrom'retriv/db/libsql'importsqliteFtsfrom'retriv/db/sqlite-fts'import{openai}from'retriv/embeddings/openai'constsearch=awaitcreateRetriv({driver: {vector: libsql({url: 'libsql://your-db.turso.io',authToken: process.env.TURSO_AUTH_TOKEN,embeddings: openai(),}),keyword: sqliteFts({path: './search.db'}),},})

How It Works

Chunking

Chunking is opt-in via retriv/chunkers/*:

import{autoChunker}from'retriv/chunkers/auto'import{markdownChunker}from'retriv/chunkers/markdown'import{codeChunker}from'retriv/chunkers/typescript'
chunking: autoChunker()// Routes by file extension
chunking: markdownChunker()// Heading-aware splitting
chunking: codeChunker()// AST-aware (TS/JS only)

The autoChunker routes by file extension — .ts, .tsx, .js, .jsx, .mjs, .mts, .cjs, .cts use AST splitting, everything else uses heading-aware markdown splitting.

Query Tokenization

Search queries are automatically expanded for code identifier matching:

Query	Expanded	Why
getUserName	get User Name getUserName	camelCase splitting
MAX_RETRY_COUNT	MAX RETRY COUNT MAX_RETRY_COUNT	snake_case splitting
React.useState	React use State useState	dotted path + camelCase
how to get user	how to get user	Natural language unchanged

This improves BM25 recall on code identifiers while being transparent for natural language queries.

Available standalone:

import{tokenizeCodeQuery}from'retriv/utils/code-tokenize'

Filtering

Narrow search results by metadata using a MongoDB-style filter DSL. Filters are applied at the SQL level (not post-search), so you get exact result counts without over-fetching.

// Attach metadata when indexingawaitsearch.index([{id: 'src/auth.ts',content: authCode,metadata: {type: 'code',lang: 'typescript'}},{id: 'src/api.ts',content: apiCode,metadata: {type: 'code',lang: 'typescript'}},{id: 'docs/guide.md',content: guide,metadata: {type: 'docs',category: 'guide'}},{id: 'docs/api-ref.md',content: apiRef,metadata: {type: 'docs',category: 'reference'}},])// Search only code filesawaitsearch.search('authentication',{filter: {type: 'code'},})// Search only docs under a path prefixawaitsearch.search('authentication',{filter: {type: 'docs',category: {$prefix: 'guide'}},})// Combine multiple conditions (AND)awaitsearch.search('handler',{filter: {type: 'code',lang: {$in: ['typescript','javascript']}},})

When chunking is enabled, chunks inherit their parent document's metadata — so filtering works on chunks too.

Operators

Operator	Example	Description
exact match	{ type: 'code' }	Equals value
$eq	{ type: { $eq: 'code' } }	Equals (explicit)
$ne	{ type: { $ne: 'draft' } }	Not equals
$gt$gte$lt$lte	{ priority: { $gt: 5 } }	Numeric comparisons
$in	{ lang: { $in: ['ts', 'js'] } }	Value in list
$prefix	{ source: { $prefix: 'src/api/' } }	String starts with
$exists	{ deprecated: { $exists: false } }	Field presence check

Multiple keys in a filter are ANDed together.

Per-driver implementation

Driver	Strategy
SQLite hybrid	Native SQL — FTS5JOIN + vec0 rowid IN subquery
SQLite FTS5	Native SQL — JOIN with metadata table
sqlite-vec	Native SQL — rowid IN subquery
pgvector	Native SQL — JSONB WHERE clauses
LibSQL	Native SQL — json_extractWHERE clauses
Upstash	Post-search filtering (4x over-fetch)
Cloudflare	Post-search filtering (4x over-fetch)

Drivers

Hybrid (BM25 + Vector)

Driver	Import	Peer Dependencies
SQLite	retriv/db/sqlite	sqlite-vec (Node.js >= 22.5)

Vector-Only

Driver	Import	Peer Dependencies
LibSQL	retriv/db/libsql	@libsql/client
Upstash	retriv/db/upstash	@upstash/vector
Cloudflare	retriv/db/cloudflare	— (uses Cloudflare bindings)
pgvector	retriv/db/pgvector	pg
sqlite-vec	retriv/db/sqlite-vec	sqlite-vec (Node.js >= 22.5)

Keyword-Only (BM25)

Driver	Import	Peer Dependencies
SQLite FTS5	retriv/db/sqlite-fts	— (Node.js >= 22.5)

Embedding Providers

All vector drivers accept an embeddings config:

Provider	Import	Peer Dependencies
OpenAI	retriv/embeddings/openai	@ai-sdk/openai ai
Google	retriv/embeddings/google	@ai-sdk/google ai
Mistral	retriv/embeddings/mistral	@ai-sdk/mistral ai
Cohere	retriv/embeddings/cohere	@ai-sdk/cohere ai
Ollama	retriv/embeddings/ollama	ollama-ai-provider-v2 ai
Transformers.js	retriv/embeddings/transformers-js	@huggingface/transformers

// Cloud (require API keys)openai({model: 'text-embedding-3-small'})google({model: 'text-embedding-004'})mistral({model: 'mistral-embed'})cohere({model: 'embed-english-v3.0'})// Local (no API key)ollama({model: 'nomic-embed-text'})transformersJs({model: 'Xenova/all-MiniLM-L6-v2'})

Transformers.js runtime options

Retriv passes device and dtype to Transformers.js. Retriv uses fp32 when you omit dtype. Transformers.js selects the device when you omit device.

transformersJs({model: 'bge-base-en-v1.5',device: 'webgpu'})transformersJs({model: 'bge-base-en-v1.5',dtype: 'q8'})

Support depends on your Transformers.js version and runtime. Test the selected combination on the target system.

API

SearchProvider

All drivers implement the same interface:

interfaceSearchProvider{index: (docs: Document[])=>Promise<{count: number}>search: (query: string,options?: SearchOptions)=>Promise<SearchResult[]>remove?: (ids: string[])=>Promise<{count: number}>clear?: ()=>Promise<void>close?: ()=>Promise<void>}

SearchOptions

interfaceSearchOptions{limit?: number// Max results (default varies by driver)returnContent?: boolean// Include original content in resultsreturnMetadata?: boolean// Include metadata in resultsreturnMeta?: boolean// Include driver-specific _metafilter?: SearchFilter// Filter by metadata fields}

SearchResult

interfaceSearchResult{id: string// Document ID (or chunk ID like "src/auth.ts#chunk-0")score: number// 0-1, higher is bettercontent?: string// If returnContent: truemetadata?: Record<string,any>// If returnMetadata: true_chunk?: ChunkInfo// When chunking enabled (see below)_meta?: SearchMeta// If returnMeta: true (driver-specific extras)}

ChunkInfo

When chunking is enabled, each result includes _chunk with source mapping and AST metadata:

interfaceChunkInfo{parentId: string// Original document IDindex: number// Chunk position (0-based)range?: [number,number]// Character range in original contentlineRange?: [number,number]// Line range in original contententities?: ChunkEntity[]// Functions, classes, methods defined in this chunkscope?: ChunkEntity[]// Containing scope chain (e.g. class this method is inside)}interfaceChunkEntity{name: string// e.g. "hashPassword"type: string// e.g. "function", "class", "method"signature?: string// e.g. "async hashPassword(raw: string): Promise<string>"isPartial?: boolean// true if entity was split across chunks}

Code chunks also produce imports and siblings on the ChunkerChunk level (available when writing custom chunkers):

interfaceChunkerChunk{text: stringlineRange?: [number,number]context?: string// Contextualized prefix for embeddingsentities?: ChunkEntity[]scope?: ChunkEntity[]imports?: ChunkImport[]// { name, source, isDefault?, isNamespace? }siblings?: ChunkSibling[]// { name, type, position: 'before'|'after', distance }}

Related

• skilld — Generate agent skills from npm package docs, uses retriv for search

Sponsors

License

Licensed under the MIT license.