Skip to content
BranchesTags

Repository files navigation

embed-drift

Detect embedding model changes and distribution shifts before they silently degrade your retrieval quality.

npm versionnpm downloadslicensenode

Description

When an embedding model changes -- OpenAI's text-embedding-ada-002 to text-embedding-3-small, a Cohere version bump, or any silent provider update -- the vectors already stored in your database become incompatible with newly produced vectors. Queries return wrong results. No error is thrown, no status code changes, no log line appears. The system looks healthy. The results are wrong.

embed-drift detects this failure before it reaches your users. It monitors embedding distributions over time through two complementary mechanisms:

Canary-based detection embeds a fixed set of reference texts, stores the resulting vectors, and later re-embeds the same texts to check whether the model has changed. This is cheap (embeds only 25 canary texts, not the entire corpus) and catches model changes on the very next check.

Statistical snapshot comparison captures the distribution of a sample of embedding vectors at time T -- centroid, per-dimension variance, pairwise similarity distribution, and more -- and compares that snapshot against a future sample. When the distributions have drifted beyond configurable thresholds, embed-drift computes a composite drift score, classifies severity, and fires alert callbacks.

Zero runtime dependencies. Pure TypeScript. All statistical computations are self-contained.

Installation

npm install embed-drift

Requires Node.js 18 or later.

Quick Start

import{createMonitor}from'embed-drift';// Create a drift monitor for your embedding modelconstmonitor=createMonitor({modelId: 'text-embedding-3-small',alertSeverity: 'high',onDrift: (report)=>{console.warn('Embedding drift detected:',report.summary);},});// Take a baseline snapshot from your current embeddingsconstbaseline=monitor.snapshot(baselineEmbeddings);monitor.setBaseline(baseline);// Later, check new embeddings against the baselineconstreport=monitor.check(newEmbeddings);console.log(report.composite.severity);// 'none' | 'low' | 'medium' | 'high' | 'critical'// Detect silent model changes using canary textsconstcanaryReport=awaitmonitor.checkCanaries(embedFn);if(canaryReport.modelChanged){console.error('Embedding model has changed!');}

Persisting Snapshots

// Save a snapshot to diskmonitor.saveSnapshot(baseline,'./snapshots/baseline.json');// Load it back laterconstloaded=monitor.loadSnapshot('./snapshots/baseline.json');monitor.setBaseline(loaded);

CI/CD Gate

import{createMonitor}from'embed-drift';constmonitor=createMonitor({modelId: 'text-embedding-3-small'});constbaseline=monitor.loadSnapshot('./snapshots/production-baseline.json');monitor.setBaseline(baseline);constreport=monitor.check(newEmbeddings);if(report.composite.severity==='high'||report.composite.severity==='critical'){console.error('Drift too high -- block deployment until re-indexing is complete.');process.exit(1);}

Features

  • Canary-based model change detection -- Embeds a fixed corpus of 25 diverse reference texts and compares their embeddings over time. Detects silent model swaps, version bumps, and provider changes within a single check cycle.

  • Five complementary drift detection methods -- Centroid shift, pairwise cosine similarity distribution, dimension-wise statistics (Cohen's d + KS-like statistic), Maximum Mean Discrepancy (MMD) approximation with random Fourier features, and canary comparison. Each method produces a normalized score in [0, 1].

  • Composite drift scoring -- Weighted average of all method scores with configurable per-method weights. Automatic weight renormalization when methods are disabled or data is unavailable.

  • Severity classification -- Composite scores are classified into five actionable bands: none, low, medium, high, critical. Model changes always produce critical severity.

  • Configurable alerting -- Set severity thresholds and per-method score thresholds. Register an onDrift callback to integrate with any monitoring system. Supports both synchronous and asynchronous callbacks.

  • Snapshot persistence -- Save and load statistical snapshots as portable JSON files. Snapshots are compact (typically 10-900 KB depending on dimensionality and sample size) and work across processes, machines, and time.

  • Zero runtime dependencies -- All statistical computations are self-contained TypeScript. No native modules, no WASM, no Python bridge.

  • Full TypeScript support -- Complete type definitions for all exports. Strict mode compatible.

API Reference

Exports

import{createMonitor,DriftError,DEFAULT_CANARY_TEXTS,}from'embed-drift';importtype{EmbedFn,DriftSeverity,MethodResult,MethodThresholds,MethodWeights,SnapshotOptions,CheckOptions,Snapshot,DriftReport,CanaryReport,DriftMonitorOptions,DriftMonitor,DriftErrorCode,}from'embed-drift';

createMonitor(options: DriftMonitorOptions): DriftMonitor

Creates a drift monitor instance. All drift detection state and configuration is encapsulated in the returned object.

Options (DriftMonitorOptions):

OptionTypeDefaultDescription
modelIdstring--Required. The embedding model identifier.
canaryTextsstring[][]Additional canary texts to append to the built-in corpus.
replaceDefaultCanariesbooleanfalseIf true, use only canaryTexts instead of built-in corpus + custom.
canaryThresholdnumber0.95Mean cosine similarity below which modelChanged is declared.
alertSeverityDriftSeverity'high'Minimum severity to fire the onDrift callback.
thresholdsPartial<MethodThresholds>{}Per-method score overrides that trigger an alert.
onDrift(report) => void | Promise<void>undefinedCallback invoked when an alert fires. Async errors are swallowed.
methodWeightsPartial<MethodWeights>see belowWeights for the composite drift score.
enabledMethods{ centroid?, pairwise?, dimensionWise?, mmd? }all trueDisable specific drift methods.
mmdRandomFeaturesnumber100Number of random Fourier features for MMD approximation.
pairwiseSamplePairsnumber500Number of random pairs sampled for pairwise similarity estimation.

Default composite weights:

{canary: 0.35,centroid: 0.15,pairwise: 0.20,dimensionWise: 0.15,mmd: 0.15}

monitor.snapshot(embeddings, options?): Snapshot

Computes a statistical snapshot of the provided embedding vectors.

Parameters:

ParameterTypeDescription
embeddingsnumber[][]At least 2 vectors of consistent dimensionality.
options.sampleSizenumberNumber of vectors to store for KS/MMD computation. Default: 50.
options.metadataRecord<string, unknown>Caller-provided key-value metadata attached to the snapshot.

Returns: A Snapshot object containing the model ID, centroid, per-dimension variance, pairwise similarity statistics, a 20-bin similarity histogram, and a random sample of vectors.

Throws:

  • DriftError('EMPTY_INPUT') if fewer than 2 vectors are given.
  • DriftError('INCONSISTENT_DIMENSIONS') if vectors have different lengths.

monitor.compare(snapshotA, snapshotB): DriftReport

Compares two snapshots and returns a DriftReport with per-method drift scores, a composite score, and severity classification.

Parameters:

ParameterTypeDescription
snapshotASnapshotThe reference (baseline) snapshot.
snapshotBSnapshotThe new snapshot to compare against the baseline.

Returns: A DriftReport with all per-method results, composite score, severity, alert status, and a human-readable summary.

Throws:

  • DriftError('INCOMPATIBLE_DIMENSIONS') if the two snapshots have different dimensionalities.

When snapshotA.modelId !== snapshotB.modelId, the report sets modelChanged: true and severity to critical.

monitor.setBaseline(snapshot): void

Stores a snapshot as the baseline for subsequent check() calls.

monitor.getBaseline(): Snapshot | undefined

Returns the currently stored baseline snapshot, or undefined if none is set.

monitor.check(embeddings, options?): DriftReport

Creates a new snapshot from embeddings and compares it against the stored baseline.

Parameters:

ParameterTypeDescription
embeddingsnumber[][]New embedding vectors to compare against the baseline.
options.snapshotOptionsSnapshotOptionsOptions forwarded to snapshot creation.

Returns: A DriftReport.

Throws:

  • DriftError('NO_BASELINE') if no baseline has been set via setBaseline().

monitor.checkCanaries(embedFn): Promise<CanaryReport>

Embeds the configured canary texts using embedFn and compares against stored reference embeddings.

Parameters:

ParameterTypeDescription
embedFn(texts: string[]) => Promise<number[][]>A function that embeds an array of texts and returns vectors.

Behavior:

  • On the first call, establishes the reference baseline. Returns a CanaryReport with isInitialBaseline: true and driftScore: 0.
  • On subsequent calls, computes per-canary cosine similarities and returns modelChanged: true when the mean similarity falls below canaryThreshold.

Returns: A CanaryReport.

Throws:

  • DriftError('EMBED_FN_FAILED') if embedFn throws.

monitor.setCanaryBaseline(canaryEmbeddings): void

Explicitly sets the canary reference embeddings without calling checkCanaries. Useful for loading a persisted canary baseline.

Parameters:

ParameterTypeDescription
canaryEmbeddingsnumber[][]Pre-computed canary embeddings (one per canary text).

Throws:

  • DriftError('EMPTY_INPUT') if the array is empty.

monitor.getCanaryTexts(): string[]

Returns the resolved canary text array (built-in + custom, or custom-only if replaceDefaultCanaries: true).

monitor.alert(report): boolean

Evaluates a DriftReport or CanaryReport against configured thresholds and returns true if an alert should fire. Does not invoke the onDrift callback.

An alert fires if:

  • The report severity meets or exceeds alertSeverity, OR
  • Any per-method score exceeds its configured threshold in thresholds.

monitor.saveSnapshot(snapshot, filePath): void

Writes a snapshot as pretty-printed JSON to the given file path.

monitor.loadSnapshot(filePath): Snapshot

Reads and validates a snapshot from a JSON file. Validates all required fields and dimensional consistency.

Throws:

  • DriftError('INVALID_SNAPSHOT') if the file is missing, not valid JSON, or fails schema validation.

DriftError

Custom error class extending Error with a code property for programmatic error handling.

import{DriftError}from'embed-drift';try{monitor.check(embeddings);}catch(err){if(errinstanceofDriftError){console.error(`Drift error [${err.code}]: ${err.message}`);}}

DEFAULT_CANARY_TEXTS

A frozen array of 25 diverse English reference texts spanning technical documentation, scientific language, legal text, casual conversation, news, medical, creative writing, mathematical, instructional, and philosophical domains. Used as the default canary corpus for model fingerprinting.

import{DEFAULT_CANARY_TEXTS}from'embed-drift';console.log(DEFAULT_CANARY_TEXTS.length);// 25

Configuration

Composite Weights

The composite drift score is a weighted average of per-method scores. Weights are renormalized when methods are disabled or their data is unavailable.

constmonitor=createMonitor({modelId: 'text-embedding-3-small',methodWeights: {canary: 0.40,// Increase canary influencecentroid: 0.10,pairwise: 0.20,dimensionWise: 0.15,mmd: 0.15,},});

Disabling Methods

Disable individual drift detection methods when they are not needed:

constmonitor=createMonitor({modelId: 'text-embedding-3-small',enabledMethods: {mmd: false,// Skip MMD computationdimensionWise: false,// Skip dimension-wise analysis},});

Disabled methods report computed: false and score 0. Their weights are redistributed to the remaining active methods.

Alert Thresholds

Alerts fire when severity meets or exceeds alertSeverity, or when any per-method score exceeds its configured threshold:

constmonitor=createMonitor({modelId: 'text-embedding-3-small',alertSeverity: 'high',thresholds: {composite: 0.40,canary: 0.05,centroid: 0.30,},onDrift: (report)=>{// Send to your monitoring systemwebhook.post('/alerts/embedding-drift',report);},});

Custom Canary Texts

Add domain-specific canary texts for increased sensitivity:

constmonitor=createMonitor({modelId: 'text-embedding-3-small',canaryTexts: ['The plaintiff alleges breach of fiduciary duty under Section 14(a).','Amortization of goodwill is calculated on a straight-line basis.',],});// Uses all 25 default canaries + 2 custom = 27 totalconstmonitorCustomOnly=createMonitor({modelId: 'text-embedding-3-small',canaryTexts: ['My custom canary text.'],replaceDefaultCanaries: true,});// Uses only the 1 custom canary text

Drift Detection Methods

embed-drift implements five complementary methods. Each produces a normalized score in [0, 1].

Centroid Shift

Measures the cosine distance between the mean embedding vectors (centroids) of two snapshots. Detects global shifts in the embedding space. Computational cost: O(n * d) where n is the sample size and d is the dimensionality.

Pairwise Cosine Similarity Distribution

Compares the distribution of pairwise cosine similarities between two snapshots. Captures changes in how embeddings are spread relative to each other -- how compact or diffuse the distribution is -- even when the centroid stays the same.

Dimension-Wise Statistics

Analyzes per-dimension statistics using Cohen's d effect size and KS-like statistics across sample vectors. The Cohen's d score identifies which specific dimensions have shifted, while the KS statistic captures distributional shape changes (bimodality, heavy tails) that mean and variance alone do not surface. The two scores are blended equally.

MMD (Maximum Mean Discrepancy)

Uses Maximum Mean Discrepancy with random Fourier features (random kitchen sinks approximation) to measure the distance between two embedding distributions in a kernel-induced feature space. The RBF kernel bandwidth is set via the median heuristic. Sensitive to all moments of the distribution difference. Configurable via mmdRandomFeatures (default: 100).

Canary Texts

Embeds a fixed corpus of diverse reference texts and compares their embeddings over time. Detects silent model changes (provider swaps, version updates) by monitoring whether the same inputs produce the same outputs. The primary and cheapest signal for model change detection.

Severity Bands

Composite ScoreSeverityRecommended Action
0.00 -- 0.05noneNo action needed. Distribution is stable.
0.05 -- 0.20lowMonitor. Normal content variation.
0.20 -- 0.40mediumInvestigate. Consider partial re-indexing.
0.40 -- 0.70highRe-embed recommended. Significant drift detected.
0.70 -- 1.00criticalRe-embed immediately.

A confirmed model change (different model IDs or canary mean similarity below threshold) always produces critical severity regardless of the composite score.

Error Handling

All errors thrown by embed-drift are instances of DriftError with a code property for programmatic handling:

CodeWhen It Is Thrown
EMPTY_INPUTEmbedding array has fewer than 2 vectors, or setCanaryBaseline receives an empty array.
INCONSISTENT_DIMENSIONSVectors in the input array have different dimensionalities.
INCOMPATIBLE_DIMENSIONSTwo snapshots being compared have different dimensionalities.
NO_BASELINEcheck() called before setBaseline().
INVALID_SNAPSHOTLoaded snapshot file is missing, not valid JSON, or fails schema validation.
NO_CANARY_BASELINECanary comparison attempted without a reference baseline.
EMBED_FN_FAILEDThe embedding function passed to checkCanaries() threw an error.
import{DriftError}from'embed-drift';try{constreport=monitor.check(newEmbeddings);}catch(err){if(errinstanceofDriftError){switch(err.code){case'NO_BASELINE':
console.error('Set a baseline before calling check().');break;case'INCOMPATIBLE_DIMENSIONS':
console.error('Snapshot dimensions do not match.');break;default:
console.error(`Unexpected drift error: ${err.code}`);}}}

Advanced Usage

Scheduled Monitoring

Run periodic drift checks as part of a cron job or background worker:

import{createMonitor}from'embed-drift';constmonitor=createMonitor({modelId: 'text-embedding-3-small',alertSeverity: 'medium',onDrift: async(report)=>{awaitsendSlackAlert(`Embedding drift detected: ${report.summary}`);},});// Load the production baselineconstbaseline=monitor.loadSnapshot('./snapshots/production-baseline.json');monitor.setBaseline(baseline);// Sample current embeddings from your vector databaseconstcurrentSample=awaitsampleFromVectorDB(1000);// Check for driftconstreport=monitor.check(currentSample);console.log(`Severity: ${report.composite.severity}, Score: ${report.composite.score}`);

Canary-Based Model Monitoring

Detect model changes with minimal API cost:

import{createMonitor}from'embed-drift';constmonitor=createMonitor({modelId: 'text-embedding-3-small',canaryThreshold: 0.95,onDrift: (report)=>{if('modelChanged'inreport&&report.modelChanged){triggerReindexingPipeline();}},});// On first run, establishes the canary baselineconstembedFn=async(texts: string[])=>{returnopenai.embeddings.create({model: 'text-embedding-3-small',input: texts}).then(res=>res.data.map(d=>d.embedding));};constreport=awaitmonitor.checkCanaries(embedFn);if(report.isInitialBaseline){console.log('Canary baseline established.');}elseif(report.modelChanged){console.error('Model changed! Drift score:',report.driftScore);}else{console.log('Model unchanged. Mean similarity:',report.meanSimilarity);}

Comparing Two Snapshots Directly

Compare snapshots without managing baseline state:

constmonitor=createMonitor({modelId: 'text-embedding-3-small'});constsnapshotA=monitor.loadSnapshot('./snapshots/2025-01-baseline.json');constsnapshotB=monitor.loadSnapshot('./snapshots/2025-03-current.json');constreport=monitor.compare(snapshotA,snapshotB);console.log('Composite score:',report.composite.score);console.log('Severity:',report.composite.severity);console.log('Centroid drift:',report.methods.centroid.score);console.log('Pairwise drift:',report.methods.pairwise.score);console.log('MMD drift:',report.methods.mmd.score);console.log('Summary:',report.summary);

Pre-Computing Canary Baselines

Load a previously saved canary baseline to avoid re-establishing on every restart:

constmonitor=createMonitor({modelId: 'text-embedding-3-small',replaceDefaultCanaries: true,canaryTexts: ['My domain-specific canary text.'],});// Load saved canary embeddingsconstsavedCanaries=JSON.parse(readFileSync('./canary-baseline.json','utf-8'));monitor.setCanaryBaseline(savedCanaries);// Now checkCanaries compares against the loaded baselineconstreport=awaitmonitor.checkCanaries(embedFn);

Types

All types are exported for use in TypeScript projects:

Snapshot

interfaceSnapshot{id: string;// UUID v4createdAt: string;// ISO 8601 timestampmodelId: string;// Embedding model identifierdimensionality: number;// Vector dimensionssampleCount: number;// Number of input vectorscentroid: number[];// Element-wise mean vectorvariance: number[];// Per-dimension variancemeanPairwiseSimilarity: number;// Mean cosine similarity across sampled pairsstdPairwiseSimilarity: number;// Std dev of pairwise cosine similaritiessimilarityHistogram: number[];// 20-bin histogram from -1.0 to 1.0sampleVectors: number[][];// Random sample of vectors for KS/MMDcanaryEmbeddings?: number[][];// Canary text embeddings (optional)metadata?: Record<string,unknown>;// Caller-provided metadata (optional)}

DriftReport

interfaceDriftReport{id: string;createdAt: string;snapshotIds: [string,string];modelIds: [string,string];modelChanged: boolean;methods: {canary: MethodResult;centroid: MethodResult;pairwise: MethodResult;dimensionWise: MethodResult;mmd: MethodResult;};composite: {score: number;// Weighted average in [0, 1]severity: DriftSeverity;// 'none' | 'low' | 'medium' | 'high' | 'critical'weights: MethodWeights;// Effective weights used};alerted: boolean;summary: string;durationMs: number;}

CanaryReport

interfaceCanaryReport{id: string;createdAt: string;canaryCount: number;meanSimilarity: number;minSimilarity: number;perCanarySimilarities: number[];driftScore: number;// 1 - meanSimilaritymodelChanged: boolean;isInitialBaseline: boolean;alerted: boolean;modelId: string;durationMs: number;}

MethodResult

interfaceMethodResult{score: number;// Drift score in [0, 1]computed: boolean;// Whether this method was runinterpretation: string;// Human-readable interpretationdetails?: Record<string,unknown>;// Method-specific details}

EmbedFn

typeEmbedFn=(texts: string[])=>Promise<number[][]>;

DriftSeverity

typeDriftSeverity='none'|'low'|'medium'|'high'|'critical';

DriftErrorCode

typeDriftErrorCode=|'EMPTY_INPUT'|'INCONSISTENT_DIMENSIONS'|'INCOMPATIBLE_DIMENSIONS'|'NO_BASELINE'|'INVALID_SNAPSHOT'|'NO_CANARY_BASELINE'|'EMBED_FN_FAILED';

License

MIT

About

Monitor embedding distribution shifts over time

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
GitHub - SiluPanda/embed-drift: Monitor embedding distribution shifts over time · GitHub
Skip to content
BranchesTags
Repository files navigation
embed-drift
Detect embedding model changes and distribution shifts before they silently degrade your retrieval quality.
npm versionnpm downloadslicensenode
Description
When an embedding model changes -- OpenAI's text-embedding-ada-002 to text-embedding-3-small, a Cohere version bump, or any silent provider update -- the vectors already stored in your database become incompatible with newly produced vectors. Queries return wrong results. No error is thrown, no status code changes, no log line appears. The system looks healthy. The results are wrong.
embed-drift detects this failure before it reaches your users. It monitors embedding distributions over time through two complementary mechanisms:
Canary-based detection embeds a fixed set of reference texts, stores the resulting vectors, and later re-embeds the same texts to check whether the model has changed. This is cheap (embeds only 25 canary texts, not the entire corpus) and catches model changes on the very next check.
Statistical snapshot comparison captures the distribution of a sample of embedding vectors at time T -- centroid, per-dimension variance, pairwise similarity distribution, and more -- and compares that snapshot against a future sample. When the distributions have drifted beyond configurable thresholds, embed-drift computes a composite drift score, classifies severity, and fires alert callbacks.
Zero runtime dependencies. Pure TypeScript. All statistical computations are self-contained.
Installation
npm install embed-drift
Requires Node.js 18 or later.
Quick Start
import{createMonitor}from'embed-drift';// Create a drift monitor for your embedding modelconstmonitor=createMonitor({modelId: 'text-embedding-3-small',alertSeverity: 'high',onDrift: (report)=>{console.warn('Embedding drift detected:',report.summary);},});// Take a baseline snapshot from your current embeddingsconstbaseline=monitor.snapshot(baselineEmbeddings);monitor.setBaseline(baseline);// Later, check new embeddings against the baselineconstreport=monitor.check(newEmbeddings);console.log(report.composite.severity);// 'none' | 'low' | 'medium' | 'high' | 'critical'// Detect silent model changes using canary textsconstcanaryReport=awaitmonitor.checkCanaries(embedFn);if(canaryReport.modelChanged){console.error('Embedding model has changed!');}
Persisting Snapshots
// Save a snapshot to diskmonitor.saveSnapshot(baseline,'./snapshots/baseline.json');// Load it back laterconstloaded=monitor.loadSnapshot('./snapshots/baseline.json');monitor.setBaseline(loaded);
CI/CD Gate
import{createMonitor}from'embed-drift';constmonitor=createMonitor({modelId: 'text-embedding-3-small'});constbaseline=monitor.loadSnapshot('./snapshots/production-baseline.json');monitor.setBaseline(baseline);constreport=monitor.check(newEmbeddings);if(report.composite.severity==='high'||report.composite.severity==='critical'){console.error('Drift too high -- block deployment until re-indexing is complete.');process.exit(1);}
Features
  • Canary-based model change detection -- Embeds a fixed corpus of 25 diverse reference texts and compares their embeddings over time. Detects silent model swaps, version bumps, and provider changes within a single check cycle.
  • Five complementary drift detection methods -- Centroid shift, pairwise cosine similarity distribution, dimension-wise statistics (Cohen's d + KS-like statistic), Maximum Mean Discrepancy (MMD) approximation with random Fourier features, and canary comparison. Each method produces a normalized score in [0, 1].
  • Composite drift scoring -- Weighted average of all method scores with configurable per-method weights. Automatic weight renormalization when methods are disabled or data is unavailable.
  • Severity classification -- Composite scores are classified into five actionable bands: none, low, medium, high, critical. Model changes always produce critical severity.
  • Configurable alerting -- Set severity thresholds and per-method score thresholds. Register an onDrift callback to integrate with any monitoring system. Supports both synchronous and asynchronous callbacks.
  • Snapshot persistence -- Save and load statistical snapshots as portable JSON files. Snapshots are compact (typically 10-900 KB depending on dimensionality and sample size) and work across processes, machines, and time.
  • Zero runtime dependencies -- All statistical computations are self-contained TypeScript. No native modules, no WASM, no Python bridge.
  • Full TypeScript support -- Complete type definitions for all exports. Strict mode compatible.
API Reference
Exports
import{createMonitor,DriftError,DEFAULT_CANARY_TEXTS,}from'embed-drift';importtype{EmbedFn,DriftSeverity,MethodResult,MethodThresholds,MethodWeights,SnapshotOptions,CheckOptions,Snapshot,DriftReport,CanaryReport,DriftMonitorOptions,DriftMonitor,DriftErrorCode,}from'embed-drift';
createMonitor(options: DriftMonitorOptions): DriftMonitor
Creates a drift monitor instance. All drift detection state and configuration is encapsulated in the returned object.
Options (DriftMonitorOptions):
OptionTypeDefaultDescription
modelIdstring--Required. The embedding model identifier.
canaryTextsstring[][]Additional canary texts to append to the built-in corpus.
replaceDefaultCanariesbooleanfalseIf true, use only canaryTexts instead of built-in corpus + custom.
canaryThresholdnumber0.95Mean cosine similarity below which modelChanged is declared.
alertSeverityDriftSeverity'high'Minimum severity to fire the onDrift callback.
thresholdsPartial<MethodThresholds>{}Per-method score overrides that trigger an alert.
onDrift(report) => void | Promise<void>undefinedCallback invoked when an alert fires. Async errors are swallowed.
methodWeightsPartial<MethodWeights>see belowWeights for the composite drift score.
enabledMethods{ centroid?, pairwise?, dimensionWise?, mmd? }all trueDisable specific drift methods.
mmdRandomFeaturesnumber100Number of random Fourier features for MMD approximation.
pairwiseSamplePairsnumber500Number of random pairs sampled for pairwise similarity estimation.
Default composite weights:
{canary: 0.35,centroid: 0.15,pairwise: 0.20,dimensionWise: 0.15,mmd: 0.15}
monitor.snapshot(embeddings, options?): Snapshot
Computes a statistical snapshot of the provided embedding vectors.
Parameters:
ParameterTypeDescription
embeddingsnumber[][]At least 2 vectors of consistent dimensionality.
options.sampleSizenumberNumber of vectors to store for KS/MMD computation. Default: 50.
options.metadataRecord<string, unknown>Caller-provided key-value metadata attached to the snapshot.
Returns: A Snapshot object containing the model ID, centroid, per-dimension variance, pairwise similarity statistics, a 20-bin similarity histogram, and a random sample of vectors.
Throws:
  • DriftError('EMPTY_INPUT') if fewer than 2 vectors are given.
  • DriftError('INCONSISTENT_DIMENSIONS') if vectors have different lengths.
monitor.compare(snapshotA, snapshotB): DriftReport
Compares two snapshots and returns a DriftReport with per-method drift scores, a composite score, and severity classification.
Parameters:
ParameterTypeDescription
snapshotASnapshotThe reference (baseline) snapshot.
snapshotBSnapshotThe new snapshot to compare against the baseline.
Returns: A DriftReport with all per-method results, composite score, severity, alert status, and a human-readable summary.
Throws:
  • DriftError('INCOMPATIBLE_DIMENSIONS') if the two snapshots have different dimensionalities.
When snapshotA.modelId !== snapshotB.modelId, the report sets modelChanged: true and severity to critical.
monitor.setBaseline(snapshot): void
Stores a snapshot as the baseline for subsequent check() calls.
monitor.getBaseline(): Snapshot | undefined
Returns the currently stored baseline snapshot, or undefined if none is set.
monitor.check(embeddings, options?): DriftReport
Creates a new snapshot from embeddings and compares it against the stored baseline.
Parameters:
ParameterTypeDescription
embeddingsnumber[][]New embedding vectors to compare against the baseline.
options.snapshotOptionsSnapshotOptionsOptions forwarded to snapshot creation.
Returns: A DriftReport.
Throws:
  • DriftError('NO_BASELINE') if no baseline has been set via setBaseline().
monitor.checkCanaries(embedFn): Promise<CanaryReport>
Embeds the configured canary texts using embedFn and compares against stored reference embeddings.
Parameters:
ParameterTypeDescription
embedFn(texts: string[]) => Promise<number[][]>A function that embeds an array of texts and returns vectors.
Behavior:
  • On the first call, establishes the reference baseline. Returns a CanaryReport with isInitialBaseline: true and driftScore: 0.
  • On subsequent calls, computes per-canary cosine similarities and returns modelChanged: true when the mean similarity falls below canaryThreshold.
Returns: A CanaryReport.
Throws:
  • DriftError('EMBED_FN_FAILED') if embedFn throws.
monitor.setCanaryBaseline(canaryEmbeddings): void
Explicitly sets the canary reference embeddings without calling checkCanaries. Useful for loading a persisted canary baseline.
Parameters:
ParameterTypeDescription
canaryEmbeddingsnumber[][]Pre-computed canary embeddings (one per canary text).
Throws:
  • DriftError('EMPTY_INPUT') if the array is empty.
monitor.getCanaryTexts(): string[]
Returns the resolved canary text array (built-in + custom, or custom-only if replaceDefaultCanaries: true).
monitor.alert(report): boolean
Evaluates a DriftReport or CanaryReport against configured thresholds and returns true if an alert should fire. Does not invoke the onDrift callback.
An alert fires if:
  • The report severity meets or exceeds alertSeverity, OR
  • Any per-method score exceeds its configured threshold in thresholds.
monitor.saveSnapshot(snapshot, filePath): void
Writes a snapshot as pretty-printed JSON to the given file path.
monitor.loadSnapshot(filePath): Snapshot
Reads and validates a snapshot from a JSON file. Validates all required fields and dimensional consistency.
Throws:
  • DriftError('INVALID_SNAPSHOT') if the file is missing, not valid JSON, or fails schema validation.
DriftError
Custom error class extending Error with a code property for programmatic error handling.
import{DriftError}from'embed-drift';try{monitor.check(embeddings);}catch(err){if(errinstanceofDriftError){console.error(`Drift error [${err.code}]: ${err.message}`);}}
DEFAULT_CANARY_TEXTS
A frozen array of 25 diverse English reference texts spanning technical documentation, scientific language, legal text, casual conversation, news, medical, creative writing, mathematical, instructional, and philosophical domains. Used as the default canary corpus for model fingerprinting.
import{DEFAULT_CANARY_TEXTS}from'embed-drift';console.log(DEFAULT_CANARY_TEXTS.length);// 25
Configuration
Composite Weights
The composite drift score is a weighted average of per-method scores. Weights are renormalized when methods are disabled or their data is unavailable.
constmonitor=createMonitor({modelId: 'text-embedding-3-small',methodWeights: {canary: 0.40,// Increase canary influencecentroid: 0.10,pairwise: 0.20,dimensionWise: 0.15,mmd: 0.15,},});
Disabling Methods
Disable individual drift detection methods when they are not needed:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',enabledMethods: {mmd: false,// Skip MMD computationdimensionWise: false,// Skip dimension-wise analysis},});
Disabled methods report computed: false and score 0. Their weights are redistributed to the remaining active methods.
Alert Thresholds
Alerts fire when severity meets or exceeds alertSeverity, or when any per-method score exceeds its configured threshold:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',alertSeverity: 'high',thresholds: {composite: 0.40,canary: 0.05,centroid: 0.30,},onDrift: (report)=>{// Send to your monitoring systemwebhook.post('/alerts/embedding-drift',report);},});
Custom Canary Texts
Add domain-specific canary texts for increased sensitivity:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',canaryTexts: ['The plaintiff alleges breach of fiduciary duty under Section 14(a).','Amortization of goodwill is calculated on a straight-line basis.',],});// Uses all 25 default canaries + 2 custom = 27 totalconstmonitorCustomOnly=createMonitor({modelId: 'text-embedding-3-small',canaryTexts: ['My custom canary text.'],replaceDefaultCanaries: true,});// Uses only the 1 custom canary text
Drift Detection Methods
embed-drift implements five complementary methods. Each produces a normalized score in [0, 1].
Centroid Shift
Measures the cosine distance between the mean embedding vectors (centroids) of two snapshots. Detects global shifts in the embedding space. Computational cost: O(n * d) where n is the sample size and d is the dimensionality.
Pairwise Cosine Similarity Distribution
Compares the distribution of pairwise cosine similarities between two snapshots. Captures changes in how embeddings are spread relative to each other -- how compact or diffuse the distribution is -- even when the centroid stays the same.
Dimension-Wise Statistics
Analyzes per-dimension statistics using Cohen's d effect size and KS-like statistics across sample vectors. The Cohen's d score identifies which specific dimensions have shifted, while the KS statistic captures distributional shape changes (bimodality, heavy tails) that mean and variance alone do not surface. The two scores are blended equally.
MMD (Maximum Mean Discrepancy)
Uses Maximum Mean Discrepancy with random Fourier features (random kitchen sinks approximation) to measure the distance between two embedding distributions in a kernel-induced feature space. The RBF kernel bandwidth is set via the median heuristic. Sensitive to all moments of the distribution difference. Configurable via mmdRandomFeatures (default: 100).
Canary Texts
Embeds a fixed corpus of diverse reference texts and compares their embeddings over time. Detects silent model changes (provider swaps, version updates) by monitoring whether the same inputs produce the same outputs. The primary and cheapest signal for model change detection.
Severity Bands
Composite ScoreSeverityRecommended Action
0.00 -- 0.05noneNo action needed. Distribution is stable.
0.05 -- 0.20lowMonitor. Normal content variation.
0.20 -- 0.40mediumInvestigate. Consider partial re-indexing.
0.40 -- 0.70highRe-embed recommended. Significant drift detected.
0.70 -- 1.00criticalRe-embed immediately.
A confirmed model change (different model IDs or canary mean similarity below threshold) always produces critical severity regardless of the composite score.
Error Handling
All errors thrown by embed-drift are instances of DriftError with a code property for programmatic handling:
CodeWhen It Is Thrown
EMPTY_INPUTEmbedding array has fewer than 2 vectors, or setCanaryBaseline receives an empty array.
INCONSISTENT_DIMENSIONSVectors in the input array have different dimensionalities.
INCOMPATIBLE_DIMENSIONSTwo snapshots being compared have different dimensionalities.
NO_BASELINEcheck() called before setBaseline().
INVALID_SNAPSHOTLoaded snapshot file is missing, not valid JSON, or fails schema validation.
NO_CANARY_BASELINECanary comparison attempted without a reference baseline.
EMBED_FN_FAILEDThe embedding function passed to checkCanaries() threw an error.
import{DriftError}from'embed-drift';try{constreport=monitor.check(newEmbeddings);}catch(err){if(errinstanceofDriftError){switch(err.code){case'NO_BASELINE':
console.error('Set a baseline before calling check().');break;case'INCOMPATIBLE_DIMENSIONS':
console.error('Snapshot dimensions do not match.');break;default:
console.error(`Unexpected drift error: ${err.code}`);}}}
Advanced Usage
Scheduled Monitoring
Run periodic drift checks as part of a cron job or background worker:
import{createMonitor}from'embed-drift';constmonitor=createMonitor({modelId: 'text-embedding-3-small',alertSeverity: 'medium',onDrift: async(report)=>{awaitsendSlackAlert(`Embedding drift detected: ${report.summary}`);},});// Load the production baselineconstbaseline=monitor.loadSnapshot('./snapshots/production-baseline.json');monitor.setBaseline(baseline);// Sample current embeddings from your vector databaseconstcurrentSample=awaitsampleFromVectorDB(1000);// Check for driftconstreport=monitor.check(currentSample);console.log(`Severity: ${report.composite.severity}, Score: ${report.composite.score}`);
Canary-Based Model Monitoring
Detect model changes with minimal API cost:
import{createMonitor}from'embed-drift';constmonitor=createMonitor({modelId: 'text-embedding-3-small',canaryThreshold: 0.95,onDrift: (report)=>{if('modelChanged'inreport&&report.modelChanged){triggerReindexingPipeline();}},});// On first run, establishes the canary baselineconstembedFn=async(texts: string[])=>{returnopenai.embeddings.create({model: 'text-embedding-3-small',input: texts}).then(res=>res.data.map(d=>d.embedding));};constreport=awaitmonitor.checkCanaries(embedFn);if(report.isInitialBaseline){console.log('Canary baseline established.');}elseif(report.modelChanged){console.error('Model changed! Drift score:',report.driftScore);}else{console.log('Model unchanged. Mean similarity:',report.meanSimilarity);}
Comparing Two Snapshots Directly
Compare snapshots without managing baseline state:
constmonitor=createMonitor({modelId: 'text-embedding-3-small'});constsnapshotA=monitor.loadSnapshot('./snapshots/2025-01-baseline.json');constsnapshotB=monitor.loadSnapshot('./snapshots/2025-03-current.json');constreport=monitor.compare(snapshotA,snapshotB);console.log('Composite score:',report.composite.score);console.log('Severity:',report.composite.severity);console.log('Centroid drift:',report.methods.centroid.score);console.log('Pairwise drift:',report.methods.pairwise.score);console.log('MMD drift:',report.methods.mmd.score);console.log('Summary:',report.summary);
Pre-Computing Canary Baselines
Load a previously saved canary baseline to avoid re-establishing on every restart:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',replaceDefaultCanaries: true,canaryTexts: ['My domain-specific canary text.'],});// Load saved canary embeddingsconstsavedCanaries=JSON.parse(readFileSync('./canary-baseline.json','utf-8'));monitor.setCanaryBaseline(savedCanaries);// Now checkCanaries compares against the loaded baselineconstreport=awaitmonitor.checkCanaries(embedFn);
Types
All types are exported for use in TypeScript projects:
Snapshot
interfaceSnapshot{id: string;// UUID v4createdAt: string;// ISO 8601 timestampmodelId: string;// Embedding model identifierdimensionality: number;// Vector dimensionssampleCount: number;// Number of input vectorscentroid: number[];// Element-wise mean vectorvariance: number[];// Per-dimension variancemeanPairwiseSimilarity: number;// Mean cosine similarity across sampled pairsstdPairwiseSimilarity: number;// Std dev of pairwise cosine similaritiessimilarityHistogram: number[];// 20-bin histogram from -1.0 to 1.0sampleVectors: number[][];// Random sample of vectors for KS/MMDcanaryEmbeddings?: number[][];// Canary text embeddings (optional)metadata?: Record<string,unknown>;// Caller-provided metadata (optional)}
DriftReport
interfaceDriftReport{id: string;createdAt: string;snapshotIds: [string,string];modelIds: [string,string];modelChanged: boolean;methods: {canary: MethodResult;centroid: MethodResult;pairwise: MethodResult;dimensionWise: MethodResult;mmd: MethodResult;};composite: {score: number;// Weighted average in [0, 1]severity: DriftSeverity;// 'none' | 'low' | 'medium' | 'high' | 'critical'weights: MethodWeights;// Effective weights used};alerted: boolean;summary: string;durationMs: number;}
CanaryReport
interfaceCanaryReport{id: string;createdAt: string;canaryCount: number;meanSimilarity: number;minSimilarity: number;perCanarySimilarities: number[];driftScore: number;// 1 - meanSimilaritymodelChanged: boolean;isInitialBaseline: boolean;alerted: boolean;modelId: string;durationMs: number;}
MethodResult
interfaceMethodResult{score: number;// Drift score in [0, 1]computed: boolean;// Whether this method was runinterpretation: string;// Human-readable interpretationdetails?: Record<string,unknown>;// Method-specific details}
EmbedFn
typeEmbedFn=(texts: string[])=>Promise<number[][]>;
DriftSeverity
typeDriftSeverity='none'|'low'|'medium'|'high'|'critical';
DriftErrorCode
typeDriftErrorCode=|'EMPTY_INPUT'|'INCONSISTENT_DIMENSIONS'|'INCOMPATIBLE_DIMENSIONS'|'NO_BASELINE'|'INVALID_SNAPSHOT'|'NO_CANARY_BASELINE'|'EMBED_FN_FAILED';
License
MIT
About
Monitor embedding distribution shifts over time
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - SiluPanda/embed-drift: Monitor embedding distribution shifts over time · GitHub
Skip to content
BranchesTags
Repository files navigation
embed-drift
Detect embedding model changes and distribution shifts before they silently degrade your retrieval quality.
npm versionnpm downloadslicensenode
Description
When an embedding model changes -- OpenAI's text-embedding-ada-002 to text-embedding-3-small, a Cohere version bump, or any silent provider update -- the vectors already stored in your database become incompatible with newly produced vectors. Queries return wrong results. No error is thrown, no status code changes, no log line appears. The system looks healthy. The results are wrong.
embed-drift detects this failure before it reaches your users. It monitors embedding distributions over time through two complementary mechanisms:
Canary-based detection embeds a fixed set of reference texts, stores the resulting vectors, and later re-embeds the same texts to check whether the model has changed. This is cheap (embeds only 25 canary texts, not the entire corpus) and catches model changes on the very next check.
Statistical snapshot comparison captures the distribution of a sample of embedding vectors at time T -- centroid, per-dimension variance, pairwise similarity distribution, and more -- and compares that snapshot against a future sample. When the distributions have drifted beyond configurable thresholds, embed-drift computes a composite drift score, classifies severity, and fires alert callbacks.
Zero runtime dependencies. Pure TypeScript. All statistical computations are self-contained.
Installation
npm install embed-drift
Requires Node.js 18 or later.
Quick Start
import{createMonitor}from'embed-drift';// Create a drift monitor for your embedding modelconstmonitor=createMonitor({modelId: 'text-embedding-3-small',alertSeverity: 'high',onDrift: (report)=>{console.warn('Embedding drift detected:',report.summary);},});// Take a baseline snapshot from your current embeddingsconstbaseline=monitor.snapshot(baselineEmbeddings);monitor.setBaseline(baseline);// Later, check new embeddings against the baselineconstreport=monitor.check(newEmbeddings);console.log(report.composite.severity);// 'none' | 'low' | 'medium' | 'high' | 'critical'// Detect silent model changes using canary textsconstcanaryReport=awaitmonitor.checkCanaries(embedFn);if(canaryReport.modelChanged){console.error('Embedding model has changed!');}
Persisting Snapshots
// Save a snapshot to diskmonitor.saveSnapshot(baseline,'./snapshots/baseline.json');// Load it back laterconstloaded=monitor.loadSnapshot('./snapshots/baseline.json');monitor.setBaseline(loaded);
CI/CD Gate
import{createMonitor}from'embed-drift';constmonitor=createMonitor({modelId: 'text-embedding-3-small'});constbaseline=monitor.loadSnapshot('./snapshots/production-baseline.json');monitor.setBaseline(baseline);constreport=monitor.check(newEmbeddings);if(report.composite.severity==='high'||report.composite.severity==='critical'){console.error('Drift too high -- block deployment until re-indexing is complete.');process.exit(1);}
Features
  • Canary-based model change detection -- Embeds a fixed corpus of 25 diverse reference texts and compares their embeddings over time. Detects silent model swaps, version bumps, and provider changes within a single check cycle.
  • Five complementary drift detection methods -- Centroid shift, pairwise cosine similarity distribution, dimension-wise statistics (Cohen's d + KS-like statistic), Maximum Mean Discrepancy (MMD) approximation with random Fourier features, and canary comparison. Each method produces a normalized score in [0, 1].
  • Composite drift scoring -- Weighted average of all method scores with configurable per-method weights. Automatic weight renormalization when methods are disabled or data is unavailable.
  • Severity classification -- Composite scores are classified into five actionable bands: none, low, medium, high, critical. Model changes always produce critical severity.
  • Configurable alerting -- Set severity thresholds and per-method score thresholds. Register an onDrift callback to integrate with any monitoring system. Supports both synchronous and asynchronous callbacks.
  • Snapshot persistence -- Save and load statistical snapshots as portable JSON files. Snapshots are compact (typically 10-900 KB depending on dimensionality and sample size) and work across processes, machines, and time.
  • Zero runtime dependencies -- All statistical computations are self-contained TypeScript. No native modules, no WASM, no Python bridge.
  • Full TypeScript support -- Complete type definitions for all exports. Strict mode compatible.
API Reference
Exports
import{createMonitor,DriftError,DEFAULT_CANARY_TEXTS,}from'embed-drift';importtype{EmbedFn,DriftSeverity,MethodResult,MethodThresholds,MethodWeights,SnapshotOptions,CheckOptions,Snapshot,DriftReport,CanaryReport,DriftMonitorOptions,DriftMonitor,DriftErrorCode,}from'embed-drift';
createMonitor(options: DriftMonitorOptions): DriftMonitor
Creates a drift monitor instance. All drift detection state and configuration is encapsulated in the returned object.
Options (DriftMonitorOptions):
OptionTypeDefaultDescription
modelIdstring--Required. The embedding model identifier.
canaryTextsstring[][]Additional canary texts to append to the built-in corpus.
replaceDefaultCanariesbooleanfalseIf true, use only canaryTexts instead of built-in corpus + custom.
canaryThresholdnumber0.95Mean cosine similarity below which modelChanged is declared.
alertSeverityDriftSeverity'high'Minimum severity to fire the onDrift callback.
thresholdsPartial<MethodThresholds>{}Per-method score overrides that trigger an alert.
onDrift(report) => void | Promise<void>undefinedCallback invoked when an alert fires. Async errors are swallowed.
methodWeightsPartial<MethodWeights>see belowWeights for the composite drift score.
enabledMethods{ centroid?, pairwise?, dimensionWise?, mmd? }all trueDisable specific drift methods.
mmdRandomFeaturesnumber100Number of random Fourier features for MMD approximation.
pairwiseSamplePairsnumber500Number of random pairs sampled for pairwise similarity estimation.
Default composite weights:
{canary: 0.35,centroid: 0.15,pairwise: 0.20,dimensionWise: 0.15,mmd: 0.15}
monitor.snapshot(embeddings, options?): Snapshot
Computes a statistical snapshot of the provided embedding vectors.
Parameters:
ParameterTypeDescription
embeddingsnumber[][]At least 2 vectors of consistent dimensionality.
options.sampleSizenumberNumber of vectors to store for KS/MMD computation. Default: 50.
options.metadataRecord<string, unknown>Caller-provided key-value metadata attached to the snapshot.
Returns: A Snapshot object containing the model ID, centroid, per-dimension variance, pairwise similarity statistics, a 20-bin similarity histogram, and a random sample of vectors.
Throws:
  • DriftError('EMPTY_INPUT') if fewer than 2 vectors are given.
  • DriftError('INCONSISTENT_DIMENSIONS') if vectors have different lengths.
monitor.compare(snapshotA, snapshotB): DriftReport
Compares two snapshots and returns a DriftReport with per-method drift scores, a composite score, and severity classification.
Parameters:
ParameterTypeDescription
snapshotASnapshotThe reference (baseline) snapshot.
snapshotBSnapshotThe new snapshot to compare against the baseline.
Returns: A DriftReport with all per-method results, composite score, severity, alert status, and a human-readable summary.
Throws:
  • DriftError('INCOMPATIBLE_DIMENSIONS') if the two snapshots have different dimensionalities.
When snapshotA.modelId !== snapshotB.modelId, the report sets modelChanged: true and severity to critical.
monitor.setBaseline(snapshot): void
Stores a snapshot as the baseline for subsequent check() calls.
monitor.getBaseline(): Snapshot | undefined
Returns the currently stored baseline snapshot, or undefined if none is set.
monitor.check(embeddings, options?): DriftReport
Creates a new snapshot from embeddings and compares it against the stored baseline.
Parameters:
ParameterTypeDescription
embeddingsnumber[][]New embedding vectors to compare against the baseline.
options.snapshotOptionsSnapshotOptionsOptions forwarded to snapshot creation.
Returns: A DriftReport.
Throws:
  • DriftError('NO_BASELINE') if no baseline has been set via setBaseline().
monitor.checkCanaries(embedFn): Promise<CanaryReport>
Embeds the configured canary texts using embedFn and compares against stored reference embeddings.
Parameters:
ParameterTypeDescription
embedFn(texts: string[]) => Promise<number[][]>A function that embeds an array of texts and returns vectors.
Behavior:
  • On the first call, establishes the reference baseline. Returns a CanaryReport with isInitialBaseline: true and driftScore: 0.
  • On subsequent calls, computes per-canary cosine similarities and returns modelChanged: true when the mean similarity falls below canaryThreshold.
Returns: A CanaryReport.
Throws:
  • DriftError('EMBED_FN_FAILED') if embedFn throws.
monitor.setCanaryBaseline(canaryEmbeddings): void
Explicitly sets the canary reference embeddings without calling checkCanaries. Useful for loading a persisted canary baseline.
Parameters:
ParameterTypeDescription
canaryEmbeddingsnumber[][]Pre-computed canary embeddings (one per canary text).
Throws:
  • DriftError('EMPTY_INPUT') if the array is empty.
monitor.getCanaryTexts(): string[]
Returns the resolved canary text array (built-in + custom, or custom-only if replaceDefaultCanaries: true).
monitor.alert(report): boolean
Evaluates a DriftReport or CanaryReport against configured thresholds and returns true if an alert should fire. Does not invoke the onDrift callback.
An alert fires if:
  • The report severity meets or exceeds alertSeverity, OR
  • Any per-method score exceeds its configured threshold in thresholds.
monitor.saveSnapshot(snapshot, filePath): void
Writes a snapshot as pretty-printed JSON to the given file path.
monitor.loadSnapshot(filePath): Snapshot
Reads and validates a snapshot from a JSON file. Validates all required fields and dimensional consistency.
Throws:
  • DriftError('INVALID_SNAPSHOT') if the file is missing, not valid JSON, or fails schema validation.
DriftError
Custom error class extending Error with a code property for programmatic error handling.
import{DriftError}from'embed-drift';try{monitor.check(embeddings);}catch(err){if(errinstanceofDriftError){console.error(`Drift error [${err.code}]: ${err.message}`);}}
DEFAULT_CANARY_TEXTS
A frozen array of 25 diverse English reference texts spanning technical documentation, scientific language, legal text, casual conversation, news, medical, creative writing, mathematical, instructional, and philosophical domains. Used as the default canary corpus for model fingerprinting.
import{DEFAULT_CANARY_TEXTS}from'embed-drift';console.log(DEFAULT_CANARY_TEXTS.length);// 25
Configuration
Composite Weights
The composite drift score is a weighted average of per-method scores. Weights are renormalized when methods are disabled or their data is unavailable.
constmonitor=createMonitor({modelId: 'text-embedding-3-small',methodWeights: {canary: 0.40,// Increase canary influencecentroid: 0.10,pairwise: 0.20,dimensionWise: 0.15,mmd: 0.15,},});
Disabling Methods
Disable individual drift detection methods when they are not needed:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',enabledMethods: {mmd: false,// Skip MMD computationdimensionWise: false,// Skip dimension-wise analysis},});
Disabled methods report computed: false and score 0. Their weights are redistributed to the remaining active methods.
Alert Thresholds
Alerts fire when severity meets or exceeds alertSeverity, or when any per-method score exceeds its configured threshold:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',alertSeverity: 'high',thresholds: {composite: 0.40,canary: 0.05,centroid: 0.30,},onDrift: (report)=>{// Send to your monitoring systemwebhook.post('/alerts/embedding-drift',report);},});
Custom Canary Texts
Add domain-specific canary texts for increased sensitivity:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',canaryTexts: ['The plaintiff alleges breach of fiduciary duty under Section 14(a).','Amortization of goodwill is calculated on a straight-line basis.',],});// Uses all 25 default canaries + 2 custom = 27 totalconstmonitorCustomOnly=createMonitor({modelId: 'text-embedding-3-small',canaryTexts: ['My custom canary text.'],replaceDefaultCanaries: true,});// Uses only the 1 custom canary text
Drift Detection Methods
embed-drift implements five complementary methods. Each produces a normalized score in [0, 1].
Centroid Shift
Measures the cosine distance between the mean embedding vectors (centroids) of two snapshots. Detects global shifts in the embedding space. Computational cost: O(n * d) where n is the sample size and d is the dimensionality.
Pairwise Cosine Similarity Distribution
Compares the distribution of pairwise cosine similarities between two snapshots. Captures changes in how embeddings are spread relative to each other -- how compact or diffuse the distribution is -- even when the centroid stays the same.
Dimension-Wise Statistics
Analyzes per-dimension statistics using Cohen's d effect size and KS-like statistics across sample vectors. The Cohen's d score identifies which specific dimensions have shifted, while the KS statistic captures distributional shape changes (bimodality, heavy tails) that mean and variance alone do not surface. The two scores are blended equally.
MMD (Maximum Mean Discrepancy)
Uses Maximum Mean Discrepancy with random Fourier features (random kitchen sinks approximation) to measure the distance between two embedding distributions in a kernel-induced feature space. The RBF kernel bandwidth is set via the median heuristic. Sensitive to all moments of the distribution difference. Configurable via mmdRandomFeatures (default: 100).
Canary Texts
Embeds a fixed corpus of diverse reference texts and compares their embeddings over time. Detects silent model changes (provider swaps, version updates) by monitoring whether the same inputs produce the same outputs. The primary and cheapest signal for model change detection.
Severity Bands
Composite ScoreSeverityRecommended Action
0.00 -- 0.05noneNo action needed. Distribution is stable.
0.05 -- 0.20lowMonitor. Normal content variation.
0.20 -- 0.40mediumInvestigate. Consider partial re-indexing.
0.40 -- 0.70highRe-embed recommended. Significant drift detected.
0.70 -- 1.00criticalRe-embed immediately.
A confirmed model change (different model IDs or canary mean similarity below threshold) always produces critical severity regardless of the composite score.
Error Handling
All errors thrown by embed-drift are instances of DriftError with a code property for programmatic handling:
CodeWhen It Is Thrown
EMPTY_INPUTEmbedding array has fewer than 2 vectors, or setCanaryBaseline receives an empty array.
INCONSISTENT_DIMENSIONSVectors in the input array have different dimensionalities.
INCOMPATIBLE_DIMENSIONSTwo snapshots being compared have different dimensionalities.
NO_BASELINEcheck() called before setBaseline().
INVALID_SNAPSHOTLoaded snapshot file is missing, not valid JSON, or fails schema validation.
NO_CANARY_BASELINECanary comparison attempted without a reference baseline.
EMBED_FN_FAILEDThe embedding function passed to checkCanaries() threw an error.
import{DriftError}from'embed-drift';try{constreport=monitor.check(newEmbeddings);}catch(err){if(errinstanceofDriftError){switch(err.code){case'NO_BASELINE':
console.error('Set a baseline before calling check().');break;case'INCOMPATIBLE_DIMENSIONS':
console.error('Snapshot dimensions do not match.');break;default:
console.error(`Unexpected drift error: ${err.code}`);}}}
Advanced Usage
Scheduled Monitoring
Run periodic drift checks as part of a cron job or background worker:
import{createMonitor}from'embed-drift';constmonitor=createMonitor({modelId: 'text-embedding-3-small',alertSeverity: 'medium',onDrift: async(report)=>{awaitsendSlackAlert(`Embedding drift detected: ${report.summary}`);},});// Load the production baselineconstbaseline=monitor.loadSnapshot('./snapshots/production-baseline.json');monitor.setBaseline(baseline);// Sample current embeddings from your vector databaseconstcurrentSample=awaitsampleFromVectorDB(1000);// Check for driftconstreport=monitor.check(currentSample);console.log(`Severity: ${report.composite.severity}, Score: ${report.composite.score}`);
Canary-Based Model Monitoring
Detect model changes with minimal API cost:
import{createMonitor}from'embed-drift';constmonitor=createMonitor({modelId: 'text-embedding-3-small',canaryThreshold: 0.95,onDrift: (report)=>{if('modelChanged'inreport&&report.modelChanged){triggerReindexingPipeline();}},});// On first run, establishes the canary baselineconstembedFn=async(texts: string[])=>{returnopenai.embeddings.create({model: 'text-embedding-3-small',input: texts}).then(res=>res.data.map(d=>d.embedding));};constreport=awaitmonitor.checkCanaries(embedFn);if(report.isInitialBaseline){console.log('Canary baseline established.');}elseif(report.modelChanged){console.error('Model changed! Drift score:',report.driftScore);}else{console.log('Model unchanged. Mean similarity:',report.meanSimilarity);}
Comparing Two Snapshots Directly
Compare snapshots without managing baseline state:
constmonitor=createMonitor({modelId: 'text-embedding-3-small'});constsnapshotA=monitor.loadSnapshot('./snapshots/2025-01-baseline.json');constsnapshotB=monitor.loadSnapshot('./snapshots/2025-03-current.json');constreport=monitor.compare(snapshotA,snapshotB);console.log('Composite score:',report.composite.score);console.log('Severity:',report.composite.severity);console.log('Centroid drift:',report.methods.centroid.score);console.log('Pairwise drift:',report.methods.pairwise.score);console.log('MMD drift:',report.methods.mmd.score);console.log('Summary:',report.summary);
Pre-Computing Canary Baselines
Load a previously saved canary baseline to avoid re-establishing on every restart:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',replaceDefaultCanaries: true,canaryTexts: ['My domain-specific canary text.'],});// Load saved canary embeddingsconstsavedCanaries=JSON.parse(readFileSync('./canary-baseline.json','utf-8'));monitor.setCanaryBaseline(savedCanaries);// Now checkCanaries compares against the loaded baselineconstreport=awaitmonitor.checkCanaries(embedFn);
Types
All types are exported for use in TypeScript projects:
Snapshot
interfaceSnapshot{id: string;// UUID v4createdAt: string;// ISO 8601 timestampmodelId: string;// Embedding model identifierdimensionality: number;// Vector dimensionssampleCount: number;// Number of input vectorscentroid: number[];// Element-wise mean vectorvariance: number[];// Per-dimension variancemeanPairwiseSimilarity: number;// Mean cosine similarity across sampled pairsstdPairwiseSimilarity: number;// Std dev of pairwise cosine similaritiessimilarityHistogram: number[];// 20-bin histogram from -1.0 to 1.0sampleVectors: number[][];// Random sample of vectors for KS/MMDcanaryEmbeddings?: number[][];// Canary text embeddings (optional)metadata?: Record<string,unknown>;// Caller-provided metadata (optional)}
DriftReport
interfaceDriftReport{id: string;createdAt: string;snapshotIds: [string,string];modelIds: [string,string];modelChanged: boolean;methods: {canary: MethodResult;centroid: MethodResult;pairwise: MethodResult;dimensionWise: MethodResult;mmd: MethodResult;};composite: {score: number;// Weighted average in [0, 1]severity: DriftSeverity;// 'none' | 'low' | 'medium' | 'high' | 'critical'weights: MethodWeights;// Effective weights used};alerted: boolean;summary: string;durationMs: number;}
CanaryReport
interfaceCanaryReport{id: string;createdAt: string;canaryCount: number;meanSimilarity: number;minSimilarity: number;perCanarySimilarities: number[];driftScore: number;// 1 - meanSimilaritymodelChanged: boolean;isInitialBaseline: boolean;alerted: boolean;modelId: string;durationMs: number;}
MethodResult
interfaceMethodResult{score: number;// Drift score in [0, 1]computed: boolean;// Whether this method was runinterpretation: string;// Human-readable interpretationdetails?: Record<string,unknown>;// Method-specific details}
EmbedFn
typeEmbedFn=(texts: string[])=>Promise<number[][]>;
DriftSeverity
typeDriftSeverity='none'|'low'|'medium'|'high'|'critical';
DriftErrorCode
typeDriftErrorCode=|'EMPTY_INPUT'|'INCONSISTENT_DIMENSIONS'|'INCOMPATIBLE_DIMENSIONS'|'NO_BASELINE'|'INVALID_SNAPSHOT'|'NO_CANARY_BASELINE'|'EMBED_FN_FAILED';
License
MIT
About
Monitor embedding distribution shifts over time
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - SiluPanda/embed-drift: Monitor embedding distribution shifts over time · GitHub
Skip to content
BranchesTags
Repository files navigation
embed-drift
Detect embedding model changes and distribution shifts before they silently degrade your retrieval quality.
npm versionnpm downloadslicensenode
Description
When an embedding model changes -- OpenAI's text-embedding-ada-002 to text-embedding-3-small, a Cohere version bump, or any silent provider update -- the vectors already stored in your database become incompatible with newly produced vectors. Queries return wrong results. No error is thrown, no status code changes, no log line appears. The system looks healthy. The results are wrong.
embed-drift detects this failure before it reaches your users. It monitors embedding distributions over time through two complementary mechanisms:
Canary-based detection embeds a fixed set of reference texts, stores the resulting vectors, and later re-embeds the same texts to check whether the model has changed. This is cheap (embeds only 25 canary texts, not the entire corpus) and catches model changes on the very next check.
Statistical snapshot comparison captures the distribution of a sample of embedding vectors at time T -- centroid, per-dimension variance, pairwise similarity distribution, and more -- and compares that snapshot against a future sample. When the distributions have drifted beyond configurable thresholds, embed-drift computes a composite drift score, classifies severity, and fires alert callbacks.
Zero runtime dependencies. Pure TypeScript. All statistical computations are self-contained.
Installation
npm install embed-drift
Requires Node.js 18 or later.
Quick Start
import{createMonitor}from'embed-drift';// Create a drift monitor for your embedding modelconstmonitor=createMonitor({modelId: 'text-embedding-3-small',alertSeverity: 'high',onDrift: (report)=>{console.warn('Embedding drift detected:',report.summary);},});// Take a baseline snapshot from your current embeddingsconstbaseline=monitor.snapshot(baselineEmbeddings);monitor.setBaseline(baseline);// Later, check new embeddings against the baselineconstreport=monitor.check(newEmbeddings);console.log(report.composite.severity);// 'none' | 'low' | 'medium' | 'high' | 'critical'// Detect silent model changes using canary textsconstcanaryReport=awaitmonitor.checkCanaries(embedFn);if(canaryReport.modelChanged){console.error('Embedding model has changed!');}
Persisting Snapshots
// Save a snapshot to diskmonitor.saveSnapshot(baseline,'./snapshots/baseline.json');// Load it back laterconstloaded=monitor.loadSnapshot('./snapshots/baseline.json');monitor.setBaseline(loaded);
CI/CD Gate
import{createMonitor}from'embed-drift';constmonitor=createMonitor({modelId: 'text-embedding-3-small'});constbaseline=monitor.loadSnapshot('./snapshots/production-baseline.json');monitor.setBaseline(baseline);constreport=monitor.check(newEmbeddings);if(report.composite.severity==='high'||report.composite.severity==='critical'){console.error('Drift too high -- block deployment until re-indexing is complete.');process.exit(1);}
Features
  • Canary-based model change detection -- Embeds a fixed corpus of 25 diverse reference texts and compares their embeddings over time. Detects silent model swaps, version bumps, and provider changes within a single check cycle.
  • Five complementary drift detection methods -- Centroid shift, pairwise cosine similarity distribution, dimension-wise statistics (Cohen's d + KS-like statistic), Maximum Mean Discrepancy (MMD) approximation with random Fourier features, and canary comparison. Each method produces a normalized score in [0, 1].
  • Composite drift scoring -- Weighted average of all method scores with configurable per-method weights. Automatic weight renormalization when methods are disabled or data is unavailable.
  • Severity classification -- Composite scores are classified into five actionable bands: none, low, medium, high, critical. Model changes always produce critical severity.
  • Configurable alerting -- Set severity thresholds and per-method score thresholds. Register an onDrift callback to integrate with any monitoring system. Supports both synchronous and asynchronous callbacks.
  • Snapshot persistence -- Save and load statistical snapshots as portable JSON files. Snapshots are compact (typically 10-900 KB depending on dimensionality and sample size) and work across processes, machines, and time.
  • Zero runtime dependencies -- All statistical computations are self-contained TypeScript. No native modules, no WASM, no Python bridge.
  • Full TypeScript support -- Complete type definitions for all exports. Strict mode compatible.
API Reference
Exports
import{createMonitor,DriftError,DEFAULT_CANARY_TEXTS,}from'embed-drift';importtype{EmbedFn,DriftSeverity,MethodResult,MethodThresholds,MethodWeights,SnapshotOptions,CheckOptions,Snapshot,DriftReport,CanaryReport,DriftMonitorOptions,DriftMonitor,DriftErrorCode,}from'embed-drift';
createMonitor(options: DriftMonitorOptions): DriftMonitor
Creates a drift monitor instance. All drift detection state and configuration is encapsulated in the returned object.
Options (DriftMonitorOptions):
OptionTypeDefaultDescription
modelIdstring--Required. The embedding model identifier.
canaryTextsstring[][]Additional canary texts to append to the built-in corpus.
replaceDefaultCanariesbooleanfalseIf true, use only canaryTexts instead of built-in corpus + custom.
canaryThresholdnumber0.95Mean cosine similarity below which modelChanged is declared.
alertSeverityDriftSeverity'high'Minimum severity to fire the onDrift callback.
thresholdsPartial<MethodThresholds>{}Per-method score overrides that trigger an alert.
onDrift(report) => void | Promise<void>undefinedCallback invoked when an alert fires. Async errors are swallowed.
methodWeightsPartial<MethodWeights>see belowWeights for the composite drift score.
enabledMethods{ centroid?, pairwise?, dimensionWise?, mmd? }all trueDisable specific drift methods.
mmdRandomFeaturesnumber100Number of random Fourier features for MMD approximation.
pairwiseSamplePairsnumber500Number of random pairs sampled for pairwise similarity estimation.
Default composite weights:
{canary: 0.35,centroid: 0.15,pairwise: 0.20,dimensionWise: 0.15,mmd: 0.15}
monitor.snapshot(embeddings, options?): Snapshot
Computes a statistical snapshot of the provided embedding vectors.
Parameters:
ParameterTypeDescription
embeddingsnumber[][]At least 2 vectors of consistent dimensionality.
options.sampleSizenumberNumber of vectors to store for KS/MMD computation. Default: 50.
options.metadataRecord<string, unknown>Caller-provided key-value metadata attached to the snapshot.
Returns: A Snapshot object containing the model ID, centroid, per-dimension variance, pairwise similarity statistics, a 20-bin similarity histogram, and a random sample of vectors.
Throws:
  • DriftError('EMPTY_INPUT') if fewer than 2 vectors are given.
  • DriftError('INCONSISTENT_DIMENSIONS') if vectors have different lengths.
monitor.compare(snapshotA, snapshotB): DriftReport
Compares two snapshots and returns a DriftReport with per-method drift scores, a composite score, and severity classification.
Parameters:
ParameterTypeDescription
snapshotASnapshotThe reference (baseline) snapshot.
snapshotBSnapshotThe new snapshot to compare against the baseline.
Returns: A DriftReport with all per-method results, composite score, severity, alert status, and a human-readable summary.
Throws:
  • DriftError('INCOMPATIBLE_DIMENSIONS') if the two snapshots have different dimensionalities.
When snapshotA.modelId !== snapshotB.modelId, the report sets modelChanged: true and severity to critical.
monitor.setBaseline(snapshot): void
Stores a snapshot as the baseline for subsequent check() calls.
monitor.getBaseline(): Snapshot | undefined
Returns the currently stored baseline snapshot, or undefined if none is set.
monitor.check(embeddings, options?): DriftReport
Creates a new snapshot from embeddings and compares it against the stored baseline.
Parameters:
ParameterTypeDescription
embeddingsnumber[][]New embedding vectors to compare against the baseline.
options.snapshotOptionsSnapshotOptionsOptions forwarded to snapshot creation.
Returns: A DriftReport.
Throws:
  • DriftError('NO_BASELINE') if no baseline has been set via setBaseline().
monitor.checkCanaries(embedFn): Promise<CanaryReport>
Embeds the configured canary texts using embedFn and compares against stored reference embeddings.
Parameters:
ParameterTypeDescription
embedFn(texts: string[]) => Promise<number[][]>A function that embeds an array of texts and returns vectors.
Behavior:
  • On the first call, establishes the reference baseline. Returns a CanaryReport with isInitialBaseline: true and driftScore: 0.
  • On subsequent calls, computes per-canary cosine similarities and returns modelChanged: true when the mean similarity falls below canaryThreshold.
Returns: A CanaryReport.
Throws:
  • DriftError('EMBED_FN_FAILED') if embedFn throws.
monitor.setCanaryBaseline(canaryEmbeddings): void
Explicitly sets the canary reference embeddings without calling checkCanaries. Useful for loading a persisted canary baseline.
Parameters:
ParameterTypeDescription
canaryEmbeddingsnumber[][]Pre-computed canary embeddings (one per canary text).
Throws:
  • DriftError('EMPTY_INPUT') if the array is empty.
monitor.getCanaryTexts(): string[]
Returns the resolved canary text array (built-in + custom, or custom-only if replaceDefaultCanaries: true).
monitor.alert(report): boolean
Evaluates a DriftReport or CanaryReport against configured thresholds and returns true if an alert should fire. Does not invoke the onDrift callback.
An alert fires if:
  • The report severity meets or exceeds alertSeverity, OR
  • Any per-method score exceeds its configured threshold in thresholds.
monitor.saveSnapshot(snapshot, filePath): void
Writes a snapshot as pretty-printed JSON to the given file path.
monitor.loadSnapshot(filePath): Snapshot
Reads and validates a snapshot from a JSON file. Validates all required fields and dimensional consistency.
Throws:
  • DriftError('INVALID_SNAPSHOT') if the file is missing, not valid JSON, or fails schema validation.
DriftError
Custom error class extending Error with a code property for programmatic error handling.
import{DriftError}from'embed-drift';try{monitor.check(embeddings);}catch(err){if(errinstanceofDriftError){console.error(`Drift error [${err.code}]: ${err.message}`);}}
DEFAULT_CANARY_TEXTS
A frozen array of 25 diverse English reference texts spanning technical documentation, scientific language, legal text, casual conversation, news, medical, creative writing, mathematical, instructional, and philosophical domains. Used as the default canary corpus for model fingerprinting.
import{DEFAULT_CANARY_TEXTS}from'embed-drift';console.log(DEFAULT_CANARY_TEXTS.length);// 25
Configuration
Composite Weights
The composite drift score is a weighted average of per-method scores. Weights are renormalized when methods are disabled or their data is unavailable.
constmonitor=createMonitor({modelId: 'text-embedding-3-small',methodWeights: {canary: 0.40,// Increase canary influencecentroid: 0.10,pairwise: 0.20,dimensionWise: 0.15,mmd: 0.15,},});
Disabling Methods
Disable individual drift detection methods when they are not needed:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',enabledMethods: {mmd: false,// Skip MMD computationdimensionWise: false,// Skip dimension-wise analysis},});
Disabled methods report computed: false and score 0. Their weights are redistributed to the remaining active methods.
Alert Thresholds
Alerts fire when severity meets or exceeds alertSeverity, or when any per-method score exceeds its configured threshold:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',alertSeverity: 'high',thresholds: {composite: 0.40,canary: 0.05,centroid: 0.30,},onDrift: (report)=>{// Send to your monitoring systemwebhook.post('/alerts/embedding-drift',report);},});
Custom Canary Texts
Add domain-specific canary texts for increased sensitivity:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',canaryTexts: ['The plaintiff alleges breach of fiduciary duty under Section 14(a).','Amortization of goodwill is calculated on a straight-line basis.',],});// Uses all 25 default canaries + 2 custom = 27 totalconstmonitorCustomOnly=createMonitor({modelId: 'text-embedding-3-small',canaryTexts: ['My custom canary text.'],replaceDefaultCanaries: true,});// Uses only the 1 custom canary text
Drift Detection Methods
embed-drift implements five complementary methods. Each produces a normalized score in [0, 1].
Centroid Shift
Measures the cosine distance between the mean embedding vectors (centroids) of two snapshots. Detects global shifts in the embedding space. Computational cost: O(n * d) where n is the sample size and d is the dimensionality.
Pairwise Cosine Similarity Distribution
Compares the distribution of pairwise cosine similarities between two snapshots. Captures changes in how embeddings are spread relative to each other -- how compact or diffuse the distribution is -- even when the centroid stays the same.
Dimension-Wise Statistics
Analyzes per-dimension statistics using Cohen's d effect size and KS-like statistics across sample vectors. The Cohen's d score identifies which specific dimensions have shifted, while the KS statistic captures distributional shape changes (bimodality, heavy tails) that mean and variance alone do not surface. The two scores are blended equally.
MMD (Maximum Mean Discrepancy)
Uses Maximum Mean Discrepancy with random Fourier features (random kitchen sinks approximation) to measure the distance between two embedding distributions in a kernel-induced feature space. The RBF kernel bandwidth is set via the median heuristic. Sensitive to all moments of the distribution difference. Configurable via mmdRandomFeatures (default: 100).
Canary Texts
Embeds a fixed corpus of diverse reference texts and compares their embeddings over time. Detects silent model changes (provider swaps, version updates) by monitoring whether the same inputs produce the same outputs. The primary and cheapest signal for model change detection.
Severity Bands
Composite ScoreSeverityRecommended Action
0.00 -- 0.05noneNo action needed. Distribution is stable.
0.05 -- 0.20lowMonitor. Normal content variation.
0.20 -- 0.40mediumInvestigate. Consider partial re-indexing.
0.40 -- 0.70highRe-embed recommended. Significant drift detected.
0.70 -- 1.00criticalRe-embed immediately.
A confirmed model change (different model IDs or canary mean similarity below threshold) always produces critical severity regardless of the composite score.
Error Handling
All errors thrown by embed-drift are instances of DriftError with a code property for programmatic handling:
CodeWhen It Is Thrown
EMPTY_INPUTEmbedding array has fewer than 2 vectors, or setCanaryBaseline receives an empty array.
INCONSISTENT_DIMENSIONSVectors in the input array have different dimensionalities.
INCOMPATIBLE_DIMENSIONSTwo snapshots being compared have different dimensionalities.
NO_BASELINEcheck() called before setBaseline().
INVALID_SNAPSHOTLoaded snapshot file is missing, not valid JSON, or fails schema validation.
NO_CANARY_BASELINECanary comparison attempted without a reference baseline.
EMBED_FN_FAILEDThe embedding function passed to checkCanaries() threw an error.
import{DriftError}from'embed-drift';try{constreport=monitor.check(newEmbeddings);}catch(err){if(errinstanceofDriftError){switch(err.code){case'NO_BASELINE':
console.error('Set a baseline before calling check().');break;case'INCOMPATIBLE_DIMENSIONS':
console.error('Snapshot dimensions do not match.');break;default:
console.error(`Unexpected drift error: ${err.code}`);}}}
Advanced Usage
Scheduled Monitoring
Run periodic drift checks as part of a cron job or background worker:
import{createMonitor}from'embed-drift';constmonitor=createMonitor({modelId: 'text-embedding-3-small',alertSeverity: 'medium',onDrift: async(report)=>{awaitsendSlackAlert(`Embedding drift detected: ${report.summary}`);},});// Load the production baselineconstbaseline=monitor.loadSnapshot('./snapshots/production-baseline.json');monitor.setBaseline(baseline);// Sample current embeddings from your vector databaseconstcurrentSample=awaitsampleFromVectorDB(1000);// Check for driftconstreport=monitor.check(currentSample);console.log(`Severity: ${report.composite.severity}, Score: ${report.composite.score}`);
Canary-Based Model Monitoring
Detect model changes with minimal API cost:
import{createMonitor}from'embed-drift';constmonitor=createMonitor({modelId: 'text-embedding-3-small',canaryThreshold: 0.95,onDrift: (report)=>{if('modelChanged'inreport&&report.modelChanged){triggerReindexingPipeline();}},});// On first run, establishes the canary baselineconstembedFn=async(texts: string[])=>{returnopenai.embeddings.create({model: 'text-embedding-3-small',input: texts}).then(res=>res.data.map(d=>d.embedding));};constreport=awaitmonitor.checkCanaries(embedFn);if(report.isInitialBaseline){console.log('Canary baseline established.');}elseif(report.modelChanged){console.error('Model changed! Drift score:',report.driftScore);}else{console.log('Model unchanged. Mean similarity:',report.meanSimilarity);}
Comparing Two Snapshots Directly
Compare snapshots without managing baseline state:
constmonitor=createMonitor({modelId: 'text-embedding-3-small'});constsnapshotA=monitor.loadSnapshot('./snapshots/2025-01-baseline.json');constsnapshotB=monitor.loadSnapshot('./snapshots/2025-03-current.json');constreport=monitor.compare(snapshotA,snapshotB);console.log('Composite score:',report.composite.score);console.log('Severity:',report.composite.severity);console.log('Centroid drift:',report.methods.centroid.score);console.log('Pairwise drift:',report.methods.pairwise.score);console.log('MMD drift:',report.methods.mmd.score);console.log('Summary:',report.summary);
Pre-Computing Canary Baselines
Load a previously saved canary baseline to avoid re-establishing on every restart:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',replaceDefaultCanaries: true,canaryTexts: ['My domain-specific canary text.'],});// Load saved canary embeddingsconstsavedCanaries=JSON.parse(readFileSync('./canary-baseline.json','utf-8'));monitor.setCanaryBaseline(savedCanaries);// Now checkCanaries compares against the loaded baselineconstreport=awaitmonitor.checkCanaries(embedFn);
Types
All types are exported for use in TypeScript projects:
Snapshot
interfaceSnapshot{id: string;// UUID v4createdAt: string;// ISO 8601 timestampmodelId: string;// Embedding model identifierdimensionality: number;// Vector dimensionssampleCount: number;// Number of input vectorscentroid: number[];// Element-wise mean vectorvariance: number[];// Per-dimension variancemeanPairwiseSimilarity: number;// Mean cosine similarity across sampled pairsstdPairwiseSimilarity: number;// Std dev of pairwise cosine similaritiessimilarityHistogram: number[];// 20-bin histogram from -1.0 to 1.0sampleVectors: number[][];// Random sample of vectors for KS/MMDcanaryEmbeddings?: number[][];// Canary text embeddings (optional)metadata?: Record<string,unknown>;// Caller-provided metadata (optional)}
DriftReport
interfaceDriftReport{id: string;createdAt: string;snapshotIds: [string,string];modelIds: [string,string];modelChanged: boolean;methods: {canary: MethodResult;centroid: MethodResult;pairwise: MethodResult;dimensionWise: MethodResult;mmd: MethodResult;};composite: {score: number;// Weighted average in [0, 1]severity: DriftSeverity;// 'none' | 'low' | 'medium' | 'high' | 'critical'weights: MethodWeights;// Effective weights used};alerted: boolean;summary: string;durationMs: number;}
CanaryReport
interfaceCanaryReport{id: string;createdAt: string;canaryCount: number;meanSimilarity: number;minSimilarity: number;perCanarySimilarities: number[];driftScore: number;// 1 - meanSimilaritymodelChanged: boolean;isInitialBaseline: boolean;alerted: boolean;modelId: string;durationMs: number;}
MethodResult
interfaceMethodResult{score: number;// Drift score in [0, 1]computed: boolean;// Whether this method was runinterpretation: string;// Human-readable interpretationdetails?: Record<string,unknown>;// Method-specific details}
EmbedFn
typeEmbedFn=(texts: string[])=>Promise<number[][]>;
DriftSeverity
typeDriftSeverity='none'|'low'|'medium'|'high'|'critical';
DriftErrorCode
typeDriftErrorCode=|'EMPTY_INPUT'|'INCONSISTENT_DIMENSIONS'|'INCOMPATIBLE_DIMENSIONS'|'NO_BASELINE'|'INVALID_SNAPSHOT'|'NO_CANARY_BASELINE'|'EMBED_FN_FAILED';
License
MIT
About
Monitor embedding distribution shifts over time
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
GitHub - SiluPanda/embed-drift: Monitor embedding distribution shifts over time · GitHub
Skip to content
BranchesTags
Repository files navigation
embed-drift
Detect embedding model changes and distribution shifts before they silently degrade your retrieval quality.
npm versionnpm downloadslicensenode
Description
When an embedding model changes -- OpenAI's text-embedding-ada-002 to text-embedding-3-small, a Cohere version bump, or any silent provider update -- the vectors already stored in your database become incompatible with newly produced vectors. Queries return wrong results. No error is thrown, no status code changes, no log line appears. The system looks healthy. The results are wrong.
embed-drift detects this failure before it reaches your users. It monitors embedding distributions over time through two complementary mechanisms:
Canary-based detection embeds a fixed set of reference texts, stores the resulting vectors, and later re-embeds the same texts to check whether the model has changed. This is cheap (embeds only 25 canary texts, not the entire corpus) and catches model changes on the very next check.
Statistical snapshot comparison captures the distribution of a sample of embedding vectors at time T -- centroid, per-dimension variance, pairwise similarity distribution, and more -- and compares that snapshot against a future sample. When the distributions have drifted beyond configurable thresholds, embed-drift computes a composite drift score, classifies severity, and fires alert callbacks.
Zero runtime dependencies. Pure TypeScript. All statistical computations are self-contained.
Installation
npm install embed-drift
Requires Node.js 18 or later.
Quick Start
import{createMonitor}from'embed-drift';// Create a drift monitor for your embedding modelconstmonitor=createMonitor({modelId: 'text-embedding-3-small',alertSeverity: 'high',onDrift: (report)=>{console.warn('Embedding drift detected:',report.summary);},});// Take a baseline snapshot from your current embeddingsconstbaseline=monitor.snapshot(baselineEmbeddings);monitor.setBaseline(baseline);// Later, check new embeddings against the baselineconstreport=monitor.check(newEmbeddings);console.log(report.composite.severity);// 'none' | 'low' | 'medium' | 'high' | 'critical'// Detect silent model changes using canary textsconstcanaryReport=awaitmonitor.checkCanaries(embedFn);if(canaryReport.modelChanged){console.error('Embedding model has changed!');}
Persisting Snapshots
// Save a snapshot to diskmonitor.saveSnapshot(baseline,'./snapshots/baseline.json');// Load it back laterconstloaded=monitor.loadSnapshot('./snapshots/baseline.json');monitor.setBaseline(loaded);
CI/CD Gate
import{createMonitor}from'embed-drift';constmonitor=createMonitor({modelId: 'text-embedding-3-small'});constbaseline=monitor.loadSnapshot('./snapshots/production-baseline.json');monitor.setBaseline(baseline);constreport=monitor.check(newEmbeddings);if(report.composite.severity==='high'||report.composite.severity==='critical'){console.error('Drift too high -- block deployment until re-indexing is complete.');process.exit(1);}
Features
  • Canary-based model change detection -- Embeds a fixed corpus of 25 diverse reference texts and compares their embeddings over time. Detects silent model swaps, version bumps, and provider changes within a single check cycle.
  • Five complementary drift detection methods -- Centroid shift, pairwise cosine similarity distribution, dimension-wise statistics (Cohen's d + KS-like statistic), Maximum Mean Discrepancy (MMD) approximation with random Fourier features, and canary comparison. Each method produces a normalized score in [0, 1].
  • Composite drift scoring -- Weighted average of all method scores with configurable per-method weights. Automatic weight renormalization when methods are disabled or data is unavailable.
  • Severity classification -- Composite scores are classified into five actionable bands: none, low, medium, high, critical. Model changes always produce critical severity.
  • Configurable alerting -- Set severity thresholds and per-method score thresholds. Register an onDrift callback to integrate with any monitoring system. Supports both synchronous and asynchronous callbacks.
  • Snapshot persistence -- Save and load statistical snapshots as portable JSON files. Snapshots are compact (typically 10-900 KB depending on dimensionality and sample size) and work across processes, machines, and time.
  • Zero runtime dependencies -- All statistical computations are self-contained TypeScript. No native modules, no WASM, no Python bridge.
  • Full TypeScript support -- Complete type definitions for all exports. Strict mode compatible.
API Reference
Exports
import{createMonitor,DriftError,DEFAULT_CANARY_TEXTS,}from'embed-drift';importtype{EmbedFn,DriftSeverity,MethodResult,MethodThresholds,MethodWeights,SnapshotOptions,CheckOptions,Snapshot,DriftReport,CanaryReport,DriftMonitorOptions,DriftMonitor,DriftErrorCode,}from'embed-drift';
createMonitor(options: DriftMonitorOptions): DriftMonitor
Creates a drift monitor instance. All drift detection state and configuration is encapsulated in the returned object.
Options (DriftMonitorOptions):
OptionTypeDefaultDescription
modelIdstring--Required. The embedding model identifier.
canaryTextsstring[][]Additional canary texts to append to the built-in corpus.
replaceDefaultCanariesbooleanfalseIf true, use only canaryTexts instead of built-in corpus + custom.
canaryThresholdnumber0.95Mean cosine similarity below which modelChanged is declared.
alertSeverityDriftSeverity'high'Minimum severity to fire the onDrift callback.
thresholdsPartial<MethodThresholds>{}Per-method score overrides that trigger an alert.
onDrift(report) => void | Promise<void>undefinedCallback invoked when an alert fires. Async errors are swallowed.
methodWeightsPartial<MethodWeights>see belowWeights for the composite drift score.
enabledMethods{ centroid?, pairwise?, dimensionWise?, mmd? }all trueDisable specific drift methods.
mmdRandomFeaturesnumber100Number of random Fourier features for MMD approximation.
pairwiseSamplePairsnumber500Number of random pairs sampled for pairwise similarity estimation.
Default composite weights:
{canary: 0.35,centroid: 0.15,pairwise: 0.20,dimensionWise: 0.15,mmd: 0.15}
monitor.snapshot(embeddings, options?): Snapshot
Computes a statistical snapshot of the provided embedding vectors.
Parameters:
ParameterTypeDescription
embeddingsnumber[][]At least 2 vectors of consistent dimensionality.
options.sampleSizenumberNumber of vectors to store for KS/MMD computation. Default: 50.
options.metadataRecord<string, unknown>Caller-provided key-value metadata attached to the snapshot.
Returns: A Snapshot object containing the model ID, centroid, per-dimension variance, pairwise similarity statistics, a 20-bin similarity histogram, and a random sample of vectors.
Throws:
  • DriftError('EMPTY_INPUT') if fewer than 2 vectors are given.
  • DriftError('INCONSISTENT_DIMENSIONS') if vectors have different lengths.
monitor.compare(snapshotA, snapshotB): DriftReport
Compares two snapshots and returns a DriftReport with per-method drift scores, a composite score, and severity classification.
Parameters:
ParameterTypeDescription
snapshotASnapshotThe reference (baseline) snapshot.
snapshotBSnapshotThe new snapshot to compare against the baseline.
Returns: A DriftReport with all per-method results, composite score, severity, alert status, and a human-readable summary.
Throws:
  • DriftError('INCOMPATIBLE_DIMENSIONS') if the two snapshots have different dimensionalities.
When snapshotA.modelId !== snapshotB.modelId, the report sets modelChanged: true and severity to critical.
monitor.setBaseline(snapshot): void
Stores a snapshot as the baseline for subsequent check() calls.
monitor.getBaseline(): Snapshot | undefined
Returns the currently stored baseline snapshot, or undefined if none is set.
monitor.check(embeddings, options?): DriftReport
Creates a new snapshot from embeddings and compares it against the stored baseline.
Parameters:
ParameterTypeDescription
embeddingsnumber[][]New embedding vectors to compare against the baseline.
options.snapshotOptionsSnapshotOptionsOptions forwarded to snapshot creation.
Returns: A DriftReport.
Throws:
  • DriftError('NO_BASELINE') if no baseline has been set via setBaseline().
monitor.checkCanaries(embedFn): Promise<CanaryReport>
Embeds the configured canary texts using embedFn and compares against stored reference embeddings.
Parameters:
ParameterTypeDescription
embedFn(texts: string[]) => Promise<number[][]>A function that embeds an array of texts and returns vectors.
Behavior:
  • On the first call, establishes the reference baseline. Returns a CanaryReport with isInitialBaseline: true and driftScore: 0.
  • On subsequent calls, computes per-canary cosine similarities and returns modelChanged: true when the mean similarity falls below canaryThreshold.
Returns: A CanaryReport.
Throws:
  • DriftError('EMBED_FN_FAILED') if embedFn throws.
monitor.setCanaryBaseline(canaryEmbeddings): void
Explicitly sets the canary reference embeddings without calling checkCanaries. Useful for loading a persisted canary baseline.
Parameters:
ParameterTypeDescription
canaryEmbeddingsnumber[][]Pre-computed canary embeddings (one per canary text).
Throws:
  • DriftError('EMPTY_INPUT') if the array is empty.
monitor.getCanaryTexts(): string[]
Returns the resolved canary text array (built-in + custom, or custom-only if replaceDefaultCanaries: true).
monitor.alert(report): boolean
Evaluates a DriftReport or CanaryReport against configured thresholds and returns true if an alert should fire. Does not invoke the onDrift callback.
An alert fires if:
  • The report severity meets or exceeds alertSeverity, OR
  • Any per-method score exceeds its configured threshold in thresholds.
monitor.saveSnapshot(snapshot, filePath): void
Writes a snapshot as pretty-printed JSON to the given file path.
monitor.loadSnapshot(filePath): Snapshot
Reads and validates a snapshot from a JSON file. Validates all required fields and dimensional consistency.
Throws:
  • DriftError('INVALID_SNAPSHOT') if the file is missing, not valid JSON, or fails schema validation.
DriftError
Custom error class extending Error with a code property for programmatic error handling.
import{DriftError}from'embed-drift';try{monitor.check(embeddings);}catch(err){if(errinstanceofDriftError){console.error(`Drift error [${err.code}]: ${err.message}`);}}
DEFAULT_CANARY_TEXTS
A frozen array of 25 diverse English reference texts spanning technical documentation, scientific language, legal text, casual conversation, news, medical, creative writing, mathematical, instructional, and philosophical domains. Used as the default canary corpus for model fingerprinting.
import{DEFAULT_CANARY_TEXTS}from'embed-drift';console.log(DEFAULT_CANARY_TEXTS.length);// 25
Configuration
Composite Weights
The composite drift score is a weighted average of per-method scores. Weights are renormalized when methods are disabled or their data is unavailable.
constmonitor=createMonitor({modelId: 'text-embedding-3-small',methodWeights: {canary: 0.40,// Increase canary influencecentroid: 0.10,pairwise: 0.20,dimensionWise: 0.15,mmd: 0.15,},});
Disabling Methods
Disable individual drift detection methods when they are not needed:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',enabledMethods: {mmd: false,// Skip MMD computationdimensionWise: false,// Skip dimension-wise analysis},});
Disabled methods report computed: false and score 0. Their weights are redistributed to the remaining active methods.
Alert Thresholds
Alerts fire when severity meets or exceeds alertSeverity, or when any per-method score exceeds its configured threshold:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',alertSeverity: 'high',thresholds: {composite: 0.40,canary: 0.05,centroid: 0.30,},onDrift: (report)=>{// Send to your monitoring systemwebhook.post('/alerts/embedding-drift',report);},});
Custom Canary Texts
Add domain-specific canary texts for increased sensitivity:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',canaryTexts: ['The plaintiff alleges breach of fiduciary duty under Section 14(a).','Amortization of goodwill is calculated on a straight-line basis.',],});// Uses all 25 default canaries + 2 custom = 27 totalconstmonitorCustomOnly=createMonitor({modelId: 'text-embedding-3-small',canaryTexts: ['My custom canary text.'],replaceDefaultCanaries: true,});// Uses only the 1 custom canary text
Drift Detection Methods
embed-drift implements five complementary methods. Each produces a normalized score in [0, 1].
Centroid Shift
Measures the cosine distance between the mean embedding vectors (centroids) of two snapshots. Detects global shifts in the embedding space. Computational cost: O(n * d) where n is the sample size and d is the dimensionality.
Pairwise Cosine Similarity Distribution
Compares the distribution of pairwise cosine similarities between two snapshots. Captures changes in how embeddings are spread relative to each other -- how compact or diffuse the distribution is -- even when the centroid stays the same.
Dimension-Wise Statistics
Analyzes per-dimension statistics using Cohen's d effect size and KS-like statistics across sample vectors. The Cohen's d score identifies which specific dimensions have shifted, while the KS statistic captures distributional shape changes (bimodality, heavy tails) that mean and variance alone do not surface. The two scores are blended equally.
MMD (Maximum Mean Discrepancy)
Uses Maximum Mean Discrepancy with random Fourier features (random kitchen sinks approximation) to measure the distance between two embedding distributions in a kernel-induced feature space. The RBF kernel bandwidth is set via the median heuristic. Sensitive to all moments of the distribution difference. Configurable via mmdRandomFeatures (default: 100).
Canary Texts
Embeds a fixed corpus of diverse reference texts and compares their embeddings over time. Detects silent model changes (provider swaps, version updates) by monitoring whether the same inputs produce the same outputs. The primary and cheapest signal for model change detection.
Severity Bands
Composite ScoreSeverityRecommended Action
0.00 -- 0.05noneNo action needed. Distribution is stable.
0.05 -- 0.20lowMonitor. Normal content variation.
0.20 -- 0.40mediumInvestigate. Consider partial re-indexing.
0.40 -- 0.70highRe-embed recommended. Significant drift detected.
0.70 -- 1.00criticalRe-embed immediately.
A confirmed model change (different model IDs or canary mean similarity below threshold) always produces critical severity regardless of the composite score.
Error Handling
All errors thrown by embed-drift are instances of DriftError with a code property for programmatic handling:
CodeWhen It Is Thrown
EMPTY_INPUTEmbedding array has fewer than 2 vectors, or setCanaryBaseline receives an empty array.
INCONSISTENT_DIMENSIONSVectors in the input array have different dimensionalities.
INCOMPATIBLE_DIMENSIONSTwo snapshots being compared have different dimensionalities.
NO_BASELINEcheck() called before setBaseline().
INVALID_SNAPSHOTLoaded snapshot file is missing, not valid JSON, or fails schema validation.
NO_CANARY_BASELINECanary comparison attempted without a reference baseline.
EMBED_FN_FAILEDThe embedding function passed to checkCanaries() threw an error.
import{DriftError}from'embed-drift';try{constreport=monitor.check(newEmbeddings);}catch(err){if(errinstanceofDriftError){switch(err.code){case'NO_BASELINE':
console.error('Set a baseline before calling check().');break;case'INCOMPATIBLE_DIMENSIONS':
console.error('Snapshot dimensions do not match.');break;default:
console.error(`Unexpected drift error: ${err.code}`);}}}
Advanced Usage
Scheduled Monitoring
Run periodic drift checks as part of a cron job or background worker:
import{createMonitor}from'embed-drift';constmonitor=createMonitor({modelId: 'text-embedding-3-small',alertSeverity: 'medium',onDrift: async(report)=>{awaitsendSlackAlert(`Embedding drift detected: ${report.summary}`);},});// Load the production baselineconstbaseline=monitor.loadSnapshot('./snapshots/production-baseline.json');monitor.setBaseline(baseline);// Sample current embeddings from your vector databaseconstcurrentSample=awaitsampleFromVectorDB(1000);// Check for driftconstreport=monitor.check(currentSample);console.log(`Severity: ${report.composite.severity}, Score: ${report.composite.score}`);
Canary-Based Model Monitoring
Detect model changes with minimal API cost:
import{createMonitor}from'embed-drift';constmonitor=createMonitor({modelId: 'text-embedding-3-small',canaryThreshold: 0.95,onDrift: (report)=>{if('modelChanged'inreport&&report.modelChanged){triggerReindexingPipeline();}},});// On first run, establishes the canary baselineconstembedFn=async(texts: string[])=>{returnopenai.embeddings.create({model: 'text-embedding-3-small',input: texts}).then(res=>res.data.map(d=>d.embedding));};constreport=awaitmonitor.checkCanaries(embedFn);if(report.isInitialBaseline){console.log('Canary baseline established.');}elseif(report.modelChanged){console.error('Model changed! Drift score:',report.driftScore);}else{console.log('Model unchanged. Mean similarity:',report.meanSimilarity);}
Comparing Two Snapshots Directly
Compare snapshots without managing baseline state:
constmonitor=createMonitor({modelId: 'text-embedding-3-small'});constsnapshotA=monitor.loadSnapshot('./snapshots/2025-01-baseline.json');constsnapshotB=monitor.loadSnapshot('./snapshots/2025-03-current.json');constreport=monitor.compare(snapshotA,snapshotB);console.log('Composite score:',report.composite.score);console.log('Severity:',report.composite.severity);console.log('Centroid drift:',report.methods.centroid.score);console.log('Pairwise drift:',report.methods.pairwise.score);console.log('MMD drift:',report.methods.mmd.score);console.log('Summary:',report.summary);
Pre-Computing Canary Baselines
Load a previously saved canary baseline to avoid re-establishing on every restart:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',replaceDefaultCanaries: true,canaryTexts: ['My domain-specific canary text.'],});// Load saved canary embeddingsconstsavedCanaries=JSON.parse(readFileSync('./canary-baseline.json','utf-8'));monitor.setCanaryBaseline(savedCanaries);// Now checkCanaries compares against the loaded baselineconstreport=awaitmonitor.checkCanaries(embedFn);
Types
All types are exported for use in TypeScript projects:
Snapshot
interfaceSnapshot{id: string;// UUID v4createdAt: string;// ISO 8601 timestampmodelId: string;// Embedding model identifierdimensionality: number;// Vector dimensionssampleCount: number;// Number of input vectorscentroid: number[];// Element-wise mean vectorvariance: number[];// Per-dimension variancemeanPairwiseSimilarity: number;// Mean cosine similarity across sampled pairsstdPairwiseSimilarity: number;// Std dev of pairwise cosine similaritiessimilarityHistogram: number[];// 20-bin histogram from -1.0 to 1.0sampleVectors: number[][];// Random sample of vectors for KS/MMDcanaryEmbeddings?: number[][];// Canary text embeddings (optional)metadata?: Record<string,unknown>;// Caller-provided metadata (optional)}
DriftReport
interfaceDriftReport{id: string;createdAt: string;snapshotIds: [string,string];modelIds: [string,string];modelChanged: boolean;methods: {canary: MethodResult;centroid: MethodResult;pairwise: MethodResult;dimensionWise: MethodResult;mmd: MethodResult;};composite: {score: number;// Weighted average in [0, 1]severity: DriftSeverity;// 'none' | 'low' | 'medium' | 'high' | 'critical'weights: MethodWeights;// Effective weights used};alerted: boolean;summary: string;durationMs: number;}
CanaryReport
interfaceCanaryReport{id: string;createdAt: string;canaryCount: number;meanSimilarity: number;minSimilarity: number;perCanarySimilarities: number[];driftScore: number;// 1 - meanSimilaritymodelChanged: boolean;isInitialBaseline: boolean;alerted: boolean;modelId: string;durationMs: number;}
MethodResult
interfaceMethodResult{score: number;// Drift score in [0, 1]computed: boolean;// Whether this method was runinterpretation: string;// Human-readable interpretationdetails?: Record<string,unknown>;// Method-specific details}
EmbedFn
typeEmbedFn=(texts: string[])=>Promise<number[][]>;
DriftSeverity
typeDriftSeverity='none'|'low'|'medium'|'high'|'critical';
DriftErrorCode
typeDriftErrorCode=|'EMPTY_INPUT'|'INCONSISTENT_DIMENSIONS'|'INCOMPATIBLE_DIMENSIONS'|'NO_BASELINE'|'INVALID_SNAPSHOT'|'NO_CANARY_BASELINE'|'EMBED_FN_FAILED';
License
MIT
About
Monitor embedding distribution shifts over time
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - SiluPanda/embed-drift: Monitor embedding distribution shifts over time · GitHub
Skip to content
BranchesTags
Repository files navigation
embed-drift
Detect embedding model changes and distribution shifts before they silently degrade your retrieval quality.
npm versionnpm downloadslicensenode
Description
When an embedding model changes -- OpenAI's text-embedding-ada-002 to text-embedding-3-small, a Cohere version bump, or any silent provider update -- the vectors already stored in your database become incompatible with newly produced vectors. Queries return wrong results. No error is thrown, no status code changes, no log line appears. The system looks healthy. The results are wrong.
embed-drift detects this failure before it reaches your users. It monitors embedding distributions over time through two complementary mechanisms:
Canary-based detection embeds a fixed set of reference texts, stores the resulting vectors, and later re-embeds the same texts to check whether the model has changed. This is cheap (embeds only 25 canary texts, not the entire corpus) and catches model changes on the very next check.
Statistical snapshot comparison captures the distribution of a sample of embedding vectors at time T -- centroid, per-dimension variance, pairwise similarity distribution, and more -- and compares that snapshot against a future sample. When the distributions have drifted beyond configurable thresholds, embed-drift computes a composite drift score, classifies severity, and fires alert callbacks.
Zero runtime dependencies. Pure TypeScript. All statistical computations are self-contained.
Installation
npm install embed-drift
Requires Node.js 18 or later.
Quick Start
import{createMonitor}from'embed-drift';// Create a drift monitor for your embedding modelconstmonitor=createMonitor({modelId: 'text-embedding-3-small',alertSeverity: 'high',onDrift: (report)=>{console.warn('Embedding drift detected:',report.summary);},});// Take a baseline snapshot from your current embeddingsconstbaseline=monitor.snapshot(baselineEmbeddings);monitor.setBaseline(baseline);// Later, check new embeddings against the baselineconstreport=monitor.check(newEmbeddings);console.log(report.composite.severity);// 'none' | 'low' | 'medium' | 'high' | 'critical'// Detect silent model changes using canary textsconstcanaryReport=awaitmonitor.checkCanaries(embedFn);if(canaryReport.modelChanged){console.error('Embedding model has changed!');}
Persisting Snapshots
// Save a snapshot to diskmonitor.saveSnapshot(baseline,'./snapshots/baseline.json');// Load it back laterconstloaded=monitor.loadSnapshot('./snapshots/baseline.json');monitor.setBaseline(loaded);
CI/CD Gate
import{createMonitor}from'embed-drift';constmonitor=createMonitor({modelId: 'text-embedding-3-small'});constbaseline=monitor.loadSnapshot('./snapshots/production-baseline.json');monitor.setBaseline(baseline);constreport=monitor.check(newEmbeddings);if(report.composite.severity==='high'||report.composite.severity==='critical'){console.error('Drift too high -- block deployment until re-indexing is complete.');process.exit(1);}
Features
  • Canary-based model change detection -- Embeds a fixed corpus of 25 diverse reference texts and compares their embeddings over time. Detects silent model swaps, version bumps, and provider changes within a single check cycle.
  • Five complementary drift detection methods -- Centroid shift, pairwise cosine similarity distribution, dimension-wise statistics (Cohen's d + KS-like statistic), Maximum Mean Discrepancy (MMD) approximation with random Fourier features, and canary comparison. Each method produces a normalized score in [0, 1].
  • Composite drift scoring -- Weighted average of all method scores with configurable per-method weights. Automatic weight renormalization when methods are disabled or data is unavailable.
  • Severity classification -- Composite scores are classified into five actionable bands: none, low, medium, high, critical. Model changes always produce critical severity.
  • Configurable alerting -- Set severity thresholds and per-method score thresholds. Register an onDrift callback to integrate with any monitoring system. Supports both synchronous and asynchronous callbacks.
  • Snapshot persistence -- Save and load statistical snapshots as portable JSON files. Snapshots are compact (typically 10-900 KB depending on dimensionality and sample size) and work across processes, machines, and time.
  • Zero runtime dependencies -- All statistical computations are self-contained TypeScript. No native modules, no WASM, no Python bridge.
  • Full TypeScript support -- Complete type definitions for all exports. Strict mode compatible.
API Reference
Exports
import{createMonitor,DriftError,DEFAULT_CANARY_TEXTS,}from'embed-drift';importtype{EmbedFn,DriftSeverity,MethodResult,MethodThresholds,MethodWeights,SnapshotOptions,CheckOptions,Snapshot,DriftReport,CanaryReport,DriftMonitorOptions,DriftMonitor,DriftErrorCode,}from'embed-drift';
createMonitor(options: DriftMonitorOptions): DriftMonitor
Creates a drift monitor instance. All drift detection state and configuration is encapsulated in the returned object.
Options (DriftMonitorOptions):
OptionTypeDefaultDescription
modelIdstring--Required. The embedding model identifier.
canaryTextsstring[][]Additional canary texts to append to the built-in corpus.
replaceDefaultCanariesbooleanfalseIf true, use only canaryTexts instead of built-in corpus + custom.
canaryThresholdnumber0.95Mean cosine similarity below which modelChanged is declared.
alertSeverityDriftSeverity'high'Minimum severity to fire the onDrift callback.
thresholdsPartial<MethodThresholds>{}Per-method score overrides that trigger an alert.
onDrift(report) => void | Promise<void>undefinedCallback invoked when an alert fires. Async errors are swallowed.
methodWeightsPartial<MethodWeights>see belowWeights for the composite drift score.
enabledMethods{ centroid?, pairwise?, dimensionWise?, mmd? }all trueDisable specific drift methods.
mmdRandomFeaturesnumber100Number of random Fourier features for MMD approximation.
pairwiseSamplePairsnumber500Number of random pairs sampled for pairwise similarity estimation.
Default composite weights:
{canary: 0.35,centroid: 0.15,pairwise: 0.20,dimensionWise: 0.15,mmd: 0.15}
monitor.snapshot(embeddings, options?): Snapshot
Computes a statistical snapshot of the provided embedding vectors.
Parameters:
ParameterTypeDescription
embeddingsnumber[][]At least 2 vectors of consistent dimensionality.
options.sampleSizenumberNumber of vectors to store for KS/MMD computation. Default: 50.
options.metadataRecord<string, unknown>Caller-provided key-value metadata attached to the snapshot.
Returns: A Snapshot object containing the model ID, centroid, per-dimension variance, pairwise similarity statistics, a 20-bin similarity histogram, and a random sample of vectors.
Throws:
  • DriftError('EMPTY_INPUT') if fewer than 2 vectors are given.
  • DriftError('INCONSISTENT_DIMENSIONS') if vectors have different lengths.
monitor.compare(snapshotA, snapshotB): DriftReport
Compares two snapshots and returns a DriftReport with per-method drift scores, a composite score, and severity classification.
Parameters:
ParameterTypeDescription
snapshotASnapshotThe reference (baseline) snapshot.
snapshotBSnapshotThe new snapshot to compare against the baseline.
Returns: A DriftReport with all per-method results, composite score, severity, alert status, and a human-readable summary.
Throws:
  • DriftError('INCOMPATIBLE_DIMENSIONS') if the two snapshots have different dimensionalities.
When snapshotA.modelId !== snapshotB.modelId, the report sets modelChanged: true and severity to critical.
monitor.setBaseline(snapshot): void
Stores a snapshot as the baseline for subsequent check() calls.
monitor.getBaseline(): Snapshot | undefined
Returns the currently stored baseline snapshot, or undefined if none is set.
monitor.check(embeddings, options?): DriftReport
Creates a new snapshot from embeddings and compares it against the stored baseline.
Parameters:
ParameterTypeDescription
embeddingsnumber[][]New embedding vectors to compare against the baseline.
options.snapshotOptionsSnapshotOptionsOptions forwarded to snapshot creation.
Returns: A DriftReport.
Throws:
  • DriftError('NO_BASELINE') if no baseline has been set via setBaseline().
monitor.checkCanaries(embedFn): Promise<CanaryReport>
Embeds the configured canary texts using embedFn and compares against stored reference embeddings.
Parameters:
ParameterTypeDescription
embedFn(texts: string[]) => Promise<number[][]>A function that embeds an array of texts and returns vectors.
Behavior:
  • On the first call, establishes the reference baseline. Returns a CanaryReport with isInitialBaseline: true and driftScore: 0.
  • On subsequent calls, computes per-canary cosine similarities and returns modelChanged: true when the mean similarity falls below canaryThreshold.
Returns: A CanaryReport.
Throws:
  • DriftError('EMBED_FN_FAILED') if embedFn throws.
monitor.setCanaryBaseline(canaryEmbeddings): void
Explicitly sets the canary reference embeddings without calling checkCanaries. Useful for loading a persisted canary baseline.
Parameters:
ParameterTypeDescription
canaryEmbeddingsnumber[][]Pre-computed canary embeddings (one per canary text).
Throws:
  • DriftError('EMPTY_INPUT') if the array is empty.
monitor.getCanaryTexts(): string[]
Returns the resolved canary text array (built-in + custom, or custom-only if replaceDefaultCanaries: true).
monitor.alert(report): boolean
Evaluates a DriftReport or CanaryReport against configured thresholds and returns true if an alert should fire. Does not invoke the onDrift callback.
An alert fires if:
  • The report severity meets or exceeds alertSeverity, OR
  • Any per-method score exceeds its configured threshold in thresholds.
monitor.saveSnapshot(snapshot, filePath): void
Writes a snapshot as pretty-printed JSON to the given file path.
monitor.loadSnapshot(filePath): Snapshot
Reads and validates a snapshot from a JSON file. Validates all required fields and dimensional consistency.
Throws:
  • DriftError('INVALID_SNAPSHOT') if the file is missing, not valid JSON, or fails schema validation.
DriftError
Custom error class extending Error with a code property for programmatic error handling.
import{DriftError}from'embed-drift';try{monitor.check(embeddings);}catch(err){if(errinstanceofDriftError){console.error(`Drift error [${err.code}]: ${err.message}`);}}
DEFAULT_CANARY_TEXTS
A frozen array of 25 diverse English reference texts spanning technical documentation, scientific language, legal text, casual conversation, news, medical, creative writing, mathematical, instructional, and philosophical domains. Used as the default canary corpus for model fingerprinting.
import{DEFAULT_CANARY_TEXTS}from'embed-drift';console.log(DEFAULT_CANARY_TEXTS.length);// 25
Configuration
Composite Weights
The composite drift score is a weighted average of per-method scores. Weights are renormalized when methods are disabled or their data is unavailable.
constmonitor=createMonitor({modelId: 'text-embedding-3-small',methodWeights: {canary: 0.40,// Increase canary influencecentroid: 0.10,pairwise: 0.20,dimensionWise: 0.15,mmd: 0.15,},});
Disabling Methods
Disable individual drift detection methods when they are not needed:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',enabledMethods: {mmd: false,// Skip MMD computationdimensionWise: false,// Skip dimension-wise analysis},});
Disabled methods report computed: false and score 0. Their weights are redistributed to the remaining active methods.
Alert Thresholds
Alerts fire when severity meets or exceeds alertSeverity, or when any per-method score exceeds its configured threshold:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',alertSeverity: 'high',thresholds: {composite: 0.40,canary: 0.05,centroid: 0.30,},onDrift: (report)=>{// Send to your monitoring systemwebhook.post('/alerts/embedding-drift',report);},});
Custom Canary Texts
Add domain-specific canary texts for increased sensitivity:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',canaryTexts: ['The plaintiff alleges breach of fiduciary duty under Section 14(a).','Amortization of goodwill is calculated on a straight-line basis.',],});// Uses all 25 default canaries + 2 custom = 27 totalconstmonitorCustomOnly=createMonitor({modelId: 'text-embedding-3-small',canaryTexts: ['My custom canary text.'],replaceDefaultCanaries: true,});// Uses only the 1 custom canary text
Drift Detection Methods
embed-drift implements five complementary methods. Each produces a normalized score in [0, 1].
Centroid Shift
Measures the cosine distance between the mean embedding vectors (centroids) of two snapshots. Detects global shifts in the embedding space. Computational cost: O(n * d) where n is the sample size and d is the dimensionality.
Pairwise Cosine Similarity Distribution
Compares the distribution of pairwise cosine similarities between two snapshots. Captures changes in how embeddings are spread relative to each other -- how compact or diffuse the distribution is -- even when the centroid stays the same.
Dimension-Wise Statistics
Analyzes per-dimension statistics using Cohen's d effect size and KS-like statistics across sample vectors. The Cohen's d score identifies which specific dimensions have shifted, while the KS statistic captures distributional shape changes (bimodality, heavy tails) that mean and variance alone do not surface. The two scores are blended equally.
MMD (Maximum Mean Discrepancy)
Uses Maximum Mean Discrepancy with random Fourier features (random kitchen sinks approximation) to measure the distance between two embedding distributions in a kernel-induced feature space. The RBF kernel bandwidth is set via the median heuristic. Sensitive to all moments of the distribution difference. Configurable via mmdRandomFeatures (default: 100).
Canary Texts
Embeds a fixed corpus of diverse reference texts and compares their embeddings over time. Detects silent model changes (provider swaps, version updates) by monitoring whether the same inputs produce the same outputs. The primary and cheapest signal for model change detection.
Severity Bands
Composite ScoreSeverityRecommended Action
0.00 -- 0.05noneNo action needed. Distribution is stable.
0.05 -- 0.20lowMonitor. Normal content variation.
0.20 -- 0.40mediumInvestigate. Consider partial re-indexing.
0.40 -- 0.70highRe-embed recommended. Significant drift detected.
0.70 -- 1.00criticalRe-embed immediately.
A confirmed model change (different model IDs or canary mean similarity below threshold) always produces critical severity regardless of the composite score.
Error Handling
All errors thrown by embed-drift are instances of DriftError with a code property for programmatic handling:
CodeWhen It Is Thrown
EMPTY_INPUTEmbedding array has fewer than 2 vectors, or setCanaryBaseline receives an empty array.
INCONSISTENT_DIMENSIONSVectors in the input array have different dimensionalities.
INCOMPATIBLE_DIMENSIONSTwo snapshots being compared have different dimensionalities.
NO_BASELINEcheck() called before setBaseline().
INVALID_SNAPSHOTLoaded snapshot file is missing, not valid JSON, or fails schema validation.
NO_CANARY_BASELINECanary comparison attempted without a reference baseline.
EMBED_FN_FAILEDThe embedding function passed to checkCanaries() threw an error.
import{DriftError}from'embed-drift';try{constreport=monitor.check(newEmbeddings);}catch(err){if(errinstanceofDriftError){switch(err.code){case'NO_BASELINE':
console.error('Set a baseline before calling check().');break;case'INCOMPATIBLE_DIMENSIONS':
console.error('Snapshot dimensions do not match.');break;default:
console.error(`Unexpected drift error: ${err.code}`);}}}
Advanced Usage
Scheduled Monitoring
Run periodic drift checks as part of a cron job or background worker:
import{createMonitor}from'embed-drift';constmonitor=createMonitor({modelId: 'text-embedding-3-small',alertSeverity: 'medium',onDrift: async(report)=>{awaitsendSlackAlert(`Embedding drift detected: ${report.summary}`);},});// Load the production baselineconstbaseline=monitor.loadSnapshot('./snapshots/production-baseline.json');monitor.setBaseline(baseline);// Sample current embeddings from your vector databaseconstcurrentSample=awaitsampleFromVectorDB(1000);// Check for driftconstreport=monitor.check(currentSample);console.log(`Severity: ${report.composite.severity}, Score: ${report.composite.score}`);
Canary-Based Model Monitoring
Detect model changes with minimal API cost:
import{createMonitor}from'embed-drift';constmonitor=createMonitor({modelId: 'text-embedding-3-small',canaryThreshold: 0.95,onDrift: (report)=>{if('modelChanged'inreport&&report.modelChanged){triggerReindexingPipeline();}},});// On first run, establishes the canary baselineconstembedFn=async(texts: string[])=>{returnopenai.embeddings.create({model: 'text-embedding-3-small',input: texts}).then(res=>res.data.map(d=>d.embedding));};constreport=awaitmonitor.checkCanaries(embedFn);if(report.isInitialBaseline){console.log('Canary baseline established.');}elseif(report.modelChanged){console.error('Model changed! Drift score:',report.driftScore);}else{console.log('Model unchanged. Mean similarity:',report.meanSimilarity);}
Comparing Two Snapshots Directly
Compare snapshots without managing baseline state:
constmonitor=createMonitor({modelId: 'text-embedding-3-small'});constsnapshotA=monitor.loadSnapshot('./snapshots/2025-01-baseline.json');constsnapshotB=monitor.loadSnapshot('./snapshots/2025-03-current.json');constreport=monitor.compare(snapshotA,snapshotB);console.log('Composite score:',report.composite.score);console.log('Severity:',report.composite.severity);console.log('Centroid drift:',report.methods.centroid.score);console.log('Pairwise drift:',report.methods.pairwise.score);console.log('MMD drift:',report.methods.mmd.score);console.log('Summary:',report.summary);
Pre-Computing Canary Baselines
Load a previously saved canary baseline to avoid re-establishing on every restart:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',replaceDefaultCanaries: true,canaryTexts: ['My domain-specific canary text.'],});// Load saved canary embeddingsconstsavedCanaries=JSON.parse(readFileSync('./canary-baseline.json','utf-8'));monitor.setCanaryBaseline(savedCanaries);// Now checkCanaries compares against the loaded baselineconstreport=awaitmonitor.checkCanaries(embedFn);
Types
All types are exported for use in TypeScript projects:
Snapshot
interfaceSnapshot{id: string;// UUID v4createdAt: string;// ISO 8601 timestampmodelId: string;// Embedding model identifierdimensionality: number;// Vector dimensionssampleCount: number;// Number of input vectorscentroid: number[];// Element-wise mean vectorvariance: number[];// Per-dimension variancemeanPairwiseSimilarity: number;// Mean cosine similarity across sampled pairsstdPairwiseSimilarity: number;// Std dev of pairwise cosine similaritiessimilarityHistogram: number[];// 20-bin histogram from -1.0 to 1.0sampleVectors: number[][];// Random sample of vectors for KS/MMDcanaryEmbeddings?: number[][];// Canary text embeddings (optional)metadata?: Record<string,unknown>;// Caller-provided metadata (optional)}
DriftReport
interfaceDriftReport{id: string;createdAt: string;snapshotIds: [string,string];modelIds: [string,string];modelChanged: boolean;methods: {canary: MethodResult;centroid: MethodResult;pairwise: MethodResult;dimensionWise: MethodResult;mmd: MethodResult;};composite: {score: number;// Weighted average in [0, 1]severity: DriftSeverity;// 'none' | 'low' | 'medium' | 'high' | 'critical'weights: MethodWeights;// Effective weights used};alerted: boolean;summary: string;durationMs: number;}
CanaryReport
interfaceCanaryReport{id: string;createdAt: string;canaryCount: number;meanSimilarity: number;minSimilarity: number;perCanarySimilarities: number[];driftScore: number;// 1 - meanSimilaritymodelChanged: boolean;isInitialBaseline: boolean;alerted: boolean;modelId: string;durationMs: number;}
MethodResult
interfaceMethodResult{score: number;// Drift score in [0, 1]computed: boolean;// Whether this method was runinterpretation: string;// Human-readable interpretationdetails?: Record<string,unknown>;// Method-specific details}
EmbedFn
typeEmbedFn=(texts: string[])=>Promise<number[][]>;
DriftSeverity
typeDriftSeverity='none'|'low'|'medium'|'high'|'critical';
DriftErrorCode
typeDriftErrorCode=|'EMPTY_INPUT'|'INCONSISTENT_DIMENSIONS'|'INCOMPATIBLE_DIMENSIONS'|'NO_BASELINE'|'INVALID_SNAPSHOT'|'NO_CANARY_BASELINE'|'EMBED_FN_FAILED';
License
MIT
About
Monitor embedding distribution shifts over time
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - SiluPanda/embed-drift: Monitor embedding distribution shifts over time · GitHub
Skip to content
BranchesTags
Repository files navigation
embed-drift
Detect embedding model changes and distribution shifts before they silently degrade your retrieval quality.
npm versionnpm downloadslicensenode
Description
When an embedding model changes -- OpenAI's text-embedding-ada-002 to text-embedding-3-small, a Cohere version bump, or any silent provider update -- the vectors already stored in your database become incompatible with newly produced vectors. Queries return wrong results. No error is thrown, no status code changes, no log line appears. The system looks healthy. The results are wrong.
embed-drift detects this failure before it reaches your users. It monitors embedding distributions over time through two complementary mechanisms:
Canary-based detection embeds a fixed set of reference texts, stores the resulting vectors, and later re-embeds the same texts to check whether the model has changed. This is cheap (embeds only 25 canary texts, not the entire corpus) and catches model changes on the very next check.
Statistical snapshot comparison captures the distribution of a sample of embedding vectors at time T -- centroid, per-dimension variance, pairwise similarity distribution, and more -- and compares that snapshot against a future sample. When the distributions have drifted beyond configurable thresholds, embed-drift computes a composite drift score, classifies severity, and fires alert callbacks.
Zero runtime dependencies. Pure TypeScript. All statistical computations are self-contained.
Installation
npm install embed-drift
Requires Node.js 18 or later.
Quick Start
import{createMonitor}from'embed-drift';// Create a drift monitor for your embedding modelconstmonitor=createMonitor({modelId: 'text-embedding-3-small',alertSeverity: 'high',onDrift: (report)=>{console.warn('Embedding drift detected:',report.summary);},});// Take a baseline snapshot from your current embeddingsconstbaseline=monitor.snapshot(baselineEmbeddings);monitor.setBaseline(baseline);// Later, check new embeddings against the baselineconstreport=monitor.check(newEmbeddings);console.log(report.composite.severity);// 'none' | 'low' | 'medium' | 'high' | 'critical'// Detect silent model changes using canary textsconstcanaryReport=awaitmonitor.checkCanaries(embedFn);if(canaryReport.modelChanged){console.error('Embedding model has changed!');}
Persisting Snapshots
// Save a snapshot to diskmonitor.saveSnapshot(baseline,'./snapshots/baseline.json');// Load it back laterconstloaded=monitor.loadSnapshot('./snapshots/baseline.json');monitor.setBaseline(loaded);
CI/CD Gate
import{createMonitor}from'embed-drift';constmonitor=createMonitor({modelId: 'text-embedding-3-small'});constbaseline=monitor.loadSnapshot('./snapshots/production-baseline.json');monitor.setBaseline(baseline);constreport=monitor.check(newEmbeddings);if(report.composite.severity==='high'||report.composite.severity==='critical'){console.error('Drift too high -- block deployment until re-indexing is complete.');process.exit(1);}
Features
  • Canary-based model change detection -- Embeds a fixed corpus of 25 diverse reference texts and compares their embeddings over time. Detects silent model swaps, version bumps, and provider changes within a single check cycle.
  • Five complementary drift detection methods -- Centroid shift, pairwise cosine similarity distribution, dimension-wise statistics (Cohen's d + KS-like statistic), Maximum Mean Discrepancy (MMD) approximation with random Fourier features, and canary comparison. Each method produces a normalized score in [0, 1].
  • Composite drift scoring -- Weighted average of all method scores with configurable per-method weights. Automatic weight renormalization when methods are disabled or data is unavailable.
  • Severity classification -- Composite scores are classified into five actionable bands: none, low, medium, high, critical. Model changes always produce critical severity.
  • Configurable alerting -- Set severity thresholds and per-method score thresholds. Register an onDrift callback to integrate with any monitoring system. Supports both synchronous and asynchronous callbacks.
  • Snapshot persistence -- Save and load statistical snapshots as portable JSON files. Snapshots are compact (typically 10-900 KB depending on dimensionality and sample size) and work across processes, machines, and time.
  • Zero runtime dependencies -- All statistical computations are self-contained TypeScript. No native modules, no WASM, no Python bridge.
  • Full TypeScript support -- Complete type definitions for all exports. Strict mode compatible.
API Reference
Exports
import{createMonitor,DriftError,DEFAULT_CANARY_TEXTS,}from'embed-drift';importtype{EmbedFn,DriftSeverity,MethodResult,MethodThresholds,MethodWeights,SnapshotOptions,CheckOptions,Snapshot,DriftReport,CanaryReport,DriftMonitorOptions,DriftMonitor,DriftErrorCode,}from'embed-drift';
createMonitor(options: DriftMonitorOptions): DriftMonitor
Creates a drift monitor instance. All drift detection state and configuration is encapsulated in the returned object.
Options (DriftMonitorOptions):
OptionTypeDefaultDescription
modelIdstring--Required. The embedding model identifier.
canaryTextsstring[][]Additional canary texts to append to the built-in corpus.
replaceDefaultCanariesbooleanfalseIf true, use only canaryTexts instead of built-in corpus + custom.
canaryThresholdnumber0.95Mean cosine similarity below which modelChanged is declared.
alertSeverityDriftSeverity'high'Minimum severity to fire the onDrift callback.
thresholdsPartial<MethodThresholds>{}Per-method score overrides that trigger an alert.
onDrift(report) => void | Promise<void>undefinedCallback invoked when an alert fires. Async errors are swallowed.
methodWeightsPartial<MethodWeights>see belowWeights for the composite drift score.
enabledMethods{ centroid?, pairwise?, dimensionWise?, mmd? }all trueDisable specific drift methods.
mmdRandomFeaturesnumber100Number of random Fourier features for MMD approximation.
pairwiseSamplePairsnumber500Number of random pairs sampled for pairwise similarity estimation.
Default composite weights:
{canary: 0.35,centroid: 0.15,pairwise: 0.20,dimensionWise: 0.15,mmd: 0.15}
monitor.snapshot(embeddings, options?): Snapshot
Computes a statistical snapshot of the provided embedding vectors.
Parameters:
ParameterTypeDescription
embeddingsnumber[][]At least 2 vectors of consistent dimensionality.
options.sampleSizenumberNumber of vectors to store for KS/MMD computation. Default: 50.
options.metadataRecord<string, unknown>Caller-provided key-value metadata attached to the snapshot.
Returns: A Snapshot object containing the model ID, centroid, per-dimension variance, pairwise similarity statistics, a 20-bin similarity histogram, and a random sample of vectors.
Throws:
  • DriftError('EMPTY_INPUT') if fewer than 2 vectors are given.
  • DriftError('INCONSISTENT_DIMENSIONS') if vectors have different lengths.
monitor.compare(snapshotA, snapshotB): DriftReport
Compares two snapshots and returns a DriftReport with per-method drift scores, a composite score, and severity classification.
Parameters:
ParameterTypeDescription
snapshotASnapshotThe reference (baseline) snapshot.
snapshotBSnapshotThe new snapshot to compare against the baseline.
Returns: A DriftReport with all per-method results, composite score, severity, alert status, and a human-readable summary.
Throws:
  • DriftError('INCOMPATIBLE_DIMENSIONS') if the two snapshots have different dimensionalities.
When snapshotA.modelId !== snapshotB.modelId, the report sets modelChanged: true and severity to critical.
monitor.setBaseline(snapshot): void
Stores a snapshot as the baseline for subsequent check() calls.
monitor.getBaseline(): Snapshot | undefined
Returns the currently stored baseline snapshot, or undefined if none is set.
monitor.check(embeddings, options?): DriftReport
Creates a new snapshot from embeddings and compares it against the stored baseline.
Parameters:
ParameterTypeDescription
embeddingsnumber[][]New embedding vectors to compare against the baseline.
options.snapshotOptionsSnapshotOptionsOptions forwarded to snapshot creation.
Returns: A DriftReport.
Throws:
  • DriftError('NO_BASELINE') if no baseline has been set via setBaseline().
monitor.checkCanaries(embedFn): Promise<CanaryReport>
Embeds the configured canary texts using embedFn and compares against stored reference embeddings.
Parameters:
ParameterTypeDescription
embedFn(texts: string[]) => Promise<number[][]>A function that embeds an array of texts and returns vectors.
Behavior:
  • On the first call, establishes the reference baseline. Returns a CanaryReport with isInitialBaseline: true and driftScore: 0.
  • On subsequent calls, computes per-canary cosine similarities and returns modelChanged: true when the mean similarity falls below canaryThreshold.
Returns: A CanaryReport.
Throws:
  • DriftError('EMBED_FN_FAILED') if embedFn throws.
monitor.setCanaryBaseline(canaryEmbeddings): void
Explicitly sets the canary reference embeddings without calling checkCanaries. Useful for loading a persisted canary baseline.
Parameters:
ParameterTypeDescription
canaryEmbeddingsnumber[][]Pre-computed canary embeddings (one per canary text).
Throws:
  • DriftError('EMPTY_INPUT') if the array is empty.
monitor.getCanaryTexts(): string[]
Returns the resolved canary text array (built-in + custom, or custom-only if replaceDefaultCanaries: true).
monitor.alert(report): boolean
Evaluates a DriftReport or CanaryReport against configured thresholds and returns true if an alert should fire. Does not invoke the onDrift callback.
An alert fires if:
  • The report severity meets or exceeds alertSeverity, OR
  • Any per-method score exceeds its configured threshold in thresholds.
monitor.saveSnapshot(snapshot, filePath): void
Writes a snapshot as pretty-printed JSON to the given file path.
monitor.loadSnapshot(filePath): Snapshot
Reads and validates a snapshot from a JSON file. Validates all required fields and dimensional consistency.
Throws:
  • DriftError('INVALID_SNAPSHOT') if the file is missing, not valid JSON, or fails schema validation.
DriftError
Custom error class extending Error with a code property for programmatic error handling.
import{DriftError}from'embed-drift';try{monitor.check(embeddings);}catch(err){if(errinstanceofDriftError){console.error(`Drift error [${err.code}]: ${err.message}`);}}
DEFAULT_CANARY_TEXTS
A frozen array of 25 diverse English reference texts spanning technical documentation, scientific language, legal text, casual conversation, news, medical, creative writing, mathematical, instructional, and philosophical domains. Used as the default canary corpus for model fingerprinting.
import{DEFAULT_CANARY_TEXTS}from'embed-drift';console.log(DEFAULT_CANARY_TEXTS.length);// 25
Configuration
Composite Weights
The composite drift score is a weighted average of per-method scores. Weights are renormalized when methods are disabled or their data is unavailable.
constmonitor=createMonitor({modelId: 'text-embedding-3-small',methodWeights: {canary: 0.40,// Increase canary influencecentroid: 0.10,pairwise: 0.20,dimensionWise: 0.15,mmd: 0.15,},});
Disabling Methods
Disable individual drift detection methods when they are not needed:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',enabledMethods: {mmd: false,// Skip MMD computationdimensionWise: false,// Skip dimension-wise analysis},});
Disabled methods report computed: false and score 0. Their weights are redistributed to the remaining active methods.
Alert Thresholds
Alerts fire when severity meets or exceeds alertSeverity, or when any per-method score exceeds its configured threshold:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',alertSeverity: 'high',thresholds: {composite: 0.40,canary: 0.05,centroid: 0.30,},onDrift: (report)=>{// Send to your monitoring systemwebhook.post('/alerts/embedding-drift',report);},});
Custom Canary Texts
Add domain-specific canary texts for increased sensitivity:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',canaryTexts: ['The plaintiff alleges breach of fiduciary duty under Section 14(a).','Amortization of goodwill is calculated on a straight-line basis.',],});// Uses all 25 default canaries + 2 custom = 27 totalconstmonitorCustomOnly=createMonitor({modelId: 'text-embedding-3-small',canaryTexts: ['My custom canary text.'],replaceDefaultCanaries: true,});// Uses only the 1 custom canary text
Drift Detection Methods
embed-drift implements five complementary methods. Each produces a normalized score in [0, 1].
Centroid Shift
Measures the cosine distance between the mean embedding vectors (centroids) of two snapshots. Detects global shifts in the embedding space. Computational cost: O(n * d) where n is the sample size and d is the dimensionality.
Pairwise Cosine Similarity Distribution
Compares the distribution of pairwise cosine similarities between two snapshots. Captures changes in how embeddings are spread relative to each other -- how compact or diffuse the distribution is -- even when the centroid stays the same.
Dimension-Wise Statistics
Analyzes per-dimension statistics using Cohen's d effect size and KS-like statistics across sample vectors. The Cohen's d score identifies which specific dimensions have shifted, while the KS statistic captures distributional shape changes (bimodality, heavy tails) that mean and variance alone do not surface. The two scores are blended equally.
MMD (Maximum Mean Discrepancy)
Uses Maximum Mean Discrepancy with random Fourier features (random kitchen sinks approximation) to measure the distance between two embedding distributions in a kernel-induced feature space. The RBF kernel bandwidth is set via the median heuristic. Sensitive to all moments of the distribution difference. Configurable via mmdRandomFeatures (default: 100).
Canary Texts
Embeds a fixed corpus of diverse reference texts and compares their embeddings over time. Detects silent model changes (provider swaps, version updates) by monitoring whether the same inputs produce the same outputs. The primary and cheapest signal for model change detection.
Severity Bands
Composite ScoreSeverityRecommended Action
0.00 -- 0.05noneNo action needed. Distribution is stable.
0.05 -- 0.20lowMonitor. Normal content variation.
0.20 -- 0.40mediumInvestigate. Consider partial re-indexing.
0.40 -- 0.70highRe-embed recommended. Significant drift detected.
0.70 -- 1.00criticalRe-embed immediately.
A confirmed model change (different model IDs or canary mean similarity below threshold) always produces critical severity regardless of the composite score.
Error Handling
All errors thrown by embed-drift are instances of DriftError with a code property for programmatic handling:
CodeWhen It Is Thrown
EMPTY_INPUTEmbedding array has fewer than 2 vectors, or setCanaryBaseline receives an empty array.
INCONSISTENT_DIMENSIONSVectors in the input array have different dimensionalities.
INCOMPATIBLE_DIMENSIONSTwo snapshots being compared have different dimensionalities.
NO_BASELINEcheck() called before setBaseline().
INVALID_SNAPSHOTLoaded snapshot file is missing, not valid JSON, or fails schema validation.
NO_CANARY_BASELINECanary comparison attempted without a reference baseline.
EMBED_FN_FAILEDThe embedding function passed to checkCanaries() threw an error.
import{DriftError}from'embed-drift';try{constreport=monitor.check(newEmbeddings);}catch(err){if(errinstanceofDriftError){switch(err.code){case'NO_BASELINE':
console.error('Set a baseline before calling check().');break;case'INCOMPATIBLE_DIMENSIONS':
console.error('Snapshot dimensions do not match.');break;default:
console.error(`Unexpected drift error: ${err.code}`);}}}
Advanced Usage
Scheduled Monitoring
Run periodic drift checks as part of a cron job or background worker:
import{createMonitor}from'embed-drift';constmonitor=createMonitor({modelId: 'text-embedding-3-small',alertSeverity: 'medium',onDrift: async(report)=>{awaitsendSlackAlert(`Embedding drift detected: ${report.summary}`);},});// Load the production baselineconstbaseline=monitor.loadSnapshot('./snapshots/production-baseline.json');monitor.setBaseline(baseline);// Sample current embeddings from your vector databaseconstcurrentSample=awaitsampleFromVectorDB(1000);// Check for driftconstreport=monitor.check(currentSample);console.log(`Severity: ${report.composite.severity}, Score: ${report.composite.score}`);
Canary-Based Model Monitoring
Detect model changes with minimal API cost:
import{createMonitor}from'embed-drift';constmonitor=createMonitor({modelId: 'text-embedding-3-small',canaryThreshold: 0.95,onDrift: (report)=>{if('modelChanged'inreport&&report.modelChanged){triggerReindexingPipeline();}},});// On first run, establishes the canary baselineconstembedFn=async(texts: string[])=>{returnopenai.embeddings.create({model: 'text-embedding-3-small',input: texts}).then(res=>res.data.map(d=>d.embedding));};constreport=awaitmonitor.checkCanaries(embedFn);if(report.isInitialBaseline){console.log('Canary baseline established.');}elseif(report.modelChanged){console.error('Model changed! Drift score:',report.driftScore);}else{console.log('Model unchanged. Mean similarity:',report.meanSimilarity);}
Comparing Two Snapshots Directly
Compare snapshots without managing baseline state:
constmonitor=createMonitor({modelId: 'text-embedding-3-small'});constsnapshotA=monitor.loadSnapshot('./snapshots/2025-01-baseline.json');constsnapshotB=monitor.loadSnapshot('./snapshots/2025-03-current.json');constreport=monitor.compare(snapshotA,snapshotB);console.log('Composite score:',report.composite.score);console.log('Severity:',report.composite.severity);console.log('Centroid drift:',report.methods.centroid.score);console.log('Pairwise drift:',report.methods.pairwise.score);console.log('MMD drift:',report.methods.mmd.score);console.log('Summary:',report.summary);
Pre-Computing Canary Baselines
Load a previously saved canary baseline to avoid re-establishing on every restart:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',replaceDefaultCanaries: true,canaryTexts: ['My domain-specific canary text.'],});// Load saved canary embeddingsconstsavedCanaries=JSON.parse(readFileSync('./canary-baseline.json','utf-8'));monitor.setCanaryBaseline(savedCanaries);// Now checkCanaries compares against the loaded baselineconstreport=awaitmonitor.checkCanaries(embedFn);
Types
All types are exported for use in TypeScript projects:
Snapshot
interfaceSnapshot{id: string;// UUID v4createdAt: string;// ISO 8601 timestampmodelId: string;// Embedding model identifierdimensionality: number;// Vector dimensionssampleCount: number;// Number of input vectorscentroid: number[];// Element-wise mean vectorvariance: number[];// Per-dimension variancemeanPairwiseSimilarity: number;// Mean cosine similarity across sampled pairsstdPairwiseSimilarity: number;// Std dev of pairwise cosine similaritiessimilarityHistogram: number[];// 20-bin histogram from -1.0 to 1.0sampleVectors: number[][];// Random sample of vectors for KS/MMDcanaryEmbeddings?: number[][];// Canary text embeddings (optional)metadata?: Record<string,unknown>;// Caller-provided metadata (optional)}
DriftReport
interfaceDriftReport{id: string;createdAt: string;snapshotIds: [string,string];modelIds: [string,string];modelChanged: boolean;methods: {canary: MethodResult;centroid: MethodResult;pairwise: MethodResult;dimensionWise: MethodResult;mmd: MethodResult;};composite: {score: number;// Weighted average in [0, 1]severity: DriftSeverity;// 'none' | 'low' | 'medium' | 'high' | 'critical'weights: MethodWeights;// Effective weights used};alerted: boolean;summary: string;durationMs: number;}
CanaryReport
interfaceCanaryReport{id: string;createdAt: string;canaryCount: number;meanSimilarity: number;minSimilarity: number;perCanarySimilarities: number[];driftScore: number;// 1 - meanSimilaritymodelChanged: boolean;isInitialBaseline: boolean;alerted: boolean;modelId: string;durationMs: number;}
MethodResult
interfaceMethodResult{score: number;// Drift score in [0, 1]computed: boolean;// Whether this method was runinterpretation: string;// Human-readable interpretationdetails?: Record<string,unknown>;// Method-specific details}
EmbedFn
typeEmbedFn=(texts: string[])=>Promise<number[][]>;
DriftSeverity
typeDriftSeverity='none'|'low'|'medium'|'high'|'critical';
DriftErrorCode
typeDriftErrorCode=|'EMPTY_INPUT'|'INCONSISTENT_DIMENSIONS'|'INCOMPATIBLE_DIMENSIONS'|'NO_BASELINE'|'INVALID_SNAPSHOT'|'NO_CANARY_BASELINE'|'EMBED_FN_FAILED';
License
MIT
About
Monitor embedding distribution shifts over time
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
GitHub - SiluPanda/embed-drift: Monitor embedding distribution shifts over time · GitHub
Skip to content
BranchesTags
Repository files navigation
embed-drift
Detect embedding model changes and distribution shifts before they silently degrade your retrieval quality.
npm versionnpm downloadslicensenode
Description
When an embedding model changes -- OpenAI's text-embedding-ada-002 to text-embedding-3-small, a Cohere version bump, or any silent provider update -- the vectors already stored in your database become incompatible with newly produced vectors. Queries return wrong results. No error is thrown, no status code changes, no log line appears. The system looks healthy. The results are wrong.
embed-drift detects this failure before it reaches your users. It monitors embedding distributions over time through two complementary mechanisms:
Canary-based detection embeds a fixed set of reference texts, stores the resulting vectors, and later re-embeds the same texts to check whether the model has changed. This is cheap (embeds only 25 canary texts, not the entire corpus) and catches model changes on the very next check.
Statistical snapshot comparison captures the distribution of a sample of embedding vectors at time T -- centroid, per-dimension variance, pairwise similarity distribution, and more -- and compares that snapshot against a future sample. When the distributions have drifted beyond configurable thresholds, embed-drift computes a composite drift score, classifies severity, and fires alert callbacks.
Zero runtime dependencies. Pure TypeScript. All statistical computations are self-contained.
Installation
npm install embed-drift
Requires Node.js 18 or later.
Quick Start
import{createMonitor}from'embed-drift';// Create a drift monitor for your embedding modelconstmonitor=createMonitor({modelId: 'text-embedding-3-small',alertSeverity: 'high',onDrift: (report)=>{console.warn('Embedding drift detected:',report.summary);},});// Take a baseline snapshot from your current embeddingsconstbaseline=monitor.snapshot(baselineEmbeddings);monitor.setBaseline(baseline);// Later, check new embeddings against the baselineconstreport=monitor.check(newEmbeddings);console.log(report.composite.severity);// 'none' | 'low' | 'medium' | 'high' | 'critical'// Detect silent model changes using canary textsconstcanaryReport=awaitmonitor.checkCanaries(embedFn);if(canaryReport.modelChanged){console.error('Embedding model has changed!');}
Persisting Snapshots
// Save a snapshot to diskmonitor.saveSnapshot(baseline,'./snapshots/baseline.json');// Load it back laterconstloaded=monitor.loadSnapshot('./snapshots/baseline.json');monitor.setBaseline(loaded);
CI/CD Gate
import{createMonitor}from'embed-drift';constmonitor=createMonitor({modelId: 'text-embedding-3-small'});constbaseline=monitor.loadSnapshot('./snapshots/production-baseline.json');monitor.setBaseline(baseline);constreport=monitor.check(newEmbeddings);if(report.composite.severity==='high'||report.composite.severity==='critical'){console.error('Drift too high -- block deployment until re-indexing is complete.');process.exit(1);}
Features
  • Canary-based model change detection -- Embeds a fixed corpus of 25 diverse reference texts and compares their embeddings over time. Detects silent model swaps, version bumps, and provider changes within a single check cycle.
  • Five complementary drift detection methods -- Centroid shift, pairwise cosine similarity distribution, dimension-wise statistics (Cohen's d + KS-like statistic), Maximum Mean Discrepancy (MMD) approximation with random Fourier features, and canary comparison. Each method produces a normalized score in [0, 1].
  • Composite drift scoring -- Weighted average of all method scores with configurable per-method weights. Automatic weight renormalization when methods are disabled or data is unavailable.
  • Severity classification -- Composite scores are classified into five actionable bands: none, low, medium, high, critical. Model changes always produce critical severity.
  • Configurable alerting -- Set severity thresholds and per-method score thresholds. Register an onDrift callback to integrate with any monitoring system. Supports both synchronous and asynchronous callbacks.
  • Snapshot persistence -- Save and load statistical snapshots as portable JSON files. Snapshots are compact (typically 10-900 KB depending on dimensionality and sample size) and work across processes, machines, and time.
  • Zero runtime dependencies -- All statistical computations are self-contained TypeScript. No native modules, no WASM, no Python bridge.
  • Full TypeScript support -- Complete type definitions for all exports. Strict mode compatible.
API Reference
Exports
import{createMonitor,DriftError,DEFAULT_CANARY_TEXTS,}from'embed-drift';importtype{EmbedFn,DriftSeverity,MethodResult,MethodThresholds,MethodWeights,SnapshotOptions,CheckOptions,Snapshot,DriftReport,CanaryReport,DriftMonitorOptions,DriftMonitor,DriftErrorCode,}from'embed-drift';
createMonitor(options: DriftMonitorOptions): DriftMonitor
Creates a drift monitor instance. All drift detection state and configuration is encapsulated in the returned object.
Options (DriftMonitorOptions):
OptionTypeDefaultDescription
modelIdstring--Required. The embedding model identifier.
canaryTextsstring[][]Additional canary texts to append to the built-in corpus.
replaceDefaultCanariesbooleanfalseIf true, use only canaryTexts instead of built-in corpus + custom.
canaryThresholdnumber0.95Mean cosine similarity below which modelChanged is declared.
alertSeverityDriftSeverity'high'Minimum severity to fire the onDrift callback.
thresholdsPartial<MethodThresholds>{}Per-method score overrides that trigger an alert.
onDrift(report) => void | Promise<void>undefinedCallback invoked when an alert fires. Async errors are swallowed.
methodWeightsPartial<MethodWeights>see belowWeights for the composite drift score.
enabledMethods{ centroid?, pairwise?, dimensionWise?, mmd? }all trueDisable specific drift methods.
mmdRandomFeaturesnumber100Number of random Fourier features for MMD approximation.
pairwiseSamplePairsnumber500Number of random pairs sampled for pairwise similarity estimation.
Default composite weights:
{canary: 0.35,centroid: 0.15,pairwise: 0.20,dimensionWise: 0.15,mmd: 0.15}
monitor.snapshot(embeddings, options?): Snapshot
Computes a statistical snapshot of the provided embedding vectors.
Parameters:
ParameterTypeDescription
embeddingsnumber[][]At least 2 vectors of consistent dimensionality.
options.sampleSizenumberNumber of vectors to store for KS/MMD computation. Default: 50.
options.metadataRecord<string, unknown>Caller-provided key-value metadata attached to the snapshot.
Returns: A Snapshot object containing the model ID, centroid, per-dimension variance, pairwise similarity statistics, a 20-bin similarity histogram, and a random sample of vectors.
Throws:
  • DriftError('EMPTY_INPUT') if fewer than 2 vectors are given.
  • DriftError('INCONSISTENT_DIMENSIONS') if vectors have different lengths.
monitor.compare(snapshotA, snapshotB): DriftReport
Compares two snapshots and returns a DriftReport with per-method drift scores, a composite score, and severity classification.
Parameters:
ParameterTypeDescription
snapshotASnapshotThe reference (baseline) snapshot.
snapshotBSnapshotThe new snapshot to compare against the baseline.
Returns: A DriftReport with all per-method results, composite score, severity, alert status, and a human-readable summary.
Throws:
  • DriftError('INCOMPATIBLE_DIMENSIONS') if the two snapshots have different dimensionalities.
When snapshotA.modelId !== snapshotB.modelId, the report sets modelChanged: true and severity to critical.
monitor.setBaseline(snapshot): void
Stores a snapshot as the baseline for subsequent check() calls.
monitor.getBaseline(): Snapshot | undefined
Returns the currently stored baseline snapshot, or undefined if none is set.
monitor.check(embeddings, options?): DriftReport
Creates a new snapshot from embeddings and compares it against the stored baseline.
Parameters:
ParameterTypeDescription
embeddingsnumber[][]New embedding vectors to compare against the baseline.
options.snapshotOptionsSnapshotOptionsOptions forwarded to snapshot creation.
Returns: A DriftReport.
Throws:
  • DriftError('NO_BASELINE') if no baseline has been set via setBaseline().
monitor.checkCanaries(embedFn): Promise<CanaryReport>
Embeds the configured canary texts using embedFn and compares against stored reference embeddings.
Parameters:
ParameterTypeDescription
embedFn(texts: string[]) => Promise<number[][]>A function that embeds an array of texts and returns vectors.
Behavior:
  • On the first call, establishes the reference baseline. Returns a CanaryReport with isInitialBaseline: true and driftScore: 0.
  • On subsequent calls, computes per-canary cosine similarities and returns modelChanged: true when the mean similarity falls below canaryThreshold.
Returns: A CanaryReport.
Throws:
  • DriftError('EMBED_FN_FAILED') if embedFn throws.
monitor.setCanaryBaseline(canaryEmbeddings): void
Explicitly sets the canary reference embeddings without calling checkCanaries. Useful for loading a persisted canary baseline.
Parameters:
ParameterTypeDescription
canaryEmbeddingsnumber[][]Pre-computed canary embeddings (one per canary text).
Throws:
  • DriftError('EMPTY_INPUT') if the array is empty.
monitor.getCanaryTexts(): string[]
Returns the resolved canary text array (built-in + custom, or custom-only if replaceDefaultCanaries: true).
monitor.alert(report): boolean
Evaluates a DriftReport or CanaryReport against configured thresholds and returns true if an alert should fire. Does not invoke the onDrift callback.
An alert fires if:
  • The report severity meets or exceeds alertSeverity, OR
  • Any per-method score exceeds its configured threshold in thresholds.
monitor.saveSnapshot(snapshot, filePath): void
Writes a snapshot as pretty-printed JSON to the given file path.
monitor.loadSnapshot(filePath): Snapshot
Reads and validates a snapshot from a JSON file. Validates all required fields and dimensional consistency.
Throws:
  • DriftError('INVALID_SNAPSHOT') if the file is missing, not valid JSON, or fails schema validation.
DriftError
Custom error class extending Error with a code property for programmatic error handling.
import{DriftError}from'embed-drift';try{monitor.check(embeddings);}catch(err){if(errinstanceofDriftError){console.error(`Drift error [${err.code}]: ${err.message}`);}}
DEFAULT_CANARY_TEXTS
A frozen array of 25 diverse English reference texts spanning technical documentation, scientific language, legal text, casual conversation, news, medical, creative writing, mathematical, instructional, and philosophical domains. Used as the default canary corpus for model fingerprinting.
import{DEFAULT_CANARY_TEXTS}from'embed-drift';console.log(DEFAULT_CANARY_TEXTS.length);// 25
Configuration
Composite Weights
The composite drift score is a weighted average of per-method scores. Weights are renormalized when methods are disabled or their data is unavailable.
constmonitor=createMonitor({modelId: 'text-embedding-3-small',methodWeights: {canary: 0.40,// Increase canary influencecentroid: 0.10,pairwise: 0.20,dimensionWise: 0.15,mmd: 0.15,},});
Disabling Methods
Disable individual drift detection methods when they are not needed:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',enabledMethods: {mmd: false,// Skip MMD computationdimensionWise: false,// Skip dimension-wise analysis},});
Disabled methods report computed: false and score 0. Their weights are redistributed to the remaining active methods.
Alert Thresholds
Alerts fire when severity meets or exceeds alertSeverity, or when any per-method score exceeds its configured threshold:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',alertSeverity: 'high',thresholds: {composite: 0.40,canary: 0.05,centroid: 0.30,},onDrift: (report)=>{// Send to your monitoring systemwebhook.post('/alerts/embedding-drift',report);},});
Custom Canary Texts
Add domain-specific canary texts for increased sensitivity:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',canaryTexts: ['The plaintiff alleges breach of fiduciary duty under Section 14(a).','Amortization of goodwill is calculated on a straight-line basis.',],});// Uses all 25 default canaries + 2 custom = 27 totalconstmonitorCustomOnly=createMonitor({modelId: 'text-embedding-3-small',canaryTexts: ['My custom canary text.'],replaceDefaultCanaries: true,});// Uses only the 1 custom canary text
Drift Detection Methods
embed-drift implements five complementary methods. Each produces a normalized score in [0, 1].
Centroid Shift
Measures the cosine distance between the mean embedding vectors (centroids) of two snapshots. Detects global shifts in the embedding space. Computational cost: O(n * d) where n is the sample size and d is the dimensionality.
Pairwise Cosine Similarity Distribution
Compares the distribution of pairwise cosine similarities between two snapshots. Captures changes in how embeddings are spread relative to each other -- how compact or diffuse the distribution is -- even when the centroid stays the same.
Dimension-Wise Statistics
Analyzes per-dimension statistics using Cohen's d effect size and KS-like statistics across sample vectors. The Cohen's d score identifies which specific dimensions have shifted, while the KS statistic captures distributional shape changes (bimodality, heavy tails) that mean and variance alone do not surface. The two scores are blended equally.
MMD (Maximum Mean Discrepancy)
Uses Maximum Mean Discrepancy with random Fourier features (random kitchen sinks approximation) to measure the distance between two embedding distributions in a kernel-induced feature space. The RBF kernel bandwidth is set via the median heuristic. Sensitive to all moments of the distribution difference. Configurable via mmdRandomFeatures (default: 100).
Canary Texts
Embeds a fixed corpus of diverse reference texts and compares their embeddings over time. Detects silent model changes (provider swaps, version updates) by monitoring whether the same inputs produce the same outputs. The primary and cheapest signal for model change detection.
Severity Bands
Composite ScoreSeverityRecommended Action
0.00 -- 0.05noneNo action needed. Distribution is stable.
0.05 -- 0.20lowMonitor. Normal content variation.
0.20 -- 0.40mediumInvestigate. Consider partial re-indexing.
0.40 -- 0.70highRe-embed recommended. Significant drift detected.
0.70 -- 1.00criticalRe-embed immediately.
A confirmed model change (different model IDs or canary mean similarity below threshold) always produces critical severity regardless of the composite score.
Error Handling
All errors thrown by embed-drift are instances of DriftError with a code property for programmatic handling:
CodeWhen It Is Thrown
EMPTY_INPUTEmbedding array has fewer than 2 vectors, or setCanaryBaseline receives an empty array.
INCONSISTENT_DIMENSIONSVectors in the input array have different dimensionalities.
INCOMPATIBLE_DIMENSIONSTwo snapshots being compared have different dimensionalities.
NO_BASELINEcheck() called before setBaseline().
INVALID_SNAPSHOTLoaded snapshot file is missing, not valid JSON, or fails schema validation.
NO_CANARY_BASELINECanary comparison attempted without a reference baseline.
EMBED_FN_FAILEDThe embedding function passed to checkCanaries() threw an error.
import{DriftError}from'embed-drift';try{constreport=monitor.check(newEmbeddings);}catch(err){if(errinstanceofDriftError){switch(err.code){case'NO_BASELINE':
console.error('Set a baseline before calling check().');break;case'INCOMPATIBLE_DIMENSIONS':
console.error('Snapshot dimensions do not match.');break;default:
console.error(`Unexpected drift error: ${err.code}`);}}}
Advanced Usage
Scheduled Monitoring
Run periodic drift checks as part of a cron job or background worker:
import{createMonitor}from'embed-drift';constmonitor=createMonitor({modelId: 'text-embedding-3-small',alertSeverity: 'medium',onDrift: async(report)=>{awaitsendSlackAlert(`Embedding drift detected: ${report.summary}`);},});// Load the production baselineconstbaseline=monitor.loadSnapshot('./snapshots/production-baseline.json');monitor.setBaseline(baseline);// Sample current embeddings from your vector databaseconstcurrentSample=awaitsampleFromVectorDB(1000);// Check for driftconstreport=monitor.check(currentSample);console.log(`Severity: ${report.composite.severity}, Score: ${report.composite.score}`);
Canary-Based Model Monitoring
Detect model changes with minimal API cost:
import{createMonitor}from'embed-drift';constmonitor=createMonitor({modelId: 'text-embedding-3-small',canaryThreshold: 0.95,onDrift: (report)=>{if('modelChanged'inreport&&report.modelChanged){triggerReindexingPipeline();}},});// On first run, establishes the canary baselineconstembedFn=async(texts: string[])=>{returnopenai.embeddings.create({model: 'text-embedding-3-small',input: texts}).then(res=>res.data.map(d=>d.embedding));};constreport=awaitmonitor.checkCanaries(embedFn);if(report.isInitialBaseline){console.log('Canary baseline established.');}elseif(report.modelChanged){console.error('Model changed! Drift score:',report.driftScore);}else{console.log('Model unchanged. Mean similarity:',report.meanSimilarity);}
Comparing Two Snapshots Directly
Compare snapshots without managing baseline state:
constmonitor=createMonitor({modelId: 'text-embedding-3-small'});constsnapshotA=monitor.loadSnapshot('./snapshots/2025-01-baseline.json');constsnapshotB=monitor.loadSnapshot('./snapshots/2025-03-current.json');constreport=monitor.compare(snapshotA,snapshotB);console.log('Composite score:',report.composite.score);console.log('Severity:',report.composite.severity);console.log('Centroid drift:',report.methods.centroid.score);console.log('Pairwise drift:',report.methods.pairwise.score);console.log('MMD drift:',report.methods.mmd.score);console.log('Summary:',report.summary);
Pre-Computing Canary Baselines
Load a previously saved canary baseline to avoid re-establishing on every restart:
constmonitor=createMonitor({modelId: 'text-embedding-3-small',replaceDefaultCanaries: true,canaryTexts: ['My domain-specific canary text.'],});// Load saved canary embeddingsconstsavedCanaries=JSON.parse(readFileSync('./canary-baseline.json','utf-8'));monitor.setCanaryBaseline(savedCanaries);// Now checkCanaries compares against the loaded baselineconstreport=awaitmonitor.checkCanaries(embedFn);
Types
All types are exported for use in TypeScript projects:
Snapshot
interfaceSnapshot{id: string;// UUID v4createdAt: string;// ISO 8601 timestampmodelId: string;// Embedding model identifierdimensionality: number;// Vector dimensionssampleCount: number;// Number of input vectorscentroid: number[];// Element-wise mean vectorvariance: number[];// Per-dimension variancemeanPairwiseSimilarity: number;// Mean cosine similarity across sampled pairsstdPairwiseSimilarity: number;// Std dev of pairwise cosine similaritiessimilarityHistogram: number[];// 20-bin histogram from -1.0 to 1.0sampleVectors: number[][];// Random sample of vectors for KS/MMDcanaryEmbeddings?: number[][];// Canary text embeddings (optional)metadata?: Record<string,unknown>;// Caller-provided metadata (optional)}
DriftReport
interfaceDriftReport{id: string;createdAt: string;snapshotIds: [string,string];modelIds: [string,string];modelChanged: boolean;methods: {canary: MethodResult;centroid: MethodResult;pairwise: MethodResult;dimensionWise: MethodResult;mmd: MethodResult;};composite: {score: number;// Weighted average in [0, 1]severity: DriftSeverity;// 'none' | 'low' | 'medium' | 'high' | 'critical'weights: MethodWeights;// Effective weights used};alerted: boolean;summary: string;durationMs: number;}
CanaryReport
interfaceCanaryReport{id: string;createdAt: string;canaryCount: number;meanSimilarity: number;minSimilarity: number;perCanarySimilarities: number[];driftScore: number;// 1 - meanSimilaritymodelChanged: boolean;isInitialBaseline: boolean;alerted: boolean;modelId: string;durationMs: number;}
MethodResult
interfaceMethodResult{score: number;// Drift score in [0, 1]computed: boolean;// Whether this method was runinterpretation: string;// Human-readable interpretationdetails?: Record<string,unknown>;// Method-specific details}
EmbedFn
typeEmbedFn=(texts: string[])=>Promise<number[][]>;
DriftSeverity
typeDriftSeverity='none'|'low'|'medium'|'high'|'critical';
DriftErrorCode
typeDriftErrorCode=|'EMPTY_INPUT'|'INCONSISTENT_DIMENSIONS'|'INCOMPATIBLE_DIMENSIONS'|'NO_BASELINE'|'INVALID_SNAPSHOT'|'NO_CANARY_BASELINE'|'EMBED_FN_FAILED';
License
MIT
About
Monitor embedding distribution shifts over time
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages