BranchesTags

Repository files navigation

ai-diff

Compare LLM outputs with word-level and line-level diffs, ANSI-colored terminal output, and AI-specific metrics.

npm versionnpm downloadslicensenodeTypeScript

ai-diff compares two or more LLM responses and produces structured diffs annotated with AI-specific metrics: token counts (input and output), estimated cost in USD (using built-in model pricing), response latency, Jaccard similarity scores, and length statistics (words, sentences, characters). It supports five diff modes -- unified, side-by-side, inline, metrics-only, and JSON structural diff -- and four output formats: terminal (ANSI-colored), JSON, Markdown, and plain text. Zero runtime dependencies.

Installation

npm install ai-diff

Quick Start

import{diff,formatDiff}from'ai-diff';constresult=diff({text: 'Paris is the capital of France.',model: 'gpt-4o',tokens: {input: 10,output: 8},latency: 1240},{text: 'The capital of France is Paris.',model: 'claude-sonnet',tokens: {input: 10,output: 8},latency: 980},);// Print a colored unified diff with metrics tableconsole.log(formatDiff(result,'terminal'));// Access structured dataconsole.log(result.identical);// falseconsole.log(result.similarity.jaccard);// 0.0 - 1.0console.log(result.metrics.latency?.delta);// -260

Features

  • Word-level and line-level diffs -- LCS-based algorithms implemented from scratch, no runtime dependencies.
  • Five diff modes -- unified (git-style), side-by-side (two-column), inline (strikethrough/underline), metrics (table only), json (structural key-level diff).
  • AI-specific metrics -- Token counts, estimated cost (USD), response latency, Jaccard similarity, word/sentence/character counts displayed in a comparison table alongside every diff.
  • Built-in model pricing -- GPT-4o, GPT-4o-mini, GPT-3.5 Turbo, GPT-4 Turbo, Claude Opus, Claude Sonnet, Claude Haiku, Gemini Pro, Gemini Flash. Override or extend with custom pricing.
  • N-way comparison -- diffOutputs() compares any number of outputs pairwise. compare() sends a prompt to multiple models via a user-provided function and diffs the results.
  • Four output formats -- terminal (ANSI colors), json (serialized result), markdown, plain.
  • Automatic token estimation -- When token counts are not provided, output tokens are estimated using a ceil(characters / 4) heuristic.
  • ANSI color detection -- Colors are enabled automatically when stdout is a TTY. Respects NO_COLOR environment variable. Override with the color option.
  • TypeScript-first -- Full type definitions exported for all interfaces, options, and result types. Strict mode enabled.
  • Zero runtime dependencies -- All diffing, similarity, formatting, and metrics logic uses only Node.js built-ins.

API Reference

diff(outputA, outputB, options?)

Compare two LLM outputs and return a DiffResult.

Parameters:

ParameterTypeDescription
outputAstring | LLMOutputFirst LLM output. Plain strings are wrapped as { text: string }.
outputBstring | LLMOutputSecond LLM output.
optionsDiffOptionsOptional configuration (see Configuration).

Returns:DiffResult

import{diff}from'ai-diff';// Compare plain stringsconstresult=diff('Output from model A','Output from model B');// Compare with full metadataconstresult=diff({text: 'Response A',model: 'gpt-4o',tokens: {input: 100,output: 50},cost: 0.005,latency: 1200},{text: 'Response B',model: 'claude-sonnet',tokens: {input: 100,output: 75},latency: 980},{mode: 'side-by-side'},);console.log(result.identical);// falseconsole.log(result.hunks.length);// number of diff hunksconsole.log(result.metrics.cost);// { a, b, delta, deltaPercent }console.log(result.similarity.jaccard);// 0.0 - 1.0

diffOutputs(outputs, options?)

Compare N LLM outputs pairwise.

Parameters:

ParameterTypeDescription
outputs(string | LLMOutput)[]Array of outputs to compare.
optionsDiffOptionsOptional configuration.

Returns:MultiDiffResult

import{diffOutputs}from'ai-diff';constresult=diffOutputs([{text: 'Output A',model: 'gpt-4o'},{text: 'Output B',model: 'claude-sonnet'},{text: 'Output C',model: 'gemini-pro'},]);console.log(result.pairwise.length);// 3 (A-B, A-C, B-C)console.log(result.metricsTable.labels);// ['gpt-4o', 'claude-sonnet', 'gemini-pro']console.log(result.metricsTable.wordCounts);// [n, n, n]

compare(prompt, models, llmFn, options?)

Send a prompt to multiple models via a user-provided function and compare the outputs.

Parameters:

ParameterTypeDescription
promptstringThe prompt to send to each model.
modelsstring[]Array of model identifiers.
llmFnLLMFnAsync function (prompt, model) => LLMOutput | string that calls the model.
optionsCompareOptionsOptional configuration (extends DiffOptions with concurrency, timeout, signal).

Returns:Promise<ComparisonResult>

import{compare}from'ai-diff';constresult=awaitcompare('Explain quantum computing in 3 sentences.',['gpt-4o','claude-sonnet'],async(prompt,model)=>{constresponse=awaitcallMyLLM(prompt,model);return{text: response.text,tokens: response.usage, model };},{concurrency: 2,timeout: 15000},);console.log(result.calls);// per-model status, output, latency, or errorconsole.log(result.pairwise);// pairwise diffs of successful outputs

formatDiff(result, format?)

Format a diff result into a displayable string.

Parameters:

ParameterTypeDefaultDescription
resultDiffResult | MultiDiffResult | ComparisonResult--The result to format.
formatOutputFormat'terminal'One of 'terminal', 'json', 'markdown', 'plain'.

Returns:string

import{diff,formatDiff}from'ai-diff';constresult=diff('hello world','hello earth');console.log(formatDiff(result,'terminal'));// ANSI-colored unified diffconsole.log(formatDiff(result,'json'));// JSON.stringify(result, null, 2)console.log(formatDiff(result,'plain'));// plain text, no ANSI codes

Similarity Functions

jaccardSimilarity(textA, textB)

Compute Jaccard similarity (word-level set overlap) between two texts. Returns a value between 0.0 and 1.0.

import{jaccardSimilarity}from'ai-diff';jaccardSimilarity('hello world','hello earth');// 0.333...jaccardSimilarity('hello world','hello world');// 1.0jaccardSimilarity('','');// 1.0

cosineSimilarity(textA, textB)

Compute cosine similarity using word-frequency vectors. Returns a value between 0.0 and 1.0.

import{cosineSimilarity}from'ai-diff';cosineSimilarity('the quick brown fox','the slow brown cat');// 0.0 - 1.0

exactMatchRatio(textA, textB)

Returns 1.0 if the two texts are identical, 0.0 otherwise.

import{exactMatchRatio}from'ai-diff';exactMatchRatio('hello','hello');// 1.0exactMatchRatio('hello','world');// 0.0

compositeSimilarity(textA, textB)

Weighted composite: Jaccard (0.5) + Cosine (0.3) + Exact Match (0.2).

import{compositeSimilarity}from'ai-diff';compositeSimilarity('hello world','hello earth');// 0.0 - 1.0

embeddingCosineSimilarity(a, b)

Compute cosine similarity between two numeric embedding vectors. Throws if vectors have different lengths.

import{embeddingCosineSimilarity}from'ai-diff';embeddingCosineSimilarity([1,0,0],[0,1,0]);// 0.0embeddingCosineSimilarity([1,0],[1,0]);// 1.0

computeLengthStats(text)

Returns { words, sentences, characters } for a given text.

import{computeLengthStats}from'ai-diff';computeLengthStats('Hello world. Goodbye.');// { words: 3, sentences: 2, characters: 21 }

Diff Utilities

diffWords(textA, textB)

Compute a word-level diff between two strings. Returns DiffSegment[].

import{diffWords}from'ai-diff';constsegments=diffWords('hello world','hello earth');// [// { text: 'hello', type: 'unchanged' },// { text: ' ', type: 'unchanged' },// { text: 'world', type: 'removed' },// { text: 'earth', type: 'added' },// ]

diffLines(textA, textB)

Compute a line-level diff between two strings. Returns DiffSegment[].

import{diffLines}from'ai-diff';constsegments=diffLines('line1\nline2','line1\nline3');

diffJson(a, b)

Compute a structural diff between two parsed JSON values. Returns JsonChange[] with dot-notation paths.

import{diffJson}from'ai-diff';constchanges=diffJson({name: 'Alice',age: 30},{name: 'Alice',age: 31,role: 'admin'},);// [// { path: 'age', type: 'changed', before: 30, after: 31 },// { path: 'role', type: 'added', after: 'admin' },// ]

computeHunks(textA, textB, contextLines?)

Compute diff hunks (contiguous groups of changes with context lines) between two texts. Returns DiffHunk[].

import{computeHunks}from'ai-diff';consthunks=computeHunks('line1\nline2\nline3','line1\nchanged\nline3',3);

tryParseJson(text)

Attempt to parse a string as JSON. Returns the parsed value on success or null on failure.

Metrics Utilities

estimateCost(output, pricingOverrides?)

Estimate the cost of an LLM output in USD. Returns the output's cost field if set, otherwise computes from model pricing and token counts. Returns undefined if insufficient data.

import{estimateCost}from'ai-diff';estimateCost({text: 'hello',model: 'gpt-4o',tokens: {input: 100,output: 50}});// 0.00075 (computed from built-in GPT-4o pricing)estimateCost({text: 'hello',model: 'custom',tokens: {input: 1000,output: 500}},{custom: {input: 0.001,output: 0.002}},);// 2.0

getModelPricing()

Returns a copy of the built-in model pricing table. Each entry maps a model name to { input: number; output: number } (per-token USD).

import{getModelPricing}from'ai-diff';constpricing=getModelPricing();// {// 'gpt-4o': { input: 0.0000025, output: 0.00001 },// 'claude-sonnet': { input: 0.000003, output: 0.000015 },// ...// }

Built-in models:gpt-4o, gpt-4o-mini, gpt-3.5-turbo, gpt-4-turbo, claude-opus, claude-sonnet, claude-haiku, gemini-pro, gemini-flash.

computeMetrics(outputA, outputB, options?)

Compute full comparative metrics between two outputs. Supports an optional embedFn for semantic similarity.

Parameters:

ParameterTypeDescription
outputALLMOutputFirst output.
outputBLLMOutputSecond output.
options.embedFn(text: string) => Promise<number[]>Optional embedding function for semantic similarity.
options.pricingRecord<string, { input: number; output: number }>Optional pricing overrides.

Returns:Promise<DiffMetrics>

Formatter Utilities

renderUnifiedDiff(result, useColor?)

Render a DiffResult as a unified diff string with optional ANSI colors. Removed lines are prefixed with - (red), added lines with + (green). Word-level changes are highlighted with bold inverse.

renderSideBySide(result, useColor?, width?)

Render a DiffResult as a two-column side-by-side display. Column width defaults to (width - 3) / 2.

renderInlineDiff(segments, useColor?)

Render DiffSegment[] as inline text. Removed words appear with strikethrough (or ~~text~~ without color), added words with underline (or __text__).

renderJsonDiff(changes, originalA, originalB, useColor?)

Render JsonChange[] as a formatted string showing added, removed, and changed keys.

renderMetricsTable(metrics, labelA, labelB, useColor?)

Render a DiffMetrics object as a Unicode box-drawing table comparing all metrics between two outputs.

shouldUseColor(override?)

Returns true if ANSI colors should be used. Checks override, then NO_COLOR env var, then process.stdout.isTTY.

Normalization Utilities

normalizeOutput(input)

Convert a string | LLMOutput to an LLMOutput object. Plain strings become { text: string }.

enrichOutput(output)

Fill in estimated fields on an LLMOutput. Adds estimated tokens.output (via ceil(text.length / 4)) when not provided.

estimateTokens(text)

Estimate token count from text length: Math.ceil(text.length / 4).

tokenizeWords(text)

Split text into word and whitespace tokens (preserving whitespace). Used internally by the LCS diff algorithm.

Configuration

DiffOptions

interfaceDiffOptions{/** Diff mode. Default: 'unified'. */mode?: 'unified'|'side-by-side'|'inline'|'metrics'|'json';/** Context lines around changes in unified mode. Default: 3. */contextLines?: number;/** Embedding function for semantic similarity. */embedFn?: (text: string)=>Promise<number[]>;/** Per-token pricing overrides in USD. Keyed by model name. */pricing?: Record<string,{input: number;output: number}>;/** Show the metrics summary table. Default: true. */showMetrics?: boolean;/** Position of the metrics table. Default: 'top'. */metricsPosition?: 'top'|'bottom';/** Which metrics to display. Default: all available. */metrics?: ('tokens'|'cost'|'latency'|'similarity'|'length'|'model')[];/** Terminal width override for side-by-side mode. Default: auto-detected. */width?: number;/** ANSI color override. Default: auto-detected (true if TTY). */color?: boolean;/** Custom labels for outputs. Default: model names or 'Output A'/'Output B'. */labels?: string[];}

CompareOptions

Extends DiffOptions with:

interfaceCompareOptionsextendsDiffOptions{/** Max concurrent model calls. Default: unlimited (all in parallel). */concurrency?: number;/** Per-call timeout in milliseconds. Default: 30000. */timeout?: number;/** AbortSignal for cancellation. */signal?: AbortSignal;}

Diff Modes

ModeDescription
unifiedGit-style unified diff with word-level highlighting (default).
side-by-sideTwo-column display with aligned content and a vertical separator.
inlineInline additions (underline) and deletions (strikethrough) within the original text.
metricsMetrics comparison table only; no text diff output.
jsonStructural diff for JSON outputs with key-level change detection. Falls back to text diff if either output is not valid JSON.

Output Formats

FormatDescription
terminalANSI-colored output for terminal display.
jsonFull result serialized as JSON.stringify(result, null, 2).
markdownMarkdown-formatted diff.
plainPlain text with no ANSI codes.

Error Handling

Invalid embedding vectors

embeddingCosineSimilarity throws if the two vectors have different lengths:

import{embeddingCosineSimilarity}from'ai-diff';embeddingCosineSimilarity([1,2],[1,2,3]);// Error: Embedding vectors must have the same length: 2 vs 3

Unknown models

When a model is not in the built-in pricing table and no pricing override is provided, estimateCost returns undefined and the cost row is omitted from the metrics table. No error is thrown.

Failed model calls in compare()

When a model call fails or times out in compare(), the failure is captured in the calls array with status: 'error' and an error message. The failed output is excluded from pairwise diffs. Remaining models continue normally.

constresult=awaitcompare('prompt',['model-a','model-b'],llmFn);for(constcallofresult.calls){if(call.status==='error'){console.error(`${call.model} failed: ${call.error}`);}}

JSON diff fallback

When mode: 'json' is used but one or both outputs are not valid JSON, the engine falls back to a standard text diff. The jsonChanges field on the result will be undefined.

Advanced Usage

Semantic similarity with custom embeddings

Provide an embedFn to compute semantic similarity alongside Jaccard:

import{computeMetrics}from'ai-diff';constmetrics=awaitcomputeMetrics({text: 'The cat sat on the mat.',model: 'gpt-4o',tokens: {output: 8}},{text: 'A feline rested on a rug.',model: 'claude-sonnet',tokens: {output: 7}},{embedFn: async(text)=>{// Call your embedding API (OpenAI, Cohere, etc.)returnawaitgetEmbedding(text);},},);console.log(metrics.similarity.semantic);// 0.0 - 1.0

Custom model pricing

Override or extend the built-in pricing table:

import{diff}from'ai-diff';constresult=diff(outputA,outputB,{pricing: {'my-custom-model': {input: 0.001,output: 0.002},'gpt-4o': {input: 0.000003,output: 0.000012},// override built-in},});

Comparing structured JSON outputs

import{diff,formatDiff}from'ai-diff';constresult=diff({text: '{"name":"Alice","age":30}',model: 'gpt-4o'},{text: '{"name":"Alice","age":31,"role":"admin"}',model: 'claude-sonnet'},{mode: 'json'},);console.log(result.jsonChanges);// [// { path: 'age', type: 'changed', before: 30, after: 31 },// { path: 'role', type: 'added', after: 'admin' },// ]console.log(formatDiff(result,'terminal'));

Concurrency-limited model comparison

import{compare}from'ai-diff';constresult=awaitcompare('Summarize this article.',['gpt-4o','gpt-4o-mini','claude-sonnet','gemini-pro'],async(prompt,model)=>callLLM(prompt,model),{concurrency: 2,timeout: 10000},);// Only 2 models called at a time; each call times out after 10s

Metrics-only comparison

import{diff,formatDiff}from'ai-diff';constresult=diff(outputA,outputB,{mode: 'metrics'});console.log(formatDiff(result,'terminal'));// Prints only the metrics comparison table, no text diff

TypeScript

All types are exported from the package root:

importtype{LLMOutput,LLMFn,DiffMode,DiffOptions,CompareOptions,OutputFormat,DiffResult,MultiDiffResult,ComparisonResult,DiffSegment,DiffHunk,DiffMetrics,LengthStats,JsonChange,}from'ai-diff';

The package is compiled with TypeScript strict mode targeting ES2022 (CommonJS output). Declaration files and source maps are included.

License

MIT

About

Compare LLM responses across models with semantic diffs

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
BranchesTags
Repository files navigation
ai-diff
Compare LLM outputs with word-level and line-level diffs, ANSI-colored terminal output, and AI-specific metrics.
npm versionnpm downloadslicensenodeTypeScript
ai-diff compares two or more LLM responses and produces structured diffs annotated with AI-specific metrics: token counts (input and output), estimated cost in USD (using built-in model pricing), response latency, Jaccard similarity scores, and length statistics (words, sentences, characters). It supports five diff modes -- unified, side-by-side, inline, metrics-only, and JSON structural diff -- and four output formats: terminal (ANSI-colored), JSON, Markdown, and plain text. Zero runtime dependencies.
Installation
npm install ai-diff
Quick Start
import{diff,formatDiff}from'ai-diff';constresult=diff({text: 'Paris is the capital of France.',model: 'gpt-4o',tokens: {input: 10,output: 8},latency: 1240},{text: 'The capital of France is Paris.',model: 'claude-sonnet',tokens: {input: 10,output: 8},latency: 980},);// Print a colored unified diff with metrics tableconsole.log(formatDiff(result,'terminal'));// Access structured dataconsole.log(result.identical);// falseconsole.log(result.similarity.jaccard);// 0.0 - 1.0console.log(result.metrics.latency?.delta);// -260
Features
  • Word-level and line-level diffs -- LCS-based algorithms implemented from scratch, no runtime dependencies.
  • Five diff modes -- unified (git-style), side-by-side (two-column), inline (strikethrough/underline), metrics (table only), json (structural key-level diff).
  • AI-specific metrics -- Token counts, estimated cost (USD), response latency, Jaccard similarity, word/sentence/character counts displayed in a comparison table alongside every diff.
  • Built-in model pricing -- GPT-4o, GPT-4o-mini, GPT-3.5 Turbo, GPT-4 Turbo, Claude Opus, Claude Sonnet, Claude Haiku, Gemini Pro, Gemini Flash. Override or extend with custom pricing.
  • N-way comparison -- diffOutputs() compares any number of outputs pairwise. compare() sends a prompt to multiple models via a user-provided function and diffs the results.
  • Four output formats -- terminal (ANSI colors), json (serialized result), markdown, plain.
  • Automatic token estimation -- When token counts are not provided, output tokens are estimated using a ceil(characters / 4) heuristic.
  • ANSI color detection -- Colors are enabled automatically when stdout is a TTY. Respects NO_COLOR environment variable. Override with the color option.
  • TypeScript-first -- Full type definitions exported for all interfaces, options, and result types. Strict mode enabled.
  • Zero runtime dependencies -- All diffing, similarity, formatting, and metrics logic uses only Node.js built-ins.
API Reference
diff(outputA, outputB, options?)
Compare two LLM outputs and return a DiffResult.
Parameters:
ParameterTypeDescription
outputAstring | LLMOutputFirst LLM output. Plain strings are wrapped as { text: string }.
outputBstring | LLMOutputSecond LLM output.
optionsDiffOptionsOptional configuration (see Configuration).
Returns:DiffResult
import{diff}from'ai-diff';// Compare plain stringsconstresult=diff('Output from model A','Output from model B');// Compare with full metadataconstresult=diff({text: 'Response A',model: 'gpt-4o',tokens: {input: 100,output: 50},cost: 0.005,latency: 1200},{text: 'Response B',model: 'claude-sonnet',tokens: {input: 100,output: 75},latency: 980},{mode: 'side-by-side'},);console.log(result.identical);// falseconsole.log(result.hunks.length);// number of diff hunksconsole.log(result.metrics.cost);// { a, b, delta, deltaPercent }console.log(result.similarity.jaccard);// 0.0 - 1.0
diffOutputs(outputs, options?)
Compare N LLM outputs pairwise.
Parameters:
ParameterTypeDescription
outputs(string | LLMOutput)[]Array of outputs to compare.
optionsDiffOptionsOptional configuration.
Returns:MultiDiffResult
import{diffOutputs}from'ai-diff';constresult=diffOutputs([{text: 'Output A',model: 'gpt-4o'},{text: 'Output B',model: 'claude-sonnet'},{text: 'Output C',model: 'gemini-pro'},]);console.log(result.pairwise.length);// 3 (A-B, A-C, B-C)console.log(result.metricsTable.labels);// ['gpt-4o', 'claude-sonnet', 'gemini-pro']console.log(result.metricsTable.wordCounts);// [n, n, n]
compare(prompt, models, llmFn, options?)
Send a prompt to multiple models via a user-provided function and compare the outputs.
Parameters:
ParameterTypeDescription
promptstringThe prompt to send to each model.
modelsstring[]Array of model identifiers.
llmFnLLMFnAsync function (prompt, model) => LLMOutput | string that calls the model.
optionsCompareOptionsOptional configuration (extends DiffOptions with concurrency, timeout, signal).
Returns:Promise<ComparisonResult>
import{compare}from'ai-diff';constresult=awaitcompare('Explain quantum computing in 3 sentences.',['gpt-4o','claude-sonnet'],async(prompt,model)=>{constresponse=awaitcallMyLLM(prompt,model);return{text: response.text,tokens: response.usage, model };},{concurrency: 2,timeout: 15000},);console.log(result.calls);// per-model status, output, latency, or errorconsole.log(result.pairwise);// pairwise diffs of successful outputs
formatDiff(result, format?)
Format a diff result into a displayable string.
Parameters:
ParameterTypeDefaultDescription
resultDiffResult | MultiDiffResult | ComparisonResult--The result to format.
formatOutputFormat'terminal'One of 'terminal', 'json', 'markdown', 'plain'.
Returns:string
import{diff,formatDiff}from'ai-diff';constresult=diff('hello world','hello earth');console.log(formatDiff(result,'terminal'));// ANSI-colored unified diffconsole.log(formatDiff(result,'json'));// JSON.stringify(result, null, 2)console.log(formatDiff(result,'plain'));// plain text, no ANSI codes
Similarity Functions
jaccardSimilarity(textA, textB)
Compute Jaccard similarity (word-level set overlap) between two texts. Returns a value between 0.0 and 1.0.
import{jaccardSimilarity}from'ai-diff';jaccardSimilarity('hello world','hello earth');// 0.333...jaccardSimilarity('hello world','hello world');// 1.0jaccardSimilarity('','');// 1.0
cosineSimilarity(textA, textB)
Compute cosine similarity using word-frequency vectors. Returns a value between 0.0 and 1.0.
import{cosineSimilarity}from'ai-diff';cosineSimilarity('the quick brown fox','the slow brown cat');// 0.0 - 1.0
exactMatchRatio(textA, textB)
Returns 1.0 if the two texts are identical, 0.0 otherwise.
import{exactMatchRatio}from'ai-diff';exactMatchRatio('hello','hello');// 1.0exactMatchRatio('hello','world');// 0.0
compositeSimilarity(textA, textB)
Weighted composite: Jaccard (0.5) + Cosine (0.3) + Exact Match (0.2).
import{compositeSimilarity}from'ai-diff';compositeSimilarity('hello world','hello earth');// 0.0 - 1.0
embeddingCosineSimilarity(a, b)
Compute cosine similarity between two numeric embedding vectors. Throws if vectors have different lengths.
import{embeddingCosineSimilarity}from'ai-diff';embeddingCosineSimilarity([1,0,0],[0,1,0]);// 0.0embeddingCosineSimilarity([1,0],[1,0]);// 1.0
computeLengthStats(text)
Returns { words, sentences, characters } for a given text.
import{computeLengthStats}from'ai-diff';computeLengthStats('Hello world. Goodbye.');// { words: 3, sentences: 2, characters: 21 }
Diff Utilities
diffWords(textA, textB)
Compute a word-level diff between two strings. Returns DiffSegment[].
import{diffWords}from'ai-diff';constsegments=diffWords('hello world','hello earth');// [// { text: 'hello', type: 'unchanged' },// { text: ' ', type: 'unchanged' },// { text: 'world', type: 'removed' },// { text: 'earth', type: 'added' },// ]
diffLines(textA, textB)
Compute a line-level diff between two strings. Returns DiffSegment[].
import{diffLines}from'ai-diff';constsegments=diffLines('line1\nline2','line1\nline3');
diffJson(a, b)
Compute a structural diff between two parsed JSON values. Returns JsonChange[] with dot-notation paths.
import{diffJson}from'ai-diff';constchanges=diffJson({name: 'Alice',age: 30},{name: 'Alice',age: 31,role: 'admin'},);// [// { path: 'age', type: 'changed', before: 30, after: 31 },// { path: 'role', type: 'added', after: 'admin' },// ]
computeHunks(textA, textB, contextLines?)
Compute diff hunks (contiguous groups of changes with context lines) between two texts. Returns DiffHunk[].
import{computeHunks}from'ai-diff';consthunks=computeHunks('line1\nline2\nline3','line1\nchanged\nline3',3);
tryParseJson(text)
Attempt to parse a string as JSON. Returns the parsed value on success or null on failure.
Metrics Utilities
estimateCost(output, pricingOverrides?)
Estimate the cost of an LLM output in USD. Returns the output's cost field if set, otherwise computes from model pricing and token counts. Returns undefined if insufficient data.
import{estimateCost}from'ai-diff';estimateCost({text: 'hello',model: 'gpt-4o',tokens: {input: 100,output: 50}});// 0.00075 (computed from built-in GPT-4o pricing)estimateCost({text: 'hello',model: 'custom',tokens: {input: 1000,output: 500}},{custom: {input: 0.001,output: 0.002}},);// 2.0
getModelPricing()
Returns a copy of the built-in model pricing table. Each entry maps a model name to { input: number; output: number } (per-token USD).
import{getModelPricing}from'ai-diff';constpricing=getModelPricing();// {// 'gpt-4o': { input: 0.0000025, output: 0.00001 },// 'claude-sonnet': { input: 0.000003, output: 0.000015 },// ...// }
Built-in models:gpt-4o, gpt-4o-mini, gpt-3.5-turbo, gpt-4-turbo, claude-opus, claude-sonnet, claude-haiku, gemini-pro, gemini-flash.
computeMetrics(outputA, outputB, options?)
Compute full comparative metrics between two outputs. Supports an optional embedFn for semantic similarity.
Parameters:
ParameterTypeDescription
outputALLMOutputFirst output.
outputBLLMOutputSecond output.
options.embedFn(text: string) => Promise<number[]>Optional embedding function for semantic similarity.
options.pricingRecord<string, { input: number; output: number }>Optional pricing overrides.
Returns:Promise<DiffMetrics>
Formatter Utilities
renderUnifiedDiff(result, useColor?)
Render a DiffResult as a unified diff string with optional ANSI colors. Removed lines are prefixed with - (red), added lines with + (green). Word-level changes are highlighted with bold inverse.
renderSideBySide(result, useColor?, width?)
Render a DiffResult as a two-column side-by-side display. Column width defaults to (width - 3) / 2.
renderInlineDiff(segments, useColor?)
Render DiffSegment[] as inline text. Removed words appear with strikethrough (or ~~text~~ without color), added words with underline (or __text__).
renderJsonDiff(changes, originalA, originalB, useColor?)
Render JsonChange[] as a formatted string showing added, removed, and changed keys.
renderMetricsTable(metrics, labelA, labelB, useColor?)
Render a DiffMetrics object as a Unicode box-drawing table comparing all metrics between two outputs.
shouldUseColor(override?)
Returns true if ANSI colors should be used. Checks override, then NO_COLOR env var, then process.stdout.isTTY.
Normalization Utilities
normalizeOutput(input)
Convert a string | LLMOutput to an LLMOutput object. Plain strings become { text: string }.
enrichOutput(output)
Fill in estimated fields on an LLMOutput. Adds estimated tokens.output (via ceil(text.length / 4)) when not provided.
estimateTokens(text)
Estimate token count from text length: Math.ceil(text.length / 4).
tokenizeWords(text)
Split text into word and whitespace tokens (preserving whitespace). Used internally by the LCS diff algorithm.
Configuration
DiffOptions
interfaceDiffOptions{/** Diff mode. Default: 'unified'. */mode?: 'unified'|'side-by-side'|'inline'|'metrics'|'json';/** Context lines around changes in unified mode. Default: 3. */contextLines?: number;/** Embedding function for semantic similarity. */embedFn?: (text: string)=>Promise<number[]>;/** Per-token pricing overrides in USD. Keyed by model name. */pricing?: Record<string,{input: number;output: number}>;/** Show the metrics summary table. Default: true. */showMetrics?: boolean;/** Position of the metrics table. Default: 'top'. */metricsPosition?: 'top'|'bottom';/** Which metrics to display. Default: all available. */metrics?: ('tokens'|'cost'|'latency'|'similarity'|'length'|'model')[];/** Terminal width override for side-by-side mode. Default: auto-detected. */width?: number;/** ANSI color override. Default: auto-detected (true if TTY). */color?: boolean;/** Custom labels for outputs. Default: model names or 'Output A'/'Output B'. */labels?: string[];}
CompareOptions
Extends DiffOptions with:
interfaceCompareOptionsextendsDiffOptions{/** Max concurrent model calls. Default: unlimited (all in parallel). */concurrency?: number;/** Per-call timeout in milliseconds. Default: 30000. */timeout?: number;/** AbortSignal for cancellation. */signal?: AbortSignal;}
Diff Modes
ModeDescription
unifiedGit-style unified diff with word-level highlighting (default).
side-by-sideTwo-column display with aligned content and a vertical separator.
inlineInline additions (underline) and deletions (strikethrough) within the original text.
metricsMetrics comparison table only; no text diff output.
jsonStructural diff for JSON outputs with key-level change detection. Falls back to text diff if either output is not valid JSON.
Output Formats
FormatDescription
terminalANSI-colored output for terminal display.
jsonFull result serialized as JSON.stringify(result, null, 2).
markdownMarkdown-formatted diff.
plainPlain text with no ANSI codes.
Error Handling
Invalid embedding vectors
embeddingCosineSimilarity throws if the two vectors have different lengths:
import{embeddingCosineSimilarity}from'ai-diff';embeddingCosineSimilarity([1,2],[1,2,3]);// Error: Embedding vectors must have the same length: 2 vs 3
Unknown models
When a model is not in the built-in pricing table and no pricing override is provided, estimateCost returns undefined and the cost row is omitted from the metrics table. No error is thrown.
Failed model calls in compare()
When a model call fails or times out in compare(), the failure is captured in the calls array with status: 'error' and an error message. The failed output is excluded from pairwise diffs. Remaining models continue normally.
constresult=awaitcompare('prompt',['model-a','model-b'],llmFn);for(constcallofresult.calls){if(call.status==='error'){console.error(`${call.model} failed: ${call.error}`);}}
JSON diff fallback
When mode: 'json' is used but one or both outputs are not valid JSON, the engine falls back to a standard text diff. The jsonChanges field on the result will be undefined.
Advanced Usage
Semantic similarity with custom embeddings
Provide an embedFn to compute semantic similarity alongside Jaccard:
import{computeMetrics}from'ai-diff';constmetrics=awaitcomputeMetrics({text: 'The cat sat on the mat.',model: 'gpt-4o',tokens: {output: 8}},{text: 'A feline rested on a rug.',model: 'claude-sonnet',tokens: {output: 7}},{embedFn: async(text)=>{// Call your embedding API (OpenAI, Cohere, etc.)returnawaitgetEmbedding(text);},},);console.log(metrics.similarity.semantic);// 0.0 - 1.0
Custom model pricing
Override or extend the built-in pricing table:
import{diff}from'ai-diff';constresult=diff(outputA,outputB,{pricing: {'my-custom-model': {input: 0.001,output: 0.002},'gpt-4o': {input: 0.000003,output: 0.000012},// override built-in},});
Comparing structured JSON outputs
import{diff,formatDiff}from'ai-diff';constresult=diff({text: '{"name":"Alice","age":30}',model: 'gpt-4o'},{text: '{"name":"Alice","age":31,"role":"admin"}',model: 'claude-sonnet'},{mode: 'json'},);console.log(result.jsonChanges);// [// { path: 'age', type: 'changed', before: 30, after: 31 },// { path: 'role', type: 'added', after: 'admin' },// ]console.log(formatDiff(result,'terminal'));
Concurrency-limited model comparison
import{compare}from'ai-diff';constresult=awaitcompare('Summarize this article.',['gpt-4o','gpt-4o-mini','claude-sonnet','gemini-pro'],async(prompt,model)=>callLLM(prompt,model),{concurrency: 2,timeout: 10000},);// Only 2 models called at a time; each call times out after 10s
Metrics-only comparison
import{diff,formatDiff}from'ai-diff';constresult=diff(outputA,outputB,{mode: 'metrics'});console.log(formatDiff(result,'terminal'));// Prints only the metrics comparison table, no text diff
TypeScript
All types are exported from the package root:
importtype{LLMOutput,LLMFn,DiffMode,DiffOptions,CompareOptions,OutputFormat,DiffResult,MultiDiffResult,ComparisonResult,DiffSegment,DiffHunk,DiffMetrics,LengthStats,JsonChange,}from'ai-diff';
The package is compiled with TypeScript strict mode targeting ES2022 (CommonJS output). Declaration files and source maps are included.
License
MIT
About
Compare LLM responses across models with semantic diffs
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
ai-diff
Compare LLM outputs with word-level and line-level diffs, ANSI-colored terminal output, and AI-specific metrics.
npm versionnpm downloadslicensenodeTypeScript
ai-diff compares two or more LLM responses and produces structured diffs annotated with AI-specific metrics: token counts (input and output), estimated cost in USD (using built-in model pricing), response latency, Jaccard similarity scores, and length statistics (words, sentences, characters). It supports five diff modes -- unified, side-by-side, inline, metrics-only, and JSON structural diff -- and four output formats: terminal (ANSI-colored), JSON, Markdown, and plain text. Zero runtime dependencies.
Installation
npm install ai-diff
Quick Start
import{diff,formatDiff}from'ai-diff';constresult=diff({text: 'Paris is the capital of France.',model: 'gpt-4o',tokens: {input: 10,output: 8},latency: 1240},{text: 'The capital of France is Paris.',model: 'claude-sonnet',tokens: {input: 10,output: 8},latency: 980},);// Print a colored unified diff with metrics tableconsole.log(formatDiff(result,'terminal'));// Access structured dataconsole.log(result.identical);// falseconsole.log(result.similarity.jaccard);// 0.0 - 1.0console.log(result.metrics.latency?.delta);// -260
Features
  • Word-level and line-level diffs -- LCS-based algorithms implemented from scratch, no runtime dependencies.
  • Five diff modes -- unified (git-style), side-by-side (two-column), inline (strikethrough/underline), metrics (table only), json (structural key-level diff).
  • AI-specific metrics -- Token counts, estimated cost (USD), response latency, Jaccard similarity, word/sentence/character counts displayed in a comparison table alongside every diff.
  • Built-in model pricing -- GPT-4o, GPT-4o-mini, GPT-3.5 Turbo, GPT-4 Turbo, Claude Opus, Claude Sonnet, Claude Haiku, Gemini Pro, Gemini Flash. Override or extend with custom pricing.
  • N-way comparison -- diffOutputs() compares any number of outputs pairwise. compare() sends a prompt to multiple models via a user-provided function and diffs the results.
  • Four output formats -- terminal (ANSI colors), json (serialized result), markdown, plain.
  • Automatic token estimation -- When token counts are not provided, output tokens are estimated using a ceil(characters / 4) heuristic.
  • ANSI color detection -- Colors are enabled automatically when stdout is a TTY. Respects NO_COLOR environment variable. Override with the color option.
  • TypeScript-first -- Full type definitions exported for all interfaces, options, and result types. Strict mode enabled.
  • Zero runtime dependencies -- All diffing, similarity, formatting, and metrics logic uses only Node.js built-ins.
API Reference
diff(outputA, outputB, options?)
Compare two LLM outputs and return a DiffResult.
Parameters:
ParameterTypeDescription
outputAstring | LLMOutputFirst LLM output. Plain strings are wrapped as { text: string }.
outputBstring | LLMOutputSecond LLM output.
optionsDiffOptionsOptional configuration (see Configuration).
Returns:DiffResult
import{diff}from'ai-diff';// Compare plain stringsconstresult=diff('Output from model A','Output from model B');// Compare with full metadataconstresult=diff({text: 'Response A',model: 'gpt-4o',tokens: {input: 100,output: 50},cost: 0.005,latency: 1200},{text: 'Response B',model: 'claude-sonnet',tokens: {input: 100,output: 75},latency: 980},{mode: 'side-by-side'},);console.log(result.identical);// falseconsole.log(result.hunks.length);// number of diff hunksconsole.log(result.metrics.cost);// { a, b, delta, deltaPercent }console.log(result.similarity.jaccard);// 0.0 - 1.0
diffOutputs(outputs, options?)
Compare N LLM outputs pairwise.
Parameters:
ParameterTypeDescription
outputs(string | LLMOutput)[]Array of outputs to compare.
optionsDiffOptionsOptional configuration.
Returns:MultiDiffResult
import{diffOutputs}from'ai-diff';constresult=diffOutputs([{text: 'Output A',model: 'gpt-4o'},{text: 'Output B',model: 'claude-sonnet'},{text: 'Output C',model: 'gemini-pro'},]);console.log(result.pairwise.length);// 3 (A-B, A-C, B-C)console.log(result.metricsTable.labels);// ['gpt-4o', 'claude-sonnet', 'gemini-pro']console.log(result.metricsTable.wordCounts);// [n, n, n]
compare(prompt, models, llmFn, options?)
Send a prompt to multiple models via a user-provided function and compare the outputs.
Parameters:
ParameterTypeDescription
promptstringThe prompt to send to each model.
modelsstring[]Array of model identifiers.
llmFnLLMFnAsync function (prompt, model) => LLMOutput | string that calls the model.
optionsCompareOptionsOptional configuration (extends DiffOptions with concurrency, timeout, signal).
Returns:Promise<ComparisonResult>
import{compare}from'ai-diff';constresult=awaitcompare('Explain quantum computing in 3 sentences.',['gpt-4o','claude-sonnet'],async(prompt,model)=>{constresponse=awaitcallMyLLM(prompt,model);return{text: response.text,tokens: response.usage, model };},{concurrency: 2,timeout: 15000},);console.log(result.calls);// per-model status, output, latency, or errorconsole.log(result.pairwise);// pairwise diffs of successful outputs
formatDiff(result, format?)
Format a diff result into a displayable string.
Parameters:
ParameterTypeDefaultDescription
resultDiffResult | MultiDiffResult | ComparisonResult--The result to format.
formatOutputFormat'terminal'One of 'terminal', 'json', 'markdown', 'plain'.
Returns:string
import{diff,formatDiff}from'ai-diff';constresult=diff('hello world','hello earth');console.log(formatDiff(result,'terminal'));// ANSI-colored unified diffconsole.log(formatDiff(result,'json'));// JSON.stringify(result, null, 2)console.log(formatDiff(result,'plain'));// plain text, no ANSI codes
Similarity Functions
jaccardSimilarity(textA, textB)
Compute Jaccard similarity (word-level set overlap) between two texts. Returns a value between 0.0 and 1.0.
import{jaccardSimilarity}from'ai-diff';jaccardSimilarity('hello world','hello earth');// 0.333...jaccardSimilarity('hello world','hello world');// 1.0jaccardSimilarity('','');// 1.0
cosineSimilarity(textA, textB)
Compute cosine similarity using word-frequency vectors. Returns a value between 0.0 and 1.0.
import{cosineSimilarity}from'ai-diff';cosineSimilarity('the quick brown fox','the slow brown cat');// 0.0 - 1.0
exactMatchRatio(textA, textB)
Returns 1.0 if the two texts are identical, 0.0 otherwise.
import{exactMatchRatio}from'ai-diff';exactMatchRatio('hello','hello');// 1.0exactMatchRatio('hello','world');// 0.0
compositeSimilarity(textA, textB)
Weighted composite: Jaccard (0.5) + Cosine (0.3) + Exact Match (0.2).
import{compositeSimilarity}from'ai-diff';compositeSimilarity('hello world','hello earth');// 0.0 - 1.0
embeddingCosineSimilarity(a, b)
Compute cosine similarity between two numeric embedding vectors. Throws if vectors have different lengths.
import{embeddingCosineSimilarity}from'ai-diff';embeddingCosineSimilarity([1,0,0],[0,1,0]);// 0.0embeddingCosineSimilarity([1,0],[1,0]);// 1.0
computeLengthStats(text)
Returns { words, sentences, characters } for a given text.
import{computeLengthStats}from'ai-diff';computeLengthStats('Hello world. Goodbye.');// { words: 3, sentences: 2, characters: 21 }
Diff Utilities
diffWords(textA, textB)
Compute a word-level diff between two strings. Returns DiffSegment[].
import{diffWords}from'ai-diff';constsegments=diffWords('hello world','hello earth');// [// { text: 'hello', type: 'unchanged' },// { text: ' ', type: 'unchanged' },// { text: 'world', type: 'removed' },// { text: 'earth', type: 'added' },// ]
diffLines(textA, textB)
Compute a line-level diff between two strings. Returns DiffSegment[].
import{diffLines}from'ai-diff';constsegments=diffLines('line1\nline2','line1\nline3');
diffJson(a, b)
Compute a structural diff between two parsed JSON values. Returns JsonChange[] with dot-notation paths.
import{diffJson}from'ai-diff';constchanges=diffJson({name: 'Alice',age: 30},{name: 'Alice',age: 31,role: 'admin'},);// [// { path: 'age', type: 'changed', before: 30, after: 31 },// { path: 'role', type: 'added', after: 'admin' },// ]
computeHunks(textA, textB, contextLines?)
Compute diff hunks (contiguous groups of changes with context lines) between two texts. Returns DiffHunk[].
import{computeHunks}from'ai-diff';consthunks=computeHunks('line1\nline2\nline3','line1\nchanged\nline3',3);
tryParseJson(text)
Attempt to parse a string as JSON. Returns the parsed value on success or null on failure.
Metrics Utilities
estimateCost(output, pricingOverrides?)
Estimate the cost of an LLM output in USD. Returns the output's cost field if set, otherwise computes from model pricing and token counts. Returns undefined if insufficient data.
import{estimateCost}from'ai-diff';estimateCost({text: 'hello',model: 'gpt-4o',tokens: {input: 100,output: 50}});// 0.00075 (computed from built-in GPT-4o pricing)estimateCost({text: 'hello',model: 'custom',tokens: {input: 1000,output: 500}},{custom: {input: 0.001,output: 0.002}},);// 2.0
getModelPricing()
Returns a copy of the built-in model pricing table. Each entry maps a model name to { input: number; output: number } (per-token USD).
import{getModelPricing}from'ai-diff';constpricing=getModelPricing();// {// 'gpt-4o': { input: 0.0000025, output: 0.00001 },// 'claude-sonnet': { input: 0.000003, output: 0.000015 },// ...// }
Built-in models:gpt-4o, gpt-4o-mini, gpt-3.5-turbo, gpt-4-turbo, claude-opus, claude-sonnet, claude-haiku, gemini-pro, gemini-flash.
computeMetrics(outputA, outputB, options?)
Compute full comparative metrics between two outputs. Supports an optional embedFn for semantic similarity.
Parameters:
ParameterTypeDescription
outputALLMOutputFirst output.
outputBLLMOutputSecond output.
options.embedFn(text: string) => Promise<number[]>Optional embedding function for semantic similarity.
options.pricingRecord<string, { input: number; output: number }>Optional pricing overrides.
Returns:Promise<DiffMetrics>
Formatter Utilities
renderUnifiedDiff(result, useColor?)
Render a DiffResult as a unified diff string with optional ANSI colors. Removed lines are prefixed with - (red), added lines with + (green). Word-level changes are highlighted with bold inverse.
renderSideBySide(result, useColor?, width?)
Render a DiffResult as a two-column side-by-side display. Column width defaults to (width - 3) / 2.
renderInlineDiff(segments, useColor?)
Render DiffSegment[] as inline text. Removed words appear with strikethrough (or ~~text~~ without color), added words with underline (or __text__).
renderJsonDiff(changes, originalA, originalB, useColor?)
Render JsonChange[] as a formatted string showing added, removed, and changed keys.
renderMetricsTable(metrics, labelA, labelB, useColor?)
Render a DiffMetrics object as a Unicode box-drawing table comparing all metrics between two outputs.
shouldUseColor(override?)
Returns true if ANSI colors should be used. Checks override, then NO_COLOR env var, then process.stdout.isTTY.
Normalization Utilities
normalizeOutput(input)
Convert a string | LLMOutput to an LLMOutput object. Plain strings become { text: string }.
enrichOutput(output)
Fill in estimated fields on an LLMOutput. Adds estimated tokens.output (via ceil(text.length / 4)) when not provided.
estimateTokens(text)
Estimate token count from text length: Math.ceil(text.length / 4).
tokenizeWords(text)
Split text into word and whitespace tokens (preserving whitespace). Used internally by the LCS diff algorithm.
Configuration
DiffOptions
interfaceDiffOptions{/** Diff mode. Default: 'unified'. */mode?: 'unified'|'side-by-side'|'inline'|'metrics'|'json';/** Context lines around changes in unified mode. Default: 3. */contextLines?: number;/** Embedding function for semantic similarity. */embedFn?: (text: string)=>Promise<number[]>;/** Per-token pricing overrides in USD. Keyed by model name. */pricing?: Record<string,{input: number;output: number}>;/** Show the metrics summary table. Default: true. */showMetrics?: boolean;/** Position of the metrics table. Default: 'top'. */metricsPosition?: 'top'|'bottom';/** Which metrics to display. Default: all available. */metrics?: ('tokens'|'cost'|'latency'|'similarity'|'length'|'model')[];/** Terminal width override for side-by-side mode. Default: auto-detected. */width?: number;/** ANSI color override. Default: auto-detected (true if TTY). */color?: boolean;/** Custom labels for outputs. Default: model names or 'Output A'/'Output B'. */labels?: string[];}
CompareOptions
Extends DiffOptions with:
interfaceCompareOptionsextendsDiffOptions{/** Max concurrent model calls. Default: unlimited (all in parallel). */concurrency?: number;/** Per-call timeout in milliseconds. Default: 30000. */timeout?: number;/** AbortSignal for cancellation. */signal?: AbortSignal;}
Diff Modes
ModeDescription
unifiedGit-style unified diff with word-level highlighting (default).
side-by-sideTwo-column display with aligned content and a vertical separator.
inlineInline additions (underline) and deletions (strikethrough) within the original text.
metricsMetrics comparison table only; no text diff output.
jsonStructural diff for JSON outputs with key-level change detection. Falls back to text diff if either output is not valid JSON.
Output Formats
FormatDescription
terminalANSI-colored output for terminal display.
jsonFull result serialized as JSON.stringify(result, null, 2).
markdownMarkdown-formatted diff.
plainPlain text with no ANSI codes.
Error Handling
Invalid embedding vectors
embeddingCosineSimilarity throws if the two vectors have different lengths:
import{embeddingCosineSimilarity}from'ai-diff';embeddingCosineSimilarity([1,2],[1,2,3]);// Error: Embedding vectors must have the same length: 2 vs 3
Unknown models
When a model is not in the built-in pricing table and no pricing override is provided, estimateCost returns undefined and the cost row is omitted from the metrics table. No error is thrown.
Failed model calls in compare()
When a model call fails or times out in compare(), the failure is captured in the calls array with status: 'error' and an error message. The failed output is excluded from pairwise diffs. Remaining models continue normally.
constresult=awaitcompare('prompt',['model-a','model-b'],llmFn);for(constcallofresult.calls){if(call.status==='error'){console.error(`${call.model} failed: ${call.error}`);}}
JSON diff fallback
When mode: 'json' is used but one or both outputs are not valid JSON, the engine falls back to a standard text diff. The jsonChanges field on the result will be undefined.
Advanced Usage
Semantic similarity with custom embeddings
Provide an embedFn to compute semantic similarity alongside Jaccard:
import{computeMetrics}from'ai-diff';constmetrics=awaitcomputeMetrics({text: 'The cat sat on the mat.',model: 'gpt-4o',tokens: {output: 8}},{text: 'A feline rested on a rug.',model: 'claude-sonnet',tokens: {output: 7}},{embedFn: async(text)=>{// Call your embedding API (OpenAI, Cohere, etc.)returnawaitgetEmbedding(text);},},);console.log(metrics.similarity.semantic);// 0.0 - 1.0
Custom model pricing
Override or extend the built-in pricing table:
import{diff}from'ai-diff';constresult=diff(outputA,outputB,{pricing: {'my-custom-model': {input: 0.001,output: 0.002},'gpt-4o': {input: 0.000003,output: 0.000012},// override built-in},});
Comparing structured JSON outputs
import{diff,formatDiff}from'ai-diff';constresult=diff({text: '{"name":"Alice","age":30}',model: 'gpt-4o'},{text: '{"name":"Alice","age":31,"role":"admin"}',model: 'claude-sonnet'},{mode: 'json'},);console.log(result.jsonChanges);// [// { path: 'age', type: 'changed', before: 30, after: 31 },// { path: 'role', type: 'added', after: 'admin' },// ]console.log(formatDiff(result,'terminal'));
Concurrency-limited model comparison
import{compare}from'ai-diff';constresult=awaitcompare('Summarize this article.',['gpt-4o','gpt-4o-mini','claude-sonnet','gemini-pro'],async(prompt,model)=>callLLM(prompt,model),{concurrency: 2,timeout: 10000},);// Only 2 models called at a time; each call times out after 10s
Metrics-only comparison
import{diff,formatDiff}from'ai-diff';constresult=diff(outputA,outputB,{mode: 'metrics'});console.log(formatDiff(result,'terminal'));// Prints only the metrics comparison table, no text diff
TypeScript
All types are exported from the package root:
importtype{LLMOutput,LLMFn,DiffMode,DiffOptions,CompareOptions,OutputFormat,DiffResult,MultiDiffResult,ComparisonResult,DiffSegment,DiffHunk,DiffMetrics,LengthStats,JsonChange,}from'ai-diff';
The package is compiled with TypeScript strict mode targeting ES2022 (CommonJS output). Declaration files and source maps are included.
License
MIT
About
Compare LLM responses across models with semantic diffs
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
ai-diff
Compare LLM outputs with word-level and line-level diffs, ANSI-colored terminal output, and AI-specific metrics.
npm versionnpm downloadslicensenodeTypeScript
ai-diff compares two or more LLM responses and produces structured diffs annotated with AI-specific metrics: token counts (input and output), estimated cost in USD (using built-in model pricing), response latency, Jaccard similarity scores, and length statistics (words, sentences, characters). It supports five diff modes -- unified, side-by-side, inline, metrics-only, and JSON structural diff -- and four output formats: terminal (ANSI-colored), JSON, Markdown, and plain text. Zero runtime dependencies.
Installation
npm install ai-diff
Quick Start
import{diff,formatDiff}from'ai-diff';constresult=diff({text: 'Paris is the capital of France.',model: 'gpt-4o',tokens: {input: 10,output: 8},latency: 1240},{text: 'The capital of France is Paris.',model: 'claude-sonnet',tokens: {input: 10,output: 8},latency: 980},);// Print a colored unified diff with metrics tableconsole.log(formatDiff(result,'terminal'));// Access structured dataconsole.log(result.identical);// falseconsole.log(result.similarity.jaccard);// 0.0 - 1.0console.log(result.metrics.latency?.delta);// -260
Features
  • Word-level and line-level diffs -- LCS-based algorithms implemented from scratch, no runtime dependencies.
  • Five diff modes -- unified (git-style), side-by-side (two-column), inline (strikethrough/underline), metrics (table only), json (structural key-level diff).
  • AI-specific metrics -- Token counts, estimated cost (USD), response latency, Jaccard similarity, word/sentence/character counts displayed in a comparison table alongside every diff.
  • Built-in model pricing -- GPT-4o, GPT-4o-mini, GPT-3.5 Turbo, GPT-4 Turbo, Claude Opus, Claude Sonnet, Claude Haiku, Gemini Pro, Gemini Flash. Override or extend with custom pricing.
  • N-way comparison -- diffOutputs() compares any number of outputs pairwise. compare() sends a prompt to multiple models via a user-provided function and diffs the results.
  • Four output formats -- terminal (ANSI colors), json (serialized result), markdown, plain.
  • Automatic token estimation -- When token counts are not provided, output tokens are estimated using a ceil(characters / 4) heuristic.
  • ANSI color detection -- Colors are enabled automatically when stdout is a TTY. Respects NO_COLOR environment variable. Override with the color option.
  • TypeScript-first -- Full type definitions exported for all interfaces, options, and result types. Strict mode enabled.
  • Zero runtime dependencies -- All diffing, similarity, formatting, and metrics logic uses only Node.js built-ins.
API Reference
diff(outputA, outputB, options?)
Compare two LLM outputs and return a DiffResult.
Parameters:
ParameterTypeDescription
outputAstring | LLMOutputFirst LLM output. Plain strings are wrapped as { text: string }.
outputBstring | LLMOutputSecond LLM output.
optionsDiffOptionsOptional configuration (see Configuration).
Returns:DiffResult
import{diff}from'ai-diff';// Compare plain stringsconstresult=diff('Output from model A','Output from model B');// Compare with full metadataconstresult=diff({text: 'Response A',model: 'gpt-4o',tokens: {input: 100,output: 50},cost: 0.005,latency: 1200},{text: 'Response B',model: 'claude-sonnet',tokens: {input: 100,output: 75},latency: 980},{mode: 'side-by-side'},);console.log(result.identical);// falseconsole.log(result.hunks.length);// number of diff hunksconsole.log(result.metrics.cost);// { a, b, delta, deltaPercent }console.log(result.similarity.jaccard);// 0.0 - 1.0
diffOutputs(outputs, options?)
Compare N LLM outputs pairwise.
Parameters:
ParameterTypeDescription
outputs(string | LLMOutput)[]Array of outputs to compare.
optionsDiffOptionsOptional configuration.
Returns:MultiDiffResult
import{diffOutputs}from'ai-diff';constresult=diffOutputs([{text: 'Output A',model: 'gpt-4o'},{text: 'Output B',model: 'claude-sonnet'},{text: 'Output C',model: 'gemini-pro'},]);console.log(result.pairwise.length);// 3 (A-B, A-C, B-C)console.log(result.metricsTable.labels);// ['gpt-4o', 'claude-sonnet', 'gemini-pro']console.log(result.metricsTable.wordCounts);// [n, n, n]
compare(prompt, models, llmFn, options?)
Send a prompt to multiple models via a user-provided function and compare the outputs.
Parameters:
ParameterTypeDescription
promptstringThe prompt to send to each model.
modelsstring[]Array of model identifiers.
llmFnLLMFnAsync function (prompt, model) => LLMOutput | string that calls the model.
optionsCompareOptionsOptional configuration (extends DiffOptions with concurrency, timeout, signal).
Returns:Promise<ComparisonResult>
import{compare}from'ai-diff';constresult=awaitcompare('Explain quantum computing in 3 sentences.',['gpt-4o','claude-sonnet'],async(prompt,model)=>{constresponse=awaitcallMyLLM(prompt,model);return{text: response.text,tokens: response.usage, model };},{concurrency: 2,timeout: 15000},);console.log(result.calls);// per-model status, output, latency, or errorconsole.log(result.pairwise);// pairwise diffs of successful outputs
formatDiff(result, format?)
Format a diff result into a displayable string.
Parameters:
ParameterTypeDefaultDescription
resultDiffResult | MultiDiffResult | ComparisonResult--The result to format.
formatOutputFormat'terminal'One of 'terminal', 'json', 'markdown', 'plain'.
Returns:string
import{diff,formatDiff}from'ai-diff';constresult=diff('hello world','hello earth');console.log(formatDiff(result,'terminal'));// ANSI-colored unified diffconsole.log(formatDiff(result,'json'));// JSON.stringify(result, null, 2)console.log(formatDiff(result,'plain'));// plain text, no ANSI codes
Similarity Functions
jaccardSimilarity(textA, textB)
Compute Jaccard similarity (word-level set overlap) between two texts. Returns a value between 0.0 and 1.0.
import{jaccardSimilarity}from'ai-diff';jaccardSimilarity('hello world','hello earth');// 0.333...jaccardSimilarity('hello world','hello world');// 1.0jaccardSimilarity('','');// 1.0
cosineSimilarity(textA, textB)
Compute cosine similarity using word-frequency vectors. Returns a value between 0.0 and 1.0.
import{cosineSimilarity}from'ai-diff';cosineSimilarity('the quick brown fox','the slow brown cat');// 0.0 - 1.0
exactMatchRatio(textA, textB)
Returns 1.0 if the two texts are identical, 0.0 otherwise.
import{exactMatchRatio}from'ai-diff';exactMatchRatio('hello','hello');// 1.0exactMatchRatio('hello','world');// 0.0
compositeSimilarity(textA, textB)
Weighted composite: Jaccard (0.5) + Cosine (0.3) + Exact Match (0.2).
import{compositeSimilarity}from'ai-diff';compositeSimilarity('hello world','hello earth');// 0.0 - 1.0
embeddingCosineSimilarity(a, b)
Compute cosine similarity between two numeric embedding vectors. Throws if vectors have different lengths.
import{embeddingCosineSimilarity}from'ai-diff';embeddingCosineSimilarity([1,0,0],[0,1,0]);// 0.0embeddingCosineSimilarity([1,0],[1,0]);// 1.0
computeLengthStats(text)
Returns { words, sentences, characters } for a given text.
import{computeLengthStats}from'ai-diff';computeLengthStats('Hello world. Goodbye.');// { words: 3, sentences: 2, characters: 21 }
Diff Utilities
diffWords(textA, textB)
Compute a word-level diff between two strings. Returns DiffSegment[].
import{diffWords}from'ai-diff';constsegments=diffWords('hello world','hello earth');// [// { text: 'hello', type: 'unchanged' },// { text: ' ', type: 'unchanged' },// { text: 'world', type: 'removed' },// { text: 'earth', type: 'added' },// ]
diffLines(textA, textB)
Compute a line-level diff between two strings. Returns DiffSegment[].
import{diffLines}from'ai-diff';constsegments=diffLines('line1\nline2','line1\nline3');
diffJson(a, b)
Compute a structural diff between two parsed JSON values. Returns JsonChange[] with dot-notation paths.
import{diffJson}from'ai-diff';constchanges=diffJson({name: 'Alice',age: 30},{name: 'Alice',age: 31,role: 'admin'},);// [// { path: 'age', type: 'changed', before: 30, after: 31 },// { path: 'role', type: 'added', after: 'admin' },// ]
computeHunks(textA, textB, contextLines?)
Compute diff hunks (contiguous groups of changes with context lines) between two texts. Returns DiffHunk[].
import{computeHunks}from'ai-diff';consthunks=computeHunks('line1\nline2\nline3','line1\nchanged\nline3',3);
tryParseJson(text)
Attempt to parse a string as JSON. Returns the parsed value on success or null on failure.
Metrics Utilities
estimateCost(output, pricingOverrides?)
Estimate the cost of an LLM output in USD. Returns the output's cost field if set, otherwise computes from model pricing and token counts. Returns undefined if insufficient data.
import{estimateCost}from'ai-diff';estimateCost({text: 'hello',model: 'gpt-4o',tokens: {input: 100,output: 50}});// 0.00075 (computed from built-in GPT-4o pricing)estimateCost({text: 'hello',model: 'custom',tokens: {input: 1000,output: 500}},{custom: {input: 0.001,output: 0.002}},);// 2.0
getModelPricing()
Returns a copy of the built-in model pricing table. Each entry maps a model name to { input: number; output: number } (per-token USD).
import{getModelPricing}from'ai-diff';constpricing=getModelPricing();// {// 'gpt-4o': { input: 0.0000025, output: 0.00001 },// 'claude-sonnet': { input: 0.000003, output: 0.000015 },// ...// }
Built-in models:gpt-4o, gpt-4o-mini, gpt-3.5-turbo, gpt-4-turbo, claude-opus, claude-sonnet, claude-haiku, gemini-pro, gemini-flash.
computeMetrics(outputA, outputB, options?)
Compute full comparative metrics between two outputs. Supports an optional embedFn for semantic similarity.
Parameters:
ParameterTypeDescription
outputALLMOutputFirst output.
outputBLLMOutputSecond output.
options.embedFn(text: string) => Promise<number[]>Optional embedding function for semantic similarity.
options.pricingRecord<string, { input: number; output: number }>Optional pricing overrides.
Returns:Promise<DiffMetrics>
Formatter Utilities
renderUnifiedDiff(result, useColor?)
Render a DiffResult as a unified diff string with optional ANSI colors. Removed lines are prefixed with - (red), added lines with + (green). Word-level changes are highlighted with bold inverse.
renderSideBySide(result, useColor?, width?)
Render a DiffResult as a two-column side-by-side display. Column width defaults to (width - 3) / 2.
renderInlineDiff(segments, useColor?)
Render DiffSegment[] as inline text. Removed words appear with strikethrough (or ~~text~~ without color), added words with underline (or __text__).
renderJsonDiff(changes, originalA, originalB, useColor?)
Render JsonChange[] as a formatted string showing added, removed, and changed keys.
renderMetricsTable(metrics, labelA, labelB, useColor?)
Render a DiffMetrics object as a Unicode box-drawing table comparing all metrics between two outputs.
shouldUseColor(override?)
Returns true if ANSI colors should be used. Checks override, then NO_COLOR env var, then process.stdout.isTTY.
Normalization Utilities
normalizeOutput(input)
Convert a string | LLMOutput to an LLMOutput object. Plain strings become { text: string }.
enrichOutput(output)
Fill in estimated fields on an LLMOutput. Adds estimated tokens.output (via ceil(text.length / 4)) when not provided.
estimateTokens(text)
Estimate token count from text length: Math.ceil(text.length / 4).
tokenizeWords(text)
Split text into word and whitespace tokens (preserving whitespace). Used internally by the LCS diff algorithm.
Configuration
DiffOptions
interfaceDiffOptions{/** Diff mode. Default: 'unified'. */mode?: 'unified'|'side-by-side'|'inline'|'metrics'|'json';/** Context lines around changes in unified mode. Default: 3. */contextLines?: number;/** Embedding function for semantic similarity. */embedFn?: (text: string)=>Promise<number[]>;/** Per-token pricing overrides in USD. Keyed by model name. */pricing?: Record<string,{input: number;output: number}>;/** Show the metrics summary table. Default: true. */showMetrics?: boolean;/** Position of the metrics table. Default: 'top'. */metricsPosition?: 'top'|'bottom';/** Which metrics to display. Default: all available. */metrics?: ('tokens'|'cost'|'latency'|'similarity'|'length'|'model')[];/** Terminal width override for side-by-side mode. Default: auto-detected. */width?: number;/** ANSI color override. Default: auto-detected (true if TTY). */color?: boolean;/** Custom labels for outputs. Default: model names or 'Output A'/'Output B'. */labels?: string[];}
CompareOptions
Extends DiffOptions with:
interfaceCompareOptionsextendsDiffOptions{/** Max concurrent model calls. Default: unlimited (all in parallel). */concurrency?: number;/** Per-call timeout in milliseconds. Default: 30000. */timeout?: number;/** AbortSignal for cancellation. */signal?: AbortSignal;}
Diff Modes
ModeDescription
unifiedGit-style unified diff with word-level highlighting (default).
side-by-sideTwo-column display with aligned content and a vertical separator.
inlineInline additions (underline) and deletions (strikethrough) within the original text.
metricsMetrics comparison table only; no text diff output.
jsonStructural diff for JSON outputs with key-level change detection. Falls back to text diff if either output is not valid JSON.
Output Formats
FormatDescription
terminalANSI-colored output for terminal display.
jsonFull result serialized as JSON.stringify(result, null, 2).
markdownMarkdown-formatted diff.
plainPlain text with no ANSI codes.
Error Handling
Invalid embedding vectors
embeddingCosineSimilarity throws if the two vectors have different lengths:
import{embeddingCosineSimilarity}from'ai-diff';embeddingCosineSimilarity([1,2],[1,2,3]);// Error: Embedding vectors must have the same length: 2 vs 3
Unknown models
When a model is not in the built-in pricing table and no pricing override is provided, estimateCost returns undefined and the cost row is omitted from the metrics table. No error is thrown.
Failed model calls in compare()
When a model call fails or times out in compare(), the failure is captured in the calls array with status: 'error' and an error message. The failed output is excluded from pairwise diffs. Remaining models continue normally.
constresult=awaitcompare('prompt',['model-a','model-b'],llmFn);for(constcallofresult.calls){if(call.status==='error'){console.error(`${call.model} failed: ${call.error}`);}}
JSON diff fallback
When mode: 'json' is used but one or both outputs are not valid JSON, the engine falls back to a standard text diff. The jsonChanges field on the result will be undefined.
Advanced Usage
Semantic similarity with custom embeddings
Provide an embedFn to compute semantic similarity alongside Jaccard:
import{computeMetrics}from'ai-diff';constmetrics=awaitcomputeMetrics({text: 'The cat sat on the mat.',model: 'gpt-4o',tokens: {output: 8}},{text: 'A feline rested on a rug.',model: 'claude-sonnet',tokens: {output: 7}},{embedFn: async(text)=>{// Call your embedding API (OpenAI, Cohere, etc.)returnawaitgetEmbedding(text);},},);console.log(metrics.similarity.semantic);// 0.0 - 1.0
Custom model pricing
Override or extend the built-in pricing table:
import{diff}from'ai-diff';constresult=diff(outputA,outputB,{pricing: {'my-custom-model': {input: 0.001,output: 0.002},'gpt-4o': {input: 0.000003,output: 0.000012},// override built-in},});
Comparing structured JSON outputs
import{diff,formatDiff}from'ai-diff';constresult=diff({text: '{"name":"Alice","age":30}',model: 'gpt-4o'},{text: '{"name":"Alice","age":31,"role":"admin"}',model: 'claude-sonnet'},{mode: 'json'},);console.log(result.jsonChanges);// [// { path: 'age', type: 'changed', before: 30, after: 31 },// { path: 'role', type: 'added', after: 'admin' },// ]console.log(formatDiff(result,'terminal'));
Concurrency-limited model comparison
import{compare}from'ai-diff';constresult=awaitcompare('Summarize this article.',['gpt-4o','gpt-4o-mini','claude-sonnet','gemini-pro'],async(prompt,model)=>callLLM(prompt,model),{concurrency: 2,timeout: 10000},);// Only 2 models called at a time; each call times out after 10s
Metrics-only comparison
import{diff,formatDiff}from'ai-diff';constresult=diff(outputA,outputB,{mode: 'metrics'});console.log(formatDiff(result,'terminal'));// Prints only the metrics comparison table, no text diff
TypeScript
All types are exported from the package root:
importtype{LLMOutput,LLMFn,DiffMode,DiffOptions,CompareOptions,OutputFormat,DiffResult,MultiDiffResult,ComparisonResult,DiffSegment,DiffHunk,DiffMetrics,LengthStats,JsonChange,}from'ai-diff';
The package is compiled with TypeScript strict mode targeting ES2022 (CommonJS output). Declaration files and source maps are included.
License
MIT
About
Compare LLM responses across models with semantic diffs
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
BranchesTags
Repository files navigation
ai-diff
Compare LLM outputs with word-level and line-level diffs, ANSI-colored terminal output, and AI-specific metrics.
npm versionnpm downloadslicensenodeTypeScript
ai-diff compares two or more LLM responses and produces structured diffs annotated with AI-specific metrics: token counts (input and output), estimated cost in USD (using built-in model pricing), response latency, Jaccard similarity scores, and length statistics (words, sentences, characters). It supports five diff modes -- unified, side-by-side, inline, metrics-only, and JSON structural diff -- and four output formats: terminal (ANSI-colored), JSON, Markdown, and plain text. Zero runtime dependencies.
Installation
npm install ai-diff
Quick Start
import{diff,formatDiff}from'ai-diff';constresult=diff({text: 'Paris is the capital of France.',model: 'gpt-4o',tokens: {input: 10,output: 8},latency: 1240},{text: 'The capital of France is Paris.',model: 'claude-sonnet',tokens: {input: 10,output: 8},latency: 980},);// Print a colored unified diff with metrics tableconsole.log(formatDiff(result,'terminal'));// Access structured dataconsole.log(result.identical);// falseconsole.log(result.similarity.jaccard);// 0.0 - 1.0console.log(result.metrics.latency?.delta);// -260
Features
  • Word-level and line-level diffs -- LCS-based algorithms implemented from scratch, no runtime dependencies.
  • Five diff modes -- unified (git-style), side-by-side (two-column), inline (strikethrough/underline), metrics (table only), json (structural key-level diff).
  • AI-specific metrics -- Token counts, estimated cost (USD), response latency, Jaccard similarity, word/sentence/character counts displayed in a comparison table alongside every diff.
  • Built-in model pricing -- GPT-4o, GPT-4o-mini, GPT-3.5 Turbo, GPT-4 Turbo, Claude Opus, Claude Sonnet, Claude Haiku, Gemini Pro, Gemini Flash. Override or extend with custom pricing.
  • N-way comparison -- diffOutputs() compares any number of outputs pairwise. compare() sends a prompt to multiple models via a user-provided function and diffs the results.
  • Four output formats -- terminal (ANSI colors), json (serialized result), markdown, plain.
  • Automatic token estimation -- When token counts are not provided, output tokens are estimated using a ceil(characters / 4) heuristic.
  • ANSI color detection -- Colors are enabled automatically when stdout is a TTY. Respects NO_COLOR environment variable. Override with the color option.
  • TypeScript-first -- Full type definitions exported for all interfaces, options, and result types. Strict mode enabled.
  • Zero runtime dependencies -- All diffing, similarity, formatting, and metrics logic uses only Node.js built-ins.
API Reference
diff(outputA, outputB, options?)
Compare two LLM outputs and return a DiffResult.
Parameters:
ParameterTypeDescription
outputAstring | LLMOutputFirst LLM output. Plain strings are wrapped as { text: string }.
outputBstring | LLMOutputSecond LLM output.
optionsDiffOptionsOptional configuration (see Configuration).
Returns:DiffResult
import{diff}from'ai-diff';// Compare plain stringsconstresult=diff('Output from model A','Output from model B');// Compare with full metadataconstresult=diff({text: 'Response A',model: 'gpt-4o',tokens: {input: 100,output: 50},cost: 0.005,latency: 1200},{text: 'Response B',model: 'claude-sonnet',tokens: {input: 100,output: 75},latency: 980},{mode: 'side-by-side'},);console.log(result.identical);// falseconsole.log(result.hunks.length);// number of diff hunksconsole.log(result.metrics.cost);// { a, b, delta, deltaPercent }console.log(result.similarity.jaccard);// 0.0 - 1.0
diffOutputs(outputs, options?)
Compare N LLM outputs pairwise.
Parameters:
ParameterTypeDescription
outputs(string | LLMOutput)[]Array of outputs to compare.
optionsDiffOptionsOptional configuration.
Returns:MultiDiffResult
import{diffOutputs}from'ai-diff';constresult=diffOutputs([{text: 'Output A',model: 'gpt-4o'},{text: 'Output B',model: 'claude-sonnet'},{text: 'Output C',model: 'gemini-pro'},]);console.log(result.pairwise.length);// 3 (A-B, A-C, B-C)console.log(result.metricsTable.labels);// ['gpt-4o', 'claude-sonnet', 'gemini-pro']console.log(result.metricsTable.wordCounts);// [n, n, n]
compare(prompt, models, llmFn, options?)
Send a prompt to multiple models via a user-provided function and compare the outputs.
Parameters:
ParameterTypeDescription
promptstringThe prompt to send to each model.
modelsstring[]Array of model identifiers.
llmFnLLMFnAsync function (prompt, model) => LLMOutput | string that calls the model.
optionsCompareOptionsOptional configuration (extends DiffOptions with concurrency, timeout, signal).
Returns:Promise<ComparisonResult>
import{compare}from'ai-diff';constresult=awaitcompare('Explain quantum computing in 3 sentences.',['gpt-4o','claude-sonnet'],async(prompt,model)=>{constresponse=awaitcallMyLLM(prompt,model);return{text: response.text,tokens: response.usage, model };},{concurrency: 2,timeout: 15000},);console.log(result.calls);// per-model status, output, latency, or errorconsole.log(result.pairwise);// pairwise diffs of successful outputs
formatDiff(result, format?)
Format a diff result into a displayable string.
Parameters:
ParameterTypeDefaultDescription
resultDiffResult | MultiDiffResult | ComparisonResult--The result to format.
formatOutputFormat'terminal'One of 'terminal', 'json', 'markdown', 'plain'.
Returns:string
import{diff,formatDiff}from'ai-diff';constresult=diff('hello world','hello earth');console.log(formatDiff(result,'terminal'));// ANSI-colored unified diffconsole.log(formatDiff(result,'json'));// JSON.stringify(result, null, 2)console.log(formatDiff(result,'plain'));// plain text, no ANSI codes
Similarity Functions
jaccardSimilarity(textA, textB)
Compute Jaccard similarity (word-level set overlap) between two texts. Returns a value between 0.0 and 1.0.
import{jaccardSimilarity}from'ai-diff';jaccardSimilarity('hello world','hello earth');// 0.333...jaccardSimilarity('hello world','hello world');// 1.0jaccardSimilarity('','');// 1.0
cosineSimilarity(textA, textB)
Compute cosine similarity using word-frequency vectors. Returns a value between 0.0 and 1.0.
import{cosineSimilarity}from'ai-diff';cosineSimilarity('the quick brown fox','the slow brown cat');// 0.0 - 1.0
exactMatchRatio(textA, textB)
Returns 1.0 if the two texts are identical, 0.0 otherwise.
import{exactMatchRatio}from'ai-diff';exactMatchRatio('hello','hello');// 1.0exactMatchRatio('hello','world');// 0.0
compositeSimilarity(textA, textB)
Weighted composite: Jaccard (0.5) + Cosine (0.3) + Exact Match (0.2).
import{compositeSimilarity}from'ai-diff';compositeSimilarity('hello world','hello earth');// 0.0 - 1.0
embeddingCosineSimilarity(a, b)
Compute cosine similarity between two numeric embedding vectors. Throws if vectors have different lengths.
import{embeddingCosineSimilarity}from'ai-diff';embeddingCosineSimilarity([1,0,0],[0,1,0]);// 0.0embeddingCosineSimilarity([1,0],[1,0]);// 1.0
computeLengthStats(text)
Returns { words, sentences, characters } for a given text.
import{computeLengthStats}from'ai-diff';computeLengthStats('Hello world. Goodbye.');// { words: 3, sentences: 2, characters: 21 }
Diff Utilities
diffWords(textA, textB)
Compute a word-level diff between two strings. Returns DiffSegment[].
import{diffWords}from'ai-diff';constsegments=diffWords('hello world','hello earth');// [// { text: 'hello', type: 'unchanged' },// { text: ' ', type: 'unchanged' },// { text: 'world', type: 'removed' },// { text: 'earth', type: 'added' },// ]
diffLines(textA, textB)
Compute a line-level diff between two strings. Returns DiffSegment[].
import{diffLines}from'ai-diff';constsegments=diffLines('line1\nline2','line1\nline3');
diffJson(a, b)
Compute a structural diff between two parsed JSON values. Returns JsonChange[] with dot-notation paths.
import{diffJson}from'ai-diff';constchanges=diffJson({name: 'Alice',age: 30},{name: 'Alice',age: 31,role: 'admin'},);// [// { path: 'age', type: 'changed', before: 30, after: 31 },// { path: 'role', type: 'added', after: 'admin' },// ]
computeHunks(textA, textB, contextLines?)
Compute diff hunks (contiguous groups of changes with context lines) between two texts. Returns DiffHunk[].
import{computeHunks}from'ai-diff';consthunks=computeHunks('line1\nline2\nline3','line1\nchanged\nline3',3);
tryParseJson(text)
Attempt to parse a string as JSON. Returns the parsed value on success or null on failure.
Metrics Utilities
estimateCost(output, pricingOverrides?)
Estimate the cost of an LLM output in USD. Returns the output's cost field if set, otherwise computes from model pricing and token counts. Returns undefined if insufficient data.
import{estimateCost}from'ai-diff';estimateCost({text: 'hello',model: 'gpt-4o',tokens: {input: 100,output: 50}});// 0.00075 (computed from built-in GPT-4o pricing)estimateCost({text: 'hello',model: 'custom',tokens: {input: 1000,output: 500}},{custom: {input: 0.001,output: 0.002}},);// 2.0
getModelPricing()
Returns a copy of the built-in model pricing table. Each entry maps a model name to { input: number; output: number } (per-token USD).
import{getModelPricing}from'ai-diff';constpricing=getModelPricing();// {// 'gpt-4o': { input: 0.0000025, output: 0.00001 },// 'claude-sonnet': { input: 0.000003, output: 0.000015 },// ...// }
Built-in models:gpt-4o, gpt-4o-mini, gpt-3.5-turbo, gpt-4-turbo, claude-opus, claude-sonnet, claude-haiku, gemini-pro, gemini-flash.
computeMetrics(outputA, outputB, options?)
Compute full comparative metrics between two outputs. Supports an optional embedFn for semantic similarity.
Parameters:
ParameterTypeDescription
outputALLMOutputFirst output.
outputBLLMOutputSecond output.
options.embedFn(text: string) => Promise<number[]>Optional embedding function for semantic similarity.
options.pricingRecord<string, { input: number; output: number }>Optional pricing overrides.
Returns:Promise<DiffMetrics>
Formatter Utilities
renderUnifiedDiff(result, useColor?)
Render a DiffResult as a unified diff string with optional ANSI colors. Removed lines are prefixed with - (red), added lines with + (green). Word-level changes are highlighted with bold inverse.
renderSideBySide(result, useColor?, width?)
Render a DiffResult as a two-column side-by-side display. Column width defaults to (width - 3) / 2.
renderInlineDiff(segments, useColor?)
Render DiffSegment[] as inline text. Removed words appear with strikethrough (or ~~text~~ without color), added words with underline (or __text__).
renderJsonDiff(changes, originalA, originalB, useColor?)
Render JsonChange[] as a formatted string showing added, removed, and changed keys.
renderMetricsTable(metrics, labelA, labelB, useColor?)
Render a DiffMetrics object as a Unicode box-drawing table comparing all metrics between two outputs.
shouldUseColor(override?)
Returns true if ANSI colors should be used. Checks override, then NO_COLOR env var, then process.stdout.isTTY.
Normalization Utilities
normalizeOutput(input)
Convert a string | LLMOutput to an LLMOutput object. Plain strings become { text: string }.
enrichOutput(output)
Fill in estimated fields on an LLMOutput. Adds estimated tokens.output (via ceil(text.length / 4)) when not provided.
estimateTokens(text)
Estimate token count from text length: Math.ceil(text.length / 4).
tokenizeWords(text)
Split text into word and whitespace tokens (preserving whitespace). Used internally by the LCS diff algorithm.
Configuration
DiffOptions
interfaceDiffOptions{/** Diff mode. Default: 'unified'. */mode?: 'unified'|'side-by-side'|'inline'|'metrics'|'json';/** Context lines around changes in unified mode. Default: 3. */contextLines?: number;/** Embedding function for semantic similarity. */embedFn?: (text: string)=>Promise<number[]>;/** Per-token pricing overrides in USD. Keyed by model name. */pricing?: Record<string,{input: number;output: number}>;/** Show the metrics summary table. Default: true. */showMetrics?: boolean;/** Position of the metrics table. Default: 'top'. */metricsPosition?: 'top'|'bottom';/** Which metrics to display. Default: all available. */metrics?: ('tokens'|'cost'|'latency'|'similarity'|'length'|'model')[];/** Terminal width override for side-by-side mode. Default: auto-detected. */width?: number;/** ANSI color override. Default: auto-detected (true if TTY). */color?: boolean;/** Custom labels for outputs. Default: model names or 'Output A'/'Output B'. */labels?: string[];}
CompareOptions
Extends DiffOptions with:
interfaceCompareOptionsextendsDiffOptions{/** Max concurrent model calls. Default: unlimited (all in parallel). */concurrency?: number;/** Per-call timeout in milliseconds. Default: 30000. */timeout?: number;/** AbortSignal for cancellation. */signal?: AbortSignal;}
Diff Modes
ModeDescription
unifiedGit-style unified diff with word-level highlighting (default).
side-by-sideTwo-column display with aligned content and a vertical separator.
inlineInline additions (underline) and deletions (strikethrough) within the original text.
metricsMetrics comparison table only; no text diff output.
jsonStructural diff for JSON outputs with key-level change detection. Falls back to text diff if either output is not valid JSON.
Output Formats
FormatDescription
terminalANSI-colored output for terminal display.
jsonFull result serialized as JSON.stringify(result, null, 2).
markdownMarkdown-formatted diff.
plainPlain text with no ANSI codes.
Error Handling
Invalid embedding vectors
embeddingCosineSimilarity throws if the two vectors have different lengths:
import{embeddingCosineSimilarity}from'ai-diff';embeddingCosineSimilarity([1,2],[1,2,3]);// Error: Embedding vectors must have the same length: 2 vs 3
Unknown models
When a model is not in the built-in pricing table and no pricing override is provided, estimateCost returns undefined and the cost row is omitted from the metrics table. No error is thrown.
Failed model calls in compare()
When a model call fails or times out in compare(), the failure is captured in the calls array with status: 'error' and an error message. The failed output is excluded from pairwise diffs. Remaining models continue normally.
constresult=awaitcompare('prompt',['model-a','model-b'],llmFn);for(constcallofresult.calls){if(call.status==='error'){console.error(`${call.model} failed: ${call.error}`);}}
JSON diff fallback
When mode: 'json' is used but one or both outputs are not valid JSON, the engine falls back to a standard text diff. The jsonChanges field on the result will be undefined.
Advanced Usage
Semantic similarity with custom embeddings
Provide an embedFn to compute semantic similarity alongside Jaccard:
import{computeMetrics}from'ai-diff';constmetrics=awaitcomputeMetrics({text: 'The cat sat on the mat.',model: 'gpt-4o',tokens: {output: 8}},{text: 'A feline rested on a rug.',model: 'claude-sonnet',tokens: {output: 7}},{embedFn: async(text)=>{// Call your embedding API (OpenAI, Cohere, etc.)returnawaitgetEmbedding(text);},},);console.log(metrics.similarity.semantic);// 0.0 - 1.0
Custom model pricing
Override or extend the built-in pricing table:
import{diff}from'ai-diff';constresult=diff(outputA,outputB,{pricing: {'my-custom-model': {input: 0.001,output: 0.002},'gpt-4o': {input: 0.000003,output: 0.000012},// override built-in},});
Comparing structured JSON outputs
import{diff,formatDiff}from'ai-diff';constresult=diff({text: '{"name":"Alice","age":30}',model: 'gpt-4o'},{text: '{"name":"Alice","age":31,"role":"admin"}',model: 'claude-sonnet'},{mode: 'json'},);console.log(result.jsonChanges);// [// { path: 'age', type: 'changed', before: 30, after: 31 },// { path: 'role', type: 'added', after: 'admin' },// ]console.log(formatDiff(result,'terminal'));
Concurrency-limited model comparison
import{compare}from'ai-diff';constresult=awaitcompare('Summarize this article.',['gpt-4o','gpt-4o-mini','claude-sonnet','gemini-pro'],async(prompt,model)=>callLLM(prompt,model),{concurrency: 2,timeout: 10000},);// Only 2 models called at a time; each call times out after 10s
Metrics-only comparison
import{diff,formatDiff}from'ai-diff';constresult=diff(outputA,outputB,{mode: 'metrics'});console.log(formatDiff(result,'terminal'));// Prints only the metrics comparison table, no text diff
TypeScript
All types are exported from the package root:
importtype{LLMOutput,LLMFn,DiffMode,DiffOptions,CompareOptions,OutputFormat,DiffResult,MultiDiffResult,ComparisonResult,DiffSegment,DiffHunk,DiffMetrics,LengthStats,JsonChange,}from'ai-diff';
The package is compiled with TypeScript strict mode targeting ES2022 (CommonJS output). Declaration files and source maps are included.
License
MIT
About
Compare LLM responses across models with semantic diffs
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
ai-diff
Compare LLM outputs with word-level and line-level diffs, ANSI-colored terminal output, and AI-specific metrics.
npm versionnpm downloadslicensenodeTypeScript
ai-diff compares two or more LLM responses and produces structured diffs annotated with AI-specific metrics: token counts (input and output), estimated cost in USD (using built-in model pricing), response latency, Jaccard similarity scores, and length statistics (words, sentences, characters). It supports five diff modes -- unified, side-by-side, inline, metrics-only, and JSON structural diff -- and four output formats: terminal (ANSI-colored), JSON, Markdown, and plain text. Zero runtime dependencies.
Installation
npm install ai-diff
Quick Start
import{diff,formatDiff}from'ai-diff';constresult=diff({text: 'Paris is the capital of France.',model: 'gpt-4o',tokens: {input: 10,output: 8},latency: 1240},{text: 'The capital of France is Paris.',model: 'claude-sonnet',tokens: {input: 10,output: 8},latency: 980},);// Print a colored unified diff with metrics tableconsole.log(formatDiff(result,'terminal'));// Access structured dataconsole.log(result.identical);// falseconsole.log(result.similarity.jaccard);// 0.0 - 1.0console.log(result.metrics.latency?.delta);// -260
Features
  • Word-level and line-level diffs -- LCS-based algorithms implemented from scratch, no runtime dependencies.
  • Five diff modes -- unified (git-style), side-by-side (two-column), inline (strikethrough/underline), metrics (table only), json (structural key-level diff).
  • AI-specific metrics -- Token counts, estimated cost (USD), response latency, Jaccard similarity, word/sentence/character counts displayed in a comparison table alongside every diff.
  • Built-in model pricing -- GPT-4o, GPT-4o-mini, GPT-3.5 Turbo, GPT-4 Turbo, Claude Opus, Claude Sonnet, Claude Haiku, Gemini Pro, Gemini Flash. Override or extend with custom pricing.
  • N-way comparison -- diffOutputs() compares any number of outputs pairwise. compare() sends a prompt to multiple models via a user-provided function and diffs the results.
  • Four output formats -- terminal (ANSI colors), json (serialized result), markdown, plain.
  • Automatic token estimation -- When token counts are not provided, output tokens are estimated using a ceil(characters / 4) heuristic.
  • ANSI color detection -- Colors are enabled automatically when stdout is a TTY. Respects NO_COLOR environment variable. Override with the color option.
  • TypeScript-first -- Full type definitions exported for all interfaces, options, and result types. Strict mode enabled.
  • Zero runtime dependencies -- All diffing, similarity, formatting, and metrics logic uses only Node.js built-ins.
API Reference
diff(outputA, outputB, options?)
Compare two LLM outputs and return a DiffResult.
Parameters:
ParameterTypeDescription
outputAstring | LLMOutputFirst LLM output. Plain strings are wrapped as { text: string }.
outputBstring | LLMOutputSecond LLM output.
optionsDiffOptionsOptional configuration (see Configuration).
Returns:DiffResult
import{diff}from'ai-diff';// Compare plain stringsconstresult=diff('Output from model A','Output from model B');// Compare with full metadataconstresult=diff({text: 'Response A',model: 'gpt-4o',tokens: {input: 100,output: 50},cost: 0.005,latency: 1200},{text: 'Response B',model: 'claude-sonnet',tokens: {input: 100,output: 75},latency: 980},{mode: 'side-by-side'},);console.log(result.identical);// falseconsole.log(result.hunks.length);// number of diff hunksconsole.log(result.metrics.cost);// { a, b, delta, deltaPercent }console.log(result.similarity.jaccard);// 0.0 - 1.0
diffOutputs(outputs, options?)
Compare N LLM outputs pairwise.
Parameters:
ParameterTypeDescription
outputs(string | LLMOutput)[]Array of outputs to compare.
optionsDiffOptionsOptional configuration.
Returns:MultiDiffResult
import{diffOutputs}from'ai-diff';constresult=diffOutputs([{text: 'Output A',model: 'gpt-4o'},{text: 'Output B',model: 'claude-sonnet'},{text: 'Output C',model: 'gemini-pro'},]);console.log(result.pairwise.length);// 3 (A-B, A-C, B-C)console.log(result.metricsTable.labels);// ['gpt-4o', 'claude-sonnet', 'gemini-pro']console.log(result.metricsTable.wordCounts);// [n, n, n]
compare(prompt, models, llmFn, options?)
Send a prompt to multiple models via a user-provided function and compare the outputs.
Parameters:
ParameterTypeDescription
promptstringThe prompt to send to each model.
modelsstring[]Array of model identifiers.
llmFnLLMFnAsync function (prompt, model) => LLMOutput | string that calls the model.
optionsCompareOptionsOptional configuration (extends DiffOptions with concurrency, timeout, signal).
Returns:Promise<ComparisonResult>
import{compare}from'ai-diff';constresult=awaitcompare('Explain quantum computing in 3 sentences.',['gpt-4o','claude-sonnet'],async(prompt,model)=>{constresponse=awaitcallMyLLM(prompt,model);return{text: response.text,tokens: response.usage, model };},{concurrency: 2,timeout: 15000},);console.log(result.calls);// per-model status, output, latency, or errorconsole.log(result.pairwise);// pairwise diffs of successful outputs
formatDiff(result, format?)
Format a diff result into a displayable string.
Parameters:
ParameterTypeDefaultDescription
resultDiffResult | MultiDiffResult | ComparisonResult--The result to format.
formatOutputFormat'terminal'One of 'terminal', 'json', 'markdown', 'plain'.
Returns:string
import{diff,formatDiff}from'ai-diff';constresult=diff('hello world','hello earth');console.log(formatDiff(result,'terminal'));// ANSI-colored unified diffconsole.log(formatDiff(result,'json'));// JSON.stringify(result, null, 2)console.log(formatDiff(result,'plain'));// plain text, no ANSI codes
Similarity Functions
jaccardSimilarity(textA, textB)
Compute Jaccard similarity (word-level set overlap) between two texts. Returns a value between 0.0 and 1.0.
import{jaccardSimilarity}from'ai-diff';jaccardSimilarity('hello world','hello earth');// 0.333...jaccardSimilarity('hello world','hello world');// 1.0jaccardSimilarity('','');// 1.0
cosineSimilarity(textA, textB)
Compute cosine similarity using word-frequency vectors. Returns a value between 0.0 and 1.0.
import{cosineSimilarity}from'ai-diff';cosineSimilarity('the quick brown fox','the slow brown cat');// 0.0 - 1.0
exactMatchRatio(textA, textB)
Returns 1.0 if the two texts are identical, 0.0 otherwise.
import{exactMatchRatio}from'ai-diff';exactMatchRatio('hello','hello');// 1.0exactMatchRatio('hello','world');// 0.0
compositeSimilarity(textA, textB)
Weighted composite: Jaccard (0.5) + Cosine (0.3) + Exact Match (0.2).
import{compositeSimilarity}from'ai-diff';compositeSimilarity('hello world','hello earth');// 0.0 - 1.0
embeddingCosineSimilarity(a, b)
Compute cosine similarity between two numeric embedding vectors. Throws if vectors have different lengths.
import{embeddingCosineSimilarity}from'ai-diff';embeddingCosineSimilarity([1,0,0],[0,1,0]);// 0.0embeddingCosineSimilarity([1,0],[1,0]);// 1.0
computeLengthStats(text)
Returns { words, sentences, characters } for a given text.
import{computeLengthStats}from'ai-diff';computeLengthStats('Hello world. Goodbye.');// { words: 3, sentences: 2, characters: 21 }
Diff Utilities
diffWords(textA, textB)
Compute a word-level diff between two strings. Returns DiffSegment[].
import{diffWords}from'ai-diff';constsegments=diffWords('hello world','hello earth');// [// { text: 'hello', type: 'unchanged' },// { text: ' ', type: 'unchanged' },// { text: 'world', type: 'removed' },// { text: 'earth', type: 'added' },// ]
diffLines(textA, textB)
Compute a line-level diff between two strings. Returns DiffSegment[].
import{diffLines}from'ai-diff';constsegments=diffLines('line1\nline2','line1\nline3');
diffJson(a, b)
Compute a structural diff between two parsed JSON values. Returns JsonChange[] with dot-notation paths.
import{diffJson}from'ai-diff';constchanges=diffJson({name: 'Alice',age: 30},{name: 'Alice',age: 31,role: 'admin'},);// [// { path: 'age', type: 'changed', before: 30, after: 31 },// { path: 'role', type: 'added', after: 'admin' },// ]
computeHunks(textA, textB, contextLines?)
Compute diff hunks (contiguous groups of changes with context lines) between two texts. Returns DiffHunk[].
import{computeHunks}from'ai-diff';consthunks=computeHunks('line1\nline2\nline3','line1\nchanged\nline3',3);
tryParseJson(text)
Attempt to parse a string as JSON. Returns the parsed value on success or null on failure.
Metrics Utilities
estimateCost(output, pricingOverrides?)
Estimate the cost of an LLM output in USD. Returns the output's cost field if set, otherwise computes from model pricing and token counts. Returns undefined if insufficient data.
import{estimateCost}from'ai-diff';estimateCost({text: 'hello',model: 'gpt-4o',tokens: {input: 100,output: 50}});// 0.00075 (computed from built-in GPT-4o pricing)estimateCost({text: 'hello',model: 'custom',tokens: {input: 1000,output: 500}},{custom: {input: 0.001,output: 0.002}},);// 2.0
getModelPricing()
Returns a copy of the built-in model pricing table. Each entry maps a model name to { input: number; output: number } (per-token USD).
import{getModelPricing}from'ai-diff';constpricing=getModelPricing();// {// 'gpt-4o': { input: 0.0000025, output: 0.00001 },// 'claude-sonnet': { input: 0.000003, output: 0.000015 },// ...// }
Built-in models:gpt-4o, gpt-4o-mini, gpt-3.5-turbo, gpt-4-turbo, claude-opus, claude-sonnet, claude-haiku, gemini-pro, gemini-flash.
computeMetrics(outputA, outputB, options?)
Compute full comparative metrics between two outputs. Supports an optional embedFn for semantic similarity.
Parameters:
ParameterTypeDescription
outputALLMOutputFirst output.
outputBLLMOutputSecond output.
options.embedFn(text: string) => Promise<number[]>Optional embedding function for semantic similarity.
options.pricingRecord<string, { input: number; output: number }>Optional pricing overrides.
Returns:Promise<DiffMetrics>
Formatter Utilities
renderUnifiedDiff(result, useColor?)
Render a DiffResult as a unified diff string with optional ANSI colors. Removed lines are prefixed with - (red), added lines with + (green). Word-level changes are highlighted with bold inverse.
renderSideBySide(result, useColor?, width?)
Render a DiffResult as a two-column side-by-side display. Column width defaults to (width - 3) / 2.
renderInlineDiff(segments, useColor?)
Render DiffSegment[] as inline text. Removed words appear with strikethrough (or ~~text~~ without color), added words with underline (or __text__).
renderJsonDiff(changes, originalA, originalB, useColor?)
Render JsonChange[] as a formatted string showing added, removed, and changed keys.
renderMetricsTable(metrics, labelA, labelB, useColor?)
Render a DiffMetrics object as a Unicode box-drawing table comparing all metrics between two outputs.
shouldUseColor(override?)
Returns true if ANSI colors should be used. Checks override, then NO_COLOR env var, then process.stdout.isTTY.
Normalization Utilities
normalizeOutput(input)
Convert a string | LLMOutput to an LLMOutput object. Plain strings become { text: string }.
enrichOutput(output)
Fill in estimated fields on an LLMOutput. Adds estimated tokens.output (via ceil(text.length / 4)) when not provided.
estimateTokens(text)
Estimate token count from text length: Math.ceil(text.length / 4).
tokenizeWords(text)
Split text into word and whitespace tokens (preserving whitespace). Used internally by the LCS diff algorithm.
Configuration
DiffOptions
interfaceDiffOptions{/** Diff mode. Default: 'unified'. */mode?: 'unified'|'side-by-side'|'inline'|'metrics'|'json';/** Context lines around changes in unified mode. Default: 3. */contextLines?: number;/** Embedding function for semantic similarity. */embedFn?: (text: string)=>Promise<number[]>;/** Per-token pricing overrides in USD. Keyed by model name. */pricing?: Record<string,{input: number;output: number}>;/** Show the metrics summary table. Default: true. */showMetrics?: boolean;/** Position of the metrics table. Default: 'top'. */metricsPosition?: 'top'|'bottom';/** Which metrics to display. Default: all available. */metrics?: ('tokens'|'cost'|'latency'|'similarity'|'length'|'model')[];/** Terminal width override for side-by-side mode. Default: auto-detected. */width?: number;/** ANSI color override. Default: auto-detected (true if TTY). */color?: boolean;/** Custom labels for outputs. Default: model names or 'Output A'/'Output B'. */labels?: string[];}
CompareOptions
Extends DiffOptions with:
interfaceCompareOptionsextendsDiffOptions{/** Max concurrent model calls. Default: unlimited (all in parallel). */concurrency?: number;/** Per-call timeout in milliseconds. Default: 30000. */timeout?: number;/** AbortSignal for cancellation. */signal?: AbortSignal;}
Diff Modes
ModeDescription
unifiedGit-style unified diff with word-level highlighting (default).
side-by-sideTwo-column display with aligned content and a vertical separator.
inlineInline additions (underline) and deletions (strikethrough) within the original text.
metricsMetrics comparison table only; no text diff output.
jsonStructural diff for JSON outputs with key-level change detection. Falls back to text diff if either output is not valid JSON.
Output Formats
FormatDescription
terminalANSI-colored output for terminal display.
jsonFull result serialized as JSON.stringify(result, null, 2).
markdownMarkdown-formatted diff.
plainPlain text with no ANSI codes.
Error Handling
Invalid embedding vectors
embeddingCosineSimilarity throws if the two vectors have different lengths:
import{embeddingCosineSimilarity}from'ai-diff';embeddingCosineSimilarity([1,2],[1,2,3]);// Error: Embedding vectors must have the same length: 2 vs 3
Unknown models
When a model is not in the built-in pricing table and no pricing override is provided, estimateCost returns undefined and the cost row is omitted from the metrics table. No error is thrown.
Failed model calls in compare()
When a model call fails or times out in compare(), the failure is captured in the calls array with status: 'error' and an error message. The failed output is excluded from pairwise diffs. Remaining models continue normally.
constresult=awaitcompare('prompt',['model-a','model-b'],llmFn);for(constcallofresult.calls){if(call.status==='error'){console.error(`${call.model} failed: ${call.error}`);}}
JSON diff fallback
When mode: 'json' is used but one or both outputs are not valid JSON, the engine falls back to a standard text diff. The jsonChanges field on the result will be undefined.
Advanced Usage
Semantic similarity with custom embeddings
Provide an embedFn to compute semantic similarity alongside Jaccard:
import{computeMetrics}from'ai-diff';constmetrics=awaitcomputeMetrics({text: 'The cat sat on the mat.',model: 'gpt-4o',tokens: {output: 8}},{text: 'A feline rested on a rug.',model: 'claude-sonnet',tokens: {output: 7}},{embedFn: async(text)=>{// Call your embedding API (OpenAI, Cohere, etc.)returnawaitgetEmbedding(text);},},);console.log(metrics.similarity.semantic);// 0.0 - 1.0
Custom model pricing
Override or extend the built-in pricing table:
import{diff}from'ai-diff';constresult=diff(outputA,outputB,{pricing: {'my-custom-model': {input: 0.001,output: 0.002},'gpt-4o': {input: 0.000003,output: 0.000012},// override built-in},});
Comparing structured JSON outputs
import{diff,formatDiff}from'ai-diff';constresult=diff({text: '{"name":"Alice","age":30}',model: 'gpt-4o'},{text: '{"name":"Alice","age":31,"role":"admin"}',model: 'claude-sonnet'},{mode: 'json'},);console.log(result.jsonChanges);// [// { path: 'age', type: 'changed', before: 30, after: 31 },// { path: 'role', type: 'added', after: 'admin' },// ]console.log(formatDiff(result,'terminal'));
Concurrency-limited model comparison
import{compare}from'ai-diff';constresult=awaitcompare('Summarize this article.',['gpt-4o','gpt-4o-mini','claude-sonnet','gemini-pro'],async(prompt,model)=>callLLM(prompt,model),{concurrency: 2,timeout: 10000},);// Only 2 models called at a time; each call times out after 10s
Metrics-only comparison
import{diff,formatDiff}from'ai-diff';constresult=diff(outputA,outputB,{mode: 'metrics'});console.log(formatDiff(result,'terminal'));// Prints only the metrics comparison table, no text diff
TypeScript
All types are exported from the package root:
importtype{LLMOutput,LLMFn,DiffMode,DiffOptions,CompareOptions,OutputFormat,DiffResult,MultiDiffResult,ComparisonResult,DiffSegment,DiffHunk,DiffMetrics,LengthStats,JsonChange,}from'ai-diff';
The package is compiled with TypeScript strict mode targeting ES2022 (CommonJS output). Declaration files and source maps are included.
License
MIT
About
Compare LLM responses across models with semantic diffs
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
ai-diff
Compare LLM outputs with word-level and line-level diffs, ANSI-colored terminal output, and AI-specific metrics.
npm versionnpm downloadslicensenodeTypeScript
ai-diff compares two or more LLM responses and produces structured diffs annotated with AI-specific metrics: token counts (input and output), estimated cost in USD (using built-in model pricing), response latency, Jaccard similarity scores, and length statistics (words, sentences, characters). It supports five diff modes -- unified, side-by-side, inline, metrics-only, and JSON structural diff -- and four output formats: terminal (ANSI-colored), JSON, Markdown, and plain text. Zero runtime dependencies.
Installation
npm install ai-diff
Quick Start
import{diff,formatDiff}from'ai-diff';constresult=diff({text: 'Paris is the capital of France.',model: 'gpt-4o',tokens: {input: 10,output: 8},latency: 1240},{text: 'The capital of France is Paris.',model: 'claude-sonnet',tokens: {input: 10,output: 8},latency: 980},);// Print a colored unified diff with metrics tableconsole.log(formatDiff(result,'terminal'));// Access structured dataconsole.log(result.identical);// falseconsole.log(result.similarity.jaccard);// 0.0 - 1.0console.log(result.metrics.latency?.delta);// -260
Features
  • Word-level and line-level diffs -- LCS-based algorithms implemented from scratch, no runtime dependencies.
  • Five diff modes -- unified (git-style), side-by-side (two-column), inline (strikethrough/underline), metrics (table only), json (structural key-level diff).
  • AI-specific metrics -- Token counts, estimated cost (USD), response latency, Jaccard similarity, word/sentence/character counts displayed in a comparison table alongside every diff.
  • Built-in model pricing -- GPT-4o, GPT-4o-mini, GPT-3.5 Turbo, GPT-4 Turbo, Claude Opus, Claude Sonnet, Claude Haiku, Gemini Pro, Gemini Flash. Override or extend with custom pricing.
  • N-way comparison -- diffOutputs() compares any number of outputs pairwise. compare() sends a prompt to multiple models via a user-provided function and diffs the results.
  • Four output formats -- terminal (ANSI colors), json (serialized result), markdown, plain.
  • Automatic token estimation -- When token counts are not provided, output tokens are estimated using a ceil(characters / 4) heuristic.
  • ANSI color detection -- Colors are enabled automatically when stdout is a TTY. Respects NO_COLOR environment variable. Override with the color option.
  • TypeScript-first -- Full type definitions exported for all interfaces, options, and result types. Strict mode enabled.
  • Zero runtime dependencies -- All diffing, similarity, formatting, and metrics logic uses only Node.js built-ins.
API Reference
diff(outputA, outputB, options?)
Compare two LLM outputs and return a DiffResult.
Parameters:
ParameterTypeDescription
outputAstring | LLMOutputFirst LLM output. Plain strings are wrapped as { text: string }.
outputBstring | LLMOutputSecond LLM output.
optionsDiffOptionsOptional configuration (see Configuration).
Returns:DiffResult
import{diff}from'ai-diff';// Compare plain stringsconstresult=diff('Output from model A','Output from model B');// Compare with full metadataconstresult=diff({text: 'Response A',model: 'gpt-4o',tokens: {input: 100,output: 50},cost: 0.005,latency: 1200},{text: 'Response B',model: 'claude-sonnet',tokens: {input: 100,output: 75},latency: 980},{mode: 'side-by-side'},);console.log(result.identical);// falseconsole.log(result.hunks.length);// number of diff hunksconsole.log(result.metrics.cost);// { a, b, delta, deltaPercent }console.log(result.similarity.jaccard);// 0.0 - 1.0
diffOutputs(outputs, options?)
Compare N LLM outputs pairwise.
Parameters:
ParameterTypeDescription
outputs(string | LLMOutput)[]Array of outputs to compare.
optionsDiffOptionsOptional configuration.
Returns:MultiDiffResult
import{diffOutputs}from'ai-diff';constresult=diffOutputs([{text: 'Output A',model: 'gpt-4o'},{text: 'Output B',model: 'claude-sonnet'},{text: 'Output C',model: 'gemini-pro'},]);console.log(result.pairwise.length);// 3 (A-B, A-C, B-C)console.log(result.metricsTable.labels);// ['gpt-4o', 'claude-sonnet', 'gemini-pro']console.log(result.metricsTable.wordCounts);// [n, n, n]
compare(prompt, models, llmFn, options?)
Send a prompt to multiple models via a user-provided function and compare the outputs.
Parameters:
ParameterTypeDescription
promptstringThe prompt to send to each model.
modelsstring[]Array of model identifiers.
llmFnLLMFnAsync function (prompt, model) => LLMOutput | string that calls the model.
optionsCompareOptionsOptional configuration (extends DiffOptions with concurrency, timeout, signal).
Returns:Promise<ComparisonResult>
import{compare}from'ai-diff';constresult=awaitcompare('Explain quantum computing in 3 sentences.',['gpt-4o','claude-sonnet'],async(prompt,model)=>{constresponse=awaitcallMyLLM(prompt,model);return{text: response.text,tokens: response.usage, model };},{concurrency: 2,timeout: 15000},);console.log(result.calls);// per-model status, output, latency, or errorconsole.log(result.pairwise);// pairwise diffs of successful outputs
formatDiff(result, format?)
Format a diff result into a displayable string.
Parameters:
ParameterTypeDefaultDescription
resultDiffResult | MultiDiffResult | ComparisonResult--The result to format.
formatOutputFormat'terminal'One of 'terminal', 'json', 'markdown', 'plain'.
Returns:string
import{diff,formatDiff}from'ai-diff';constresult=diff('hello world','hello earth');console.log(formatDiff(result,'terminal'));// ANSI-colored unified diffconsole.log(formatDiff(result,'json'));// JSON.stringify(result, null, 2)console.log(formatDiff(result,'plain'));// plain text, no ANSI codes
Similarity Functions
jaccardSimilarity(textA, textB)
Compute Jaccard similarity (word-level set overlap) between two texts. Returns a value between 0.0 and 1.0.
import{jaccardSimilarity}from'ai-diff';jaccardSimilarity('hello world','hello earth');// 0.333...jaccardSimilarity('hello world','hello world');// 1.0jaccardSimilarity('','');// 1.0
cosineSimilarity(textA, textB)
Compute cosine similarity using word-frequency vectors. Returns a value between 0.0 and 1.0.
import{cosineSimilarity}from'ai-diff';cosineSimilarity('the quick brown fox','the slow brown cat');// 0.0 - 1.0
exactMatchRatio(textA, textB)
Returns 1.0 if the two texts are identical, 0.0 otherwise.
import{exactMatchRatio}from'ai-diff';exactMatchRatio('hello','hello');// 1.0exactMatchRatio('hello','world');// 0.0
compositeSimilarity(textA, textB)
Weighted composite: Jaccard (0.5) + Cosine (0.3) + Exact Match (0.2).
import{compositeSimilarity}from'ai-diff';compositeSimilarity('hello world','hello earth');// 0.0 - 1.0
embeddingCosineSimilarity(a, b)
Compute cosine similarity between two numeric embedding vectors. Throws if vectors have different lengths.
import{embeddingCosineSimilarity}from'ai-diff';embeddingCosineSimilarity([1,0,0],[0,1,0]);// 0.0embeddingCosineSimilarity([1,0],[1,0]);// 1.0
computeLengthStats(text)
Returns { words, sentences, characters } for a given text.
import{computeLengthStats}from'ai-diff';computeLengthStats('Hello world. Goodbye.');// { words: 3, sentences: 2, characters: 21 }
Diff Utilities
diffWords(textA, textB)
Compute a word-level diff between two strings. Returns DiffSegment[].
import{diffWords}from'ai-diff';constsegments=diffWords('hello world','hello earth');// [// { text: 'hello', type: 'unchanged' },// { text: ' ', type: 'unchanged' },// { text: 'world', type: 'removed' },// { text: 'earth', type: 'added' },// ]
diffLines(textA, textB)
Compute a line-level diff between two strings. Returns DiffSegment[].
import{diffLines}from'ai-diff';constsegments=diffLines('line1\nline2','line1\nline3');
diffJson(a, b)
Compute a structural diff between two parsed JSON values. Returns JsonChange[] with dot-notation paths.
import{diffJson}from'ai-diff';constchanges=diffJson({name: 'Alice',age: 30},{name: 'Alice',age: 31,role: 'admin'},);// [// { path: 'age', type: 'changed', before: 30, after: 31 },// { path: 'role', type: 'added', after: 'admin' },// ]
computeHunks(textA, textB, contextLines?)
Compute diff hunks (contiguous groups of changes with context lines) between two texts. Returns DiffHunk[].
import{computeHunks}from'ai-diff';consthunks=computeHunks('line1\nline2\nline3','line1\nchanged\nline3',3);
tryParseJson(text)
Attempt to parse a string as JSON. Returns the parsed value on success or null on failure.
Metrics Utilities
estimateCost(output, pricingOverrides?)
Estimate the cost of an LLM output in USD. Returns the output's cost field if set, otherwise computes from model pricing and token counts. Returns undefined if insufficient data.
import{estimateCost}from'ai-diff';estimateCost({text: 'hello',model: 'gpt-4o',tokens: {input: 100,output: 50}});// 0.00075 (computed from built-in GPT-4o pricing)estimateCost({text: 'hello',model: 'custom',tokens: {input: 1000,output: 500}},{custom: {input: 0.001,output: 0.002}},);// 2.0
getModelPricing()
Returns a copy of the built-in model pricing table. Each entry maps a model name to { input: number; output: number } (per-token USD).
import{getModelPricing}from'ai-diff';constpricing=getModelPricing();// {// 'gpt-4o': { input: 0.0000025, output: 0.00001 },// 'claude-sonnet': { input: 0.000003, output: 0.000015 },// ...// }
Built-in models:gpt-4o, gpt-4o-mini, gpt-3.5-turbo, gpt-4-turbo, claude-opus, claude-sonnet, claude-haiku, gemini-pro, gemini-flash.
computeMetrics(outputA, outputB, options?)
Compute full comparative metrics between two outputs. Supports an optional embedFn for semantic similarity.
Parameters:
ParameterTypeDescription
outputALLMOutputFirst output.
outputBLLMOutputSecond output.
options.embedFn(text: string) => Promise<number[]>Optional embedding function for semantic similarity.
options.pricingRecord<string, { input: number; output: number }>Optional pricing overrides.
Returns:Promise<DiffMetrics>
Formatter Utilities
renderUnifiedDiff(result, useColor?)
Render a DiffResult as a unified diff string with optional ANSI colors. Removed lines are prefixed with - (red), added lines with + (green). Word-level changes are highlighted with bold inverse.
renderSideBySide(result, useColor?, width?)
Render a DiffResult as a two-column side-by-side display. Column width defaults to (width - 3) / 2.
renderInlineDiff(segments, useColor?)
Render DiffSegment[] as inline text. Removed words appear with strikethrough (or ~~text~~ without color), added words with underline (or __text__).
renderJsonDiff(changes, originalA, originalB, useColor?)
Render JsonChange[] as a formatted string showing added, removed, and changed keys.
renderMetricsTable(metrics, labelA, labelB, useColor?)
Render a DiffMetrics object as a Unicode box-drawing table comparing all metrics between two outputs.
shouldUseColor(override?)
Returns true if ANSI colors should be used. Checks override, then NO_COLOR env var, then process.stdout.isTTY.
Normalization Utilities
normalizeOutput(input)
Convert a string | LLMOutput to an LLMOutput object. Plain strings become { text: string }.
enrichOutput(output)
Fill in estimated fields on an LLMOutput. Adds estimated tokens.output (via ceil(text.length / 4)) when not provided.
estimateTokens(text)
Estimate token count from text length: Math.ceil(text.length / 4).
tokenizeWords(text)
Split text into word and whitespace tokens (preserving whitespace). Used internally by the LCS diff algorithm.
Configuration
DiffOptions
interfaceDiffOptions{/** Diff mode. Default: 'unified'. */mode?: 'unified'|'side-by-side'|'inline'|'metrics'|'json';/** Context lines around changes in unified mode. Default: 3. */contextLines?: number;/** Embedding function for semantic similarity. */embedFn?: (text: string)=>Promise<number[]>;/** Per-token pricing overrides in USD. Keyed by model name. */pricing?: Record<string,{input: number;output: number}>;/** Show the metrics summary table. Default: true. */showMetrics?: boolean;/** Position of the metrics table. Default: 'top'. */metricsPosition?: 'top'|'bottom';/** Which metrics to display. Default: all available. */metrics?: ('tokens'|'cost'|'latency'|'similarity'|'length'|'model')[];/** Terminal width override for side-by-side mode. Default: auto-detected. */width?: number;/** ANSI color override. Default: auto-detected (true if TTY). */color?: boolean;/** Custom labels for outputs. Default: model names or 'Output A'/'Output B'. */labels?: string[];}
CompareOptions
Extends DiffOptions with:
interfaceCompareOptionsextendsDiffOptions{/** Max concurrent model calls. Default: unlimited (all in parallel). */concurrency?: number;/** Per-call timeout in milliseconds. Default: 30000. */timeout?: number;/** AbortSignal for cancellation. */signal?: AbortSignal;}
Diff Modes
ModeDescription
unifiedGit-style unified diff with word-level highlighting (default).
side-by-sideTwo-column display with aligned content and a vertical separator.
inlineInline additions (underline) and deletions (strikethrough) within the original text.
metricsMetrics comparison table only; no text diff output.
jsonStructural diff for JSON outputs with key-level change detection. Falls back to text diff if either output is not valid JSON.
Output Formats
FormatDescription
terminalANSI-colored output for terminal display.
jsonFull result serialized as JSON.stringify(result, null, 2).
markdownMarkdown-formatted diff.
plainPlain text with no ANSI codes.
Error Handling
Invalid embedding vectors
embeddingCosineSimilarity throws if the two vectors have different lengths:
import{embeddingCosineSimilarity}from'ai-diff';embeddingCosineSimilarity([1,2],[1,2,3]);// Error: Embedding vectors must have the same length: 2 vs 3
Unknown models
When a model is not in the built-in pricing table and no pricing override is provided, estimateCost returns undefined and the cost row is omitted from the metrics table. No error is thrown.
Failed model calls in compare()
When a model call fails or times out in compare(), the failure is captured in the calls array with status: 'error' and an error message. The failed output is excluded from pairwise diffs. Remaining models continue normally.
constresult=awaitcompare('prompt',['model-a','model-b'],llmFn);for(constcallofresult.calls){if(call.status==='error'){console.error(`${call.model} failed: ${call.error}`);}}
JSON diff fallback
When mode: 'json' is used but one or both outputs are not valid JSON, the engine falls back to a standard text diff. The jsonChanges field on the result will be undefined.
Advanced Usage
Semantic similarity with custom embeddings
Provide an embedFn to compute semantic similarity alongside Jaccard:
import{computeMetrics}from'ai-diff';constmetrics=awaitcomputeMetrics({text: 'The cat sat on the mat.',model: 'gpt-4o',tokens: {output: 8}},{text: 'A feline rested on a rug.',model: 'claude-sonnet',tokens: {output: 7}},{embedFn: async(text)=>{// Call your embedding API (OpenAI, Cohere, etc.)returnawaitgetEmbedding(text);},},);console.log(metrics.similarity.semantic);// 0.0 - 1.0
Custom model pricing
Override or extend the built-in pricing table:
import{diff}from'ai-diff';constresult=diff(outputA,outputB,{pricing: {'my-custom-model': {input: 0.001,output: 0.002},'gpt-4o': {input: 0.000003,output: 0.000012},// override built-in},});
Comparing structured JSON outputs
import{diff,formatDiff}from'ai-diff';constresult=diff({text: '{"name":"Alice","age":30}',model: 'gpt-4o'},{text: '{"name":"Alice","age":31,"role":"admin"}',model: 'claude-sonnet'},{mode: 'json'},);console.log(result.jsonChanges);// [// { path: 'age', type: 'changed', before: 30, after: 31 },// { path: 'role', type: 'added', after: 'admin' },// ]console.log(formatDiff(result,'terminal'));
Concurrency-limited model comparison
import{compare}from'ai-diff';constresult=awaitcompare('Summarize this article.',['gpt-4o','gpt-4o-mini','claude-sonnet','gemini-pro'],async(prompt,model)=>callLLM(prompt,model),{concurrency: 2,timeout: 10000},);// Only 2 models called at a time; each call times out after 10s
Metrics-only comparison
import{diff,formatDiff}from'ai-diff';constresult=diff(outputA,outputB,{mode: 'metrics'});console.log(formatDiff(result,'terminal'));// Prints only the metrics comparison table, no text diff
TypeScript
All types are exported from the package root:
importtype{LLMOutput,LLMFn,DiffMode,DiffOptions,CompareOptions,OutputFormat,DiffResult,MultiDiffResult,ComparisonResult,DiffSegment,DiffHunk,DiffMetrics,LengthStats,JsonChange,}from'ai-diff';
The package is compiled with TypeScript strict mode targeting ES2022 (CommonJS output). Declaration files and source maps are included.
License
MIT
About
Compare LLM responses across models with semantic diffs
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
BranchesTags
Repository files navigation
ai-diff
Compare LLM outputs with word-level and line-level diffs, ANSI-colored terminal output, and AI-specific metrics.
npm versionnpm downloadslicensenodeTypeScript
ai-diff compares two or more LLM responses and produces structured diffs annotated with AI-specific metrics: token counts (input and output), estimated cost in USD (using built-in model pricing), response latency, Jaccard similarity scores, and length statistics (words, sentences, characters). It supports five diff modes -- unified, side-by-side, inline, metrics-only, and JSON structural diff -- and four output formats: terminal (ANSI-colored), JSON, Markdown, and plain text. Zero runtime dependencies.
Installation
npm install ai-diff
Quick Start
import{diff,formatDiff}from'ai-diff';constresult=diff({text: 'Paris is the capital of France.',model: 'gpt-4o',tokens: {input: 10,output: 8},latency: 1240},{text: 'The capital of France is Paris.',model: 'claude-sonnet',tokens: {input: 10,output: 8},latency: 980},);// Print a colored unified diff with metrics tableconsole.log(formatDiff(result,'terminal'));// Access structured dataconsole.log(result.identical);// falseconsole.log(result.similarity.jaccard);// 0.0 - 1.0console.log(result.metrics.latency?.delta);// -260
Features
  • Word-level and line-level diffs -- LCS-based algorithms implemented from scratch, no runtime dependencies.
  • Five diff modes -- unified (git-style), side-by-side (two-column), inline (strikethrough/underline), metrics (table only), json (structural key-level diff).
  • AI-specific metrics -- Token counts, estimated cost (USD), response latency, Jaccard similarity, word/sentence/character counts displayed in a comparison table alongside every diff.
  • Built-in model pricing -- GPT-4o, GPT-4o-mini, GPT-3.5 Turbo, GPT-4 Turbo, Claude Opus, Claude Sonnet, Claude Haiku, Gemini Pro, Gemini Flash. Override or extend with custom pricing.
  • N-way comparison -- diffOutputs() compares any number of outputs pairwise. compare() sends a prompt to multiple models via a user-provided function and diffs the results.
  • Four output formats -- terminal (ANSI colors), json (serialized result), markdown, plain.
  • Automatic token estimation -- When token counts are not provided, output tokens are estimated using a ceil(characters / 4) heuristic.
  • ANSI color detection -- Colors are enabled automatically when stdout is a TTY. Respects NO_COLOR environment variable. Override with the color option.
  • TypeScript-first -- Full type definitions exported for all interfaces, options, and result types. Strict mode enabled.
  • Zero runtime dependencies -- All diffing, similarity, formatting, and metrics logic uses only Node.js built-ins.
API Reference
diff(outputA, outputB, options?)
Compare two LLM outputs and return a DiffResult.
Parameters:
ParameterTypeDescription
outputAstring | LLMOutputFirst LLM output. Plain strings are wrapped as { text: string }.
outputBstring | LLMOutputSecond LLM output.
optionsDiffOptionsOptional configuration (see Configuration).
Returns:DiffResult
import{diff}from'ai-diff';// Compare plain stringsconstresult=diff('Output from model A','Output from model B');// Compare with full metadataconstresult=diff({text: 'Response A',model: 'gpt-4o',tokens: {input: 100,output: 50},cost: 0.005,latency: 1200},{text: 'Response B',model: 'claude-sonnet',tokens: {input: 100,output: 75},latency: 980},{mode: 'side-by-side'},);console.log(result.identical);// falseconsole.log(result.hunks.length);// number of diff hunksconsole.log(result.metrics.cost);// { a, b, delta, deltaPercent }console.log(result.similarity.jaccard);// 0.0 - 1.0
diffOutputs(outputs, options?)
Compare N LLM outputs pairwise.
Parameters:
ParameterTypeDescription
outputs(string | LLMOutput)[]Array of outputs to compare.
optionsDiffOptionsOptional configuration.
Returns:MultiDiffResult
import{diffOutputs}from'ai-diff';constresult=diffOutputs([{text: 'Output A',model: 'gpt-4o'},{text: 'Output B',model: 'claude-sonnet'},{text: 'Output C',model: 'gemini-pro'},]);console.log(result.pairwise.length);// 3 (A-B, A-C, B-C)console.log(result.metricsTable.labels);// ['gpt-4o', 'claude-sonnet', 'gemini-pro']console.log(result.metricsTable.wordCounts);// [n, n, n]
compare(prompt, models, llmFn, options?)
Send a prompt to multiple models via a user-provided function and compare the outputs.
Parameters:
ParameterTypeDescription
promptstringThe prompt to send to each model.
modelsstring[]Array of model identifiers.
llmFnLLMFnAsync function (prompt, model) => LLMOutput | string that calls the model.
optionsCompareOptionsOptional configuration (extends DiffOptions with concurrency, timeout, signal).
Returns:Promise<ComparisonResult>
import{compare}from'ai-diff';constresult=awaitcompare('Explain quantum computing in 3 sentences.',['gpt-4o','claude-sonnet'],async(prompt,model)=>{constresponse=awaitcallMyLLM(prompt,model);return{text: response.text,tokens: response.usage, model };},{concurrency: 2,timeout: 15000},);console.log(result.calls);// per-model status, output, latency, or errorconsole.log(result.pairwise);// pairwise diffs of successful outputs
formatDiff(result, format?)
Format a diff result into a displayable string.
Parameters:
ParameterTypeDefaultDescription
resultDiffResult | MultiDiffResult | ComparisonResult--The result to format.
formatOutputFormat'terminal'One of 'terminal', 'json', 'markdown', 'plain'.
Returns:string
import{diff,formatDiff}from'ai-diff';constresult=diff('hello world','hello earth');console.log(formatDiff(result,'terminal'));// ANSI-colored unified diffconsole.log(formatDiff(result,'json'));// JSON.stringify(result, null, 2)console.log(formatDiff(result,'plain'));// plain text, no ANSI codes
Similarity Functions
jaccardSimilarity(textA, textB)
Compute Jaccard similarity (word-level set overlap) between two texts. Returns a value between 0.0 and 1.0.
import{jaccardSimilarity}from'ai-diff';jaccardSimilarity('hello world','hello earth');// 0.333...jaccardSimilarity('hello world','hello world');// 1.0jaccardSimilarity('','');// 1.0
cosineSimilarity(textA, textB)
Compute cosine similarity using word-frequency vectors. Returns a value between 0.0 and 1.0.
import{cosineSimilarity}from'ai-diff';cosineSimilarity('the quick brown fox','the slow brown cat');// 0.0 - 1.0
exactMatchRatio(textA, textB)
Returns 1.0 if the two texts are identical, 0.0 otherwise.
import{exactMatchRatio}from'ai-diff';exactMatchRatio('hello','hello');// 1.0exactMatchRatio('hello','world');// 0.0
compositeSimilarity(textA, textB)
Weighted composite: Jaccard (0.5) + Cosine (0.3) + Exact Match (0.2).
import{compositeSimilarity}from'ai-diff';compositeSimilarity('hello world','hello earth');// 0.0 - 1.0
embeddingCosineSimilarity(a, b)
Compute cosine similarity between two numeric embedding vectors. Throws if vectors have different lengths.
import{embeddingCosineSimilarity}from'ai-diff';embeddingCosineSimilarity([1,0,0],[0,1,0]);// 0.0embeddingCosineSimilarity([1,0],[1,0]);// 1.0
computeLengthStats(text)
Returns { words, sentences, characters } for a given text.
import{computeLengthStats}from'ai-diff';computeLengthStats('Hello world. Goodbye.');// { words: 3, sentences: 2, characters: 21 }
Diff Utilities
diffWords(textA, textB)
Compute a word-level diff between two strings. Returns DiffSegment[].
import{diffWords}from'ai-diff';constsegments=diffWords('hello world','hello earth');// [// { text: 'hello', type: 'unchanged' },// { text: ' ', type: 'unchanged' },// { text: 'world', type: 'removed' },// { text: 'earth', type: 'added' },// ]
diffLines(textA, textB)
Compute a line-level diff between two strings. Returns DiffSegment[].
import{diffLines}from'ai-diff';constsegments=diffLines('line1\nline2','line1\nline3');
diffJson(a, b)
Compute a structural diff between two parsed JSON values. Returns JsonChange[] with dot-notation paths.
import{diffJson}from'ai-diff';constchanges=diffJson({name: 'Alice',age: 30},{name: 'Alice',age: 31,role: 'admin'},);// [// { path: 'age', type: 'changed', before: 30, after: 31 },// { path: 'role', type: 'added', after: 'admin' },// ]
computeHunks(textA, textB, contextLines?)
Compute diff hunks (contiguous groups of changes with context lines) between two texts. Returns DiffHunk[].
import{computeHunks}from'ai-diff';consthunks=computeHunks('line1\nline2\nline3','line1\nchanged\nline3',3);
tryParseJson(text)
Attempt to parse a string as JSON. Returns the parsed value on success or null on failure.
Metrics Utilities
estimateCost(output, pricingOverrides?)
Estimate the cost of an LLM output in USD. Returns the output's cost field if set, otherwise computes from model pricing and token counts. Returns undefined if insufficient data.
import{estimateCost}from'ai-diff';estimateCost({text: 'hello',model: 'gpt-4o',tokens: {input: 100,output: 50}});// 0.00075 (computed from built-in GPT-4o pricing)estimateCost({text: 'hello',model: 'custom',tokens: {input: 1000,output: 500}},{custom: {input: 0.001,output: 0.002}},);// 2.0
getModelPricing()
Returns a copy of the built-in model pricing table. Each entry maps a model name to { input: number; output: number } (per-token USD).
import{getModelPricing}from'ai-diff';constpricing=getModelPricing();// {// 'gpt-4o': { input: 0.0000025, output: 0.00001 },// 'claude-sonnet': { input: 0.000003, output: 0.000015 },// ...// }
Built-in models:gpt-4o, gpt-4o-mini, gpt-3.5-turbo, gpt-4-turbo, claude-opus, claude-sonnet, claude-haiku, gemini-pro, gemini-flash.
computeMetrics(outputA, outputB, options?)
Compute full comparative metrics between two outputs. Supports an optional embedFn for semantic similarity.
Parameters:
ParameterTypeDescription
outputALLMOutputFirst output.
outputBLLMOutputSecond output.
options.embedFn(text: string) => Promise<number[]>Optional embedding function for semantic similarity.
options.pricingRecord<string, { input: number; output: number }>Optional pricing overrides.
Returns:Promise<DiffMetrics>
Formatter Utilities
renderUnifiedDiff(result, useColor?)
Render a DiffResult as a unified diff string with optional ANSI colors. Removed lines are prefixed with - (red), added lines with + (green). Word-level changes are highlighted with bold inverse.
renderSideBySide(result, useColor?, width?)
Render a DiffResult as a two-column side-by-side display. Column width defaults to (width - 3) / 2.
renderInlineDiff(segments, useColor?)
Render DiffSegment[] as inline text. Removed words appear with strikethrough (or ~~text~~ without color), added words with underline (or __text__).
renderJsonDiff(changes, originalA, originalB, useColor?)
Render JsonChange[] as a formatted string showing added, removed, and changed keys.
renderMetricsTable(metrics, labelA, labelB, useColor?)
Render a DiffMetrics object as a Unicode box-drawing table comparing all metrics between two outputs.
shouldUseColor(override?)
Returns true if ANSI colors should be used. Checks override, then NO_COLOR env var, then process.stdout.isTTY.
Normalization Utilities
normalizeOutput(input)
Convert a string | LLMOutput to an LLMOutput object. Plain strings become { text: string }.
enrichOutput(output)
Fill in estimated fields on an LLMOutput. Adds estimated tokens.output (via ceil(text.length / 4)) when not provided.
estimateTokens(text)
Estimate token count from text length: Math.ceil(text.length / 4).
tokenizeWords(text)
Split text into word and whitespace tokens (preserving whitespace). Used internally by the LCS diff algorithm.
Configuration
DiffOptions
interfaceDiffOptions{/** Diff mode. Default: 'unified'. */mode?: 'unified'|'side-by-side'|'inline'|'metrics'|'json';/** Context lines around changes in unified mode. Default: 3. */contextLines?: number;/** Embedding function for semantic similarity. */embedFn?: (text: string)=>Promise<number[]>;/** Per-token pricing overrides in USD. Keyed by model name. */pricing?: Record<string,{input: number;output: number}>;/** Show the metrics summary table. Default: true. */showMetrics?: boolean;/** Position of the metrics table. Default: 'top'. */metricsPosition?: 'top'|'bottom';/** Which metrics to display. Default: all available. */metrics?: ('tokens'|'cost'|'latency'|'similarity'|'length'|'model')[];/** Terminal width override for side-by-side mode. Default: auto-detected. */width?: number;/** ANSI color override. Default: auto-detected (true if TTY). */color?: boolean;/** Custom labels for outputs. Default: model names or 'Output A'/'Output B'. */labels?: string[];}
CompareOptions
Extends DiffOptions with:
interfaceCompareOptionsextendsDiffOptions{/** Max concurrent model calls. Default: unlimited (all in parallel). */concurrency?: number;/** Per-call timeout in milliseconds. Default: 30000. */timeout?: number;/** AbortSignal for cancellation. */signal?: AbortSignal;}
Diff Modes
ModeDescription
unifiedGit-style unified diff with word-level highlighting (default).
side-by-sideTwo-column display with aligned content and a vertical separator.
inlineInline additions (underline) and deletions (strikethrough) within the original text.
metricsMetrics comparison table only; no text diff output.
jsonStructural diff for JSON outputs with key-level change detection. Falls back to text diff if either output is not valid JSON.
Output Formats
FormatDescription
terminalANSI-colored output for terminal display.
jsonFull result serialized as JSON.stringify(result, null, 2).
markdownMarkdown-formatted diff.
plainPlain text with no ANSI codes.
Error Handling
Invalid embedding vectors
embeddingCosineSimilarity throws if the two vectors have different lengths:
import{embeddingCosineSimilarity}from'ai-diff';embeddingCosineSimilarity([1,2],[1,2,3]);// Error: Embedding vectors must have the same length: 2 vs 3
Unknown models
When a model is not in the built-in pricing table and no pricing override is provided, estimateCost returns undefined and the cost row is omitted from the metrics table. No error is thrown.
Failed model calls in compare()
When a model call fails or times out in compare(), the failure is captured in the calls array with status: 'error' and an error message. The failed output is excluded from pairwise diffs. Remaining models continue normally.
constresult=awaitcompare('prompt',['model-a','model-b'],llmFn);for(constcallofresult.calls){if(call.status==='error'){console.error(`${call.model} failed: ${call.error}`);}}
JSON diff fallback
When mode: 'json' is used but one or both outputs are not valid JSON, the engine falls back to a standard text diff. The jsonChanges field on the result will be undefined.
Advanced Usage
Semantic similarity with custom embeddings
Provide an embedFn to compute semantic similarity alongside Jaccard:
import{computeMetrics}from'ai-diff';constmetrics=awaitcomputeMetrics({text: 'The cat sat on the mat.',model: 'gpt-4o',tokens: {output: 8}},{text: 'A feline rested on a rug.',model: 'claude-sonnet',tokens: {output: 7}},{embedFn: async(text)=>{// Call your embedding API (OpenAI, Cohere, etc.)returnawaitgetEmbedding(text);},},);console.log(metrics.similarity.semantic);// 0.0 - 1.0
Custom model pricing
Override or extend the built-in pricing table:
import{diff}from'ai-diff';constresult=diff(outputA,outputB,{pricing: {'my-custom-model': {input: 0.001,output: 0.002},'gpt-4o': {input: 0.000003,output: 0.000012},// override built-in},});
Comparing structured JSON outputs
import{diff,formatDiff}from'ai-diff';constresult=diff({text: '{"name":"Alice","age":30}',model: 'gpt-4o'},{text: '{"name":"Alice","age":31,"role":"admin"}',model: 'claude-sonnet'},{mode: 'json'},);console.log(result.jsonChanges);// [// { path: 'age', type: 'changed', before: 30, after: 31 },// { path: 'role', type: 'added', after: 'admin' },// ]console.log(formatDiff(result,'terminal'));
Concurrency-limited model comparison
import{compare}from'ai-diff';constresult=awaitcompare('Summarize this article.',['gpt-4o','gpt-4o-mini','claude-sonnet','gemini-pro'],async(prompt,model)=>callLLM(prompt,model),{concurrency: 2,timeout: 10000},);// Only 2 models called at a time; each call times out after 10s
Metrics-only comparison
import{diff,formatDiff}from'ai-diff';constresult=diff(outputA,outputB,{mode: 'metrics'});console.log(formatDiff(result,'terminal'));// Prints only the metrics comparison table, no text diff
TypeScript
All types are exported from the package root:
importtype{LLMOutput,LLMFn,DiffMode,DiffOptions,CompareOptions,OutputFormat,DiffResult,MultiDiffResult,ComparisonResult,DiffSegment,DiffHunk,DiffMetrics,LengthStats,JsonChange,}from'ai-diff';
The package is compiled with TypeScript strict mode targeting ES2022 (CommonJS output). Declaration files and source maps are included.
License
MIT
About
Compare LLM responses across models with semantic diffs
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages