Skip to content
BranchesTags

Repository files navigation

eval-dataset

Version-controlled eval dataset manager for LLM testing.

npm versionnpm downloadslicensenode

eval-dataset manages the lifecycle of evaluation datasets for LLM testing. It loads, validates, splits, samples, deduplicates, and exports collections of test cases across formats (JSON, JSONL, CSV). All transformation methods return new immutable Dataset instances, all randomization is seeded for reproducibility, and the entire API is fully typed in TypeScript.

Every LLM evaluation framework expects test data -- inputs, expected outputs, context documents, and metadata -- but none of them manage the dataset itself. eval-dataset fills this gap by providing a single package that handles loading from multiple formats, splitting with reproducible seeded randomness, sampling with stratification, deduplicating with configurable similarity, validating schema completeness, and computing statistics. Zero external runtime dependencies.

Installation

npm install eval-dataset

Requires Node.js 18 or later.

Quick Start

import{createDataset,loadDataset}from'eval-dataset';// Create a dataset from test casesconstds=createDataset({name: 'qa-eval',version: '1.0.0',cases: [{id: '1',input: 'What is 2+2?',expected: '4',category: 'math',tags: ['arithmetic']},{id: '2',input: 'Capital of France?',expected: 'Paris',category: 'geography'},{id: '3',input: 'Who wrote Hamlet?',expected: 'Shakespeare',category: 'literature'},],});console.log(ds.size);// 3console.log(ds.categories());// ['math', 'geography', 'literature']// Split into train/test setsconstsplits=ds.split({ratios: {train: 0.7,test: 0.3},seed: 42});console.log(splits.train.size);// 2console.log(splits.test.size);// 1// Export to JSON Linesconstjsonl=ds.export('jsonl');// Load from a JSON stringconstds2=awaitloadDataset('[{"id":"1","input":"hello","expected":"world"}]',{name: 'loaded',format: 'json',});

Features

  • Immutable Dataset objects -- Every transformation method (filter, map, add, remove, split, sample, dedup) returns a new Dataset. The original is never modified.
  • Seeded randomization -- Splitting, sampling, and shuffling use a Mulberry32 PRNG with configurable seeds. The same seed always produces the same result.
  • Multi-format loading -- Load test cases from JSON arrays, JSON Lines, CSV strings, or in-memory TestCase[] arrays. Format auto-detection inspects content structure when not explicitly specified.
  • Multi-format export -- Export datasets to JSON (pretty or compact), JSON Lines, or CSV with configurable column order.
  • Splitting -- Random and stratified splitting into named partitions with configurable ratios. Stratified splits maintain proportional category representation in each partition.
  • Sampling -- Random and stratified sampling with configurable sample size. Supports sampling with replacement.
  • Deduplication -- Exact match, normalized match (case-insensitive, whitespace-collapsed), and near-duplicate detection via Jaccard token similarity.
  • Validation -- Detects empty inputs, duplicate IDs, and empty datasets.
  • Statistics -- Computes case counts, expected output coverage, context coverage, category and tag distributions, and input length statistics (min, max, mean).
  • Zero runtime dependencies -- Built entirely on Node.js built-ins. Only development dependencies are used for building and testing.
  • Full TypeScript support -- All public types, interfaces, and function signatures are exported with declaration files.

API Reference

createDataset(options)

Creates a new Dataset from the provided options.

functioncreateDataset(options: CreateOptions): Dataset;

Parameters:

ParameterTypeRequiredDefaultDescription
options.namestringYes--Name of the dataset
options.versionstringNo'0.1.0'Semver version string
options.casesTestCase[]No[]Initial test cases

Returns: A Dataset instance.

constds=createDataset({name: 'my-eval',version: '1.0.0',cases: [{id: '1',input: 'What is 2+2?',expected: '4',category: 'math'},],});

loadDataset(source, options?)

Loads a dataset from a string (JSON, JSONL, or CSV content) or an in-memory TestCase[] array. Returns a Promise<Dataset>.

functionloadDataset(source: string|TestCase[],options?: LoadOptions): Promise<Dataset>;

Parameters:

ParameterTypeRequiredDefaultDescription
sourcestring | TestCase[]Yes--Content string or array of test cases
options.format'json' | 'jsonl' | 'csv' | 'auto'No'auto'Format of the source string. Ignored when source is an array.
options.namestringNo'dataset'Dataset name
options.versionstringNo'0.1.0'Dataset version

When format is 'auto', the loader inspects the content to determine the format:

  • Strings starting with [ or { are parsed as JSON.
  • Strings where every non-empty line is a JSON object are parsed as JSONL.
  • All other strings are parsed as CSV.
// Load from JSON stringconstds=awaitloadDataset('[{"id":"1","input":"hello"}]',{name: 'test'});// Load from JSONL stringconstds2=awaitloadDataset('{"id":"1","input":"hello"}\n{"id":"2","input":"world"}',{name: 'test',format: 'jsonl'},);// Load from CSV stringconstds3=awaitloadDataset('id,input,expected,category\n1,Hello,World,test\n2,Foo,Bar,test',{name: 'test',format: 'csv'},);// Load from in-memory arrayconstds4=awaitloadDataset([{id: '1',input: 'hello',expected: 'world'}],{name: 'test'},);

Field mapping during loading:

When loading from JSON, JSONL, or CSV, the loader maps common field names to the internal TestCase schema:

  • input or question maps to input
  • expected maps to expected
  • category maps to category
  • difficulty is parsed as a number
  • context is parsed as a string array
  • tags is parsed as a string array (pipe-delimited | in CSV)
  • metadata is parsed as a JSON object

Test cases without an id are assigned an auto-generated 8-character UUID.

Dataset Interface

The Dataset interface represents a named, versioned, immutable collection of test cases. All transformation methods return new Dataset instances.

Properties:

PropertyTypeDescription
namestring (readonly)Dataset name
versionstring (readonly)Semver version string
casesreadonly TestCase[] (readonly)Frozen array of test cases
sizenumber (readonly)Number of test cases

dataset.filter(fn)

Returns a new Dataset containing only test cases for which the predicate returns true.

filter(fn: (tc: TestCase)=>boolean): Dataset;
constmathOnly=ds.filter((tc)=>tc.category==='math');constwithExpected=ds.filter((tc)=>tc.expected!==undefined);

dataset.map(fn)

Returns a new Dataset with each test case transformed by the provided function.

map(fn: (tc: TestCase)=>TestCase): Dataset;
constuppercased=ds.map((tc)=>({ ...tc,input: tc.input.toUpperCase()}));

dataset.add(tc)

Returns a new Dataset with the test case appended. If id is not provided, one is auto-generated. If input is not provided, it defaults to an empty string.

add(tc: Partial<TestCase>): Dataset;
constds2=ds.add({input: 'New question?',expected: 'New answer',category: 'general'});// ds2.size === ds.size + 1

dataset.remove(id)

Returns a new Dataset with the test case matching the given id removed.

remove(id: string): Dataset;
constds2=ds.remove('1');// ds2.has('1') === false

dataset.update(id, changes)

Returns a new Dataset with the test case matching id updated by merging the provided changes. The id field itself cannot be changed.

update(id: string,changes: Partial<TestCase>): Dataset;
constds2=ds.update('1',{expected: 'four',category: 'arithmetic'});// ds2.get('1')?.expected === 'four'// ds2.get('1')?.id === '1' (unchanged)

dataset.get(id)

Returns the test case with the given id, or undefined if not found.

get(id: string): TestCase|undefined;

dataset.has(id)

Returns true if a test case with the given id exists in the dataset.

has(id: string): boolean;

dataset.ids()

Returns an array of all test case IDs, in order.

ids(): string[];

dataset.categories()

Returns an array of unique category values across all test cases. Test cases without a category are excluded.

categories(): string[];

dataset.tagSet()

Returns an array of unique tags across all test cases.

tagSet(): string[];

dataset.slice(start, end?)

Returns a new Dataset with a positional slice of the cases array, using the same semantics as Array.prototype.slice.

slice(start: number,end?: number): Dataset;
constfirst10=ds.slice(0,10);constlastHalf=ds.slice(Math.floor(ds.size/2));

dataset.concat(other)

Returns a new Dataset merging cases from another dataset. Test cases from other whose IDs already exist in the current dataset are skipped (deduplication by ID).

concat(other: Dataset): Dataset;
constmerged=ds1.concat(ds2);

dataset.shuffle(seed?)

Returns a new Dataset with cases shuffled using the Mulberry32 seeded PRNG. Default seed is 42.

shuffle(seed?: number): Dataset;
constshuffled=ds.shuffle(123);// Same seed always produces the same orderconstshuffled2=ds.shuffle(123);// shuffled.ids() deep-equals shuffled2.ids()

dataset.split(config)

Splits the dataset into named, non-overlapping partitions. Returns a SplitResult (a Record<string, Dataset> keyed by partition name).

split(config: SplitConfig): SplitResult;

SplitConfig:

FieldTypeRequiredDefaultDescription
ratiosRecord<string, number>Yes--Partition names mapped to their ratios. Ratios are normalized to sum to 1.0.
mode'random' | 'stratified'No'random'Split mode
seednumberNo42PRNG seed for deterministic splits
stratifyBykeyof TestCaseNo'category'Field to stratify by (only used when mode is 'stratified')

Ratios do not need to sum to exactly 1.0 -- they are normalized automatically. For example, { train: 3, test: 1 } produces a 75/25 split.

// Random 80/20 splitconst{ train, test }=ds.split({ratios: {train: 0.8,test: 0.2},seed: 42,});// Three-way stratified split preserving category proportionsconstsplits=ds.split({ratios: {train: 0.7,val: 0.15,test: 0.15},mode: 'stratified',stratifyBy: 'category',seed: 42,});

dataset.sample(n, options?)

Returns a new Dataset containing n randomly selected test cases. When n exceeds the dataset size and replace is false, all cases are returned.

sample(n: number,options?: SampleOptions): Dataset;

SampleOptions:

FieldTypeRequiredDefaultDescription
mode'random' | 'stratified'No'random'Sampling mode
seednumberNo42PRNG seed for deterministic sampling
stratifyBystringNo'category'Field to stratify by (only used when mode is 'stratified')
replacebooleanNofalseWhether to sample with replacement
// Random sample of 20 casesconstsampled=ds.sample(20,{seed: 42});// Stratified sample preserving category proportionsconstsampled2=ds.sample(20,{mode: 'stratified',stratifyBy: 'category',seed: 42});// Sample with replacement (can return more than ds.size cases)constsampled3=ds.sample(100,{seed: 42,replace: true});

dataset.dedup(options?)

Returns a new Dataset with duplicate test cases removed.

dedup(options?: DedupOptions): Dataset;

DedupOptions:

FieldTypeRequiredDefaultDescription
mode'exact' | 'normalized' | 'jaccard'No'exact'Deduplication strategy
fieldstringNo'input'Field to compare for duplicates
thresholdnumberNo0.9Jaccard similarity threshold (only used when mode is 'jaccard')
keep'first' | 'last'No'first'Which occurrence to keep (only used for 'exact' and 'normalized' modes)

Deduplication modes:

  • exact -- Removes test cases with identical field values. Case-sensitive, whitespace-sensitive.
  • normalized -- Lowercases the value, trims whitespace, and collapses multiple spaces to a single space before comparing. "Hello World" and " hello world " are considered duplicates.
  • jaccard -- Tokenizes values by whitespace, computes Jaccard similarity (|A intersect B| / |A union B|), and treats pairs exceeding the threshold as duplicates. The first occurrence is kept.
// Exact dedup on the input fieldconstdeduped=ds.dedup();// Normalized dedup (case-insensitive, whitespace-collapsed)constdeduped2=ds.dedup({mode: 'normalized'});// Near-duplicate detection with Jaccard similarityconstdeduped3=ds.dedup({mode: 'jaccard',threshold: 0.85});// Dedup on a different field, keep last occurrenceconstdeduped4=ds.dedup({field: 'expected',keep: 'last'});

dataset.export(format, options?)

Serializes the dataset to a string in the specified format.

export(format: ExportFormat,options?: ExportOptions): string;

ExportFormat:'json' | 'jsonl' | 'csv'

ExportOptions:

FieldTypeRequiredDefaultDescription
prettybooleanNotruePretty-print JSON output with 2-space indentation
includeMetadatabooleanNotrue (JSON) / false (CSV)Include the metadata field in output
columnOrderstring[]No--Custom column order for CSV export
// Pretty-printed JSONconstjson=ds.export('json');// Compact JSONconstcompact=ds.export('json',{pretty: false});// JSON without metadataconstnoMeta=ds.export('json',{includeMetadata: false});// JSON Lines (one JSON object per line)constjsonl=ds.export('jsonl');// CSV with default column orderconstcsv=ds.export('csv');// CSV with custom column orderconstcsv2=ds.export('csv',{columnOrder: ['id','input','expected','category']});

CSV export details:

  • Array fields (tags, context) are serialized as pipe-delimited values.
  • Fields containing commas, quotes, or newlines are enclosed in double quotes with proper escaping.
  • Column order defaults to: id, input, expected, category, difficulty, tags, context, followed by any additional fields in alphabetical order.

dataset.stats()

Computes and returns statistics about the dataset.

stats(): DatasetStats;

DatasetStats:

FieldTypeDescription
totalCasesnumberTotal number of test cases
withExpectednumberNumber of cases with an expected value
withContextnumberNumber of cases with a non-empty context array
categoriesRecord<string, number>Category value to count mapping
tagsRecord<string, number>Tag to count mapping (across all cases)
inputLength{ min, max, mean }Input string length statistics
consts=ds.stats();// {// totalCases: 100,// withExpected: 85,// withContext: 30,// categories: { math: 40, reading: 60 },// tags: { hard: 20, easy: 50 },// inputLength: { min: 5, max: 200, mean: 42.3 }// }

For an empty dataset, inputLength returns { min: 0, max: 0, mean: 0 }.

dataset.validate()

Validates the dataset and returns a result with errors and warnings.

validate(): ValidationResult;

ValidationResult:

FieldTypeDescription
validbooleantrue if no errors were found
errorsArray<{ type, caseId?, message }>Validation errors
warningsArray<{ type, message }>Validation warnings

Detected errors:

  • missing_input -- A test case has an empty or whitespace-only input field.
  • duplicate_id -- Two or more test cases share the same id.

Detected warnings:

  • empty_dataset -- The dataset contains no test cases.
constresult=ds.validate();if(!result.valid){for(consterrofresult.errors){console.error(`[${err.type}] ${err.message}`);}}

dataset.toJSON()

Returns a plain JSON-serializable object representation of the dataset.

toJSON(): Record<string,unknown>;

The returned object contains name, version, cases (as a mutable array copy), and size.

constobj=ds.toJSON();// { name: 'qa-eval', version: '1.0.0', cases: [...], size: 100 }// Serialize to JSON stringconststr=JSON.stringify(ds.toJSON(),null,2);

TestCase Interface

The universal test case schema used throughout the package.

interfaceTestCase{id: string;input: string;expected?: string;context?: string[];metadata?: Record<string,unknown>;tags?: string[];difficulty?: number;category?: string;}
FieldTypeRequiredDescription
idstringYesUnique identifier. Auto-generated (8-character UUID prefix) if not provided when adding cases.
inputstringYesThe prompt, question, or query to send to the LLM
expectedstringNoExpected output / ground truth answer
contextstring[]NoContext documents for RAG evaluation
metadataRecord<string, unknown>NoArbitrary key-value metadata
tagsstring[]NoLabels for filtering and stratification
difficultynumberNoNumeric difficulty rating
categorystringNoPrimary classification label for stratification

Supporting Types

interfaceSplitConfig{ratios: Record<string,number>;mode?: 'random'|'stratified';seed?: number;stratifyBy?: keyofTestCase;}typeSplitResult=Record<string,Dataset>;interfaceSampleOptions{mode?: 'random'|'stratified';seed?: number;stratifyBy?: string;replace?: boolean;}interfaceDedupOptions{mode?: 'exact'|'normalized'|'jaccard';field?: string;threshold?: number;keep?: 'first'|'last';}typeExportFormat='json'|'jsonl'|'csv';interfaceExportOptions{pretty?: boolean;includeMetadata?: boolean;columnOrder?: string[];}interfaceDatasetStats{totalCases: number;withExpected: number;withContext: number;categories: Record<string,number>;tags: Record<string,number>;inputLength: {min: number;max: number;mean: number};}interfaceValidationResult{valid: boolean;errors: Array<{type: string;caseId?: string;message: string}>;warnings: Array<{type: string;message: string}>;}interfaceCreateOptions{name: string;version?: string;cases?: TestCase[];}interfaceLoadOptions{format?: 'json'|'jsonl'|'csv'|'auto';name?: string;version?: string;}

Configuration

Split Ratios

Split ratios are normalized automatically. The following are equivalent:

ds.split({ratios: {train: 0.8,test: 0.2}});ds.split({ratios: {train: 4,test: 1}});ds.split({ratios: {train: 80,test: 20}});

The last partition absorbs any rounding remainder to ensure all cases are assigned.

Seeded Randomization

All random operations default to seed 42. Pass an explicit seed to control the random sequence:

consta=ds.shuffle(1).ids();constb=ds.shuffle(1).ids();// a deep-equals bconstc=ds.shuffle(2).ids();// a does not deep-equal c

The Mulberry32 PRNG is used for all randomization. It produces deterministic results across platforms without relying on Math.random().

Error Handling

loadDataset throws standard JavaScript errors for invalid input:

  • SyntaxError -- When JSON or JSONL content is malformed.
  • Invalid CSV -- When the CSV string has fewer than 2 lines (no header + data), an empty array is returned rather than throwing.

dataset.validate() does not throw. It returns a ValidationResult object with structured errors and warnings that can be inspected programmatically:

constresult=ds.validate();if(!result.valid){result.errors.forEach((e)=>console.error(`${e.type}: ${e.message}`));}result.warnings.forEach((w)=>console.warn(`${w.type}: ${w.message}`));

Advanced Usage

Chaining Transformations

Because every method returns a new Dataset, transformations can be chained:

constresult=ds.filter((tc)=>tc.category==='math').dedup({mode: 'normalized'}).shuffle(42).sample(50,{seed: 7}).export('jsonl');

Building Datasets Incrementally

letds=createDataset({name: 'growing-eval',version: '1.0.0'});ds=ds.add({input: 'What is 2+2?',expected: '4',category: 'math'});ds=ds.add({input: 'Capital of France?',expected: 'Paris',category: 'geography'});ds=ds.add({input: 'Who wrote Hamlet?',expected: 'Shakespeare',category: 'literature'});console.log(ds.size);// 3

Reproducible Evaluation Pipelines

constds=awaitloadDataset(jsonString,{name: 'qa-eval',version: '2.0.0'});// Always produces the same train/test split for this datasetconst{ train, test }=ds.split({ratios: {train: 0.8,test: 0.2},seed: 42,});// Always selects the same 20 cases from the training setconstdevSample=train.sample(20,{seed: 7});

Cross-Format Round-Tripping

// Load from CSVconstds=awaitloadDataset(csvString,{name: 'test',format: 'csv'});// Export to JSON Linesconstjsonl=ds.export('jsonl');// Reload from JSON Linesconstds2=awaitloadDataset(jsonl,{name: 'test',format: 'jsonl'});// ds2 contains the same cases as ds

Merging Datasets

constds1=createDataset({name: 'batch-1',cases: firstBatch});constds2=createDataset({name: 'batch-2',cases: secondBatch});// Merge, deduplicating by IDconstmerged=ds1.concat(ds2);// Dedup by input contentconstclean=merged.dedup({mode: 'normalized'});

Stratified Splitting for Balanced Evaluation

constds=createDataset({name: 'eval',cases: [{id: '1',input: 'q1',category: 'math'},{id: '2',input: 'q2',category: 'math'},{id: '3',input: 'q3',category: 'reading'},{id: '4',input: 'q4',category: 'reading'},{id: '5',input: 'q5',category: 'coding'},{id: '6',input: 'q6',category: 'coding'},],});// Each split preserves the category distributionconstsplits=ds.split({ratios: {train: 0.67,test: 0.33},mode: 'stratified',stratifyBy: 'category',seed: 42,});

TypeScript

eval-dataset is written in TypeScript and ships with complete declaration files. All public types are exported from the package root:

importtype{TestCase,Dataset,SplitConfig,SplitResult,SampleOptions,DedupOptions,ExportFormat,ExportOptions,DatasetStats,ValidationResult,CreateOptions,LoadOptions,}from'eval-dataset';

The package targets ES2022 and uses CommonJS modules. TypeScript declaration maps are included for IDE navigation into source files.

License

MIT

About

Version-controlled eval dataset manager for LLM testing

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
GitHub - SiluPanda/eval-dataset: Version-controlled eval dataset manager for LLM testing · GitHub
Skip to content
BranchesTags
Repository files navigation
eval-dataset
Version-controlled eval dataset manager for LLM testing.
npm versionnpm downloadslicensenode
eval-dataset manages the lifecycle of evaluation datasets for LLM testing. It loads, validates, splits, samples, deduplicates, and exports collections of test cases across formats (JSON, JSONL, CSV). All transformation methods return new immutable Dataset instances, all randomization is seeded for reproducibility, and the entire API is fully typed in TypeScript.
Every LLM evaluation framework expects test data -- inputs, expected outputs, context documents, and metadata -- but none of them manage the dataset itself. eval-dataset fills this gap by providing a single package that handles loading from multiple formats, splitting with reproducible seeded randomness, sampling with stratification, deduplicating with configurable similarity, validating schema completeness, and computing statistics. Zero external runtime dependencies.
Installation
npm install eval-dataset
Requires Node.js 18 or later.
Quick Start
import{createDataset,loadDataset}from'eval-dataset';// Create a dataset from test casesconstds=createDataset({name: 'qa-eval',version: '1.0.0',cases: [{id: '1',input: 'What is 2+2?',expected: '4',category: 'math',tags: ['arithmetic']},{id: '2',input: 'Capital of France?',expected: 'Paris',category: 'geography'},{id: '3',input: 'Who wrote Hamlet?',expected: 'Shakespeare',category: 'literature'},],});console.log(ds.size);// 3console.log(ds.categories());// ['math', 'geography', 'literature']// Split into train/test setsconstsplits=ds.split({ratios: {train: 0.7,test: 0.3},seed: 42});console.log(splits.train.size);// 2console.log(splits.test.size);// 1// Export to JSON Linesconstjsonl=ds.export('jsonl');// Load from a JSON stringconstds2=awaitloadDataset('[{"id":"1","input":"hello","expected":"world"}]',{name: 'loaded',format: 'json',});
Features
  • Immutable Dataset objects -- Every transformation method (filter, map, add, remove, split, sample, dedup) returns a new Dataset. The original is never modified.
  • Seeded randomization -- Splitting, sampling, and shuffling use a Mulberry32 PRNG with configurable seeds. The same seed always produces the same result.
  • Multi-format loading -- Load test cases from JSON arrays, JSON Lines, CSV strings, or in-memory TestCase[] arrays. Format auto-detection inspects content structure when not explicitly specified.
  • Multi-format export -- Export datasets to JSON (pretty or compact), JSON Lines, or CSV with configurable column order.
  • Splitting -- Random and stratified splitting into named partitions with configurable ratios. Stratified splits maintain proportional category representation in each partition.
  • Sampling -- Random and stratified sampling with configurable sample size. Supports sampling with replacement.
  • Deduplication -- Exact match, normalized match (case-insensitive, whitespace-collapsed), and near-duplicate detection via Jaccard token similarity.
  • Validation -- Detects empty inputs, duplicate IDs, and empty datasets.
  • Statistics -- Computes case counts, expected output coverage, context coverage, category and tag distributions, and input length statistics (min, max, mean).
  • Zero runtime dependencies -- Built entirely on Node.js built-ins. Only development dependencies are used for building and testing.
  • Full TypeScript support -- All public types, interfaces, and function signatures are exported with declaration files.
API Reference
createDataset(options)
Creates a new Dataset from the provided options.
functioncreateDataset(options: CreateOptions): Dataset;
Parameters:
ParameterTypeRequiredDefaultDescription
options.namestringYes--Name of the dataset
options.versionstringNo'0.1.0'Semver version string
options.casesTestCase[]No[]Initial test cases
Returns: A Dataset instance.
constds=createDataset({name: 'my-eval',version: '1.0.0',cases: [{id: '1',input: 'What is 2+2?',expected: '4',category: 'math'},],});
loadDataset(source, options?)
Loads a dataset from a string (JSON, JSONL, or CSV content) or an in-memory TestCase[] array. Returns a Promise<Dataset>.
functionloadDataset(source: string|TestCase[],options?: LoadOptions): Promise<Dataset>;
Parameters:
ParameterTypeRequiredDefaultDescription
sourcestring | TestCase[]Yes--Content string or array of test cases
options.format'json' | 'jsonl' | 'csv' | 'auto'No'auto'Format of the source string. Ignored when source is an array.
options.namestringNo'dataset'Dataset name
options.versionstringNo'0.1.0'Dataset version
When format is 'auto', the loader inspects the content to determine the format:
  • Strings starting with [ or { are parsed as JSON.
  • Strings where every non-empty line is a JSON object are parsed as JSONL.
  • All other strings are parsed as CSV.
// Load from JSON stringconstds=awaitloadDataset('[{"id":"1","input":"hello"}]',{name: 'test'});// Load from JSONL stringconstds2=awaitloadDataset('{"id":"1","input":"hello"}\n{"id":"2","input":"world"}',{name: 'test',format: 'jsonl'},);// Load from CSV stringconstds3=awaitloadDataset('id,input,expected,category\n1,Hello,World,test\n2,Foo,Bar,test',{name: 'test',format: 'csv'},);// Load from in-memory arrayconstds4=awaitloadDataset([{id: '1',input: 'hello',expected: 'world'}],{name: 'test'},);
Field mapping during loading:
When loading from JSON, JSONL, or CSV, the loader maps common field names to the internal TestCase schema:
  • input or question maps to input
  • expected maps to expected
  • category maps to category
  • difficulty is parsed as a number
  • context is parsed as a string array
  • tags is parsed as a string array (pipe-delimited | in CSV)
  • metadata is parsed as a JSON object
Test cases without an id are assigned an auto-generated 8-character UUID.
Dataset Interface
The Dataset interface represents a named, versioned, immutable collection of test cases. All transformation methods return new Dataset instances.
Properties:
PropertyTypeDescription
namestring (readonly)Dataset name
versionstring (readonly)Semver version string
casesreadonly TestCase[] (readonly)Frozen array of test cases
sizenumber (readonly)Number of test cases
dataset.filter(fn)
Returns a new Dataset containing only test cases for which the predicate returns true.
filter(fn: (tc: TestCase)=>boolean): Dataset;
constmathOnly=ds.filter((tc)=>tc.category==='math');constwithExpected=ds.filter((tc)=>tc.expected!==undefined);
dataset.map(fn)
Returns a new Dataset with each test case transformed by the provided function.
map(fn: (tc: TestCase)=>TestCase): Dataset;
constuppercased=ds.map((tc)=>({ ...tc,input: tc.input.toUpperCase()}));
dataset.add(tc)
Returns a new Dataset with the test case appended. If id is not provided, one is auto-generated. If input is not provided, it defaults to an empty string.
add(tc: Partial<TestCase>): Dataset;
constds2=ds.add({input: 'New question?',expected: 'New answer',category: 'general'});// ds2.size === ds.size + 1
dataset.remove(id)
Returns a new Dataset with the test case matching the given id removed.
remove(id: string): Dataset;
constds2=ds.remove('1');// ds2.has('1') === false
dataset.update(id, changes)
Returns a new Dataset with the test case matching id updated by merging the provided changes. The id field itself cannot be changed.
update(id: string,changes: Partial<TestCase>): Dataset;
constds2=ds.update('1',{expected: 'four',category: 'arithmetic'});// ds2.get('1')?.expected === 'four'// ds2.get('1')?.id === '1' (unchanged)
dataset.get(id)
Returns the test case with the given id, or undefined if not found.
get(id: string): TestCase|undefined;
dataset.has(id)
Returns true if a test case with the given id exists in the dataset.
has(id: string): boolean;
dataset.ids()
Returns an array of all test case IDs, in order.
ids(): string[];
dataset.categories()
Returns an array of unique category values across all test cases. Test cases without a category are excluded.
categories(): string[];
dataset.tagSet()
Returns an array of unique tags across all test cases.
tagSet(): string[];
dataset.slice(start, end?)
Returns a new Dataset with a positional slice of the cases array, using the same semantics as Array.prototype.slice.
slice(start: number,end?: number): Dataset;
constfirst10=ds.slice(0,10);constlastHalf=ds.slice(Math.floor(ds.size/2));
dataset.concat(other)
Returns a new Dataset merging cases from another dataset. Test cases from other whose IDs already exist in the current dataset are skipped (deduplication by ID).
concat(other: Dataset): Dataset;
constmerged=ds1.concat(ds2);
dataset.shuffle(seed?)
Returns a new Dataset with cases shuffled using the Mulberry32 seeded PRNG. Default seed is 42.
shuffle(seed?: number): Dataset;
constshuffled=ds.shuffle(123);// Same seed always produces the same orderconstshuffled2=ds.shuffle(123);// shuffled.ids() deep-equals shuffled2.ids()
dataset.split(config)
Splits the dataset into named, non-overlapping partitions. Returns a SplitResult (a Record<string, Dataset> keyed by partition name).
split(config: SplitConfig): SplitResult;
SplitConfig:
FieldTypeRequiredDefaultDescription
ratiosRecord<string, number>Yes--Partition names mapped to their ratios. Ratios are normalized to sum to 1.0.
mode'random' | 'stratified'No'random'Split mode
seednumberNo42PRNG seed for deterministic splits
stratifyBykeyof TestCaseNo'category'Field to stratify by (only used when mode is 'stratified')
Ratios do not need to sum to exactly 1.0 -- they are normalized automatically. For example, { train: 3, test: 1 } produces a 75/25 split.
// Random 80/20 splitconst{ train, test }=ds.split({ratios: {train: 0.8,test: 0.2},seed: 42,});// Three-way stratified split preserving category proportionsconstsplits=ds.split({ratios: {train: 0.7,val: 0.15,test: 0.15},mode: 'stratified',stratifyBy: 'category',seed: 42,});
dataset.sample(n, options?)
Returns a new Dataset containing n randomly selected test cases. When n exceeds the dataset size and replace is false, all cases are returned.
sample(n: number,options?: SampleOptions): Dataset;
SampleOptions:
FieldTypeRequiredDefaultDescription
mode'random' | 'stratified'No'random'Sampling mode
seednumberNo42PRNG seed for deterministic sampling
stratifyBystringNo'category'Field to stratify by (only used when mode is 'stratified')
replacebooleanNofalseWhether to sample with replacement
// Random sample of 20 casesconstsampled=ds.sample(20,{seed: 42});// Stratified sample preserving category proportionsconstsampled2=ds.sample(20,{mode: 'stratified',stratifyBy: 'category',seed: 42});// Sample with replacement (can return more than ds.size cases)constsampled3=ds.sample(100,{seed: 42,replace: true});
dataset.dedup(options?)
Returns a new Dataset with duplicate test cases removed.
dedup(options?: DedupOptions): Dataset;
DedupOptions:
FieldTypeRequiredDefaultDescription
mode'exact' | 'normalized' | 'jaccard'No'exact'Deduplication strategy
fieldstringNo'input'Field to compare for duplicates
thresholdnumberNo0.9Jaccard similarity threshold (only used when mode is 'jaccard')
keep'first' | 'last'No'first'Which occurrence to keep (only used for 'exact' and 'normalized' modes)
Deduplication modes:
  • exact -- Removes test cases with identical field values. Case-sensitive, whitespace-sensitive.
  • normalized -- Lowercases the value, trims whitespace, and collapses multiple spaces to a single space before comparing. "Hello World" and " hello world " are considered duplicates.
  • jaccard -- Tokenizes values by whitespace, computes Jaccard similarity (|A intersect B| / |A union B|), and treats pairs exceeding the threshold as duplicates. The first occurrence is kept.
// Exact dedup on the input fieldconstdeduped=ds.dedup();// Normalized dedup (case-insensitive, whitespace-collapsed)constdeduped2=ds.dedup({mode: 'normalized'});// Near-duplicate detection with Jaccard similarityconstdeduped3=ds.dedup({mode: 'jaccard',threshold: 0.85});// Dedup on a different field, keep last occurrenceconstdeduped4=ds.dedup({field: 'expected',keep: 'last'});
dataset.export(format, options?)
Serializes the dataset to a string in the specified format.
export(format: ExportFormat,options?: ExportOptions): string;
ExportFormat:'json' | 'jsonl' | 'csv'
ExportOptions:
FieldTypeRequiredDefaultDescription
prettybooleanNotruePretty-print JSON output with 2-space indentation
includeMetadatabooleanNotrue (JSON) / false (CSV)Include the metadata field in output
columnOrderstring[]No--Custom column order for CSV export
// Pretty-printed JSONconstjson=ds.export('json');// Compact JSONconstcompact=ds.export('json',{pretty: false});// JSON without metadataconstnoMeta=ds.export('json',{includeMetadata: false});// JSON Lines (one JSON object per line)constjsonl=ds.export('jsonl');// CSV with default column orderconstcsv=ds.export('csv');// CSV with custom column orderconstcsv2=ds.export('csv',{columnOrder: ['id','input','expected','category']});
CSV export details:
  • Array fields (tags, context) are serialized as pipe-delimited values.
  • Fields containing commas, quotes, or newlines are enclosed in double quotes with proper escaping.
  • Column order defaults to: id, input, expected, category, difficulty, tags, context, followed by any additional fields in alphabetical order.
dataset.stats()
Computes and returns statistics about the dataset.
stats(): DatasetStats;
DatasetStats:
FieldTypeDescription
totalCasesnumberTotal number of test cases
withExpectednumberNumber of cases with an expected value
withContextnumberNumber of cases with a non-empty context array
categoriesRecord<string, number>Category value to count mapping
tagsRecord<string, number>Tag to count mapping (across all cases)
inputLength{ min, max, mean }Input string length statistics
consts=ds.stats();// {// totalCases: 100,// withExpected: 85,// withContext: 30,// categories: { math: 40, reading: 60 },// tags: { hard: 20, easy: 50 },// inputLength: { min: 5, max: 200, mean: 42.3 }// }
For an empty dataset, inputLength returns { min: 0, max: 0, mean: 0 }.
dataset.validate()
Validates the dataset and returns a result with errors and warnings.
validate(): ValidationResult;
ValidationResult:
FieldTypeDescription
validbooleantrue if no errors were found
errorsArray<{ type, caseId?, message }>Validation errors
warningsArray<{ type, message }>Validation warnings
Detected errors:
  • missing_input -- A test case has an empty or whitespace-only input field.
  • duplicate_id -- Two or more test cases share the same id.
Detected warnings:
  • empty_dataset -- The dataset contains no test cases.
constresult=ds.validate();if(!result.valid){for(consterrofresult.errors){console.error(`[${err.type}] ${err.message}`);}}
dataset.toJSON()
Returns a plain JSON-serializable object representation of the dataset.
toJSON(): Record<string,unknown>;
The returned object contains name, version, cases (as a mutable array copy), and size.
constobj=ds.toJSON();// { name: 'qa-eval', version: '1.0.0', cases: [...], size: 100 }// Serialize to JSON stringconststr=JSON.stringify(ds.toJSON(),null,2);
TestCase Interface
The universal test case schema used throughout the package.
interfaceTestCase{id: string;input: string;expected?: string;context?: string[];metadata?: Record<string,unknown>;tags?: string[];difficulty?: number;category?: string;}
FieldTypeRequiredDescription
idstringYesUnique identifier. Auto-generated (8-character UUID prefix) if not provided when adding cases.
inputstringYesThe prompt, question, or query to send to the LLM
expectedstringNoExpected output / ground truth answer
contextstring[]NoContext documents for RAG evaluation
metadataRecord<string, unknown>NoArbitrary key-value metadata
tagsstring[]NoLabels for filtering and stratification
difficultynumberNoNumeric difficulty rating
categorystringNoPrimary classification label for stratification
Supporting Types
interfaceSplitConfig{ratios: Record<string,number>;mode?: 'random'|'stratified';seed?: number;stratifyBy?: keyofTestCase;}typeSplitResult=Record<string,Dataset>;interfaceSampleOptions{mode?: 'random'|'stratified';seed?: number;stratifyBy?: string;replace?: boolean;}interfaceDedupOptions{mode?: 'exact'|'normalized'|'jaccard';field?: string;threshold?: number;keep?: 'first'|'last';}typeExportFormat='json'|'jsonl'|'csv';interfaceExportOptions{pretty?: boolean;includeMetadata?: boolean;columnOrder?: string[];}interfaceDatasetStats{totalCases: number;withExpected: number;withContext: number;categories: Record<string,number>;tags: Record<string,number>;inputLength: {min: number;max: number;mean: number};}interfaceValidationResult{valid: boolean;errors: Array<{type: string;caseId?: string;message: string}>;warnings: Array<{type: string;message: string}>;}interfaceCreateOptions{name: string;version?: string;cases?: TestCase[];}interfaceLoadOptions{format?: 'json'|'jsonl'|'csv'|'auto';name?: string;version?: string;}
Configuration
Split Ratios
Split ratios are normalized automatically. The following are equivalent:
ds.split({ratios: {train: 0.8,test: 0.2}});ds.split({ratios: {train: 4,test: 1}});ds.split({ratios: {train: 80,test: 20}});
The last partition absorbs any rounding remainder to ensure all cases are assigned.
Seeded Randomization
All random operations default to seed 42. Pass an explicit seed to control the random sequence:
consta=ds.shuffle(1).ids();constb=ds.shuffle(1).ids();// a deep-equals bconstc=ds.shuffle(2).ids();// a does not deep-equal c
The Mulberry32 PRNG is used for all randomization. It produces deterministic results across platforms without relying on Math.random().
Error Handling
loadDataset throws standard JavaScript errors for invalid input:
  • SyntaxError -- When JSON or JSONL content is malformed.
  • Invalid CSV -- When the CSV string has fewer than 2 lines (no header + data), an empty array is returned rather than throwing.
dataset.validate() does not throw. It returns a ValidationResult object with structured errors and warnings that can be inspected programmatically:
constresult=ds.validate();if(!result.valid){result.errors.forEach((e)=>console.error(`${e.type}: ${e.message}`));}result.warnings.forEach((w)=>console.warn(`${w.type}: ${w.message}`));
Advanced Usage
Chaining Transformations
Because every method returns a new Dataset, transformations can be chained:
constresult=ds.filter((tc)=>tc.category==='math').dedup({mode: 'normalized'}).shuffle(42).sample(50,{seed: 7}).export('jsonl');
Building Datasets Incrementally
letds=createDataset({name: 'growing-eval',version: '1.0.0'});ds=ds.add({input: 'What is 2+2?',expected: '4',category: 'math'});ds=ds.add({input: 'Capital of France?',expected: 'Paris',category: 'geography'});ds=ds.add({input: 'Who wrote Hamlet?',expected: 'Shakespeare',category: 'literature'});console.log(ds.size);// 3
Reproducible Evaluation Pipelines
constds=awaitloadDataset(jsonString,{name: 'qa-eval',version: '2.0.0'});// Always produces the same train/test split for this datasetconst{ train, test }=ds.split({ratios: {train: 0.8,test: 0.2},seed: 42,});// Always selects the same 20 cases from the training setconstdevSample=train.sample(20,{seed: 7});
Cross-Format Round-Tripping
// Load from CSVconstds=awaitloadDataset(csvString,{name: 'test',format: 'csv'});// Export to JSON Linesconstjsonl=ds.export('jsonl');// Reload from JSON Linesconstds2=awaitloadDataset(jsonl,{name: 'test',format: 'jsonl'});// ds2 contains the same cases as ds
Merging Datasets
constds1=createDataset({name: 'batch-1',cases: firstBatch});constds2=createDataset({name: 'batch-2',cases: secondBatch});// Merge, deduplicating by IDconstmerged=ds1.concat(ds2);// Dedup by input contentconstclean=merged.dedup({mode: 'normalized'});
Stratified Splitting for Balanced Evaluation
constds=createDataset({name: 'eval',cases: [{id: '1',input: 'q1',category: 'math'},{id: '2',input: 'q2',category: 'math'},{id: '3',input: 'q3',category: 'reading'},{id: '4',input: 'q4',category: 'reading'},{id: '5',input: 'q5',category: 'coding'},{id: '6',input: 'q6',category: 'coding'},],});// Each split preserves the category distributionconstsplits=ds.split({ratios: {train: 0.67,test: 0.33},mode: 'stratified',stratifyBy: 'category',seed: 42,});
TypeScript
eval-dataset is written in TypeScript and ships with complete declaration files. All public types are exported from the package root:
importtype{TestCase,Dataset,SplitConfig,SplitResult,SampleOptions,DedupOptions,ExportFormat,ExportOptions,DatasetStats,ValidationResult,CreateOptions,LoadOptions,}from'eval-dataset';
The package targets ES2022 and uses CommonJS modules. TypeScript declaration maps are included for IDE navigation into source files.
License
MIT
About
Version-controlled eval dataset manager for LLM testing
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - SiluPanda/eval-dataset: Version-controlled eval dataset manager for LLM testing · GitHub
Skip to content
BranchesTags
Repository files navigation
eval-dataset
Version-controlled eval dataset manager for LLM testing.
npm versionnpm downloadslicensenode
eval-dataset manages the lifecycle of evaluation datasets for LLM testing. It loads, validates, splits, samples, deduplicates, and exports collections of test cases across formats (JSON, JSONL, CSV). All transformation methods return new immutable Dataset instances, all randomization is seeded for reproducibility, and the entire API is fully typed in TypeScript.
Every LLM evaluation framework expects test data -- inputs, expected outputs, context documents, and metadata -- but none of them manage the dataset itself. eval-dataset fills this gap by providing a single package that handles loading from multiple formats, splitting with reproducible seeded randomness, sampling with stratification, deduplicating with configurable similarity, validating schema completeness, and computing statistics. Zero external runtime dependencies.
Installation
npm install eval-dataset
Requires Node.js 18 or later.
Quick Start
import{createDataset,loadDataset}from'eval-dataset';// Create a dataset from test casesconstds=createDataset({name: 'qa-eval',version: '1.0.0',cases: [{id: '1',input: 'What is 2+2?',expected: '4',category: 'math',tags: ['arithmetic']},{id: '2',input: 'Capital of France?',expected: 'Paris',category: 'geography'},{id: '3',input: 'Who wrote Hamlet?',expected: 'Shakespeare',category: 'literature'},],});console.log(ds.size);// 3console.log(ds.categories());// ['math', 'geography', 'literature']// Split into train/test setsconstsplits=ds.split({ratios: {train: 0.7,test: 0.3},seed: 42});console.log(splits.train.size);// 2console.log(splits.test.size);// 1// Export to JSON Linesconstjsonl=ds.export('jsonl');// Load from a JSON stringconstds2=awaitloadDataset('[{"id":"1","input":"hello","expected":"world"}]',{name: 'loaded',format: 'json',});
Features
  • Immutable Dataset objects -- Every transformation method (filter, map, add, remove, split, sample, dedup) returns a new Dataset. The original is never modified.
  • Seeded randomization -- Splitting, sampling, and shuffling use a Mulberry32 PRNG with configurable seeds. The same seed always produces the same result.
  • Multi-format loading -- Load test cases from JSON arrays, JSON Lines, CSV strings, or in-memory TestCase[] arrays. Format auto-detection inspects content structure when not explicitly specified.
  • Multi-format export -- Export datasets to JSON (pretty or compact), JSON Lines, or CSV with configurable column order.
  • Splitting -- Random and stratified splitting into named partitions with configurable ratios. Stratified splits maintain proportional category representation in each partition.
  • Sampling -- Random and stratified sampling with configurable sample size. Supports sampling with replacement.
  • Deduplication -- Exact match, normalized match (case-insensitive, whitespace-collapsed), and near-duplicate detection via Jaccard token similarity.
  • Validation -- Detects empty inputs, duplicate IDs, and empty datasets.
  • Statistics -- Computes case counts, expected output coverage, context coverage, category and tag distributions, and input length statistics (min, max, mean).
  • Zero runtime dependencies -- Built entirely on Node.js built-ins. Only development dependencies are used for building and testing.
  • Full TypeScript support -- All public types, interfaces, and function signatures are exported with declaration files.
API Reference
createDataset(options)
Creates a new Dataset from the provided options.
functioncreateDataset(options: CreateOptions): Dataset;
Parameters:
ParameterTypeRequiredDefaultDescription
options.namestringYes--Name of the dataset
options.versionstringNo'0.1.0'Semver version string
options.casesTestCase[]No[]Initial test cases
Returns: A Dataset instance.
constds=createDataset({name: 'my-eval',version: '1.0.0',cases: [{id: '1',input: 'What is 2+2?',expected: '4',category: 'math'},],});
loadDataset(source, options?)
Loads a dataset from a string (JSON, JSONL, or CSV content) or an in-memory TestCase[] array. Returns a Promise<Dataset>.
functionloadDataset(source: string|TestCase[],options?: LoadOptions): Promise<Dataset>;
Parameters:
ParameterTypeRequiredDefaultDescription
sourcestring | TestCase[]Yes--Content string or array of test cases
options.format'json' | 'jsonl' | 'csv' | 'auto'No'auto'Format of the source string. Ignored when source is an array.
options.namestringNo'dataset'Dataset name
options.versionstringNo'0.1.0'Dataset version
When format is 'auto', the loader inspects the content to determine the format:
  • Strings starting with [ or { are parsed as JSON.
  • Strings where every non-empty line is a JSON object are parsed as JSONL.
  • All other strings are parsed as CSV.
// Load from JSON stringconstds=awaitloadDataset('[{"id":"1","input":"hello"}]',{name: 'test'});// Load from JSONL stringconstds2=awaitloadDataset('{"id":"1","input":"hello"}\n{"id":"2","input":"world"}',{name: 'test',format: 'jsonl'},);// Load from CSV stringconstds3=awaitloadDataset('id,input,expected,category\n1,Hello,World,test\n2,Foo,Bar,test',{name: 'test',format: 'csv'},);// Load from in-memory arrayconstds4=awaitloadDataset([{id: '1',input: 'hello',expected: 'world'}],{name: 'test'},);
Field mapping during loading:
When loading from JSON, JSONL, or CSV, the loader maps common field names to the internal TestCase schema:
  • input or question maps to input
  • expected maps to expected
  • category maps to category
  • difficulty is parsed as a number
  • context is parsed as a string array
  • tags is parsed as a string array (pipe-delimited | in CSV)
  • metadata is parsed as a JSON object
Test cases without an id are assigned an auto-generated 8-character UUID.
Dataset Interface
The Dataset interface represents a named, versioned, immutable collection of test cases. All transformation methods return new Dataset instances.
Properties:
PropertyTypeDescription
namestring (readonly)Dataset name
versionstring (readonly)Semver version string
casesreadonly TestCase[] (readonly)Frozen array of test cases
sizenumber (readonly)Number of test cases
dataset.filter(fn)
Returns a new Dataset containing only test cases for which the predicate returns true.
filter(fn: (tc: TestCase)=>boolean): Dataset;
constmathOnly=ds.filter((tc)=>tc.category==='math');constwithExpected=ds.filter((tc)=>tc.expected!==undefined);
dataset.map(fn)
Returns a new Dataset with each test case transformed by the provided function.
map(fn: (tc: TestCase)=>TestCase): Dataset;
constuppercased=ds.map((tc)=>({ ...tc,input: tc.input.toUpperCase()}));
dataset.add(tc)
Returns a new Dataset with the test case appended. If id is not provided, one is auto-generated. If input is not provided, it defaults to an empty string.
add(tc: Partial<TestCase>): Dataset;
constds2=ds.add({input: 'New question?',expected: 'New answer',category: 'general'});// ds2.size === ds.size + 1
dataset.remove(id)
Returns a new Dataset with the test case matching the given id removed.
remove(id: string): Dataset;
constds2=ds.remove('1');// ds2.has('1') === false
dataset.update(id, changes)
Returns a new Dataset with the test case matching id updated by merging the provided changes. The id field itself cannot be changed.
update(id: string,changes: Partial<TestCase>): Dataset;
constds2=ds.update('1',{expected: 'four',category: 'arithmetic'});// ds2.get('1')?.expected === 'four'// ds2.get('1')?.id === '1' (unchanged)
dataset.get(id)
Returns the test case with the given id, or undefined if not found.
get(id: string): TestCase|undefined;
dataset.has(id)
Returns true if a test case with the given id exists in the dataset.
has(id: string): boolean;
dataset.ids()
Returns an array of all test case IDs, in order.
ids(): string[];
dataset.categories()
Returns an array of unique category values across all test cases. Test cases without a category are excluded.
categories(): string[];
dataset.tagSet()
Returns an array of unique tags across all test cases.
tagSet(): string[];
dataset.slice(start, end?)
Returns a new Dataset with a positional slice of the cases array, using the same semantics as Array.prototype.slice.
slice(start: number,end?: number): Dataset;
constfirst10=ds.slice(0,10);constlastHalf=ds.slice(Math.floor(ds.size/2));
dataset.concat(other)
Returns a new Dataset merging cases from another dataset. Test cases from other whose IDs already exist in the current dataset are skipped (deduplication by ID).
concat(other: Dataset): Dataset;
constmerged=ds1.concat(ds2);
dataset.shuffle(seed?)
Returns a new Dataset with cases shuffled using the Mulberry32 seeded PRNG. Default seed is 42.
shuffle(seed?: number): Dataset;
constshuffled=ds.shuffle(123);// Same seed always produces the same orderconstshuffled2=ds.shuffle(123);// shuffled.ids() deep-equals shuffled2.ids()
dataset.split(config)
Splits the dataset into named, non-overlapping partitions. Returns a SplitResult (a Record<string, Dataset> keyed by partition name).
split(config: SplitConfig): SplitResult;
SplitConfig:
FieldTypeRequiredDefaultDescription
ratiosRecord<string, number>Yes--Partition names mapped to their ratios. Ratios are normalized to sum to 1.0.
mode'random' | 'stratified'No'random'Split mode
seednumberNo42PRNG seed for deterministic splits
stratifyBykeyof TestCaseNo'category'Field to stratify by (only used when mode is 'stratified')
Ratios do not need to sum to exactly 1.0 -- they are normalized automatically. For example, { train: 3, test: 1 } produces a 75/25 split.
// Random 80/20 splitconst{ train, test }=ds.split({ratios: {train: 0.8,test: 0.2},seed: 42,});// Three-way stratified split preserving category proportionsconstsplits=ds.split({ratios: {train: 0.7,val: 0.15,test: 0.15},mode: 'stratified',stratifyBy: 'category',seed: 42,});
dataset.sample(n, options?)
Returns a new Dataset containing n randomly selected test cases. When n exceeds the dataset size and replace is false, all cases are returned.
sample(n: number,options?: SampleOptions): Dataset;
SampleOptions:
FieldTypeRequiredDefaultDescription
mode'random' | 'stratified'No'random'Sampling mode
seednumberNo42PRNG seed for deterministic sampling
stratifyBystringNo'category'Field to stratify by (only used when mode is 'stratified')
replacebooleanNofalseWhether to sample with replacement
// Random sample of 20 casesconstsampled=ds.sample(20,{seed: 42});// Stratified sample preserving category proportionsconstsampled2=ds.sample(20,{mode: 'stratified',stratifyBy: 'category',seed: 42});// Sample with replacement (can return more than ds.size cases)constsampled3=ds.sample(100,{seed: 42,replace: true});
dataset.dedup(options?)
Returns a new Dataset with duplicate test cases removed.
dedup(options?: DedupOptions): Dataset;
DedupOptions:
FieldTypeRequiredDefaultDescription
mode'exact' | 'normalized' | 'jaccard'No'exact'Deduplication strategy
fieldstringNo'input'Field to compare for duplicates
thresholdnumberNo0.9Jaccard similarity threshold (only used when mode is 'jaccard')
keep'first' | 'last'No'first'Which occurrence to keep (only used for 'exact' and 'normalized' modes)
Deduplication modes:
  • exact -- Removes test cases with identical field values. Case-sensitive, whitespace-sensitive.
  • normalized -- Lowercases the value, trims whitespace, and collapses multiple spaces to a single space before comparing. "Hello World" and " hello world " are considered duplicates.
  • jaccard -- Tokenizes values by whitespace, computes Jaccard similarity (|A intersect B| / |A union B|), and treats pairs exceeding the threshold as duplicates. The first occurrence is kept.
// Exact dedup on the input fieldconstdeduped=ds.dedup();// Normalized dedup (case-insensitive, whitespace-collapsed)constdeduped2=ds.dedup({mode: 'normalized'});// Near-duplicate detection with Jaccard similarityconstdeduped3=ds.dedup({mode: 'jaccard',threshold: 0.85});// Dedup on a different field, keep last occurrenceconstdeduped4=ds.dedup({field: 'expected',keep: 'last'});
dataset.export(format, options?)
Serializes the dataset to a string in the specified format.
export(format: ExportFormat,options?: ExportOptions): string;
ExportFormat:'json' | 'jsonl' | 'csv'
ExportOptions:
FieldTypeRequiredDefaultDescription
prettybooleanNotruePretty-print JSON output with 2-space indentation
includeMetadatabooleanNotrue (JSON) / false (CSV)Include the metadata field in output
columnOrderstring[]No--Custom column order for CSV export
// Pretty-printed JSONconstjson=ds.export('json');// Compact JSONconstcompact=ds.export('json',{pretty: false});// JSON without metadataconstnoMeta=ds.export('json',{includeMetadata: false});// JSON Lines (one JSON object per line)constjsonl=ds.export('jsonl');// CSV with default column orderconstcsv=ds.export('csv');// CSV with custom column orderconstcsv2=ds.export('csv',{columnOrder: ['id','input','expected','category']});
CSV export details:
  • Array fields (tags, context) are serialized as pipe-delimited values.
  • Fields containing commas, quotes, or newlines are enclosed in double quotes with proper escaping.
  • Column order defaults to: id, input, expected, category, difficulty, tags, context, followed by any additional fields in alphabetical order.
dataset.stats()
Computes and returns statistics about the dataset.
stats(): DatasetStats;
DatasetStats:
FieldTypeDescription
totalCasesnumberTotal number of test cases
withExpectednumberNumber of cases with an expected value
withContextnumberNumber of cases with a non-empty context array
categoriesRecord<string, number>Category value to count mapping
tagsRecord<string, number>Tag to count mapping (across all cases)
inputLength{ min, max, mean }Input string length statistics
consts=ds.stats();// {// totalCases: 100,// withExpected: 85,// withContext: 30,// categories: { math: 40, reading: 60 },// tags: { hard: 20, easy: 50 },// inputLength: { min: 5, max: 200, mean: 42.3 }// }
For an empty dataset, inputLength returns { min: 0, max: 0, mean: 0 }.
dataset.validate()
Validates the dataset and returns a result with errors and warnings.
validate(): ValidationResult;
ValidationResult:
FieldTypeDescription
validbooleantrue if no errors were found
errorsArray<{ type, caseId?, message }>Validation errors
warningsArray<{ type, message }>Validation warnings
Detected errors:
  • missing_input -- A test case has an empty or whitespace-only input field.
  • duplicate_id -- Two or more test cases share the same id.
Detected warnings:
  • empty_dataset -- The dataset contains no test cases.
constresult=ds.validate();if(!result.valid){for(consterrofresult.errors){console.error(`[${err.type}] ${err.message}`);}}
dataset.toJSON()
Returns a plain JSON-serializable object representation of the dataset.
toJSON(): Record<string,unknown>;
The returned object contains name, version, cases (as a mutable array copy), and size.
constobj=ds.toJSON();// { name: 'qa-eval', version: '1.0.0', cases: [...], size: 100 }// Serialize to JSON stringconststr=JSON.stringify(ds.toJSON(),null,2);
TestCase Interface
The universal test case schema used throughout the package.
interfaceTestCase{id: string;input: string;expected?: string;context?: string[];metadata?: Record<string,unknown>;tags?: string[];difficulty?: number;category?: string;}
FieldTypeRequiredDescription
idstringYesUnique identifier. Auto-generated (8-character UUID prefix) if not provided when adding cases.
inputstringYesThe prompt, question, or query to send to the LLM
expectedstringNoExpected output / ground truth answer
contextstring[]NoContext documents for RAG evaluation
metadataRecord<string, unknown>NoArbitrary key-value metadata
tagsstring[]NoLabels for filtering and stratification
difficultynumberNoNumeric difficulty rating
categorystringNoPrimary classification label for stratification
Supporting Types
interfaceSplitConfig{ratios: Record<string,number>;mode?: 'random'|'stratified';seed?: number;stratifyBy?: keyofTestCase;}typeSplitResult=Record<string,Dataset>;interfaceSampleOptions{mode?: 'random'|'stratified';seed?: number;stratifyBy?: string;replace?: boolean;}interfaceDedupOptions{mode?: 'exact'|'normalized'|'jaccard';field?: string;threshold?: number;keep?: 'first'|'last';}typeExportFormat='json'|'jsonl'|'csv';interfaceExportOptions{pretty?: boolean;includeMetadata?: boolean;columnOrder?: string[];}interfaceDatasetStats{totalCases: number;withExpected: number;withContext: number;categories: Record<string,number>;tags: Record<string,number>;inputLength: {min: number;max: number;mean: number};}interfaceValidationResult{valid: boolean;errors: Array<{type: string;caseId?: string;message: string}>;warnings: Array<{type: string;message: string}>;}interfaceCreateOptions{name: string;version?: string;cases?: TestCase[];}interfaceLoadOptions{format?: 'json'|'jsonl'|'csv'|'auto';name?: string;version?: string;}
Configuration
Split Ratios
Split ratios are normalized automatically. The following are equivalent:
ds.split({ratios: {train: 0.8,test: 0.2}});ds.split({ratios: {train: 4,test: 1}});ds.split({ratios: {train: 80,test: 20}});
The last partition absorbs any rounding remainder to ensure all cases are assigned.
Seeded Randomization
All random operations default to seed 42. Pass an explicit seed to control the random sequence:
consta=ds.shuffle(1).ids();constb=ds.shuffle(1).ids();// a deep-equals bconstc=ds.shuffle(2).ids();// a does not deep-equal c
The Mulberry32 PRNG is used for all randomization. It produces deterministic results across platforms without relying on Math.random().
Error Handling
loadDataset throws standard JavaScript errors for invalid input:
  • SyntaxError -- When JSON or JSONL content is malformed.
  • Invalid CSV -- When the CSV string has fewer than 2 lines (no header + data), an empty array is returned rather than throwing.
dataset.validate() does not throw. It returns a ValidationResult object with structured errors and warnings that can be inspected programmatically:
constresult=ds.validate();if(!result.valid){result.errors.forEach((e)=>console.error(`${e.type}: ${e.message}`));}result.warnings.forEach((w)=>console.warn(`${w.type}: ${w.message}`));
Advanced Usage
Chaining Transformations
Because every method returns a new Dataset, transformations can be chained:
constresult=ds.filter((tc)=>tc.category==='math').dedup({mode: 'normalized'}).shuffle(42).sample(50,{seed: 7}).export('jsonl');
Building Datasets Incrementally
letds=createDataset({name: 'growing-eval',version: '1.0.0'});ds=ds.add({input: 'What is 2+2?',expected: '4',category: 'math'});ds=ds.add({input: 'Capital of France?',expected: 'Paris',category: 'geography'});ds=ds.add({input: 'Who wrote Hamlet?',expected: 'Shakespeare',category: 'literature'});console.log(ds.size);// 3
Reproducible Evaluation Pipelines
constds=awaitloadDataset(jsonString,{name: 'qa-eval',version: '2.0.0'});// Always produces the same train/test split for this datasetconst{ train, test }=ds.split({ratios: {train: 0.8,test: 0.2},seed: 42,});// Always selects the same 20 cases from the training setconstdevSample=train.sample(20,{seed: 7});
Cross-Format Round-Tripping
// Load from CSVconstds=awaitloadDataset(csvString,{name: 'test',format: 'csv'});// Export to JSON Linesconstjsonl=ds.export('jsonl');// Reload from JSON Linesconstds2=awaitloadDataset(jsonl,{name: 'test',format: 'jsonl'});// ds2 contains the same cases as ds
Merging Datasets
constds1=createDataset({name: 'batch-1',cases: firstBatch});constds2=createDataset({name: 'batch-2',cases: secondBatch});// Merge, deduplicating by IDconstmerged=ds1.concat(ds2);// Dedup by input contentconstclean=merged.dedup({mode: 'normalized'});
Stratified Splitting for Balanced Evaluation
constds=createDataset({name: 'eval',cases: [{id: '1',input: 'q1',category: 'math'},{id: '2',input: 'q2',category: 'math'},{id: '3',input: 'q3',category: 'reading'},{id: '4',input: 'q4',category: 'reading'},{id: '5',input: 'q5',category: 'coding'},{id: '6',input: 'q6',category: 'coding'},],});// Each split preserves the category distributionconstsplits=ds.split({ratios: {train: 0.67,test: 0.33},mode: 'stratified',stratifyBy: 'category',seed: 42,});
TypeScript
eval-dataset is written in TypeScript and ships with complete declaration files. All public types are exported from the package root:
importtype{TestCase,Dataset,SplitConfig,SplitResult,SampleOptions,DedupOptions,ExportFormat,ExportOptions,DatasetStats,ValidationResult,CreateOptions,LoadOptions,}from'eval-dataset';
The package targets ES2022 and uses CommonJS modules. TypeScript declaration maps are included for IDE navigation into source files.
License
MIT
About
Version-controlled eval dataset manager for LLM testing
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - SiluPanda/eval-dataset: Version-controlled eval dataset manager for LLM testing · GitHub
Skip to content
BranchesTags
Repository files navigation
eval-dataset
Version-controlled eval dataset manager for LLM testing.
npm versionnpm downloadslicensenode
eval-dataset manages the lifecycle of evaluation datasets for LLM testing. It loads, validates, splits, samples, deduplicates, and exports collections of test cases across formats (JSON, JSONL, CSV). All transformation methods return new immutable Dataset instances, all randomization is seeded for reproducibility, and the entire API is fully typed in TypeScript.
Every LLM evaluation framework expects test data -- inputs, expected outputs, context documents, and metadata -- but none of them manage the dataset itself. eval-dataset fills this gap by providing a single package that handles loading from multiple formats, splitting with reproducible seeded randomness, sampling with stratification, deduplicating with configurable similarity, validating schema completeness, and computing statistics. Zero external runtime dependencies.
Installation
npm install eval-dataset
Requires Node.js 18 or later.
Quick Start
import{createDataset,loadDataset}from'eval-dataset';// Create a dataset from test casesconstds=createDataset({name: 'qa-eval',version: '1.0.0',cases: [{id: '1',input: 'What is 2+2?',expected: '4',category: 'math',tags: ['arithmetic']},{id: '2',input: 'Capital of France?',expected: 'Paris',category: 'geography'},{id: '3',input: 'Who wrote Hamlet?',expected: 'Shakespeare',category: 'literature'},],});console.log(ds.size);// 3console.log(ds.categories());// ['math', 'geography', 'literature']// Split into train/test setsconstsplits=ds.split({ratios: {train: 0.7,test: 0.3},seed: 42});console.log(splits.train.size);// 2console.log(splits.test.size);// 1// Export to JSON Linesconstjsonl=ds.export('jsonl');// Load from a JSON stringconstds2=awaitloadDataset('[{"id":"1","input":"hello","expected":"world"}]',{name: 'loaded',format: 'json',});
Features
  • Immutable Dataset objects -- Every transformation method (filter, map, add, remove, split, sample, dedup) returns a new Dataset. The original is never modified.
  • Seeded randomization -- Splitting, sampling, and shuffling use a Mulberry32 PRNG with configurable seeds. The same seed always produces the same result.
  • Multi-format loading -- Load test cases from JSON arrays, JSON Lines, CSV strings, or in-memory TestCase[] arrays. Format auto-detection inspects content structure when not explicitly specified.
  • Multi-format export -- Export datasets to JSON (pretty or compact), JSON Lines, or CSV with configurable column order.
  • Splitting -- Random and stratified splitting into named partitions with configurable ratios. Stratified splits maintain proportional category representation in each partition.
  • Sampling -- Random and stratified sampling with configurable sample size. Supports sampling with replacement.
  • Deduplication -- Exact match, normalized match (case-insensitive, whitespace-collapsed), and near-duplicate detection via Jaccard token similarity.
  • Validation -- Detects empty inputs, duplicate IDs, and empty datasets.
  • Statistics -- Computes case counts, expected output coverage, context coverage, category and tag distributions, and input length statistics (min, max, mean).
  • Zero runtime dependencies -- Built entirely on Node.js built-ins. Only development dependencies are used for building and testing.
  • Full TypeScript support -- All public types, interfaces, and function signatures are exported with declaration files.
API Reference
createDataset(options)
Creates a new Dataset from the provided options.
functioncreateDataset(options: CreateOptions): Dataset;
Parameters:
ParameterTypeRequiredDefaultDescription
options.namestringYes--Name of the dataset
options.versionstringNo'0.1.0'Semver version string
options.casesTestCase[]No[]Initial test cases
Returns: A Dataset instance.
constds=createDataset({name: 'my-eval',version: '1.0.0',cases: [{id: '1',input: 'What is 2+2?',expected: '4',category: 'math'},],});
loadDataset(source, options?)
Loads a dataset from a string (JSON, JSONL, or CSV content) or an in-memory TestCase[] array. Returns a Promise<Dataset>.
functionloadDataset(source: string|TestCase[],options?: LoadOptions): Promise<Dataset>;
Parameters:
ParameterTypeRequiredDefaultDescription
sourcestring | TestCase[]Yes--Content string or array of test cases
options.format'json' | 'jsonl' | 'csv' | 'auto'No'auto'Format of the source string. Ignored when source is an array.
options.namestringNo'dataset'Dataset name
options.versionstringNo'0.1.0'Dataset version
When format is 'auto', the loader inspects the content to determine the format:
  • Strings starting with [ or { are parsed as JSON.
  • Strings where every non-empty line is a JSON object are parsed as JSONL.
  • All other strings are parsed as CSV.
// Load from JSON stringconstds=awaitloadDataset('[{"id":"1","input":"hello"}]',{name: 'test'});// Load from JSONL stringconstds2=awaitloadDataset('{"id":"1","input":"hello"}\n{"id":"2","input":"world"}',{name: 'test',format: 'jsonl'},);// Load from CSV stringconstds3=awaitloadDataset('id,input,expected,category\n1,Hello,World,test\n2,Foo,Bar,test',{name: 'test',format: 'csv'},);// Load from in-memory arrayconstds4=awaitloadDataset([{id: '1',input: 'hello',expected: 'world'}],{name: 'test'},);
Field mapping during loading:
When loading from JSON, JSONL, or CSV, the loader maps common field names to the internal TestCase schema:
  • input or question maps to input
  • expected maps to expected
  • category maps to category
  • difficulty is parsed as a number
  • context is parsed as a string array
  • tags is parsed as a string array (pipe-delimited | in CSV)
  • metadata is parsed as a JSON object
Test cases without an id are assigned an auto-generated 8-character UUID.
Dataset Interface
The Dataset interface represents a named, versioned, immutable collection of test cases. All transformation methods return new Dataset instances.
Properties:
PropertyTypeDescription
namestring (readonly)Dataset name
versionstring (readonly)Semver version string
casesreadonly TestCase[] (readonly)Frozen array of test cases
sizenumber (readonly)Number of test cases
dataset.filter(fn)
Returns a new Dataset containing only test cases for which the predicate returns true.
filter(fn: (tc: TestCase)=>boolean): Dataset;
constmathOnly=ds.filter((tc)=>tc.category==='math');constwithExpected=ds.filter((tc)=>tc.expected!==undefined);
dataset.map(fn)
Returns a new Dataset with each test case transformed by the provided function.
map(fn: (tc: TestCase)=>TestCase): Dataset;
constuppercased=ds.map((tc)=>({ ...tc,input: tc.input.toUpperCase()}));
dataset.add(tc)
Returns a new Dataset with the test case appended. If id is not provided, one is auto-generated. If input is not provided, it defaults to an empty string.
add(tc: Partial<TestCase>): Dataset;
constds2=ds.add({input: 'New question?',expected: 'New answer',category: 'general'});// ds2.size === ds.size + 1
dataset.remove(id)
Returns a new Dataset with the test case matching the given id removed.
remove(id: string): Dataset;
constds2=ds.remove('1');// ds2.has('1') === false
dataset.update(id, changes)
Returns a new Dataset with the test case matching id updated by merging the provided changes. The id field itself cannot be changed.
update(id: string,changes: Partial<TestCase>): Dataset;
constds2=ds.update('1',{expected: 'four',category: 'arithmetic'});// ds2.get('1')?.expected === 'four'// ds2.get('1')?.id === '1' (unchanged)
dataset.get(id)
Returns the test case with the given id, or undefined if not found.
get(id: string): TestCase|undefined;
dataset.has(id)
Returns true if a test case with the given id exists in the dataset.
has(id: string): boolean;
dataset.ids()
Returns an array of all test case IDs, in order.
ids(): string[];
dataset.categories()
Returns an array of unique category values across all test cases. Test cases without a category are excluded.
categories(): string[];
dataset.tagSet()
Returns an array of unique tags across all test cases.
tagSet(): string[];
dataset.slice(start, end?)
Returns a new Dataset with a positional slice of the cases array, using the same semantics as Array.prototype.slice.
slice(start: number,end?: number): Dataset;
constfirst10=ds.slice(0,10);constlastHalf=ds.slice(Math.floor(ds.size/2));
dataset.concat(other)
Returns a new Dataset merging cases from another dataset. Test cases from other whose IDs already exist in the current dataset are skipped (deduplication by ID).
concat(other: Dataset): Dataset;
constmerged=ds1.concat(ds2);
dataset.shuffle(seed?)
Returns a new Dataset with cases shuffled using the Mulberry32 seeded PRNG. Default seed is 42.
shuffle(seed?: number): Dataset;
constshuffled=ds.shuffle(123);// Same seed always produces the same orderconstshuffled2=ds.shuffle(123);// shuffled.ids() deep-equals shuffled2.ids()
dataset.split(config)
Splits the dataset into named, non-overlapping partitions. Returns a SplitResult (a Record<string, Dataset> keyed by partition name).
split(config: SplitConfig): SplitResult;
SplitConfig:
FieldTypeRequiredDefaultDescription
ratiosRecord<string, number>Yes--Partition names mapped to their ratios. Ratios are normalized to sum to 1.0.
mode'random' | 'stratified'No'random'Split mode
seednumberNo42PRNG seed for deterministic splits
stratifyBykeyof TestCaseNo'category'Field to stratify by (only used when mode is 'stratified')
Ratios do not need to sum to exactly 1.0 -- they are normalized automatically. For example, { train: 3, test: 1 } produces a 75/25 split.
// Random 80/20 splitconst{ train, test }=ds.split({ratios: {train: 0.8,test: 0.2},seed: 42,});// Three-way stratified split preserving category proportionsconstsplits=ds.split({ratios: {train: 0.7,val: 0.15,test: 0.15},mode: 'stratified',stratifyBy: 'category',seed: 42,});
dataset.sample(n, options?)
Returns a new Dataset containing n randomly selected test cases. When n exceeds the dataset size and replace is false, all cases are returned.
sample(n: number,options?: SampleOptions): Dataset;
SampleOptions:
FieldTypeRequiredDefaultDescription
mode'random' | 'stratified'No'random'Sampling mode
seednumberNo42PRNG seed for deterministic sampling
stratifyBystringNo'category'Field to stratify by (only used when mode is 'stratified')
replacebooleanNofalseWhether to sample with replacement
// Random sample of 20 casesconstsampled=ds.sample(20,{seed: 42});// Stratified sample preserving category proportionsconstsampled2=ds.sample(20,{mode: 'stratified',stratifyBy: 'category',seed: 42});// Sample with replacement (can return more than ds.size cases)constsampled3=ds.sample(100,{seed: 42,replace: true});
dataset.dedup(options?)
Returns a new Dataset with duplicate test cases removed.
dedup(options?: DedupOptions): Dataset;
DedupOptions:
FieldTypeRequiredDefaultDescription
mode'exact' | 'normalized' | 'jaccard'No'exact'Deduplication strategy
fieldstringNo'input'Field to compare for duplicates
thresholdnumberNo0.9Jaccard similarity threshold (only used when mode is 'jaccard')
keep'first' | 'last'No'first'Which occurrence to keep (only used for 'exact' and 'normalized' modes)
Deduplication modes:
  • exact -- Removes test cases with identical field values. Case-sensitive, whitespace-sensitive.
  • normalized -- Lowercases the value, trims whitespace, and collapses multiple spaces to a single space before comparing. "Hello World" and " hello world " are considered duplicates.
  • jaccard -- Tokenizes values by whitespace, computes Jaccard similarity (|A intersect B| / |A union B|), and treats pairs exceeding the threshold as duplicates. The first occurrence is kept.
// Exact dedup on the input fieldconstdeduped=ds.dedup();// Normalized dedup (case-insensitive, whitespace-collapsed)constdeduped2=ds.dedup({mode: 'normalized'});// Near-duplicate detection with Jaccard similarityconstdeduped3=ds.dedup({mode: 'jaccard',threshold: 0.85});// Dedup on a different field, keep last occurrenceconstdeduped4=ds.dedup({field: 'expected',keep: 'last'});
dataset.export(format, options?)
Serializes the dataset to a string in the specified format.
export(format: ExportFormat,options?: ExportOptions): string;
ExportFormat:'json' | 'jsonl' | 'csv'
ExportOptions:
FieldTypeRequiredDefaultDescription
prettybooleanNotruePretty-print JSON output with 2-space indentation
includeMetadatabooleanNotrue (JSON) / false (CSV)Include the metadata field in output
columnOrderstring[]No--Custom column order for CSV export
// Pretty-printed JSONconstjson=ds.export('json');// Compact JSONconstcompact=ds.export('json',{pretty: false});// JSON without metadataconstnoMeta=ds.export('json',{includeMetadata: false});// JSON Lines (one JSON object per line)constjsonl=ds.export('jsonl');// CSV with default column orderconstcsv=ds.export('csv');// CSV with custom column orderconstcsv2=ds.export('csv',{columnOrder: ['id','input','expected','category']});
CSV export details:
  • Array fields (tags, context) are serialized as pipe-delimited values.
  • Fields containing commas, quotes, or newlines are enclosed in double quotes with proper escaping.
  • Column order defaults to: id, input, expected, category, difficulty, tags, context, followed by any additional fields in alphabetical order.
dataset.stats()
Computes and returns statistics about the dataset.
stats(): DatasetStats;
DatasetStats:
FieldTypeDescription
totalCasesnumberTotal number of test cases
withExpectednumberNumber of cases with an expected value
withContextnumberNumber of cases with a non-empty context array
categoriesRecord<string, number>Category value to count mapping
tagsRecord<string, number>Tag to count mapping (across all cases)
inputLength{ min, max, mean }Input string length statistics
consts=ds.stats();// {// totalCases: 100,// withExpected: 85,// withContext: 30,// categories: { math: 40, reading: 60 },// tags: { hard: 20, easy: 50 },// inputLength: { min: 5, max: 200, mean: 42.3 }// }
For an empty dataset, inputLength returns { min: 0, max: 0, mean: 0 }.
dataset.validate()
Validates the dataset and returns a result with errors and warnings.
validate(): ValidationResult;
ValidationResult:
FieldTypeDescription
validbooleantrue if no errors were found
errorsArray<{ type, caseId?, message }>Validation errors
warningsArray<{ type, message }>Validation warnings
Detected errors:
  • missing_input -- A test case has an empty or whitespace-only input field.
  • duplicate_id -- Two or more test cases share the same id.
Detected warnings:
  • empty_dataset -- The dataset contains no test cases.
constresult=ds.validate();if(!result.valid){for(consterrofresult.errors){console.error(`[${err.type}] ${err.message}`);}}
dataset.toJSON()
Returns a plain JSON-serializable object representation of the dataset.
toJSON(): Record<string,unknown>;
The returned object contains name, version, cases (as a mutable array copy), and size.
constobj=ds.toJSON();// { name: 'qa-eval', version: '1.0.0', cases: [...], size: 100 }// Serialize to JSON stringconststr=JSON.stringify(ds.toJSON(),null,2);
TestCase Interface
The universal test case schema used throughout the package.
interfaceTestCase{id: string;input: string;expected?: string;context?: string[];metadata?: Record<string,unknown>;tags?: string[];difficulty?: number;category?: string;}
FieldTypeRequiredDescription
idstringYesUnique identifier. Auto-generated (8-character UUID prefix) if not provided when adding cases.
inputstringYesThe prompt, question, or query to send to the LLM
expectedstringNoExpected output / ground truth answer
contextstring[]NoContext documents for RAG evaluation
metadataRecord<string, unknown>NoArbitrary key-value metadata
tagsstring[]NoLabels for filtering and stratification
difficultynumberNoNumeric difficulty rating
categorystringNoPrimary classification label for stratification
Supporting Types
interfaceSplitConfig{ratios: Record<string,number>;mode?: 'random'|'stratified';seed?: number;stratifyBy?: keyofTestCase;}typeSplitResult=Record<string,Dataset>;interfaceSampleOptions{mode?: 'random'|'stratified';seed?: number;stratifyBy?: string;replace?: boolean;}interfaceDedupOptions{mode?: 'exact'|'normalized'|'jaccard';field?: string;threshold?: number;keep?: 'first'|'last';}typeExportFormat='json'|'jsonl'|'csv';interfaceExportOptions{pretty?: boolean;includeMetadata?: boolean;columnOrder?: string[];}interfaceDatasetStats{totalCases: number;withExpected: number;withContext: number;categories: Record<string,number>;tags: Record<string,number>;inputLength: {min: number;max: number;mean: number};}interfaceValidationResult{valid: boolean;errors: Array<{type: string;caseId?: string;message: string}>;warnings: Array<{type: string;message: string}>;}interfaceCreateOptions{name: string;version?: string;cases?: TestCase[];}interfaceLoadOptions{format?: 'json'|'jsonl'|'csv'|'auto';name?: string;version?: string;}
Configuration
Split Ratios
Split ratios are normalized automatically. The following are equivalent:
ds.split({ratios: {train: 0.8,test: 0.2}});ds.split({ratios: {train: 4,test: 1}});ds.split({ratios: {train: 80,test: 20}});
The last partition absorbs any rounding remainder to ensure all cases are assigned.
Seeded Randomization
All random operations default to seed 42. Pass an explicit seed to control the random sequence:
consta=ds.shuffle(1).ids();constb=ds.shuffle(1).ids();// a deep-equals bconstc=ds.shuffle(2).ids();// a does not deep-equal c
The Mulberry32 PRNG is used for all randomization. It produces deterministic results across platforms without relying on Math.random().
Error Handling
loadDataset throws standard JavaScript errors for invalid input:
  • SyntaxError -- When JSON or JSONL content is malformed.
  • Invalid CSV -- When the CSV string has fewer than 2 lines (no header + data), an empty array is returned rather than throwing.
dataset.validate() does not throw. It returns a ValidationResult object with structured errors and warnings that can be inspected programmatically:
constresult=ds.validate();if(!result.valid){result.errors.forEach((e)=>console.error(`${e.type}: ${e.message}`));}result.warnings.forEach((w)=>console.warn(`${w.type}: ${w.message}`));
Advanced Usage
Chaining Transformations
Because every method returns a new Dataset, transformations can be chained:
constresult=ds.filter((tc)=>tc.category==='math').dedup({mode: 'normalized'}).shuffle(42).sample(50,{seed: 7}).export('jsonl');
Building Datasets Incrementally
letds=createDataset({name: 'growing-eval',version: '1.0.0'});ds=ds.add({input: 'What is 2+2?',expected: '4',category: 'math'});ds=ds.add({input: 'Capital of France?',expected: 'Paris',category: 'geography'});ds=ds.add({input: 'Who wrote Hamlet?',expected: 'Shakespeare',category: 'literature'});console.log(ds.size);// 3
Reproducible Evaluation Pipelines
constds=awaitloadDataset(jsonString,{name: 'qa-eval',version: '2.0.0'});// Always produces the same train/test split for this datasetconst{ train, test }=ds.split({ratios: {train: 0.8,test: 0.2},seed: 42,});// Always selects the same 20 cases from the training setconstdevSample=train.sample(20,{seed: 7});
Cross-Format Round-Tripping
// Load from CSVconstds=awaitloadDataset(csvString,{name: 'test',format: 'csv'});// Export to JSON Linesconstjsonl=ds.export('jsonl');// Reload from JSON Linesconstds2=awaitloadDataset(jsonl,{name: 'test',format: 'jsonl'});// ds2 contains the same cases as ds
Merging Datasets
constds1=createDataset({name: 'batch-1',cases: firstBatch});constds2=createDataset({name: 'batch-2',cases: secondBatch});// Merge, deduplicating by IDconstmerged=ds1.concat(ds2);// Dedup by input contentconstclean=merged.dedup({mode: 'normalized'});
Stratified Splitting for Balanced Evaluation
constds=createDataset({name: 'eval',cases: [{id: '1',input: 'q1',category: 'math'},{id: '2',input: 'q2',category: 'math'},{id: '3',input: 'q3',category: 'reading'},{id: '4',input: 'q4',category: 'reading'},{id: '5',input: 'q5',category: 'coding'},{id: '6',input: 'q6',category: 'coding'},],});// Each split preserves the category distributionconstsplits=ds.split({ratios: {train: 0.67,test: 0.33},mode: 'stratified',stratifyBy: 'category',seed: 42,});
TypeScript
eval-dataset is written in TypeScript and ships with complete declaration files. All public types are exported from the package root:
importtype{TestCase,Dataset,SplitConfig,SplitResult,SampleOptions,DedupOptions,ExportFormat,ExportOptions,DatasetStats,ValidationResult,CreateOptions,LoadOptions,}from'eval-dataset';
The package targets ES2022 and uses CommonJS modules. TypeScript declaration maps are included for IDE navigation into source files.
License
MIT
About
Version-controlled eval dataset manager for LLM testing
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
GitHub - SiluPanda/eval-dataset: Version-controlled eval dataset manager for LLM testing · GitHub
Skip to content
BranchesTags
Repository files navigation
eval-dataset
Version-controlled eval dataset manager for LLM testing.
npm versionnpm downloadslicensenode
eval-dataset manages the lifecycle of evaluation datasets for LLM testing. It loads, validates, splits, samples, deduplicates, and exports collections of test cases across formats (JSON, JSONL, CSV). All transformation methods return new immutable Dataset instances, all randomization is seeded for reproducibility, and the entire API is fully typed in TypeScript.
Every LLM evaluation framework expects test data -- inputs, expected outputs, context documents, and metadata -- but none of them manage the dataset itself. eval-dataset fills this gap by providing a single package that handles loading from multiple formats, splitting with reproducible seeded randomness, sampling with stratification, deduplicating with configurable similarity, validating schema completeness, and computing statistics. Zero external runtime dependencies.
Installation
npm install eval-dataset
Requires Node.js 18 or later.
Quick Start
import{createDataset,loadDataset}from'eval-dataset';// Create a dataset from test casesconstds=createDataset({name: 'qa-eval',version: '1.0.0',cases: [{id: '1',input: 'What is 2+2?',expected: '4',category: 'math',tags: ['arithmetic']},{id: '2',input: 'Capital of France?',expected: 'Paris',category: 'geography'},{id: '3',input: 'Who wrote Hamlet?',expected: 'Shakespeare',category: 'literature'},],});console.log(ds.size);// 3console.log(ds.categories());// ['math', 'geography', 'literature']// Split into train/test setsconstsplits=ds.split({ratios: {train: 0.7,test: 0.3},seed: 42});console.log(splits.train.size);// 2console.log(splits.test.size);// 1// Export to JSON Linesconstjsonl=ds.export('jsonl');// Load from a JSON stringconstds2=awaitloadDataset('[{"id":"1","input":"hello","expected":"world"}]',{name: 'loaded',format: 'json',});
Features
  • Immutable Dataset objects -- Every transformation method (filter, map, add, remove, split, sample, dedup) returns a new Dataset. The original is never modified.
  • Seeded randomization -- Splitting, sampling, and shuffling use a Mulberry32 PRNG with configurable seeds. The same seed always produces the same result.
  • Multi-format loading -- Load test cases from JSON arrays, JSON Lines, CSV strings, or in-memory TestCase[] arrays. Format auto-detection inspects content structure when not explicitly specified.
  • Multi-format export -- Export datasets to JSON (pretty or compact), JSON Lines, or CSV with configurable column order.
  • Splitting -- Random and stratified splitting into named partitions with configurable ratios. Stratified splits maintain proportional category representation in each partition.
  • Sampling -- Random and stratified sampling with configurable sample size. Supports sampling with replacement.
  • Deduplication -- Exact match, normalized match (case-insensitive, whitespace-collapsed), and near-duplicate detection via Jaccard token similarity.
  • Validation -- Detects empty inputs, duplicate IDs, and empty datasets.
  • Statistics -- Computes case counts, expected output coverage, context coverage, category and tag distributions, and input length statistics (min, max, mean).
  • Zero runtime dependencies -- Built entirely on Node.js built-ins. Only development dependencies are used for building and testing.
  • Full TypeScript support -- All public types, interfaces, and function signatures are exported with declaration files.
API Reference
createDataset(options)
Creates a new Dataset from the provided options.
functioncreateDataset(options: CreateOptions): Dataset;
Parameters:
ParameterTypeRequiredDefaultDescription
options.namestringYes--Name of the dataset
options.versionstringNo'0.1.0'Semver version string
options.casesTestCase[]No[]Initial test cases
Returns: A Dataset instance.
constds=createDataset({name: 'my-eval',version: '1.0.0',cases: [{id: '1',input: 'What is 2+2?',expected: '4',category: 'math'},],});
loadDataset(source, options?)
Loads a dataset from a string (JSON, JSONL, or CSV content) or an in-memory TestCase[] array. Returns a Promise<Dataset>.
functionloadDataset(source: string|TestCase[],options?: LoadOptions): Promise<Dataset>;
Parameters:
ParameterTypeRequiredDefaultDescription
sourcestring | TestCase[]Yes--Content string or array of test cases
options.format'json' | 'jsonl' | 'csv' | 'auto'No'auto'Format of the source string. Ignored when source is an array.
options.namestringNo'dataset'Dataset name
options.versionstringNo'0.1.0'Dataset version
When format is 'auto', the loader inspects the content to determine the format:
  • Strings starting with [ or { are parsed as JSON.
  • Strings where every non-empty line is a JSON object are parsed as JSONL.
  • All other strings are parsed as CSV.
// Load from JSON stringconstds=awaitloadDataset('[{"id":"1","input":"hello"}]',{name: 'test'});// Load from JSONL stringconstds2=awaitloadDataset('{"id":"1","input":"hello"}\n{"id":"2","input":"world"}',{name: 'test',format: 'jsonl'},);// Load from CSV stringconstds3=awaitloadDataset('id,input,expected,category\n1,Hello,World,test\n2,Foo,Bar,test',{name: 'test',format: 'csv'},);// Load from in-memory arrayconstds4=awaitloadDataset([{id: '1',input: 'hello',expected: 'world'}],{name: 'test'},);
Field mapping during loading:
When loading from JSON, JSONL, or CSV, the loader maps common field names to the internal TestCase schema:
  • input or question maps to input
  • expected maps to expected
  • category maps to category
  • difficulty is parsed as a number
  • context is parsed as a string array
  • tags is parsed as a string array (pipe-delimited | in CSV)
  • metadata is parsed as a JSON object
Test cases without an id are assigned an auto-generated 8-character UUID.
Dataset Interface
The Dataset interface represents a named, versioned, immutable collection of test cases. All transformation methods return new Dataset instances.
Properties:
PropertyTypeDescription
namestring (readonly)Dataset name
versionstring (readonly)Semver version string
casesreadonly TestCase[] (readonly)Frozen array of test cases
sizenumber (readonly)Number of test cases
dataset.filter(fn)
Returns a new Dataset containing only test cases for which the predicate returns true.
filter(fn: (tc: TestCase)=>boolean): Dataset;
constmathOnly=ds.filter((tc)=>tc.category==='math');constwithExpected=ds.filter((tc)=>tc.expected!==undefined);
dataset.map(fn)
Returns a new Dataset with each test case transformed by the provided function.
map(fn: (tc: TestCase)=>TestCase): Dataset;
constuppercased=ds.map((tc)=>({ ...tc,input: tc.input.toUpperCase()}));
dataset.add(tc)
Returns a new Dataset with the test case appended. If id is not provided, one is auto-generated. If input is not provided, it defaults to an empty string.
add(tc: Partial<TestCase>): Dataset;
constds2=ds.add({input: 'New question?',expected: 'New answer',category: 'general'});// ds2.size === ds.size + 1
dataset.remove(id)
Returns a new Dataset with the test case matching the given id removed.
remove(id: string): Dataset;
constds2=ds.remove('1');// ds2.has('1') === false
dataset.update(id, changes)
Returns a new Dataset with the test case matching id updated by merging the provided changes. The id field itself cannot be changed.
update(id: string,changes: Partial<TestCase>): Dataset;
constds2=ds.update('1',{expected: 'four',category: 'arithmetic'});// ds2.get('1')?.expected === 'four'// ds2.get('1')?.id === '1' (unchanged)
dataset.get(id)
Returns the test case with the given id, or undefined if not found.
get(id: string): TestCase|undefined;
dataset.has(id)
Returns true if a test case with the given id exists in the dataset.
has(id: string): boolean;
dataset.ids()
Returns an array of all test case IDs, in order.
ids(): string[];
dataset.categories()
Returns an array of unique category values across all test cases. Test cases without a category are excluded.
categories(): string[];
dataset.tagSet()
Returns an array of unique tags across all test cases.
tagSet(): string[];
dataset.slice(start, end?)
Returns a new Dataset with a positional slice of the cases array, using the same semantics as Array.prototype.slice.
slice(start: number,end?: number): Dataset;
constfirst10=ds.slice(0,10);constlastHalf=ds.slice(Math.floor(ds.size/2));
dataset.concat(other)
Returns a new Dataset merging cases from another dataset. Test cases from other whose IDs already exist in the current dataset are skipped (deduplication by ID).
concat(other: Dataset): Dataset;
constmerged=ds1.concat(ds2);
dataset.shuffle(seed?)
Returns a new Dataset with cases shuffled using the Mulberry32 seeded PRNG. Default seed is 42.
shuffle(seed?: number): Dataset;
constshuffled=ds.shuffle(123);// Same seed always produces the same orderconstshuffled2=ds.shuffle(123);// shuffled.ids() deep-equals shuffled2.ids()
dataset.split(config)
Splits the dataset into named, non-overlapping partitions. Returns a SplitResult (a Record<string, Dataset> keyed by partition name).
split(config: SplitConfig): SplitResult;
SplitConfig:
FieldTypeRequiredDefaultDescription
ratiosRecord<string, number>Yes--Partition names mapped to their ratios. Ratios are normalized to sum to 1.0.
mode'random' | 'stratified'No'random'Split mode
seednumberNo42PRNG seed for deterministic splits
stratifyBykeyof TestCaseNo'category'Field to stratify by (only used when mode is 'stratified')
Ratios do not need to sum to exactly 1.0 -- they are normalized automatically. For example, { train: 3, test: 1 } produces a 75/25 split.
// Random 80/20 splitconst{ train, test }=ds.split({ratios: {train: 0.8,test: 0.2},seed: 42,});// Three-way stratified split preserving category proportionsconstsplits=ds.split({ratios: {train: 0.7,val: 0.15,test: 0.15},mode: 'stratified',stratifyBy: 'category',seed: 42,});
dataset.sample(n, options?)
Returns a new Dataset containing n randomly selected test cases. When n exceeds the dataset size and replace is false, all cases are returned.
sample(n: number,options?: SampleOptions): Dataset;
SampleOptions:
FieldTypeRequiredDefaultDescription
mode'random' | 'stratified'No'random'Sampling mode
seednumberNo42PRNG seed for deterministic sampling
stratifyBystringNo'category'Field to stratify by (only used when mode is 'stratified')
replacebooleanNofalseWhether to sample with replacement
// Random sample of 20 casesconstsampled=ds.sample(20,{seed: 42});// Stratified sample preserving category proportionsconstsampled2=ds.sample(20,{mode: 'stratified',stratifyBy: 'category',seed: 42});// Sample with replacement (can return more than ds.size cases)constsampled3=ds.sample(100,{seed: 42,replace: true});
dataset.dedup(options?)
Returns a new Dataset with duplicate test cases removed.
dedup(options?: DedupOptions): Dataset;
DedupOptions:
FieldTypeRequiredDefaultDescription
mode'exact' | 'normalized' | 'jaccard'No'exact'Deduplication strategy
fieldstringNo'input'Field to compare for duplicates
thresholdnumberNo0.9Jaccard similarity threshold (only used when mode is 'jaccard')
keep'first' | 'last'No'first'Which occurrence to keep (only used for 'exact' and 'normalized' modes)
Deduplication modes:
  • exact -- Removes test cases with identical field values. Case-sensitive, whitespace-sensitive.
  • normalized -- Lowercases the value, trims whitespace, and collapses multiple spaces to a single space before comparing. "Hello World" and " hello world " are considered duplicates.
  • jaccard -- Tokenizes values by whitespace, computes Jaccard similarity (|A intersect B| / |A union B|), and treats pairs exceeding the threshold as duplicates. The first occurrence is kept.
// Exact dedup on the input fieldconstdeduped=ds.dedup();// Normalized dedup (case-insensitive, whitespace-collapsed)constdeduped2=ds.dedup({mode: 'normalized'});// Near-duplicate detection with Jaccard similarityconstdeduped3=ds.dedup({mode: 'jaccard',threshold: 0.85});// Dedup on a different field, keep last occurrenceconstdeduped4=ds.dedup({field: 'expected',keep: 'last'});
dataset.export(format, options?)
Serializes the dataset to a string in the specified format.
export(format: ExportFormat,options?: ExportOptions): string;
ExportFormat:'json' | 'jsonl' | 'csv'
ExportOptions:
FieldTypeRequiredDefaultDescription
prettybooleanNotruePretty-print JSON output with 2-space indentation
includeMetadatabooleanNotrue (JSON) / false (CSV)Include the metadata field in output
columnOrderstring[]No--Custom column order for CSV export
// Pretty-printed JSONconstjson=ds.export('json');// Compact JSONconstcompact=ds.export('json',{pretty: false});// JSON without metadataconstnoMeta=ds.export('json',{includeMetadata: false});// JSON Lines (one JSON object per line)constjsonl=ds.export('jsonl');// CSV with default column orderconstcsv=ds.export('csv');// CSV with custom column orderconstcsv2=ds.export('csv',{columnOrder: ['id','input','expected','category']});
CSV export details:
  • Array fields (tags, context) are serialized as pipe-delimited values.
  • Fields containing commas, quotes, or newlines are enclosed in double quotes with proper escaping.
  • Column order defaults to: id, input, expected, category, difficulty, tags, context, followed by any additional fields in alphabetical order.
dataset.stats()
Computes and returns statistics about the dataset.
stats(): DatasetStats;
DatasetStats:
FieldTypeDescription
totalCasesnumberTotal number of test cases
withExpectednumberNumber of cases with an expected value
withContextnumberNumber of cases with a non-empty context array
categoriesRecord<string, number>Category value to count mapping
tagsRecord<string, number>Tag to count mapping (across all cases)
inputLength{ min, max, mean }Input string length statistics
consts=ds.stats();// {// totalCases: 100,// withExpected: 85,// withContext: 30,// categories: { math: 40, reading: 60 },// tags: { hard: 20, easy: 50 },// inputLength: { min: 5, max: 200, mean: 42.3 }// }
For an empty dataset, inputLength returns { min: 0, max: 0, mean: 0 }.
dataset.validate()
Validates the dataset and returns a result with errors and warnings.
validate(): ValidationResult;
ValidationResult:
FieldTypeDescription
validbooleantrue if no errors were found
errorsArray<{ type, caseId?, message }>Validation errors
warningsArray<{ type, message }>Validation warnings
Detected errors:
  • missing_input -- A test case has an empty or whitespace-only input field.
  • duplicate_id -- Two or more test cases share the same id.
Detected warnings:
  • empty_dataset -- The dataset contains no test cases.
constresult=ds.validate();if(!result.valid){for(consterrofresult.errors){console.error(`[${err.type}] ${err.message}`);}}
dataset.toJSON()
Returns a plain JSON-serializable object representation of the dataset.
toJSON(): Record<string,unknown>;
The returned object contains name, version, cases (as a mutable array copy), and size.
constobj=ds.toJSON();// { name: 'qa-eval', version: '1.0.0', cases: [...], size: 100 }// Serialize to JSON stringconststr=JSON.stringify(ds.toJSON(),null,2);
TestCase Interface
The universal test case schema used throughout the package.
interfaceTestCase{id: string;input: string;expected?: string;context?: string[];metadata?: Record<string,unknown>;tags?: string[];difficulty?: number;category?: string;}
FieldTypeRequiredDescription
idstringYesUnique identifier. Auto-generated (8-character UUID prefix) if not provided when adding cases.
inputstringYesThe prompt, question, or query to send to the LLM
expectedstringNoExpected output / ground truth answer
contextstring[]NoContext documents for RAG evaluation
metadataRecord<string, unknown>NoArbitrary key-value metadata
tagsstring[]NoLabels for filtering and stratification
difficultynumberNoNumeric difficulty rating
categorystringNoPrimary classification label for stratification
Supporting Types
interfaceSplitConfig{ratios: Record<string,number>;mode?: 'random'|'stratified';seed?: number;stratifyBy?: keyofTestCase;}typeSplitResult=Record<string,Dataset>;interfaceSampleOptions{mode?: 'random'|'stratified';seed?: number;stratifyBy?: string;replace?: boolean;}interfaceDedupOptions{mode?: 'exact'|'normalized'|'jaccard';field?: string;threshold?: number;keep?: 'first'|'last';}typeExportFormat='json'|'jsonl'|'csv';interfaceExportOptions{pretty?: boolean;includeMetadata?: boolean;columnOrder?: string[];}interfaceDatasetStats{totalCases: number;withExpected: number;withContext: number;categories: Record<string,number>;tags: Record<string,number>;inputLength: {min: number;max: number;mean: number};}interfaceValidationResult{valid: boolean;errors: Array<{type: string;caseId?: string;message: string}>;warnings: Array<{type: string;message: string}>;}interfaceCreateOptions{name: string;version?: string;cases?: TestCase[];}interfaceLoadOptions{format?: 'json'|'jsonl'|'csv'|'auto';name?: string;version?: string;}
Configuration
Split Ratios
Split ratios are normalized automatically. The following are equivalent:
ds.split({ratios: {train: 0.8,test: 0.2}});ds.split({ratios: {train: 4,test: 1}});ds.split({ratios: {train: 80,test: 20}});
The last partition absorbs any rounding remainder to ensure all cases are assigned.
Seeded Randomization
All random operations default to seed 42. Pass an explicit seed to control the random sequence:
consta=ds.shuffle(1).ids();constb=ds.shuffle(1).ids();// a deep-equals bconstc=ds.shuffle(2).ids();// a does not deep-equal c
The Mulberry32 PRNG is used for all randomization. It produces deterministic results across platforms without relying on Math.random().
Error Handling
loadDataset throws standard JavaScript errors for invalid input:
  • SyntaxError -- When JSON or JSONL content is malformed.
  • Invalid CSV -- When the CSV string has fewer than 2 lines (no header + data), an empty array is returned rather than throwing.
dataset.validate() does not throw. It returns a ValidationResult object with structured errors and warnings that can be inspected programmatically:
constresult=ds.validate();if(!result.valid){result.errors.forEach((e)=>console.error(`${e.type}: ${e.message}`));}result.warnings.forEach((w)=>console.warn(`${w.type}: ${w.message}`));
Advanced Usage
Chaining Transformations
Because every method returns a new Dataset, transformations can be chained:
constresult=ds.filter((tc)=>tc.category==='math').dedup({mode: 'normalized'}).shuffle(42).sample(50,{seed: 7}).export('jsonl');
Building Datasets Incrementally
letds=createDataset({name: 'growing-eval',version: '1.0.0'});ds=ds.add({input: 'What is 2+2?',expected: '4',category: 'math'});ds=ds.add({input: 'Capital of France?',expected: 'Paris',category: 'geography'});ds=ds.add({input: 'Who wrote Hamlet?',expected: 'Shakespeare',category: 'literature'});console.log(ds.size);// 3
Reproducible Evaluation Pipelines
constds=awaitloadDataset(jsonString,{name: 'qa-eval',version: '2.0.0'});// Always produces the same train/test split for this datasetconst{ train, test }=ds.split({ratios: {train: 0.8,test: 0.2},seed: 42,});// Always selects the same 20 cases from the training setconstdevSample=train.sample(20,{seed: 7});
Cross-Format Round-Tripping
// Load from CSVconstds=awaitloadDataset(csvString,{name: 'test',format: 'csv'});// Export to JSON Linesconstjsonl=ds.export('jsonl');// Reload from JSON Linesconstds2=awaitloadDataset(jsonl,{name: 'test',format: 'jsonl'});// ds2 contains the same cases as ds
Merging Datasets
constds1=createDataset({name: 'batch-1',cases: firstBatch});constds2=createDataset({name: 'batch-2',cases: secondBatch});// Merge, deduplicating by IDconstmerged=ds1.concat(ds2);// Dedup by input contentconstclean=merged.dedup({mode: 'normalized'});
Stratified Splitting for Balanced Evaluation
constds=createDataset({name: 'eval',cases: [{id: '1',input: 'q1',category: 'math'},{id: '2',input: 'q2',category: 'math'},{id: '3',input: 'q3',category: 'reading'},{id: '4',input: 'q4',category: 'reading'},{id: '5',input: 'q5',category: 'coding'},{id: '6',input: 'q6',category: 'coding'},],});// Each split preserves the category distributionconstsplits=ds.split({ratios: {train: 0.67,test: 0.33},mode: 'stratified',stratifyBy: 'category',seed: 42,});
TypeScript
eval-dataset is written in TypeScript and ships with complete declaration files. All public types are exported from the package root:
importtype{TestCase,Dataset,SplitConfig,SplitResult,SampleOptions,DedupOptions,ExportFormat,ExportOptions,DatasetStats,ValidationResult,CreateOptions,LoadOptions,}from'eval-dataset';
The package targets ES2022 and uses CommonJS modules. TypeScript declaration maps are included for IDE navigation into source files.
License
MIT
About
Version-controlled eval dataset manager for LLM testing
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - SiluPanda/eval-dataset: Version-controlled eval dataset manager for LLM testing · GitHub
Skip to content
BranchesTags
Repository files navigation
eval-dataset
Version-controlled eval dataset manager for LLM testing.
npm versionnpm downloadslicensenode
eval-dataset manages the lifecycle of evaluation datasets for LLM testing. It loads, validates, splits, samples, deduplicates, and exports collections of test cases across formats (JSON, JSONL, CSV). All transformation methods return new immutable Dataset instances, all randomization is seeded for reproducibility, and the entire API is fully typed in TypeScript.
Every LLM evaluation framework expects test data -- inputs, expected outputs, context documents, and metadata -- but none of them manage the dataset itself. eval-dataset fills this gap by providing a single package that handles loading from multiple formats, splitting with reproducible seeded randomness, sampling with stratification, deduplicating with configurable similarity, validating schema completeness, and computing statistics. Zero external runtime dependencies.
Installation
npm install eval-dataset
Requires Node.js 18 or later.
Quick Start
import{createDataset,loadDataset}from'eval-dataset';// Create a dataset from test casesconstds=createDataset({name: 'qa-eval',version: '1.0.0',cases: [{id: '1',input: 'What is 2+2?',expected: '4',category: 'math',tags: ['arithmetic']},{id: '2',input: 'Capital of France?',expected: 'Paris',category: 'geography'},{id: '3',input: 'Who wrote Hamlet?',expected: 'Shakespeare',category: 'literature'},],});console.log(ds.size);// 3console.log(ds.categories());// ['math', 'geography', 'literature']// Split into train/test setsconstsplits=ds.split({ratios: {train: 0.7,test: 0.3},seed: 42});console.log(splits.train.size);// 2console.log(splits.test.size);// 1// Export to JSON Linesconstjsonl=ds.export('jsonl');// Load from a JSON stringconstds2=awaitloadDataset('[{"id":"1","input":"hello","expected":"world"}]',{name: 'loaded',format: 'json',});
Features
  • Immutable Dataset objects -- Every transformation method (filter, map, add, remove, split, sample, dedup) returns a new Dataset. The original is never modified.
  • Seeded randomization -- Splitting, sampling, and shuffling use a Mulberry32 PRNG with configurable seeds. The same seed always produces the same result.
  • Multi-format loading -- Load test cases from JSON arrays, JSON Lines, CSV strings, or in-memory TestCase[] arrays. Format auto-detection inspects content structure when not explicitly specified.
  • Multi-format export -- Export datasets to JSON (pretty or compact), JSON Lines, or CSV with configurable column order.
  • Splitting -- Random and stratified splitting into named partitions with configurable ratios. Stratified splits maintain proportional category representation in each partition.
  • Sampling -- Random and stratified sampling with configurable sample size. Supports sampling with replacement.
  • Deduplication -- Exact match, normalized match (case-insensitive, whitespace-collapsed), and near-duplicate detection via Jaccard token similarity.
  • Validation -- Detects empty inputs, duplicate IDs, and empty datasets.
  • Statistics -- Computes case counts, expected output coverage, context coverage, category and tag distributions, and input length statistics (min, max, mean).
  • Zero runtime dependencies -- Built entirely on Node.js built-ins. Only development dependencies are used for building and testing.
  • Full TypeScript support -- All public types, interfaces, and function signatures are exported with declaration files.
API Reference
createDataset(options)
Creates a new Dataset from the provided options.
functioncreateDataset(options: CreateOptions): Dataset;
Parameters:
ParameterTypeRequiredDefaultDescription
options.namestringYes--Name of the dataset
options.versionstringNo'0.1.0'Semver version string
options.casesTestCase[]No[]Initial test cases
Returns: A Dataset instance.
constds=createDataset({name: 'my-eval',version: '1.0.0',cases: [{id: '1',input: 'What is 2+2?',expected: '4',category: 'math'},],});
loadDataset(source, options?)
Loads a dataset from a string (JSON, JSONL, or CSV content) or an in-memory TestCase[] array. Returns a Promise<Dataset>.
functionloadDataset(source: string|TestCase[],options?: LoadOptions): Promise<Dataset>;
Parameters:
ParameterTypeRequiredDefaultDescription
sourcestring | TestCase[]Yes--Content string or array of test cases
options.format'json' | 'jsonl' | 'csv' | 'auto'No'auto'Format of the source string. Ignored when source is an array.
options.namestringNo'dataset'Dataset name
options.versionstringNo'0.1.0'Dataset version
When format is 'auto', the loader inspects the content to determine the format:
  • Strings starting with [ or { are parsed as JSON.
  • Strings where every non-empty line is a JSON object are parsed as JSONL.
  • All other strings are parsed as CSV.
// Load from JSON stringconstds=awaitloadDataset('[{"id":"1","input":"hello"}]',{name: 'test'});// Load from JSONL stringconstds2=awaitloadDataset('{"id":"1","input":"hello"}\n{"id":"2","input":"world"}',{name: 'test',format: 'jsonl'},);// Load from CSV stringconstds3=awaitloadDataset('id,input,expected,category\n1,Hello,World,test\n2,Foo,Bar,test',{name: 'test',format: 'csv'},);// Load from in-memory arrayconstds4=awaitloadDataset([{id: '1',input: 'hello',expected: 'world'}],{name: 'test'},);
Field mapping during loading:
When loading from JSON, JSONL, or CSV, the loader maps common field names to the internal TestCase schema:
  • input or question maps to input
  • expected maps to expected
  • category maps to category
  • difficulty is parsed as a number
  • context is parsed as a string array
  • tags is parsed as a string array (pipe-delimited | in CSV)
  • metadata is parsed as a JSON object
Test cases without an id are assigned an auto-generated 8-character UUID.
Dataset Interface
The Dataset interface represents a named, versioned, immutable collection of test cases. All transformation methods return new Dataset instances.
Properties:
PropertyTypeDescription
namestring (readonly)Dataset name
versionstring (readonly)Semver version string
casesreadonly TestCase[] (readonly)Frozen array of test cases
sizenumber (readonly)Number of test cases
dataset.filter(fn)
Returns a new Dataset containing only test cases for which the predicate returns true.
filter(fn: (tc: TestCase)=>boolean): Dataset;
constmathOnly=ds.filter((tc)=>tc.category==='math');constwithExpected=ds.filter((tc)=>tc.expected!==undefined);
dataset.map(fn)
Returns a new Dataset with each test case transformed by the provided function.
map(fn: (tc: TestCase)=>TestCase): Dataset;
constuppercased=ds.map((tc)=>({ ...tc,input: tc.input.toUpperCase()}));
dataset.add(tc)
Returns a new Dataset with the test case appended. If id is not provided, one is auto-generated. If input is not provided, it defaults to an empty string.
add(tc: Partial<TestCase>): Dataset;
constds2=ds.add({input: 'New question?',expected: 'New answer',category: 'general'});// ds2.size === ds.size + 1
dataset.remove(id)
Returns a new Dataset with the test case matching the given id removed.
remove(id: string): Dataset;
constds2=ds.remove('1');// ds2.has('1') === false
dataset.update(id, changes)
Returns a new Dataset with the test case matching id updated by merging the provided changes. The id field itself cannot be changed.
update(id: string,changes: Partial<TestCase>): Dataset;
constds2=ds.update('1',{expected: 'four',category: 'arithmetic'});// ds2.get('1')?.expected === 'four'// ds2.get('1')?.id === '1' (unchanged)
dataset.get(id)
Returns the test case with the given id, or undefined if not found.
get(id: string): TestCase|undefined;
dataset.has(id)
Returns true if a test case with the given id exists in the dataset.
has(id: string): boolean;
dataset.ids()
Returns an array of all test case IDs, in order.
ids(): string[];
dataset.categories()
Returns an array of unique category values across all test cases. Test cases without a category are excluded.
categories(): string[];
dataset.tagSet()
Returns an array of unique tags across all test cases.
tagSet(): string[];
dataset.slice(start, end?)
Returns a new Dataset with a positional slice of the cases array, using the same semantics as Array.prototype.slice.
slice(start: number,end?: number): Dataset;
constfirst10=ds.slice(0,10);constlastHalf=ds.slice(Math.floor(ds.size/2));
dataset.concat(other)
Returns a new Dataset merging cases from another dataset. Test cases from other whose IDs already exist in the current dataset are skipped (deduplication by ID).
concat(other: Dataset): Dataset;
constmerged=ds1.concat(ds2);
dataset.shuffle(seed?)
Returns a new Dataset with cases shuffled using the Mulberry32 seeded PRNG. Default seed is 42.
shuffle(seed?: number): Dataset;
constshuffled=ds.shuffle(123);// Same seed always produces the same orderconstshuffled2=ds.shuffle(123);// shuffled.ids() deep-equals shuffled2.ids()
dataset.split(config)
Splits the dataset into named, non-overlapping partitions. Returns a SplitResult (a Record<string, Dataset> keyed by partition name).
split(config: SplitConfig): SplitResult;
SplitConfig:
FieldTypeRequiredDefaultDescription
ratiosRecord<string, number>Yes--Partition names mapped to their ratios. Ratios are normalized to sum to 1.0.
mode'random' | 'stratified'No'random'Split mode
seednumberNo42PRNG seed for deterministic splits
stratifyBykeyof TestCaseNo'category'Field to stratify by (only used when mode is 'stratified')
Ratios do not need to sum to exactly 1.0 -- they are normalized automatically. For example, { train: 3, test: 1 } produces a 75/25 split.
// Random 80/20 splitconst{ train, test }=ds.split({ratios: {train: 0.8,test: 0.2},seed: 42,});// Three-way stratified split preserving category proportionsconstsplits=ds.split({ratios: {train: 0.7,val: 0.15,test: 0.15},mode: 'stratified',stratifyBy: 'category',seed: 42,});
dataset.sample(n, options?)
Returns a new Dataset containing n randomly selected test cases. When n exceeds the dataset size and replace is false, all cases are returned.
sample(n: number,options?: SampleOptions): Dataset;
SampleOptions:
FieldTypeRequiredDefaultDescription
mode'random' | 'stratified'No'random'Sampling mode
seednumberNo42PRNG seed for deterministic sampling
stratifyBystringNo'category'Field to stratify by (only used when mode is 'stratified')
replacebooleanNofalseWhether to sample with replacement
// Random sample of 20 casesconstsampled=ds.sample(20,{seed: 42});// Stratified sample preserving category proportionsconstsampled2=ds.sample(20,{mode: 'stratified',stratifyBy: 'category',seed: 42});// Sample with replacement (can return more than ds.size cases)constsampled3=ds.sample(100,{seed: 42,replace: true});
dataset.dedup(options?)
Returns a new Dataset with duplicate test cases removed.
dedup(options?: DedupOptions): Dataset;
DedupOptions:
FieldTypeRequiredDefaultDescription
mode'exact' | 'normalized' | 'jaccard'No'exact'Deduplication strategy
fieldstringNo'input'Field to compare for duplicates
thresholdnumberNo0.9Jaccard similarity threshold (only used when mode is 'jaccard')
keep'first' | 'last'No'first'Which occurrence to keep (only used for 'exact' and 'normalized' modes)
Deduplication modes:
  • exact -- Removes test cases with identical field values. Case-sensitive, whitespace-sensitive.
  • normalized -- Lowercases the value, trims whitespace, and collapses multiple spaces to a single space before comparing. "Hello World" and " hello world " are considered duplicates.
  • jaccard -- Tokenizes values by whitespace, computes Jaccard similarity (|A intersect B| / |A union B|), and treats pairs exceeding the threshold as duplicates. The first occurrence is kept.
// Exact dedup on the input fieldconstdeduped=ds.dedup();// Normalized dedup (case-insensitive, whitespace-collapsed)constdeduped2=ds.dedup({mode: 'normalized'});// Near-duplicate detection with Jaccard similarityconstdeduped3=ds.dedup({mode: 'jaccard',threshold: 0.85});// Dedup on a different field, keep last occurrenceconstdeduped4=ds.dedup({field: 'expected',keep: 'last'});
dataset.export(format, options?)
Serializes the dataset to a string in the specified format.
export(format: ExportFormat,options?: ExportOptions): string;
ExportFormat:'json' | 'jsonl' | 'csv'
ExportOptions:
FieldTypeRequiredDefaultDescription
prettybooleanNotruePretty-print JSON output with 2-space indentation
includeMetadatabooleanNotrue (JSON) / false (CSV)Include the metadata field in output
columnOrderstring[]No--Custom column order for CSV export
// Pretty-printed JSONconstjson=ds.export('json');// Compact JSONconstcompact=ds.export('json',{pretty: false});// JSON without metadataconstnoMeta=ds.export('json',{includeMetadata: false});// JSON Lines (one JSON object per line)constjsonl=ds.export('jsonl');// CSV with default column orderconstcsv=ds.export('csv');// CSV with custom column orderconstcsv2=ds.export('csv',{columnOrder: ['id','input','expected','category']});
CSV export details:
  • Array fields (tags, context) are serialized as pipe-delimited values.
  • Fields containing commas, quotes, or newlines are enclosed in double quotes with proper escaping.
  • Column order defaults to: id, input, expected, category, difficulty, tags, context, followed by any additional fields in alphabetical order.
dataset.stats()
Computes and returns statistics about the dataset.
stats(): DatasetStats;
DatasetStats:
FieldTypeDescription
totalCasesnumberTotal number of test cases
withExpectednumberNumber of cases with an expected value
withContextnumberNumber of cases with a non-empty context array
categoriesRecord<string, number>Category value to count mapping
tagsRecord<string, number>Tag to count mapping (across all cases)
inputLength{ min, max, mean }Input string length statistics
consts=ds.stats();// {// totalCases: 100,// withExpected: 85,// withContext: 30,// categories: { math: 40, reading: 60 },// tags: { hard: 20, easy: 50 },// inputLength: { min: 5, max: 200, mean: 42.3 }// }
For an empty dataset, inputLength returns { min: 0, max: 0, mean: 0 }.
dataset.validate()
Validates the dataset and returns a result with errors and warnings.
validate(): ValidationResult;
ValidationResult:
FieldTypeDescription
validbooleantrue if no errors were found
errorsArray<{ type, caseId?, message }>Validation errors
warningsArray<{ type, message }>Validation warnings
Detected errors:
  • missing_input -- A test case has an empty or whitespace-only input field.
  • duplicate_id -- Two or more test cases share the same id.
Detected warnings:
  • empty_dataset -- The dataset contains no test cases.
constresult=ds.validate();if(!result.valid){for(consterrofresult.errors){console.error(`[${err.type}] ${err.message}`);}}
dataset.toJSON()
Returns a plain JSON-serializable object representation of the dataset.
toJSON(): Record<string,unknown>;
The returned object contains name, version, cases (as a mutable array copy), and size.
constobj=ds.toJSON();// { name: 'qa-eval', version: '1.0.0', cases: [...], size: 100 }// Serialize to JSON stringconststr=JSON.stringify(ds.toJSON(),null,2);
TestCase Interface
The universal test case schema used throughout the package.
interfaceTestCase{id: string;input: string;expected?: string;context?: string[];metadata?: Record<string,unknown>;tags?: string[];difficulty?: number;category?: string;}
FieldTypeRequiredDescription
idstringYesUnique identifier. Auto-generated (8-character UUID prefix) if not provided when adding cases.
inputstringYesThe prompt, question, or query to send to the LLM
expectedstringNoExpected output / ground truth answer
contextstring[]NoContext documents for RAG evaluation
metadataRecord<string, unknown>NoArbitrary key-value metadata
tagsstring[]NoLabels for filtering and stratification
difficultynumberNoNumeric difficulty rating
categorystringNoPrimary classification label for stratification
Supporting Types
interfaceSplitConfig{ratios: Record<string,number>;mode?: 'random'|'stratified';seed?: number;stratifyBy?: keyofTestCase;}typeSplitResult=Record<string,Dataset>;interfaceSampleOptions{mode?: 'random'|'stratified';seed?: number;stratifyBy?: string;replace?: boolean;}interfaceDedupOptions{mode?: 'exact'|'normalized'|'jaccard';field?: string;threshold?: number;keep?: 'first'|'last';}typeExportFormat='json'|'jsonl'|'csv';interfaceExportOptions{pretty?: boolean;includeMetadata?: boolean;columnOrder?: string[];}interfaceDatasetStats{totalCases: number;withExpected: number;withContext: number;categories: Record<string,number>;tags: Record<string,number>;inputLength: {min: number;max: number;mean: number};}interfaceValidationResult{valid: boolean;errors: Array<{type: string;caseId?: string;message: string}>;warnings: Array<{type: string;message: string}>;}interfaceCreateOptions{name: string;version?: string;cases?: TestCase[];}interfaceLoadOptions{format?: 'json'|'jsonl'|'csv'|'auto';name?: string;version?: string;}
Configuration
Split Ratios
Split ratios are normalized automatically. The following are equivalent:
ds.split({ratios: {train: 0.8,test: 0.2}});ds.split({ratios: {train: 4,test: 1}});ds.split({ratios: {train: 80,test: 20}});
The last partition absorbs any rounding remainder to ensure all cases are assigned.
Seeded Randomization
All random operations default to seed 42. Pass an explicit seed to control the random sequence:
consta=ds.shuffle(1).ids();constb=ds.shuffle(1).ids();// a deep-equals bconstc=ds.shuffle(2).ids();// a does not deep-equal c
The Mulberry32 PRNG is used for all randomization. It produces deterministic results across platforms without relying on Math.random().
Error Handling
loadDataset throws standard JavaScript errors for invalid input:
  • SyntaxError -- When JSON or JSONL content is malformed.
  • Invalid CSV -- When the CSV string has fewer than 2 lines (no header + data), an empty array is returned rather than throwing.
dataset.validate() does not throw. It returns a ValidationResult object with structured errors and warnings that can be inspected programmatically:
constresult=ds.validate();if(!result.valid){result.errors.forEach((e)=>console.error(`${e.type}: ${e.message}`));}result.warnings.forEach((w)=>console.warn(`${w.type}: ${w.message}`));
Advanced Usage
Chaining Transformations
Because every method returns a new Dataset, transformations can be chained:
constresult=ds.filter((tc)=>tc.category==='math').dedup({mode: 'normalized'}).shuffle(42).sample(50,{seed: 7}).export('jsonl');
Building Datasets Incrementally
letds=createDataset({name: 'growing-eval',version: '1.0.0'});ds=ds.add({input: 'What is 2+2?',expected: '4',category: 'math'});ds=ds.add({input: 'Capital of France?',expected: 'Paris',category: 'geography'});ds=ds.add({input: 'Who wrote Hamlet?',expected: 'Shakespeare',category: 'literature'});console.log(ds.size);// 3
Reproducible Evaluation Pipelines
constds=awaitloadDataset(jsonString,{name: 'qa-eval',version: '2.0.0'});// Always produces the same train/test split for this datasetconst{ train, test }=ds.split({ratios: {train: 0.8,test: 0.2},seed: 42,});// Always selects the same 20 cases from the training setconstdevSample=train.sample(20,{seed: 7});
Cross-Format Round-Tripping
// Load from CSVconstds=awaitloadDataset(csvString,{name: 'test',format: 'csv'});// Export to JSON Linesconstjsonl=ds.export('jsonl');// Reload from JSON Linesconstds2=awaitloadDataset(jsonl,{name: 'test',format: 'jsonl'});// ds2 contains the same cases as ds
Merging Datasets
constds1=createDataset({name: 'batch-1',cases: firstBatch});constds2=createDataset({name: 'batch-2',cases: secondBatch});// Merge, deduplicating by IDconstmerged=ds1.concat(ds2);// Dedup by input contentconstclean=merged.dedup({mode: 'normalized'});
Stratified Splitting for Balanced Evaluation
constds=createDataset({name: 'eval',cases: [{id: '1',input: 'q1',category: 'math'},{id: '2',input: 'q2',category: 'math'},{id: '3',input: 'q3',category: 'reading'},{id: '4',input: 'q4',category: 'reading'},{id: '5',input: 'q5',category: 'coding'},{id: '6',input: 'q6',category: 'coding'},],});// Each split preserves the category distributionconstsplits=ds.split({ratios: {train: 0.67,test: 0.33},mode: 'stratified',stratifyBy: 'category',seed: 42,});
TypeScript
eval-dataset is written in TypeScript and ships with complete declaration files. All public types are exported from the package root:
importtype{TestCase,Dataset,SplitConfig,SplitResult,SampleOptions,DedupOptions,ExportFormat,ExportOptions,DatasetStats,ValidationResult,CreateOptions,LoadOptions,}from'eval-dataset';
The package targets ES2022 and uses CommonJS modules. TypeScript declaration maps are included for IDE navigation into source files.
License
MIT
About
Version-controlled eval dataset manager for LLM testing
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - SiluPanda/eval-dataset: Version-controlled eval dataset manager for LLM testing · GitHub
Skip to content
BranchesTags
Repository files navigation
eval-dataset
Version-controlled eval dataset manager for LLM testing.
npm versionnpm downloadslicensenode
eval-dataset manages the lifecycle of evaluation datasets for LLM testing. It loads, validates, splits, samples, deduplicates, and exports collections of test cases across formats (JSON, JSONL, CSV). All transformation methods return new immutable Dataset instances, all randomization is seeded for reproducibility, and the entire API is fully typed in TypeScript.
Every LLM evaluation framework expects test data -- inputs, expected outputs, context documents, and metadata -- but none of them manage the dataset itself. eval-dataset fills this gap by providing a single package that handles loading from multiple formats, splitting with reproducible seeded randomness, sampling with stratification, deduplicating with configurable similarity, validating schema completeness, and computing statistics. Zero external runtime dependencies.
Installation
npm install eval-dataset
Requires Node.js 18 or later.
Quick Start
import{createDataset,loadDataset}from'eval-dataset';// Create a dataset from test casesconstds=createDataset({name: 'qa-eval',version: '1.0.0',cases: [{id: '1',input: 'What is 2+2?',expected: '4',category: 'math',tags: ['arithmetic']},{id: '2',input: 'Capital of France?',expected: 'Paris',category: 'geography'},{id: '3',input: 'Who wrote Hamlet?',expected: 'Shakespeare',category: 'literature'},],});console.log(ds.size);// 3console.log(ds.categories());// ['math', 'geography', 'literature']// Split into train/test setsconstsplits=ds.split({ratios: {train: 0.7,test: 0.3},seed: 42});console.log(splits.train.size);// 2console.log(splits.test.size);// 1// Export to JSON Linesconstjsonl=ds.export('jsonl');// Load from a JSON stringconstds2=awaitloadDataset('[{"id":"1","input":"hello","expected":"world"}]',{name: 'loaded',format: 'json',});
Features
  • Immutable Dataset objects -- Every transformation method (filter, map, add, remove, split, sample, dedup) returns a new Dataset. The original is never modified.
  • Seeded randomization -- Splitting, sampling, and shuffling use a Mulberry32 PRNG with configurable seeds. The same seed always produces the same result.
  • Multi-format loading -- Load test cases from JSON arrays, JSON Lines, CSV strings, or in-memory TestCase[] arrays. Format auto-detection inspects content structure when not explicitly specified.
  • Multi-format export -- Export datasets to JSON (pretty or compact), JSON Lines, or CSV with configurable column order.
  • Splitting -- Random and stratified splitting into named partitions with configurable ratios. Stratified splits maintain proportional category representation in each partition.
  • Sampling -- Random and stratified sampling with configurable sample size. Supports sampling with replacement.
  • Deduplication -- Exact match, normalized match (case-insensitive, whitespace-collapsed), and near-duplicate detection via Jaccard token similarity.
  • Validation -- Detects empty inputs, duplicate IDs, and empty datasets.
  • Statistics -- Computes case counts, expected output coverage, context coverage, category and tag distributions, and input length statistics (min, max, mean).
  • Zero runtime dependencies -- Built entirely on Node.js built-ins. Only development dependencies are used for building and testing.
  • Full TypeScript support -- All public types, interfaces, and function signatures are exported with declaration files.
API Reference
createDataset(options)
Creates a new Dataset from the provided options.
functioncreateDataset(options: CreateOptions): Dataset;
Parameters:
ParameterTypeRequiredDefaultDescription
options.namestringYes--Name of the dataset
options.versionstringNo'0.1.0'Semver version string
options.casesTestCase[]No[]Initial test cases
Returns: A Dataset instance.
constds=createDataset({name: 'my-eval',version: '1.0.0',cases: [{id: '1',input: 'What is 2+2?',expected: '4',category: 'math'},],});
loadDataset(source, options?)
Loads a dataset from a string (JSON, JSONL, or CSV content) or an in-memory TestCase[] array. Returns a Promise<Dataset>.
functionloadDataset(source: string|TestCase[],options?: LoadOptions): Promise<Dataset>;
Parameters:
ParameterTypeRequiredDefaultDescription
sourcestring | TestCase[]Yes--Content string or array of test cases
options.format'json' | 'jsonl' | 'csv' | 'auto'No'auto'Format of the source string. Ignored when source is an array.
options.namestringNo'dataset'Dataset name
options.versionstringNo'0.1.0'Dataset version
When format is 'auto', the loader inspects the content to determine the format:
  • Strings starting with [ or { are parsed as JSON.
  • Strings where every non-empty line is a JSON object are parsed as JSONL.
  • All other strings are parsed as CSV.
// Load from JSON stringconstds=awaitloadDataset('[{"id":"1","input":"hello"}]',{name: 'test'});// Load from JSONL stringconstds2=awaitloadDataset('{"id":"1","input":"hello"}\n{"id":"2","input":"world"}',{name: 'test',format: 'jsonl'},);// Load from CSV stringconstds3=awaitloadDataset('id,input,expected,category\n1,Hello,World,test\n2,Foo,Bar,test',{name: 'test',format: 'csv'},);// Load from in-memory arrayconstds4=awaitloadDataset([{id: '1',input: 'hello',expected: 'world'}],{name: 'test'},);
Field mapping during loading:
When loading from JSON, JSONL, or CSV, the loader maps common field names to the internal TestCase schema:
  • input or question maps to input
  • expected maps to expected
  • category maps to category
  • difficulty is parsed as a number
  • context is parsed as a string array
  • tags is parsed as a string array (pipe-delimited | in CSV)
  • metadata is parsed as a JSON object
Test cases without an id are assigned an auto-generated 8-character UUID.
Dataset Interface
The Dataset interface represents a named, versioned, immutable collection of test cases. All transformation methods return new Dataset instances.
Properties:
PropertyTypeDescription
namestring (readonly)Dataset name
versionstring (readonly)Semver version string
casesreadonly TestCase[] (readonly)Frozen array of test cases
sizenumber (readonly)Number of test cases
dataset.filter(fn)
Returns a new Dataset containing only test cases for which the predicate returns true.
filter(fn: (tc: TestCase)=>boolean): Dataset;
constmathOnly=ds.filter((tc)=>tc.category==='math');constwithExpected=ds.filter((tc)=>tc.expected!==undefined);
dataset.map(fn)
Returns a new Dataset with each test case transformed by the provided function.
map(fn: (tc: TestCase)=>TestCase): Dataset;
constuppercased=ds.map((tc)=>({ ...tc,input: tc.input.toUpperCase()}));
dataset.add(tc)
Returns a new Dataset with the test case appended. If id is not provided, one is auto-generated. If input is not provided, it defaults to an empty string.
add(tc: Partial<TestCase>): Dataset;
constds2=ds.add({input: 'New question?',expected: 'New answer',category: 'general'});// ds2.size === ds.size + 1
dataset.remove(id)
Returns a new Dataset with the test case matching the given id removed.
remove(id: string): Dataset;
constds2=ds.remove('1');// ds2.has('1') === false
dataset.update(id, changes)
Returns a new Dataset with the test case matching id updated by merging the provided changes. The id field itself cannot be changed.
update(id: string,changes: Partial<TestCase>): Dataset;
constds2=ds.update('1',{expected: 'four',category: 'arithmetic'});// ds2.get('1')?.expected === 'four'// ds2.get('1')?.id === '1' (unchanged)
dataset.get(id)
Returns the test case with the given id, or undefined if not found.
get(id: string): TestCase|undefined;
dataset.has(id)
Returns true if a test case with the given id exists in the dataset.
has(id: string): boolean;
dataset.ids()
Returns an array of all test case IDs, in order.
ids(): string[];
dataset.categories()
Returns an array of unique category values across all test cases. Test cases without a category are excluded.
categories(): string[];
dataset.tagSet()
Returns an array of unique tags across all test cases.
tagSet(): string[];
dataset.slice(start, end?)
Returns a new Dataset with a positional slice of the cases array, using the same semantics as Array.prototype.slice.
slice(start: number,end?: number): Dataset;
constfirst10=ds.slice(0,10);constlastHalf=ds.slice(Math.floor(ds.size/2));
dataset.concat(other)
Returns a new Dataset merging cases from another dataset. Test cases from other whose IDs already exist in the current dataset are skipped (deduplication by ID).
concat(other: Dataset): Dataset;
constmerged=ds1.concat(ds2);
dataset.shuffle(seed?)
Returns a new Dataset with cases shuffled using the Mulberry32 seeded PRNG. Default seed is 42.
shuffle(seed?: number): Dataset;
constshuffled=ds.shuffle(123);// Same seed always produces the same orderconstshuffled2=ds.shuffle(123);// shuffled.ids() deep-equals shuffled2.ids()
dataset.split(config)
Splits the dataset into named, non-overlapping partitions. Returns a SplitResult (a Record<string, Dataset> keyed by partition name).
split(config: SplitConfig): SplitResult;
SplitConfig:
FieldTypeRequiredDefaultDescription
ratiosRecord<string, number>Yes--Partition names mapped to their ratios. Ratios are normalized to sum to 1.0.
mode'random' | 'stratified'No'random'Split mode
seednumberNo42PRNG seed for deterministic splits
stratifyBykeyof TestCaseNo'category'Field to stratify by (only used when mode is 'stratified')
Ratios do not need to sum to exactly 1.0 -- they are normalized automatically. For example, { train: 3, test: 1 } produces a 75/25 split.
// Random 80/20 splitconst{ train, test }=ds.split({ratios: {train: 0.8,test: 0.2},seed: 42,});// Three-way stratified split preserving category proportionsconstsplits=ds.split({ratios: {train: 0.7,val: 0.15,test: 0.15},mode: 'stratified',stratifyBy: 'category',seed: 42,});
dataset.sample(n, options?)
Returns a new Dataset containing n randomly selected test cases. When n exceeds the dataset size and replace is false, all cases are returned.
sample(n: number,options?: SampleOptions): Dataset;
SampleOptions:
FieldTypeRequiredDefaultDescription
mode'random' | 'stratified'No'random'Sampling mode
seednumberNo42PRNG seed for deterministic sampling
stratifyBystringNo'category'Field to stratify by (only used when mode is 'stratified')
replacebooleanNofalseWhether to sample with replacement
// Random sample of 20 casesconstsampled=ds.sample(20,{seed: 42});// Stratified sample preserving category proportionsconstsampled2=ds.sample(20,{mode: 'stratified',stratifyBy: 'category',seed: 42});// Sample with replacement (can return more than ds.size cases)constsampled3=ds.sample(100,{seed: 42,replace: true});
dataset.dedup(options?)
Returns a new Dataset with duplicate test cases removed.
dedup(options?: DedupOptions): Dataset;
DedupOptions:
FieldTypeRequiredDefaultDescription
mode'exact' | 'normalized' | 'jaccard'No'exact'Deduplication strategy
fieldstringNo'input'Field to compare for duplicates
thresholdnumberNo0.9Jaccard similarity threshold (only used when mode is 'jaccard')
keep'first' | 'last'No'first'Which occurrence to keep (only used for 'exact' and 'normalized' modes)
Deduplication modes:
  • exact -- Removes test cases with identical field values. Case-sensitive, whitespace-sensitive.
  • normalized -- Lowercases the value, trims whitespace, and collapses multiple spaces to a single space before comparing. "Hello World" and " hello world " are considered duplicates.
  • jaccard -- Tokenizes values by whitespace, computes Jaccard similarity (|A intersect B| / |A union B|), and treats pairs exceeding the threshold as duplicates. The first occurrence is kept.
// Exact dedup on the input fieldconstdeduped=ds.dedup();// Normalized dedup (case-insensitive, whitespace-collapsed)constdeduped2=ds.dedup({mode: 'normalized'});// Near-duplicate detection with Jaccard similarityconstdeduped3=ds.dedup({mode: 'jaccard',threshold: 0.85});// Dedup on a different field, keep last occurrenceconstdeduped4=ds.dedup({field: 'expected',keep: 'last'});
dataset.export(format, options?)
Serializes the dataset to a string in the specified format.
export(format: ExportFormat,options?: ExportOptions): string;
ExportFormat:'json' | 'jsonl' | 'csv'
ExportOptions:
FieldTypeRequiredDefaultDescription
prettybooleanNotruePretty-print JSON output with 2-space indentation
includeMetadatabooleanNotrue (JSON) / false (CSV)Include the metadata field in output
columnOrderstring[]No--Custom column order for CSV export
// Pretty-printed JSONconstjson=ds.export('json');// Compact JSONconstcompact=ds.export('json',{pretty: false});// JSON without metadataconstnoMeta=ds.export('json',{includeMetadata: false});// JSON Lines (one JSON object per line)constjsonl=ds.export('jsonl');// CSV with default column orderconstcsv=ds.export('csv');// CSV with custom column orderconstcsv2=ds.export('csv',{columnOrder: ['id','input','expected','category']});
CSV export details:
  • Array fields (tags, context) are serialized as pipe-delimited values.
  • Fields containing commas, quotes, or newlines are enclosed in double quotes with proper escaping.
  • Column order defaults to: id, input, expected, category, difficulty, tags, context, followed by any additional fields in alphabetical order.
dataset.stats()
Computes and returns statistics about the dataset.
stats(): DatasetStats;
DatasetStats:
FieldTypeDescription
totalCasesnumberTotal number of test cases
withExpectednumberNumber of cases with an expected value
withContextnumberNumber of cases with a non-empty context array
categoriesRecord<string, number>Category value to count mapping
tagsRecord<string, number>Tag to count mapping (across all cases)
inputLength{ min, max, mean }Input string length statistics
consts=ds.stats();// {// totalCases: 100,// withExpected: 85,// withContext: 30,// categories: { math: 40, reading: 60 },// tags: { hard: 20, easy: 50 },// inputLength: { min: 5, max: 200, mean: 42.3 }// }
For an empty dataset, inputLength returns { min: 0, max: 0, mean: 0 }.
dataset.validate()
Validates the dataset and returns a result with errors and warnings.
validate(): ValidationResult;
ValidationResult:
FieldTypeDescription
validbooleantrue if no errors were found
errorsArray<{ type, caseId?, message }>Validation errors
warningsArray<{ type, message }>Validation warnings
Detected errors:
  • missing_input -- A test case has an empty or whitespace-only input field.
  • duplicate_id -- Two or more test cases share the same id.
Detected warnings:
  • empty_dataset -- The dataset contains no test cases.
constresult=ds.validate();if(!result.valid){for(consterrofresult.errors){console.error(`[${err.type}] ${err.message}`);}}
dataset.toJSON()
Returns a plain JSON-serializable object representation of the dataset.
toJSON(): Record<string,unknown>;
The returned object contains name, version, cases (as a mutable array copy), and size.
constobj=ds.toJSON();// { name: 'qa-eval', version: '1.0.0', cases: [...], size: 100 }// Serialize to JSON stringconststr=JSON.stringify(ds.toJSON(),null,2);
TestCase Interface
The universal test case schema used throughout the package.
interfaceTestCase{id: string;input: string;expected?: string;context?: string[];metadata?: Record<string,unknown>;tags?: string[];difficulty?: number;category?: string;}
FieldTypeRequiredDescription
idstringYesUnique identifier. Auto-generated (8-character UUID prefix) if not provided when adding cases.
inputstringYesThe prompt, question, or query to send to the LLM
expectedstringNoExpected output / ground truth answer
contextstring[]NoContext documents for RAG evaluation
metadataRecord<string, unknown>NoArbitrary key-value metadata
tagsstring[]NoLabels for filtering and stratification
difficultynumberNoNumeric difficulty rating
categorystringNoPrimary classification label for stratification
Supporting Types
interfaceSplitConfig{ratios: Record<string,number>;mode?: 'random'|'stratified';seed?: number;stratifyBy?: keyofTestCase;}typeSplitResult=Record<string,Dataset>;interfaceSampleOptions{mode?: 'random'|'stratified';seed?: number;stratifyBy?: string;replace?: boolean;}interfaceDedupOptions{mode?: 'exact'|'normalized'|'jaccard';field?: string;threshold?: number;keep?: 'first'|'last';}typeExportFormat='json'|'jsonl'|'csv';interfaceExportOptions{pretty?: boolean;includeMetadata?: boolean;columnOrder?: string[];}interfaceDatasetStats{totalCases: number;withExpected: number;withContext: number;categories: Record<string,number>;tags: Record<string,number>;inputLength: {min: number;max: number;mean: number};}interfaceValidationResult{valid: boolean;errors: Array<{type: string;caseId?: string;message: string}>;warnings: Array<{type: string;message: string}>;}interfaceCreateOptions{name: string;version?: string;cases?: TestCase[];}interfaceLoadOptions{format?: 'json'|'jsonl'|'csv'|'auto';name?: string;version?: string;}
Configuration
Split Ratios
Split ratios are normalized automatically. The following are equivalent:
ds.split({ratios: {train: 0.8,test: 0.2}});ds.split({ratios: {train: 4,test: 1}});ds.split({ratios: {train: 80,test: 20}});
The last partition absorbs any rounding remainder to ensure all cases are assigned.
Seeded Randomization
All random operations default to seed 42. Pass an explicit seed to control the random sequence:
consta=ds.shuffle(1).ids();constb=ds.shuffle(1).ids();// a deep-equals bconstc=ds.shuffle(2).ids();// a does not deep-equal c
The Mulberry32 PRNG is used for all randomization. It produces deterministic results across platforms without relying on Math.random().
Error Handling
loadDataset throws standard JavaScript errors for invalid input:
  • SyntaxError -- When JSON or JSONL content is malformed.
  • Invalid CSV -- When the CSV string has fewer than 2 lines (no header + data), an empty array is returned rather than throwing.
dataset.validate() does not throw. It returns a ValidationResult object with structured errors and warnings that can be inspected programmatically:
constresult=ds.validate();if(!result.valid){result.errors.forEach((e)=>console.error(`${e.type}: ${e.message}`));}result.warnings.forEach((w)=>console.warn(`${w.type}: ${w.message}`));
Advanced Usage
Chaining Transformations
Because every method returns a new Dataset, transformations can be chained:
constresult=ds.filter((tc)=>tc.category==='math').dedup({mode: 'normalized'}).shuffle(42).sample(50,{seed: 7}).export('jsonl');
Building Datasets Incrementally
letds=createDataset({name: 'growing-eval',version: '1.0.0'});ds=ds.add({input: 'What is 2+2?',expected: '4',category: 'math'});ds=ds.add({input: 'Capital of France?',expected: 'Paris',category: 'geography'});ds=ds.add({input: 'Who wrote Hamlet?',expected: 'Shakespeare',category: 'literature'});console.log(ds.size);// 3
Reproducible Evaluation Pipelines
constds=awaitloadDataset(jsonString,{name: 'qa-eval',version: '2.0.0'});// Always produces the same train/test split for this datasetconst{ train, test }=ds.split({ratios: {train: 0.8,test: 0.2},seed: 42,});// Always selects the same 20 cases from the training setconstdevSample=train.sample(20,{seed: 7});
Cross-Format Round-Tripping
// Load from CSVconstds=awaitloadDataset(csvString,{name: 'test',format: 'csv'});// Export to JSON Linesconstjsonl=ds.export('jsonl');// Reload from JSON Linesconstds2=awaitloadDataset(jsonl,{name: 'test',format: 'jsonl'});// ds2 contains the same cases as ds
Merging Datasets
constds1=createDataset({name: 'batch-1',cases: firstBatch});constds2=createDataset({name: 'batch-2',cases: secondBatch});// Merge, deduplicating by IDconstmerged=ds1.concat(ds2);// Dedup by input contentconstclean=merged.dedup({mode: 'normalized'});
Stratified Splitting for Balanced Evaluation
constds=createDataset({name: 'eval',cases: [{id: '1',input: 'q1',category: 'math'},{id: '2',input: 'q2',category: 'math'},{id: '3',input: 'q3',category: 'reading'},{id: '4',input: 'q4',category: 'reading'},{id: '5',input: 'q5',category: 'coding'},{id: '6',input: 'q6',category: 'coding'},],});// Each split preserves the category distributionconstsplits=ds.split({ratios: {train: 0.67,test: 0.33},mode: 'stratified',stratifyBy: 'category',seed: 42,});
TypeScript
eval-dataset is written in TypeScript and ships with complete declaration files. All public types are exported from the package root:
importtype{TestCase,Dataset,SplitConfig,SplitResult,SampleOptions,DedupOptions,ExportFormat,ExportOptions,DatasetStats,ValidationResult,CreateOptions,LoadOptions,}from'eval-dataset';
The package targets ES2022 and uses CommonJS modules. TypeScript declaration maps are included for IDE navigation into source files.
License
MIT
About
Version-controlled eval dataset manager for LLM testing
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
GitHub - SiluPanda/eval-dataset: Version-controlled eval dataset manager for LLM testing · GitHub
Skip to content
BranchesTags
Repository files navigation
eval-dataset
Version-controlled eval dataset manager for LLM testing.
npm versionnpm downloadslicensenode
eval-dataset manages the lifecycle of evaluation datasets for LLM testing. It loads, validates, splits, samples, deduplicates, and exports collections of test cases across formats (JSON, JSONL, CSV). All transformation methods return new immutable Dataset instances, all randomization is seeded for reproducibility, and the entire API is fully typed in TypeScript.
Every LLM evaluation framework expects test data -- inputs, expected outputs, context documents, and metadata -- but none of them manage the dataset itself. eval-dataset fills this gap by providing a single package that handles loading from multiple formats, splitting with reproducible seeded randomness, sampling with stratification, deduplicating with configurable similarity, validating schema completeness, and computing statistics. Zero external runtime dependencies.
Installation
npm install eval-dataset
Requires Node.js 18 or later.
Quick Start
import{createDataset,loadDataset}from'eval-dataset';// Create a dataset from test casesconstds=createDataset({name: 'qa-eval',version: '1.0.0',cases: [{id: '1',input: 'What is 2+2?',expected: '4',category: 'math',tags: ['arithmetic']},{id: '2',input: 'Capital of France?',expected: 'Paris',category: 'geography'},{id: '3',input: 'Who wrote Hamlet?',expected: 'Shakespeare',category: 'literature'},],});console.log(ds.size);// 3console.log(ds.categories());// ['math', 'geography', 'literature']// Split into train/test setsconstsplits=ds.split({ratios: {train: 0.7,test: 0.3},seed: 42});console.log(splits.train.size);// 2console.log(splits.test.size);// 1// Export to JSON Linesconstjsonl=ds.export('jsonl');// Load from a JSON stringconstds2=awaitloadDataset('[{"id":"1","input":"hello","expected":"world"}]',{name: 'loaded',format: 'json',});
Features
  • Immutable Dataset objects -- Every transformation method (filter, map, add, remove, split, sample, dedup) returns a new Dataset. The original is never modified.
  • Seeded randomization -- Splitting, sampling, and shuffling use a Mulberry32 PRNG with configurable seeds. The same seed always produces the same result.
  • Multi-format loading -- Load test cases from JSON arrays, JSON Lines, CSV strings, or in-memory TestCase[] arrays. Format auto-detection inspects content structure when not explicitly specified.
  • Multi-format export -- Export datasets to JSON (pretty or compact), JSON Lines, or CSV with configurable column order.
  • Splitting -- Random and stratified splitting into named partitions with configurable ratios. Stratified splits maintain proportional category representation in each partition.
  • Sampling -- Random and stratified sampling with configurable sample size. Supports sampling with replacement.
  • Deduplication -- Exact match, normalized match (case-insensitive, whitespace-collapsed), and near-duplicate detection via Jaccard token similarity.
  • Validation -- Detects empty inputs, duplicate IDs, and empty datasets.
  • Statistics -- Computes case counts, expected output coverage, context coverage, category and tag distributions, and input length statistics (min, max, mean).
  • Zero runtime dependencies -- Built entirely on Node.js built-ins. Only development dependencies are used for building and testing.
  • Full TypeScript support -- All public types, interfaces, and function signatures are exported with declaration files.
API Reference
createDataset(options)
Creates a new Dataset from the provided options.
functioncreateDataset(options: CreateOptions): Dataset;
Parameters:
ParameterTypeRequiredDefaultDescription
options.namestringYes--Name of the dataset
options.versionstringNo'0.1.0'Semver version string
options.casesTestCase[]No[]Initial test cases
Returns: A Dataset instance.
constds=createDataset({name: 'my-eval',version: '1.0.0',cases: [{id: '1',input: 'What is 2+2?',expected: '4',category: 'math'},],});
loadDataset(source, options?)
Loads a dataset from a string (JSON, JSONL, or CSV content) or an in-memory TestCase[] array. Returns a Promise<Dataset>.
functionloadDataset(source: string|TestCase[],options?: LoadOptions): Promise<Dataset>;
Parameters:
ParameterTypeRequiredDefaultDescription
sourcestring | TestCase[]Yes--Content string or array of test cases
options.format'json' | 'jsonl' | 'csv' | 'auto'No'auto'Format of the source string. Ignored when source is an array.
options.namestringNo'dataset'Dataset name
options.versionstringNo'0.1.0'Dataset version
When format is 'auto', the loader inspects the content to determine the format:
  • Strings starting with [ or { are parsed as JSON.
  • Strings where every non-empty line is a JSON object are parsed as JSONL.
  • All other strings are parsed as CSV.
// Load from JSON stringconstds=awaitloadDataset('[{"id":"1","input":"hello"}]',{name: 'test'});// Load from JSONL stringconstds2=awaitloadDataset('{"id":"1","input":"hello"}\n{"id":"2","input":"world"}',{name: 'test',format: 'jsonl'},);// Load from CSV stringconstds3=awaitloadDataset('id,input,expected,category\n1,Hello,World,test\n2,Foo,Bar,test',{name: 'test',format: 'csv'},);// Load from in-memory arrayconstds4=awaitloadDataset([{id: '1',input: 'hello',expected: 'world'}],{name: 'test'},);
Field mapping during loading:
When loading from JSON, JSONL, or CSV, the loader maps common field names to the internal TestCase schema:
  • input or question maps to input
  • expected maps to expected
  • category maps to category
  • difficulty is parsed as a number
  • context is parsed as a string array
  • tags is parsed as a string array (pipe-delimited | in CSV)
  • metadata is parsed as a JSON object
Test cases without an id are assigned an auto-generated 8-character UUID.
Dataset Interface
The Dataset interface represents a named, versioned, immutable collection of test cases. All transformation methods return new Dataset instances.
Properties:
PropertyTypeDescription
namestring (readonly)Dataset name
versionstring (readonly)Semver version string
casesreadonly TestCase[] (readonly)Frozen array of test cases
sizenumber (readonly)Number of test cases
dataset.filter(fn)
Returns a new Dataset containing only test cases for which the predicate returns true.
filter(fn: (tc: TestCase)=>boolean): Dataset;
constmathOnly=ds.filter((tc)=>tc.category==='math');constwithExpected=ds.filter((tc)=>tc.expected!==undefined);
dataset.map(fn)
Returns a new Dataset with each test case transformed by the provided function.
map(fn: (tc: TestCase)=>TestCase): Dataset;
constuppercased=ds.map((tc)=>({ ...tc,input: tc.input.toUpperCase()}));
dataset.add(tc)
Returns a new Dataset with the test case appended. If id is not provided, one is auto-generated. If input is not provided, it defaults to an empty string.
add(tc: Partial<TestCase>): Dataset;
constds2=ds.add({input: 'New question?',expected: 'New answer',category: 'general'});// ds2.size === ds.size + 1
dataset.remove(id)
Returns a new Dataset with the test case matching the given id removed.
remove(id: string): Dataset;
constds2=ds.remove('1');// ds2.has('1') === false
dataset.update(id, changes)
Returns a new Dataset with the test case matching id updated by merging the provided changes. The id field itself cannot be changed.
update(id: string,changes: Partial<TestCase>): Dataset;
constds2=ds.update('1',{expected: 'four',category: 'arithmetic'});// ds2.get('1')?.expected === 'four'// ds2.get('1')?.id === '1' (unchanged)
dataset.get(id)
Returns the test case with the given id, or undefined if not found.
get(id: string): TestCase|undefined;
dataset.has(id)
Returns true if a test case with the given id exists in the dataset.
has(id: string): boolean;
dataset.ids()
Returns an array of all test case IDs, in order.
ids(): string[];
dataset.categories()
Returns an array of unique category values across all test cases. Test cases without a category are excluded.
categories(): string[];
dataset.tagSet()
Returns an array of unique tags across all test cases.
tagSet(): string[];
dataset.slice(start, end?)
Returns a new Dataset with a positional slice of the cases array, using the same semantics as Array.prototype.slice.
slice(start: number,end?: number): Dataset;
constfirst10=ds.slice(0,10);constlastHalf=ds.slice(Math.floor(ds.size/2));
dataset.concat(other)
Returns a new Dataset merging cases from another dataset. Test cases from other whose IDs already exist in the current dataset are skipped (deduplication by ID).
concat(other: Dataset): Dataset;
constmerged=ds1.concat(ds2);
dataset.shuffle(seed?)
Returns a new Dataset with cases shuffled using the Mulberry32 seeded PRNG. Default seed is 42.
shuffle(seed?: number): Dataset;
constshuffled=ds.shuffle(123);// Same seed always produces the same orderconstshuffled2=ds.shuffle(123);// shuffled.ids() deep-equals shuffled2.ids()
dataset.split(config)
Splits the dataset into named, non-overlapping partitions. Returns a SplitResult (a Record<string, Dataset> keyed by partition name).
split(config: SplitConfig): SplitResult;
SplitConfig:
FieldTypeRequiredDefaultDescription
ratiosRecord<string, number>Yes--Partition names mapped to their ratios. Ratios are normalized to sum to 1.0.
mode'random' | 'stratified'No'random'Split mode
seednumberNo42PRNG seed for deterministic splits
stratifyBykeyof TestCaseNo'category'Field to stratify by (only used when mode is 'stratified')
Ratios do not need to sum to exactly 1.0 -- they are normalized automatically. For example, { train: 3, test: 1 } produces a 75/25 split.
// Random 80/20 splitconst{ train, test }=ds.split({ratios: {train: 0.8,test: 0.2},seed: 42,});// Three-way stratified split preserving category proportionsconstsplits=ds.split({ratios: {train: 0.7,val: 0.15,test: 0.15},mode: 'stratified',stratifyBy: 'category',seed: 42,});
dataset.sample(n, options?)
Returns a new Dataset containing n randomly selected test cases. When n exceeds the dataset size and replace is false, all cases are returned.
sample(n: number,options?: SampleOptions): Dataset;
SampleOptions:
FieldTypeRequiredDefaultDescription
mode'random' | 'stratified'No'random'Sampling mode
seednumberNo42PRNG seed for deterministic sampling
stratifyBystringNo'category'Field to stratify by (only used when mode is 'stratified')
replacebooleanNofalseWhether to sample with replacement
// Random sample of 20 casesconstsampled=ds.sample(20,{seed: 42});// Stratified sample preserving category proportionsconstsampled2=ds.sample(20,{mode: 'stratified',stratifyBy: 'category',seed: 42});// Sample with replacement (can return more than ds.size cases)constsampled3=ds.sample(100,{seed: 42,replace: true});
dataset.dedup(options?)
Returns a new Dataset with duplicate test cases removed.
dedup(options?: DedupOptions): Dataset;
DedupOptions:
FieldTypeRequiredDefaultDescription
mode'exact' | 'normalized' | 'jaccard'No'exact'Deduplication strategy
fieldstringNo'input'Field to compare for duplicates
thresholdnumberNo0.9Jaccard similarity threshold (only used when mode is 'jaccard')
keep'first' | 'last'No'first'Which occurrence to keep (only used for 'exact' and 'normalized' modes)
Deduplication modes:
  • exact -- Removes test cases with identical field values. Case-sensitive, whitespace-sensitive.
  • normalized -- Lowercases the value, trims whitespace, and collapses multiple spaces to a single space before comparing. "Hello World" and " hello world " are considered duplicates.
  • jaccard -- Tokenizes values by whitespace, computes Jaccard similarity (|A intersect B| / |A union B|), and treats pairs exceeding the threshold as duplicates. The first occurrence is kept.
// Exact dedup on the input fieldconstdeduped=ds.dedup();// Normalized dedup (case-insensitive, whitespace-collapsed)constdeduped2=ds.dedup({mode: 'normalized'});// Near-duplicate detection with Jaccard similarityconstdeduped3=ds.dedup({mode: 'jaccard',threshold: 0.85});// Dedup on a different field, keep last occurrenceconstdeduped4=ds.dedup({field: 'expected',keep: 'last'});
dataset.export(format, options?)
Serializes the dataset to a string in the specified format.
export(format: ExportFormat,options?: ExportOptions): string;
ExportFormat:'json' | 'jsonl' | 'csv'
ExportOptions:
FieldTypeRequiredDefaultDescription
prettybooleanNotruePretty-print JSON output with 2-space indentation
includeMetadatabooleanNotrue (JSON) / false (CSV)Include the metadata field in output
columnOrderstring[]No--Custom column order for CSV export
// Pretty-printed JSONconstjson=ds.export('json');// Compact JSONconstcompact=ds.export('json',{pretty: false});// JSON without metadataconstnoMeta=ds.export('json',{includeMetadata: false});// JSON Lines (one JSON object per line)constjsonl=ds.export('jsonl');// CSV with default column orderconstcsv=ds.export('csv');// CSV with custom column orderconstcsv2=ds.export('csv',{columnOrder: ['id','input','expected','category']});
CSV export details:
  • Array fields (tags, context) are serialized as pipe-delimited values.
  • Fields containing commas, quotes, or newlines are enclosed in double quotes with proper escaping.
  • Column order defaults to: id, input, expected, category, difficulty, tags, context, followed by any additional fields in alphabetical order.
dataset.stats()
Computes and returns statistics about the dataset.
stats(): DatasetStats;
DatasetStats:
FieldTypeDescription
totalCasesnumberTotal number of test cases
withExpectednumberNumber of cases with an expected value
withContextnumberNumber of cases with a non-empty context array
categoriesRecord<string, number>Category value to count mapping
tagsRecord<string, number>Tag to count mapping (across all cases)
inputLength{ min, max, mean }Input string length statistics
consts=ds.stats();// {// totalCases: 100,// withExpected: 85,// withContext: 30,// categories: { math: 40, reading: 60 },// tags: { hard: 20, easy: 50 },// inputLength: { min: 5, max: 200, mean: 42.3 }// }
For an empty dataset, inputLength returns { min: 0, max: 0, mean: 0 }.
dataset.validate()
Validates the dataset and returns a result with errors and warnings.
validate(): ValidationResult;
ValidationResult:
FieldTypeDescription
validbooleantrue if no errors were found
errorsArray<{ type, caseId?, message }>Validation errors
warningsArray<{ type, message }>Validation warnings
Detected errors:
  • missing_input -- A test case has an empty or whitespace-only input field.
  • duplicate_id -- Two or more test cases share the same id.
Detected warnings:
  • empty_dataset -- The dataset contains no test cases.
constresult=ds.validate();if(!result.valid){for(consterrofresult.errors){console.error(`[${err.type}] ${err.message}`);}}
dataset.toJSON()
Returns a plain JSON-serializable object representation of the dataset.
toJSON(): Record<string,unknown>;
The returned object contains name, version, cases (as a mutable array copy), and size.
constobj=ds.toJSON();// { name: 'qa-eval', version: '1.0.0', cases: [...], size: 100 }// Serialize to JSON stringconststr=JSON.stringify(ds.toJSON(),null,2);
TestCase Interface
The universal test case schema used throughout the package.
interfaceTestCase{id: string;input: string;expected?: string;context?: string[];metadata?: Record<string,unknown>;tags?: string[];difficulty?: number;category?: string;}
FieldTypeRequiredDescription
idstringYesUnique identifier. Auto-generated (8-character UUID prefix) if not provided when adding cases.
inputstringYesThe prompt, question, or query to send to the LLM
expectedstringNoExpected output / ground truth answer
contextstring[]NoContext documents for RAG evaluation
metadataRecord<string, unknown>NoArbitrary key-value metadata
tagsstring[]NoLabels for filtering and stratification
difficultynumberNoNumeric difficulty rating
categorystringNoPrimary classification label for stratification
Supporting Types
interfaceSplitConfig{ratios: Record<string,number>;mode?: 'random'|'stratified';seed?: number;stratifyBy?: keyofTestCase;}typeSplitResult=Record<string,Dataset>;interfaceSampleOptions{mode?: 'random'|'stratified';seed?: number;stratifyBy?: string;replace?: boolean;}interfaceDedupOptions{mode?: 'exact'|'normalized'|'jaccard';field?: string;threshold?: number;keep?: 'first'|'last';}typeExportFormat='json'|'jsonl'|'csv';interfaceExportOptions{pretty?: boolean;includeMetadata?: boolean;columnOrder?: string[];}interfaceDatasetStats{totalCases: number;withExpected: number;withContext: number;categories: Record<string,number>;tags: Record<string,number>;inputLength: {min: number;max: number;mean: number};}interfaceValidationResult{valid: boolean;errors: Array<{type: string;caseId?: string;message: string}>;warnings: Array<{type: string;message: string}>;}interfaceCreateOptions{name: string;version?: string;cases?: TestCase[];}interfaceLoadOptions{format?: 'json'|'jsonl'|'csv'|'auto';name?: string;version?: string;}
Configuration
Split Ratios
Split ratios are normalized automatically. The following are equivalent:
ds.split({ratios: {train: 0.8,test: 0.2}});ds.split({ratios: {train: 4,test: 1}});ds.split({ratios: {train: 80,test: 20}});
The last partition absorbs any rounding remainder to ensure all cases are assigned.
Seeded Randomization
All random operations default to seed 42. Pass an explicit seed to control the random sequence:
consta=ds.shuffle(1).ids();constb=ds.shuffle(1).ids();// a deep-equals bconstc=ds.shuffle(2).ids();// a does not deep-equal c
The Mulberry32 PRNG is used for all randomization. It produces deterministic results across platforms without relying on Math.random().
Error Handling
loadDataset throws standard JavaScript errors for invalid input:
  • SyntaxError -- When JSON or JSONL content is malformed.
  • Invalid CSV -- When the CSV string has fewer than 2 lines (no header + data), an empty array is returned rather than throwing.
dataset.validate() does not throw. It returns a ValidationResult object with structured errors and warnings that can be inspected programmatically:
constresult=ds.validate();if(!result.valid){result.errors.forEach((e)=>console.error(`${e.type}: ${e.message}`));}result.warnings.forEach((w)=>console.warn(`${w.type}: ${w.message}`));
Advanced Usage
Chaining Transformations
Because every method returns a new Dataset, transformations can be chained:
constresult=ds.filter((tc)=>tc.category==='math').dedup({mode: 'normalized'}).shuffle(42).sample(50,{seed: 7}).export('jsonl');
Building Datasets Incrementally
letds=createDataset({name: 'growing-eval',version: '1.0.0'});ds=ds.add({input: 'What is 2+2?',expected: '4',category: 'math'});ds=ds.add({input: 'Capital of France?',expected: 'Paris',category: 'geography'});ds=ds.add({input: 'Who wrote Hamlet?',expected: 'Shakespeare',category: 'literature'});console.log(ds.size);// 3
Reproducible Evaluation Pipelines
constds=awaitloadDataset(jsonString,{name: 'qa-eval',version: '2.0.0'});// Always produces the same train/test split for this datasetconst{ train, test }=ds.split({ratios: {train: 0.8,test: 0.2},seed: 42,});// Always selects the same 20 cases from the training setconstdevSample=train.sample(20,{seed: 7});
Cross-Format Round-Tripping
// Load from CSVconstds=awaitloadDataset(csvString,{name: 'test',format: 'csv'});// Export to JSON Linesconstjsonl=ds.export('jsonl');// Reload from JSON Linesconstds2=awaitloadDataset(jsonl,{name: 'test',format: 'jsonl'});// ds2 contains the same cases as ds
Merging Datasets
constds1=createDataset({name: 'batch-1',cases: firstBatch});constds2=createDataset({name: 'batch-2',cases: secondBatch});// Merge, deduplicating by IDconstmerged=ds1.concat(ds2);// Dedup by input contentconstclean=merged.dedup({mode: 'normalized'});
Stratified Splitting for Balanced Evaluation
constds=createDataset({name: 'eval',cases: [{id: '1',input: 'q1',category: 'math'},{id: '2',input: 'q2',category: 'math'},{id: '3',input: 'q3',category: 'reading'},{id: '4',input: 'q4',category: 'reading'},{id: '5',input: 'q5',category: 'coding'},{id: '6',input: 'q6',category: 'coding'},],});// Each split preserves the category distributionconstsplits=ds.split({ratios: {train: 0.67,test: 0.33},mode: 'stratified',stratifyBy: 'category',seed: 42,});
TypeScript
eval-dataset is written in TypeScript and ships with complete declaration files. All public types are exported from the package root:
importtype{TestCase,Dataset,SplitConfig,SplitResult,SampleOptions,DedupOptions,ExportFormat,ExportOptions,DatasetStats,ValidationResult,CreateOptions,LoadOptions,}from'eval-dataset';
The package targets ES2022 and uses CommonJS modules. TypeScript declaration maps are included for IDE navigation into source files.
License
MIT
About
Version-controlled eval dataset manager for LLM testing
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages