FastData

Description

FastData is a code generator that analyzes your data and creates high-performance, read-only data structures for static data with support for key/value and membership queries. It supports many output languages (C#, C++, Rust, etc.), ready for inclusion in your project with zero dependencies.

Download

Name	Link
Executable	GitHub Releases
C# source generator
C# library
.NET Tool
PowerShell

Getting started

Using the executable

1. Download the executable
2. Run FastData <lang> dogs.txt

<lang> can be one of rust, cpp, or csharp.

Using the .NET CLI tool

1. Install the Genbox.FastData.Cli tool: dotnet tool install --global Genbox.FastData.Cli
2. Run FastData <lang> dogs.txt

Using the PowerShell module

1. Install the PowerShell module: Install-Module -Name Genbox.FastData
2. Run Invoke-FastData -Language <lang> -InputFile dogs.txt

Using the .NET Source Generator

1. Add the Genbox.FastData.SourceGenerator package to your project
2. Add FastDataAttribute as an assembly level attribute.

usingGenbox.FastData.SourceGenerator.Attributes;[assembly:FastData<string>("Dogs",["Labrador","German Shepherd","Golden Retriever"])]internalstaticclassProgram{privatestaticvoidMain(){Console.WriteLine(Dogs.Contains("Labrador"));Console.WriteLine(Dogs.Contains("Beagle"));}}

Using the C# library

1. Add the Genbox.FastData.Generator.CSharp NuGet package to your project.
2. Use the FastDataGenerator.Generate() method:

internalstaticclassProgram{privatestaticvoidMain(){StringDataConfigconfig=newStringDataConfig();CSharpCodeGeneratorgenerator=newCSharpCodeGenerator(newCSharpCodeGeneratorConfig("Dogs"));stringsource=FastDataGenerator.Generate(["Labrador","German Shepherd","Golden Retriever"],config,generator);Console.WriteLine(source);}}

Why use FastData?

Generic data structures like arrays, hash tables, etc. are not optimized for your data. If you only need read-only access to a dataset, FastData can provide up to 14x better performance and less memory overhead.

Here is a classic example on just using an array:

string[]dogs=["Labrador","German Shepherd","Golden Retriever"];if(dogs.Contains("Beagle"))Console.WriteLine("It contains Beagle");

We know our data at compile-time, so why not let a program analyze it and come up with a better way?

FastData produces the following code:

internalstaticclassDogs{publicstaticboolContains(stringvalue){if((49280UL&(1UL<<(value.Length-1)))==0)returnfalse;switch(value){case"Labrador":case"German Shepherd":case"Golden Retriever":returntrue;default:returnfalse;}}publicconstintItemCount=3;publicconstintMinLength=8;publicconstintMaxLength=16;}

Benefits of the generated code:

• Early exit: A single-register bitmap of string lengths allows early termination for string lengths that cannot be in the set.
• Efficient lookups: A switch-based data structure which is faster for small datasets.

As a bonus, we also get some metadata about the dataset as constants, which, when used, allows for better code generation by optimizing compiler.

Features

• Data analysis: Optimizes the algorithms on the inherent properties of the dataset.
• Many data structures: FastData automatically chooses the best data structure for your data.
• Fast hashing: Strings are analyzed and the hash function is specially tailored to the data.
• Zero dependencies: The generated code has no dependencies, making it easy to integrate into your project.
• Minimal memory usage: The generated data structures are memory-efficient, using only the necessary amount of memory for the dataset.
• High-performance: The generated data structures are generated without unnecessary branching or virtualization making the compiler produce optimal code.
• Key/Value support: FastData can produce key/value lookup data structures

For more details about the data structures, see data structures.

FastData supports several output programming languages.

• C#: FastData csharp <input-file>
• C++: FastData cpp <input-file>
• Rust: FastData rust <input-file>

Each output language has different settings. Run FastData <lang> --help to see the options.

Common CLI options include:

• --key-type, -k: input key type. Defaults to string.
• --values-file, -v: input values file for key/value lookup generation.
• --value-type: input value type. Defaults to string.
• --structure-type, -s: force a structure instead of automatic selection.
• --ignore-case, -ic: enable ordinal-ignore-case lookups for string keys.
• --analysis-level: string-hash analysis level. Use disabled, fast, balanced, or aggressive.
• --allow-approximate: allow approximate membership lookups such as Bloom filters, which can return false positives.
• --required-functions: require generated support for membership, key-value-lookup, enumeration, or directaccess.
• --class-name, -cn: generated class or struct name.
• --output-file, -o: write generated source to a file instead of standard output.

C# output also supports --namespace, --class-visibility, and --class-type.

How FastData chooses a structure

FastData analyzes the input before emitting code. Numeric keys are analyzed for ranges, density, missing bits, and hash behavior. String keys are analyzed for length distribution, ASCII compatibility, edge-unit distributions, and optional string hash candidates.

The selected structure depends on that analysis and on the configured limits. Small sets often become direct conditionals or single-value checks, dense integer ranges can become range or bitset lookups, sparse integer sets can use succinct bit vectors, and larger or irregular sets fall back to hash-table variants.

For a deeper explanation, see how it works, data structures, and optimizations.

Benchmarks

A benchmark of .NET's Array, HashSet<T> and FrozenSet<T> versus FastData's auto-generated data structure really illustrates the difference.

Membership queries

Method	Categories	Mean	Factor
Array	InSet	6.5198 ns	-
HashSet	InSet	6.2191 ns	1.05x
FrozenSet	InSet	1.6010 ns	4.07x
FastData	InSet	0.9378 ns	6.95x
Array	NotInSet	7.4015 ns	-
HashSet	NotInSet	4.6013 ns	1.61x
FrozenSet	NotInSet	1.5816 ns	4.68x
FastData	NotInSet	0.5284 ns	14.01x

Keyed queries

Method	Categories	Mean	Factor
Dictionary	InSet	6.890 ns	-
FrozenDictionary	InSet	1.484 ns	4.64x
FastData	InSet	1.375 ns	5.01x
Dictionary	NotInSet	5.832 ns	-
FrozenDictionary	NotInSet	1.376 ns	4.24x
FastData	NotInSet	1.349 ns	4.32x

General FAQ

Does FastData use less memory than runtime structures?

Yes and no. For some data structures like Array, it uses the same amount of memory. For others, like HashTable, depending on the data, it can use considerably less memory.

Does it support key/value lookup?

Yes, you can specify key/value arrays as input data and FastData will generate a efficient key lookup function that returns a value.

Does it support case-insensitive lookups?

Yes. Set StringDataConfig.IgnoreCase = true (or IgnoreCase = true on the source generator attribute, or --ignore-case in the CLI) to use OrdinalIgnoreCase on ASCII string keys.

Does it support custom equality comparers?

No, not yet.

Are there any best practices for using FastData?

• Put the most often queried items first in the input data. It can speed up query speed for some data structures.
• Enable string analysis when using string keys to produce a more efficient hash function.

Can I use it for dynamic data?

No, FastData is designed for static data only. It generates code at compile time, so the data must be known beforehand.

C# FAQ

Why not use System.Collections.Frozen?

There are several reasons:

• Frozen comes with considerable runtime overhead
• Frozen is only available in .NET 8.0+
• Frozen only provides a few of the optimizations provided in FastData
• Frozen is only available in C#. FastData can produce data structures in many languages.