Hyper API for Rust

A pure-Rust implementation of the Hyper database API, using the PostgreSQL wire protocol with Hyper-specific extensions. Create, read, and manipulate Hyper database files (.hyper) without any C library dependencies.

Project Status — 0.4.x, AI-Assisted

This crate is AI-assisted but human-directed: much of the code was written by AI coding assistants under close review, with the design, architecture, and engineering trade-offs decided by an experienced developer. The pre-1.0 (0.x) line will probably undergo more large breaking changes; the public API won't settle until the 1.0.0 release.

Contributors and reviewers should, at a minimum, run an AI code reviewer over any changes, following the conventions, layering rules, and patterns captured in AGENTS.md (and the subdirectory hyperdb-api-node/AGENTS.md). Those files are the authoritative guidance for AI assistants working in this repository.

Key Features

• Pure Rust — no C library dependencies, standard cargo build
• High Performance — 22-24M rows/sec inserts, 18M rows/sec queries (100M row benchmark)
• Memory Safe — streaming by default, constant memory for billion-row results
• Dual Architecture — sync (Connection) and async (AsyncConnection) APIs
• Typed Row Mapping — #[derive(FromRow)] structs, including streaming stream_as for constant-memory typed queries
• Compile-time SQL Validation — opt-in query_as! macro checks SQL against your schema at build time (red squigglies in VS Code)
• Connection Pooling — async pooling via deadpool for high-concurrency applications
• Key-Value Store — string-native KvStore / AsyncKvStore backed by a single fixed table
• Arrow Integration — insert and read data in Arrow IPC stream format
• gRPC Transport — read-only access with Arrow IPC and load balancing support
• Full Type Support — all Hyper types including Numeric, Geography, Intervals
• Salesforce Auth — OAuth 2.0 and JWT Bearer Token flows for Data Cloud
• TLS — via rustls (always-on, pure Rust)
• Formal Verification — Kani proof harnesses for model-checked correctness

Quick Start

Build from Source

Install Rust via rustup.rs, install protoc for your platform, download the hyperd executable with make download-hyperd (bundled helper — see hyperdb-bootstrap), then build:

Platform	Install protoc	Build
macOS	brew install protobuf	make build
Linux (Debian/Ubuntu)	sudo apt-get install -y protobuf-compiler build-essential	make build
Linux (Fedora/RHEL)	sudo dnf install protobuf-compiler	make build
Windows	choco install protoc (also install VS Build Tools with the "Desktop development with C++" workload for the MSVC linker)	.\build.ps1 build

# Linux / macOS
make download-hyperd # downloads hyperd into .hyperd/current/ (first time only)
make build # or `make build-release` for optimized builds
make test# runs unit + integration tests, test-release for release build
make doc # builds the Hyper Rust documentation# Windows (PowerShell)
.\build.ps1 download-hyperd
.\build.ps1 build # or `.\build.ps1 build-release`
.\build.ps1 test# or test-release for release build
.\build.ps1 doc # builds the Hyper Rust documentation

The Makefile and build.ps1 wrappers auto-discover the downloaded hyperd at .hyperd/current/hyperd, and — if nothing is found on disk — auto-run download-hyperd the first time you invoke a target that actually needs hyperd (build, test, examples, doc). So make test from a clean checkout Just Works; subsequent runs are cache hits. If you already have a hyperd elsewhere, set HYPERD_PATH=/path/to/hyperd and the downloader stays inert — nothing is fetched, and no build step touches the network. Plain cargo build / cargo test also work as long as one of those is true.

See DEVELOPMENT.md for the full build guide including WSL, cross-compilation, benchmarks, and per-platform troubleshooting.

Installation

Add to your Cargo.toml:

[dependencies]
hyperdb-api = { path = "hyperdb-api" }

Installing the CLIs

hyperdb-mcp and hyperdb-bootstrap ship two ways:

Via npm (recommended for hyperdb-mcp; bundles a matching hyperd):

npm install -g hyperdb-mcp

Supported platforms: macOS ARM64 (Apple Silicon), Linux x64 (glibc), Windows x64. Intel macOS is built-from-source only at the moment — see the platform table in hyperdb-mcp/README.md.

Via crates.io (compiles from source; no bundled hyperd):

cargo install hyperdb-mcp
cargo install hyperdb-bootstrap

hyperdb-bootstrap will then download a compatible hyperd for you:

hyperdb-bootstrap download

Environment Setup

The hyperd executable (Hyper database server) must be available. The simplest path is:

make download-hyperd # or `.\build.ps1 download-hyperd` on Windows

This installs hyperd under .hyperd/current/ in the repo and is auto-discovered by the Makefile / build.ps1. If you already have a hyperd elsewhere, export HYPERD_PATH instead:

export HYPERD_PATH=/path/to/hyperd

Sync Example

use hyperdb_api::{Catalog,Connection,CreateMode,HyperProcess,Inserter,Result,SqlType,TableDefinition,};fnmain() -> Result<()>{let hyper = HyperProcess::new(None,None)?;let conn = Connection::new(&hyper,"example.hyper",CreateMode::CreateIfNotExists)?;// Create a tablelet table_def = TableDefinition::from("users").add_required_column("id",SqlType::int()).add_required_column("name",SqlType::text());Catalog::new(&conn).create_table(&table_def)?;// Insert data (COPY protocol, 22M+ rows/sec){letmut inserter = Inserter::new(&conn,&table_def)?;
inserter.add_row(&[&1i32,&"Alice"])?;
inserter.add_row(&[&2i32,&"Bob"])?;
inserter.execute()?;}// Query datalet result = conn.execute_query("SELECT * FROM users")?;for row in result.rows(){let row = row?;let id:Option<i32> = row.get(0);let name:Option<String> = row.get(1);println!("{:?} - {:?}", id, name);}Ok(())}

Async Example

use hyperdb_api::{AsyncConnection,CreateMode,HyperProcess,Result};#[tokio::main]asyncfnmain() -> Result<()>{let hyper = HyperProcess::new(None,None)?;let endpoint = hyper.require_endpoint()?;let conn = AsyncConnection::connect(
endpoint,"example_async.hyper",CreateMode::CreateIfNotExists,).await?;
conn.execute_command("CREATE TABLE users (id INT, name TEXT)").await?;
conn.execute_command("INSERT INTO users VALUES (1, 'Alice')").await?;
conn.close().await?;Ok(())}

Crate Overview

Crate	Purpose	Published
hyperdb-api	High-level API — connections, inserters, catalog, Arrow, pooling	crates.io
hyperdb-api-core	Internal implementation details (types, protocol, client). Not a public API — depend on hyperdb-api instead.	crates.io
hyperdb-api-salesforce	Salesforce Data Cloud OAuth authentication	crates.io
hyperdb-mcp	MCP server for LLM-driven SQL analytics on .hyper files	crates.io
sea-query-hyperdb	HyperDB dialect backend for sea-query	crates.io
hyperdb-api-node	Node.js/TypeScript bindings via napi-rs	npm
hyperdb-bootstrap	Download the hyperd executable from Tableau's release packages	crates.io

Examples

The API ships 14 examples in hyperdb-api/examples/ plus 2 companion crate examples.

Core Examples

Example	Description
insert_data_into_single_table	Create a table and insert data using Inserter
insert_data_into_multiple_tables	Multiple related tables
create_hyper_file_from_csv	Load CSV data into a Hyper table
delete_data_in_existing_hyper_file	Delete data with SQL DELETE
update_data_in_existing_hyper_file	Update data with SQL UPDATE
read_and_print_data_from_existing_hyper_file	Read table definitions and query data
insert_data_with_expressions	Column mappings with MappedInserter
insert_geospatial_data_to_a_hyper_file	Insert geospatial data

Rust-Specific Examples

Example	Description
arrow	Read/write Arrow RecordBatch data
async_usage	AsyncConnection and Tokio patterns
threaded_inserter	Multi-threaded bulk insertion with InsertChunk/ChunkSender
grpc_query	gRPC transport, Arrow IPC results
connection_pool	Async connection pooling with deadpool
transactions	RAII guards, multi-table rollback, DDL, reconnect semantics

Running Examples

export HYPERD_PATH=/path/to/hyperd
# Run individual examples
cargo run -p hyperdb-api --example insert_data_into_single_table
cargo run -p hyperdb-api --example arrow
cargo run -p hyperdb-api --example connection_pool
# Companion crate examples
cargo run -p sea-query-hyperdb --example basic_usage
cargo run -p hyperdb-api-salesforce --example salesforce_auth_example
# Run all examples
./run_all_examples.sh

Companion Crates

sea-query-hyperdb

HyperDB dialect backend for sea-query — use for window functions, CTEs, complex JOINs, and type-safe query composition:

[dependencies]
sea-query = "0.32"sea-query-hyperdb = { path = "sea-query-hyperdb" }

use sea_query::{Query,Expr,Iden};use sea_query_hyperdb::HyperQueryBuilder;let sql = Query::select().column(Users::Name).from(Users::Table).and_where(Expr::col(Users::Age).gt(18)).to_string(HyperQueryBuilder);let result = conn.fetch_all(&sql)?;

hyperdb-api-salesforce

Salesforce Data Cloud OAuth authentication — JWT Bearer Token, Username-Password, and Refresh Token flows:

[dependencies]
hyperdb-api-salesforce = { path = "hyperdb-api-salesforce" }

use hyperdb_api_salesforce::{SalesforceAuthConfig,AuthMode,SharedTokenProvider};let auth_config = SalesforceAuthConfig::new("https://login.salesforce.com","your-connected-app-consumer-key",)?
.auth_mode(AuthMode::private_key("user@example.com",&private_key_pem)?);let token_provider = SharedTokenProvider::new(auth_config)?;

See hyperdb-api-salesforce/README.md for full setup guide.

Node.js Bindings

The hyperdb-api-node package provides Node.js and TypeScript bindings built with napi-rs:

const{ HyperProcess, Connection, CreateMode }=require('hyperdb-api-node');consthyper=newHyperProcess();constconn=awaitConnection.connect(hyper.endpoint,'my.hyper',CreateMode.CreateAndReplace);// Tagged template literals — SQL injection safeconstrows=awaitconn.sql`SELECT * FROM users WHERE age > ${18}`;awaitconn.close();hyper.close();

See hyperdb-api-node/README.md for full documentation.

Platform Support

Platform	Status	Build Tool
Linux (x86_64)	Supported	make build
macOS (ARM & x64)	Supported	make build
Windows	Supported	.\build.ps1 build
WSL	Supported	make build

MSRV: Check rust-version in Cargo.toml.

Documentation

Resource	Description
hyperdb-api/README.md	Full user guide for the hyperdb-api crate
docs/WHATS_NEW_0.4.md	Highlights of the 0.4.0 release
docs/ROW_MAPPING.md	The five ways to map result rows into Rust values
hyperdb-api-derive/README.md	#[derive(FromRow)], #[derive(Table)], and compile-time SQL validation
DEVELOPMENT.md	Architecture, building, testing, benchmarks — for contributors
CONTRIBUTING.md	How to contribute
docs/GITHUB_OPERATIONS.md	CI/release workflows and how maintainers cut a release
docs/TRANSACTIONS.md	Transaction API design
docs/BENCHMARK_GUIDE.md	How to run benchmarks

Per-crate documentation: each crate has its own README.md (see Crate Overview).

Generate API docs locally:

make doc # or: cargo doc --no-deps --open

Contributing

See CONTRIBUTING.md for the governance model, contribution checklist, commit message format, and pull request process.

Acknowledgments

This project includes code adapted from sfackler/rust-postgres (the postgres-protocol, tokio-postgres, and postgres-types crates by Steven Fackler, MIT or Apache-2.0). See NOTICE for the full third-party attribution list and the upstream license text.

License

Licensed under either of MIT or Apache-2.0 at your option.