VortexLake

A high-performance columnar database built on Apache Vortex, enhanced with vector indexing capabilities for efficient RAG (Retrieval-Augmented Generation) workloads. Combines OLAP analytical processing with vector similarity search in a unified system.

Features

• OLAP-Compatible Storage: Built on Vortex columnar storage engine, optimized for analytical workloads
• Vector Enhancement: Advanced vector indexing (IVF-PQ, DiskANN) for similarity search
• Unified Data Model: Single system for both structured analytics and vector operations
• SQL Interface: Full SQL support via Apache DataFusion integration
• Python Bindings: Easy Python integration with PyO3
• Document Ingestion: Flexible document processing pipelines for RAG applications
• Hybrid Queries: Combine traditional SQL analytics with vector similarity search

Architecture

VortexLake is designed as a vector-enhanced columnar database that unifies OLAP analytics and vector similarity search:

┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Python API │ │ SQL Analytics │ │ REST/gRPC │
│ │ │ (DataFusion) │ │ Server │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │ │
└───────────────────────┼───────────────────────┘
│
┌─────────────────┐
│ Vector Index │
│ (IVF-PQ, │
│ DiskANN) │
└─────────────────┘
│
┌─────────────────┐
│ Columnar │
│ Storage │
│ (Vortex OLAP) │
└─────────────────┘

Key Design Principles:

• OLAP Foundation: Vortex provides the columnar storage optimized for analytical queries
• Vector Extension: Adds similarity search capabilities on top of OLAP storage
• Unified Schema: Single table can contain both analytical columns and vector columns
• Hybrid Queries: SQL queries can combine aggregations with vector similarity search

Quick Start

Installation

# Install from source
cargo build --release
# Or install Python package (when available)
pip install vortexlake

Basic Usage

use vortexlake_core::{VortexLake,Schema,Writer};use vortexlake_index::VectorIndex;// Create a new databaselet db = VortexLake::new("/path/to/database")?;let schema = Schema::new(vec![Field::new("id",DataType::Utf8,false),Field::new("vector",DataType::FixedSizeList(Arc::new(Field::new("item",DataType::Float32,false)),384),false),Field::new("text",DataType::Utf8,false),])?;// Create tables and indexes
db.create_table("documents", schema)?;
db.create_vector_index("documents","vector",IndexType::IVFPQ{num_clusters:1000,num_subvectors:32,})?;// Ingest dataletmut writer = db.writer("documents")?;
writer.write_batch(batch)?;
writer.commit()?;// Query datalet results = db.vector_search("documents", query_vector,10)?;

Python API

importvortexlake# Connect to databasedb=vortexlake.connect("/path/to/database")
# Create tabledb.create_table("documents", {
"id": "string",
"vector": f"fixed_size_list[float32; 384]",
"text": "string"
})
# Ingest documentsdocuments= [
{"id": "doc1", "vector": embedding1, "text": "Document content..."},
{"id": "doc2", "vector": embedding2, "text": "More content..."}
]
db.insert("documents", documents)
# Vector searchresults=db.vector_search("documents", query_embedding, top_k=10)

Project Structure

• crates/vortexlake-core/: Core storage engine with Vortex integration
• crates/vortexlake-index/: Vector indexing implementations (IVF-PQ, DiskANN)
• crates/vortexlake-sql/: SQL interface via DataFusion
• crates/vortexlake-ingest/: Document ingestion pipelines
• crates/vortexlake-server/: Optional gRPC/REST server
• crates/vortexlake-python/: Python bindings

Development

Prerequisites

• Rust nightly-2025-11-21 or later
• Python 3.8+ (for Python bindings)

Building

# Build all crates
cargo build
# Build with optimizations
cargo build --release
# Run tests
cargo test# Build Python extensioncd crates/vortexlake-python
maturin develop

Examples

See the examples/ directory for complete usage examples:

• rag_ingest.rs: Complete RAG ingestion pipeline
• hybrid_query.rs: Hybrid vector + keyword search
• benchmark_compress.rs: Compression benchmark

Performance

VortexLake inherits Vortex's OLAP-optimized performance while adding vector capabilities:

• OLAP Performance: Vortex columnar storage provides analytical query performance comparable to modern OLAP systems
• Vector Search: Sub-millisecond similarity search with IVF-PQ indexing
• Compression: Vortex's advanced compression (up to 10x better than traditional formats)
• Memory Efficiency: Columnar storage with lazy loading for both analytics and vectors
• Scalability: Handles billions of rows with mixed analytical and vector data
• Hybrid Workloads: Efficiently processes both traditional analytics and RAG applications

Contributing

Contributions are welcome! Please see our contributing guidelines and code of conduct.

License

Licensed under either of:

• Apache License, Version 2.0 (LICENSE-APACHE)
• MIT license (LICENSE-MIT)

at your option.