Datavera

Datavera is a production-grade, open-source distributed data management platform built in Scala. It provides large-scale ingestion, processing, validation, and storage of structured and semi-structured data with a functional, type-safe architecture.

Features

• Configurable by design - All behavior externalized via HOCON configuration with dev/staging/production profiles
• Pluggable architecture - Extension points for ingestion connectors, transformations, storage backends, and auth providers
• Batch & streaming ingestion - CSV, JSON, Parquet files and Kafka streaming
• Functional pipelines - Composable transformations (map, filter, fold) defined in configuration
• Multi-backend storage - PostgreSQL for relational/metadata storage, S3-compatible object storage for large datasets
• REST API - http4s-based API with JWT authentication and role-based access control
• Observability - Structured logging, Prometheus metrics, OpenTelemetry tracing
• Production deployment - Docker containerization and Kubernetes manifests with HPA

Architecture

┌─────────────────────────────────────────────────────────────┐
│ API Layer (http4s) │
│ REST endpoints · Auth · RBAC │
├──────────────┬──────────────────┬───────────────────────────┤
│ Ingestion │ Processing │ Storage │
│ Connectors │ Pipeline Engine │ PostgreSQL · S3 │
│ Validation │ Transformations │ Metadata · Versioning │
├──────────────┴──────────────────┴───────────────────────────┤
│ Config · Observability · Plugin Registry │
└─────────────────────────────────────────────────────────────┘

Quick Start

Prerequisites

• JDK 17+
• sbt 1.10+
• PostgreSQL 16+ (optional, for persistent storage)

Build

sbt compile
sbt test
sbt "project app" stage

Run locally

export DATAVERA_ENV=dev
sbt "project app" run

The API will be available at http://localhost:8080/api/v1.

Docker

sbt "project app" stage
docker compose -f docker/docker-compose.yml up --build

Configuration

Configuration files live in modules/config/src/main/resources/:

File	Purpose
application.conf	Base production-safe defaults
application-dev.conf	Development overrides
application-staging.conf	Staging environment
application-prod.conf	Production with env var substitution

Set the active profile:

export DATAVERA_ENV=dev # dev | staging | prod

Key environment variables:

Variable	Description
DATAVERA_ENV	Active configuration profile
POSTGRES_HOST	PostgreSQL hostname
POSTGRES_PASSWORD	Database password
JWT_SECRET	JWT signing secret
KAFKA_BOOTSTRAP_SERVERS	Kafka broker addresses
S3_ENDPOINT	S3-compatible endpoint
OTEL_ENDPOINT	OpenTelemetry collector endpoint

API Endpoints

Method	Path	Description
GET	/api/v1/health	Health check
GET	/api/v1/datasets	List datasets
POST	/api/v1/datasets	Create dataset
GET	/api/v1/datasets/{id}	Get dataset metadata
POST	/api/v1/datasets/{id}/ingest	Ingest records
GET	/api/v1/datasets/{id}/query	Query dataset records
POST	/api/v1/pipelines/{id}/run	Run transformation pipeline
GET	/api/v1/metadata/pipelines	List configured pipelines

Authentication uses Bearer JWT tokens when datavera.auth.enabled = true.

Extending Datavera

Custom Ingestion Connector

Implement IngestionConnector[F] and register via configuration:

datavera.ingestion.connectors += {
id = "my-connector"type = "my-connector-type"enabled = true# custom settings...
}

Custom Transformation

Implement Transformation[F] and register in TransformationRegistry.

Custom Storage Backend

Implement StorageBackend[F] and configure in datavera.storage.backends.

Project Structure

datavera/
├── modules/
│ ├── core/ # Domain types, interfaces, plugin system
│ ├── config/ # HOCON configuration loading & validation
│ ├── ingestion/ # Batch & streaming connectors
│ ├── processing/ # Pipeline engine & transformations
│ ├── storage/ # PostgreSQL & S3 backends
│ ├── observability/ # Logging, metrics, tracing
│ ├── api/ # http4s REST API
│ ├── app/ # Application entry point
│ └── integration/ # Integration tests
├── docker/ # Dockerfile & docker-compose
├── k8s/ # Kubernetes manifests
└── build.sbt

Testing

sbt test# All unit tests
sbt "project integration"test# Integration tests (requires Docker)

Tests include:

• Unit tests for core logic and configuration validation
• Property-based tests with ScalaCheck for data model invariants
• Integration tests with Testcontainers (PostgreSQL, LocalStack)

Kubernetes Deployment

kubectl apply -f k8s/deployment.yaml

The deployment includes 3 replicas by default with Horizontal Pod Autoscaler (2-10 pods based on CPU).

License

Apache License 2.0 - See LICENSE for details.

Contributing

Contributions are welcome! Please read the architecture docs above and ensure:

1. All behavior is configurable, not hardcoded
2. New features implement the appropriate plugin interface
3. Tests cover critical paths
4. Configuration changes include validation