Chitral is a local semantic-layer + synthetic-data workbench: connect a database, import tables onto a canvas, design relationships, and generate privacy-preserving synthetic data from statistical profiles.
Chitral supports two complementary workflows:
- Relational synthesis — generate deterministic, relationship-valid snapshots from existing tables.
- Temporal and journey synthesis — optionally describe how table rows represent events, constrain their timestamps, and model probabilistic customer journeys without requiring Kafka or a separate event store.
| Area | What Chitral provides |
|---|---|
| Visual semantic canvas | Import or create datasets, inspect fields, draw relationships, edit metadata, search, filter by source, auto-layout the graph, and focus on upstream/downstream lineage. |
| Database profiling | Learn aggregate column distributions, null and distinct counts, quantiles, string patterns, boolean rates, JSON structure, uniqueness, and relationship fan-out. |
| Deterministic synthesis | A fixed seed produces reproducible output. Generation streams in bounded batches and does not train a model or retain all output in memory. |
| Relationship integrity | Preserve primary keys, unique values, foreign keys, child-per-parent fan-out, childless-parent rates, and constraint columns copied from the selected parent row. |
| Fidelity controls | Configure ordering rules, composite unique keys, discriminator-based distributions, correlated fan-out, cross-table conditioning, and optional LLM-assisted JSON templates. |
| Quality gates | Evaluate schema, row counts, distributions, null rates, uniqueness, foreign keys, ordering, temporal rules, transitions, delays, correlation, and privacy before accepting output. |
| Temporal datasets | Map events to tables or discriminator values and enforce constraints such as within, not-before, not-after, and same-or-after. |
| Customer journeys | Define actor mappings, occurrence bounds, transition probabilities, exclusive branches, termination, bounded repeats, delays, and correlation behavior. |
| Versioned generation plans | Create mutable drafts, validate them, publish immutable versions, preview bounded samples, and execute the exact compiled version. |
| Durable job control | Submit asynchronous work, monitor stages, inspect privacy-safe events, cancel cooperatively, enforce deadlines and capacity, and retry deterministically. |
| Portable and Lakehouse output | Download CSV or Parquet. Optionally write supported database sinks or publish to Apache Iceberg on S3 through a registered REST catalog. |
Connect source
-> Import tables and fields
-> Review or design relationships on the canvas
-> Configure synthesis fidelity rules
-> Profile aggregate statistics
-> Generate deterministic relational data
-> Apply optional temporal and journey rules
-> Run quality gates
-> Download or publish accepted output
Security posture — read this first
Chitral is a local, single-user developer tool. There is no authentication. The API trusts whoever can reach it. Do not expose the backend to the public internet without putting it behind your own auth/gateway.
Saved data-source credentials are encrypted at rest (Fernet) in
backend/chitral.db. List/read APIs never return passwords or raw config— only a non-secretconfig_summary. ProtectDATASOURCE_ENCRYPTION_KEY(orbackend/.datasource_key); anyone with both the DB file and the key can decrypt. Treat the machine as a trusted workstation.If you don't set DATASOURCE_ENCRYPTION_KEY, a key is auto-generated at backend/.datasource_key. Saved datasource credentials are not portable across machines or fresh clones unless you pin that env var (or copy the key file along with the database).
| Tool | Version |
|---|---|
| Python | ≥ 3.12 |
| uv | latest |
| Node.js | ≥ 18 |
| npm | ≥ 9 |
| Docker (optional) | for connector integration-test DBs |
curl -LsSf https://astral.sh/uv/install.sh | sh# Install dependencies
make install
# equivalent to:# cd backend && uv sync --extra connectors# cd frontend && npm ci# Terminal 1 — API (http://localhost:8000)cd backend
uv run run.py
# Terminal 2 — UI (http://localhost:5173)cd frontend
npm run devOpen http://localhost:5173. Interactive API docs: http://localhost:8000/docs.
| Service | Default URL |
|---|---|
| Frontend (Vite) | http://localhost:5173 |
| Backend API | http://localhost:8000 |
| Health check | http://localhost:8000/health |
The relational workflow is the default and does not require any temporal or journey feature flags.
- Select Connect Source in the canvas toolbar.
- Choose SQLite, PostgreSQL, MySQL, MongoDB, or Snowflake.
- Enter the connection information and test or browse the source.
- Select the tables or collections to import.
- Import them onto the canvas.
- Save the connection with the Source role when prompted. Generation from the UI uses a saved source so credentials stay server-side.
Imported tables become dataset nodes with their fields and source identity. Relational imports are atomic: if a required part of the import fails, Chitral rolls the import back instead of leaving a partial canvas.
Use the canvas to verify the imported model before generating data:
- Drag from one dataset to another to create a relationship.
- Select the relationship type and map one or more column pairs.
- Mark a pair as a join key to preserve FK integrity or as a constraint to copy the value from the exact selected parent row.
- Open a relationship to enable correlated fan-out or select a parent conditioning column.
- Open a dataset's Edit metadata action to configure ordering rules, composite unique keys, or a discriminator column.
- Use Show Lineage to inspect upstream and downstream dependencies.
- Use search, source filters, and auto-layout when working with a larger canvas.
Chitral generates related tables in dependency order. Unsupported cycles are rejected by the strict workflow; the compatibility workflow can retain its legacy input-order behavior.
- Select Generate Synthetic Data.
- Choose the saved source that contains the imported tables.
- Select the tables to generate. The server-advertised table limit is shown in the panel.
- Optionally enter an exact output row count for each table. If omitted, Chitral uses the profiled row count multiplied by the scale factor.
- Review the readiness summary for relationships, constraints, conditioning, and boundary warnings.
- Choose Download only or a supported saved Sink connection.
- Open Advanced to set the scale factor, deterministic seed, relationship preservation, and optional JSON-template settings.
- Select Generate Data.
If only one side of a relationship is selected, Chitral warns about the boundary. A missing referenced parent falls back to value generation for that FK rather than silently pretending the relationship was preserved.
The generation panel reports the current job stage and any warnings. When the job succeeds, download the result as:
- Parquet ZIP for efficient typed data exchange; or
- CSV ZIP for broad tool compatibility.
Downloads are assembled from disk-backed Parquet batches and remain available until the local artifact TTL expires. When a database sink is selected, Chitral also writes the generated tables after generation; a sink warning does not remove the downloadable result.
Temporal and journey synthesis are implemented but disabled by default. They are server-controlled capabilities intended for an explicitly enabled local or isolated deployment.
| Mode | Use it when |
|---|---|
| Relational snapshot | You need statistically similar tables with keys and relationships preserved. |
| Temporal dataset | Related rows must obey timestamp constraints, such as an order occurring after account creation. |
| Journey scenario | Physical tables or discriminator rows represent logical events connected by probabilities, delays, occurrence rules, and actor identity. |
Journey modeling is an overlay on the relational canvas. Relational edges continue to represent physical data integrity; journey edges separately represent event transitions. A journey can therefore produce ordinary database or Lakehouse tables—Kafka is not required.
Set only the capabilities you intend to evaluate in backend/.env, then restart the backend:
GENERATION_SPECS_ENABLED=trueTEMPORAL_GENERATION_ENABLED=trueJOURNEY_GENERATION_ENABLED=trueGENERATION_PREVIEW_ENABLED=true# Keep disabled unless a registered Iceberg REST/S3 destination has passed# the deployment-specific atomicity and security gates.LAKEHOUSE_PUBLICATION_ENABLED=falseThe frontend reads the server's /api/capabilities response. Hiding or showing a UI control is not the security boundary; the backend independently enforces every capability.
- Import the physical tables and verify their PK/FK relationships.
- Open Journey and create a specification from the current canvas.
- Choose Relational snapshot, Temporal dataset, or Journey scenario.
- For temporal or journey modes, define logical events:
- map an event to a dedicated table or a discriminator value;
- select its timestamp column;
- map actor keys through exactly one directed relational path;
- optionally select correlation columns and occurrence bounds.
- Add temporal rules and choose the null behavior for missing anchors.
- For a journey, add transitions, probabilities, delays, exclusive groups, termination behavior, and bounded repeats.
- Save the draft and resolve any revision conflict explicitly.
- Select Validate and fix every blocking diagnostic.
- Select Publish exact revision. Published versions are immutable.
- Generate a bounded preview and review samples, UTC histograms, resource projections, diagnostics, and the quality report.
- Select a saved source and start the job.
- Monitor its stage, cancel if necessary, and download only when publication is accepted.
Preview and full execution use the same immutable compiled plan. Seeds, logical IDs, transition choices, occurrence counts, delays, and temporal values are independent of batch boundaries and scheduling, making retries reproducible.
For the focused workflow and operating limits, see Journey and Temporal Generation User Guide.
Chitral's current engine is profile-driven. It queries aggregate statistics and generates new values from those profiles; it does not train a generative model or simply resample complete source rows.
The engine supports:
- deterministic numeric, categorical, boolean, date/time, string, UUID, and JSON generation;
- collision-free primary and unique values by construction;
- exact or quantile-based relationship fan-out, including childless parents;
- top-level JSON key presence, type mixtures, and value distributions;
- optional bounded LLM template pools for selected JSON columns;
- per-table discriminator groups for row-coherent conditional distributions;
- cross-table conditioning based on one declared parent attribute;
- ordering constraints such as
created_at <= completed_at; - composite unique tuples generated without retaining an unbounded seen-set; and
- independent quality collectors that verify the generated result rather than trusting the generator.
- Columns without an explicit discriminator, ordering, conditioning, or relationship rule are generated independently.
- Cross-table conditioning uses one declared parent attribute rather than the parent's entire row jointly.
- Composite keys across tables and multiple discriminator columns are not modeled.
- JSON fidelity is strongest at the top level; nested values preserve discovered structure and type but not full nested distributions.
- Empty tables can be imported, but generation requires at least one source row for every generated table.
| Destination | Support | Behavior |
|---|---|---|
| CSV ZIP | Supported | Generated from the accepted disk-backed Parquet artifact. |
| Parquet ZIP | Supported | Default typed, portable output; supports bounded multipart generation. |
| SQLite sink | Supported | Same-named tables are replaced after generation. |
| Snowflake sink | Supported | Uses batched INSERT; staged bulk copy is not yet implemented. |
| PostgreSQL/MySQL/MongoDB sink | Not yet | Use downloadable output instead. |
| Iceberg REST catalog + S3 | Implemented, disabled by default | Supports atomic append and replace-snapshot bundles for existing tables through one catalog transaction. |
Accepted Parquet publication uses private staging, schema and checksum validation, a signed quality attestation, an immutable manifest, and atomic promotion. Retries resolve to the same deterministic manifest instead of creating duplicate output.
Iceberg table creation remains disabled unless the exact registered catalog and version proves atomic multi-table creation. Chitral never falls back to sequential per-table creation because that could expose a partial journey bundle. See Iceberg REST/S3 Output Architecture.
| Connector | Source (profile / import) | Sink (write synthetic tables) | Automated tests |
|---|---|---|---|
| SQLite | Supported | Supported | Strong unit tests |
| Snowflake | Supported | Supported (batched INSERT; no staged COPY yet) | Unit / mocked only — no live integration suite in CI |
| PostgreSQL | Supported | Not yet (download-only; 422 if selected as sink) | Unit + Docker integration |
| MySQL | Supported | Not yet | Unit + Docker integration |
| MongoDB | Supported (collections; no FK inference) | Not yet | Unit + Docker integration |
Empty tables/collections can be imported but generation requires ≥ 1 source row.
make test-db-up # Postgres :5433, MySQL :3307, MongoDB :27018
python scripts/seed_e2e_dbs.py
make test-postgres # or test-mysql / test-mongodb
make test-db-downDefault URLs (also in .env.test.example):
postgresql://chitral:chitral@localhost:5433/chitral_testmysql://chitral:chitral@localhost:3307/chitral_testmongodb://chitral:chitral@localhost:27018/?authSource=admin
Without these env vars, make test skips ~88 integration tests on purpose.
- Chitral currently runs generation in one API process with a bounded thread executor. Do not add web workers to scale generation; the repository does not yet provide an external queue/worker runtime.
- Default admission limits are 50 tables, 100 million logical rows, approximately 50 GiB, eight active jobs, and a one-hour deadline. Deployments can reduce these limits.
- A process restart fails interrupted embedded-worker jobs closed. Resubmit the immutable version with the same seed for a deterministic replay.
- Local artifacts expire after one hour by default; durable job metadata and events expire after 24 hours.
- Job events and diagnostics are designed to exclude credentials, connector URIs, and source values. Do not add sensitive samples to logs or bug reports.
- The project does not provide Kafka publication, arbitrary user scripting, public self-service, built-in RBAC, or a multi-tenant identity boundary.
Operators should read the Generation Operations Runbook before enabling advanced capabilities outside a developer workstation.
By default the API allows only localhost frontends:
http://localhost:5173/http://127.0.0.1:5173http://localhost:3000/http://127.0.0.1:3000
To allow other origins, set in backend/.env:
CORS_ORIGINS=["http://localhost:5173","https://your-frontend.example"]Do not set ["*"] on a networked deployment without auth.
| Command | Purpose |
|---|---|
make install | Backend + frontend deps |
make test | Backend unit tests (uv run pytest) |
make lint | Backend ruff + frontend ESLint |
make build | Frontend production build |
make test:e2e | Playwright browser tests |
make test-db-up | Start Docker test DBs |
More detail: developers.md, AGENTS.md, backend/ARCHITECTURE.md.
| Document | Purpose |
|---|---|
| Journey and Temporal Generation User Guide | Concise workflow for versioned relational, temporal, and journey generation. |
| Generation Operations Runbook | Admission, observability, cancellation, restart, cleanup, and escalation procedures. |
| Backend Architecture | Detailed APIs, contracts, synthesis pipeline, quality, jobs, and output architecture. |
| Iceberg REST/S3 Output Architecture | Lakehouse registration, atomicity, recovery, and approval requirements. |
| Temporal/Journey Product Requirements | Complete phased product and engineering requirements. |
| Production Readiness Plan | Deferred hardening work and release gates. |