Skip to content

[Feature] Warehouse driver auto-installation and connection validation #61

Description

@anandgupta42

Problem

When users run /discover or manually add warehouse connections, the system creates connection entries even when the required Python driver isn't installed. This leads to a broken state where connections exist but every operation fails with generic errors like "DuckDB is not installed in this environment."

The error messages don't specify which Python environment to install into, making recovery difficult.

Desired Behavior

  1. DuckDB should always work out-of-box — it's the local development default and should be bundled or auto-installed during first use
  2. Connection validation before creation/discover and warehouse_add should verify driver availability before creating a connection entry
  3. Auto-install option — when a driver is missing, offer to install it into the engine's venv automatically
  4. Clear install instructions — if auto-install isn't possible, show the exact pip command with the correct Python path

Implementation Notes

Key Files

  • packages/altimate-engine/ — Python engine and its venv where drivers need to be installed
  • packages/altimate-code/src/tool/warehouse-test.ts — warehouse connection testing
  • packages/altimate-code/src/tool/warehouse-add.ts — warehouse addition (should validate before saving)
  • packages/altimate-code/src/bridge/client.ts — bridge calls that surface driver errors

Approach

Phase 1: Validate before creating connections

  • Before warehouse_add saves a connection, call a bridge method to check if the required driver is importable
  • If not importable, show: Driver 'duckdb' not found. Install with: <exact pip command with venv path>
  • /discover should skip creating entries for connections it can't actually use

Phase 2: Auto-install drivers

  • Add a bridge method driver_install(driver_name) that runs pip install in the engine's venv
  • Prompt user: "DuckDB driver not installed. Install now? [Y/n]"
  • Support drivers: duckdb, psycopg2-binary, snowflake-connector-python, google-cloud-bigquery, databricks-sql-connector

Phase 3: Bundle DuckDB

  • Add duckdb as a default dependency in the engine's pyproject.toml
  • Ensure it's always available for local development without extra steps

Industry Patterns

  • SQLMesh: Bundles DuckDB as zero-config default for local dev
  • Meltano: Auto-installs plugins into dedicated venvs on first use (--install flag, default on)
  • dbt: Clear error "No data platform adapters installed!" with exact install commands per adapter

Acceptance Criteria

  • DuckDB connections work without manual driver installation
  • warehouse_add validates driver availability before saving
  • /discover doesn't create connections that can't work
  • Error messages include the exact pip command with correct Python environment path
  • Auto-install prompt when driver is missing (with user confirmation)

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestpriority:highHigh priorityuxUser experience improvementswarehouseWarehouse connectivity and drivers

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions