Skip to content

Capture lineage for previously-dropped and opaque statements (CREATE/DROP/ALTER/LOAD/DO) - #88

Merged
lustefaniak merged 7 commits into
mainfrom
lukasz-create-statement-lineage
Jun 2, 2026
Merged

Capture lineage for previously-dropped and opaque statements (CREATE/DROP/ALTER/LOAD/DO)#88
lustefaniak merged 7 commits into
mainfrom
lukasz-create-statement-lineage

Conversation

@lustefaniak

@lustefaniaklustefaniak commented Jun 2, 2026•
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

Replace the parser's "silently wrong" fallbacks — statements that were dropped to a fake COMMENT/SET node or rejected outright — with honest, lineage-focused AST. Every statement that carries a query body, a source/target table, or a procedural body now preserves it (or captures the raw text verbatim) so column-level lineage can reach the references. All captured identifiers carry source spans (WithSpan).

7 commits, each validated independently (zero corpus regressions per commit).

1. Lineage AST for previously-dropped CREATE statements (2c2d5dec)

ConstructDialectCaptured
CREATE TASK … AS <sql>Snowflakename, AFTER deps, WHEN, body as a real Statement
CREATE STREAM … ON TABLE/VIEW/STAGESnowflakesource→stream CDC edge (incl. ON STAGE)
CREATE PIPE … AS COPY INTOSnowflakeingest body (stage→table)
CREATE MODEL … AS <query>BigQuery MLtraining-input query
CREATE EXTERNAL SCHEMA … FROM …Redshiftfederated database/schema source
struct<>/array<>/map<> columnsRedshift Spectrumnested schema in external tables
CREATE FOREIGN TABLEPostgresexternal-table columns
CREATE [OR REFRESH] STREAMING TABLEDatabricksAS <query> (graceful fallback for DLT clauses)

Unmodelled CREATE → CreateUnsupported instead of a fake COMMENT ON <type>; ExecuteTask now carries the task-name span.

2. CREATE MATERIALIZED VIEW pre-AS option clauses (324cd57b)

Skips unmodelled pre-AS clauses (Oracle BUILD/REFRESH/PCTFREE, Databricks SCHEDULE/TRIGGER, Trino GRACE PERIOD, BigQuery ANNOTATIONS, …) to reach the AS <query>. ClickHouse excluded (its [APPEND] TO <table> output edge is parsed precisely).

3. Honest opaque nodes for unmodelled DROP/ALTER and SYSTEM/LOCK (c088d2d9)

  • DROP/ALTER <unmodelled> (e.g. DROP WAREHOUSE, ALTER TASK) previously errored and failed the surrounding block → now DropUnsupported / AlterUnsupported { object_type, body }.
  • ClickHouse SYSTEM, Postgres LOCK TABLE → UnsupportedStatement { keyword, body } (previously a fake SET <keyword> that dropped the LOCK table ref).
  • CreateUnsupported gains a body string (raw trailing text).

4. Standalone DO body + DuckDbLoad rename (70deca9c)

  • Postgres DO [LANGUAGE <lang>] $$ … $$ → Statement::Do with the body captured as a re-parseable FunctionDefinition (same body_start handling as CREATE PROCEDURE, via a shared parse_function_body_definition helper), so embedded DML stays recoverable. Previously a fake SET DO.
  • Renamed Statement::Load → Statement::DuckDbLoad to disambiguate the DuckDB LOAD <extension> statement from LOAD DATA … (Statement::LoadData), matching DuckDbDialect casing.

5. LOAD DATA … INPATH … INTO TABLE (5546b3c2)

Hive/Spark/Databricks LOAD DATA [LOCAL] INPATH '<path>' [OVERWRITE] INTO TABLE <t> [PARTITION …] now parses — captures inpath source + target table (LoadData gains local/inpath). BigQuery LOAD DATA … FROM FILES form unaffected.

6. Redshift CREATE MODEL bare-table FROM (7dce6079)

Redshift ML CREATE MODEL … FROM <table> / FROM (<query>) now records the training input (new from_table field; mutually exclusive with query). Previously the bare-table source was dropped.

7. Docs (4391c6ed)

CLAUDE.md learnings: the opaque/unsupported-node helpers, dummy spans under parse_sql (use tokenize_with_location in tests), AST node naming convention, and the file-by-file corpus regression check.

Validation

  • 1180 unit tests pass (assertion-based on documented SQL, incl. span/body_start location coverage).
  • ~+41 corpus fixes, zero regressions (8 Redshift Spectrum/external-schema + 27 materialized-view across Oracle/Databricks/Trino/Doris/Trino-customer + 4 DROP/ALTER blocks + 2 Databricks LOAD DATA). Verified file-by-file against isolated baselines (the symlinked corpus reprocessed several times mid-development).

Snowflake/BigQuery TASK/STREAM/PIPE/MODEL show no pass/fail delta because those statements were already fake-COMMENT "passes" — the win is real lineage AST, invisible to a parse-success metric.

@lustefaniak
feat: capture lineage for previously-dropped CREATE statements
2c2d5de
The generic CREATE classifier skipped any object type it didn't dispatch
explicitly to the end of the statement and returned a fake `COMMENT ON <type>`
node with an empty object name. Several lineage-bearing statements fell into
this hole, losing their query bodies and source references entirely.
Add real, lineage-focused AST + parsing for:
- Snowflake CREATE TASK ... AS <sql>: captures the task name, AFTER
predecessors, WHEN predicate, and the AS body parsed as a real Statement
(INSERT/MERGE/CALL/BEGIN..END/...). Opaque bodies (EXECUTE DBT PROJECT)
degrade to None instead of corrupting the statement.
- Snowflake CREATE STREAM ... ON {TABLE|VIEW|STAGE|EXTERNAL TABLE} <src>:
captures the source-to-stream CDC edge.
- Snowflake CREATE PIPE ... AS COPY INTO: captures the COPY INTO ingest body.
- BigQuery ML CREATE MODEL ... AS <query> and Redshift ML CREATE MODEL ...
FROM (<query>): captures the training-input query.
- Redshift CREATE EXTERNAL SCHEMA ... FROM { DATA CATALOG | HIVE METASTORE |
REDSHIFT | ... }: captures the federated database/schema source.
- Redshift Spectrum struct<>/array<>/map<> nested column types in
CREATE EXTERNAL TABLE (Hive-style colon-separated fields).
- Postgres CREATE FOREIGN TABLE and Databricks CREATE [OR REFRESH] STREAMING
TABLE are routed through the existing table parsers; STREAMING TABLE falls
back gracefully for Delta Live Tables clauses (FLOW/WATERMARK/EXPECT/SCD).
Unmodelled CREATEs now return a dedicated CreateUnsupported node instead of a
fake COMMENT, so consumers can tell them apart. ExecuteTask now carries the
task name span. All captured identifiers carry source locations (WithSpan).
Fixes 8 corpus test failures (Redshift).
@lustefaniak
fix: skip non-lineage pre-AS option clauses in CREATE MATERIALIZED VIEW
324cd57
Oracle / Databricks / Trino / BigQuery materialized views carry option
clauses between the name and the AS query that the view parser didn't model
(Oracle BUILD/REFRESH/PCTFREE, Databricks SCHEDULE/TRIGGER, Trino GRACE
PERIOD, BigQuery ANNOTATIONS, ...), causing an "Expected AS" parse error.
Skip any leftover pre-AS tokens (respecting parens) up to the AS query body
so the lineage-bearing query is preserved. ClickHouse is excluded because its
`[APPEND] TO <table>` output target and ENGINE/REFRESH clauses are parsed
precisely and a blind skip would drop the output edge.
Fixes 27 corpus test failures (Oracle, Databricks, Trino, Doris).
@github-actions

github-actionsBot commented Jun 2, 2026•
edited
Loading

Copy link
Copy Markdown

Corpus Parsing Report

Total: 191708 passed, 1921 failed (99.0% pass rate)

✨ No changes in test results

By Dialect

DialectPassedFailedTotalPass RateDelta
ansi5116958088.1%-
athena3713897.4%-
bigquery423241134243799.7%-
clickhouse2723129285295.5%-
databricks2962185314794.1%+7
doris28124070.0%+6
dremio27027100.0%-
duckdb112445116996.2%-
exasol5476188.5%-
fabric606100.0%-
generic17385530.9%-
hive35104577.8%-
materialize6142030.0%-
mssql2276407268384.8%+3
mysql1513718880.3%-
oracle1046352139874.8%+12
postgres1172111128391.3%-
presto5586387.3%-
redshift40503654056899.8%+8
singlestore141915094.0%-
snowflake947391389487799.9%+1
spark902011081.8%-
sqlite51166776.1%-
starrocks2943387.9%-
teradata23204353.5%-
trino141377149094.8%+4
tsql1653419982.9%-

@lustefaniak
feat: honest opaque nodes for unmodelled DROP/ALTER and SYSTEM/LOCK/DO
c088d2d
Replace the remaining masquerade fallbacks that returned a structurally wrong
statement node with honest opaque nodes that preserve the unmatched text:
- CREATE <unmodelled> now also captures the trailing text in a `body` field
(CreateUnsupported gains `body`).
- DROP <unmodelled> (e.g. DROP WAREHOUSE / DROP STREAM / DROP PIPE) returned an
"Expected ... after DROP" error and failed the surrounding block; it now
returns DropUnsupported { object_type, body }.
- ALTER <unmodelled> (e.g. ALTER WAREHOUSE / ALTER TASK) likewise now returns
AlterUnsupported { object_type, body } instead of erroring.
- ClickHouse SYSTEM, PostgreSQL LOCK TABLE and DO blocks were misrepresented as
a no-op `SET <keyword>` statement (dropping the LOCK table reference / the DO
procedural body); they now return UnsupportedStatement { keyword, body } with
the body preserved verbatim.
The captured `body` is the raw token text up to the statement end, so consumers
can inspect or re-parse it.
Fixes 4 corpus test failures (MSSQL, Snowflake).
@lustefaniak
feat(postgres): parse standalone DO block body as re-parseable defini…
70deca9
…tion
The PostgreSQL standalone anonymous code block `DO [LANGUAGE <lang>] $$ ... $$`
now captures its body as a re-parseable FunctionDefinition (the same shape used
by CREATE PROCEDURE), preserving the body's source start location so lineage
extraction can recover embedded DML/DDL and remap body-relative spans. Adds a
new Statement::Do node and extracts the shared body-capture logic into
Parser::parse_function_body_definition (used by both DO and CREATE PROCEDURE),
so the body_start computation is identical.
Also renames Statement::Load to Statement::DuckDbLoad to disambiguate the DuckDB
`LOAD <extension>` statement from the Hive/Spark/BigQuery `LOAD DATA ...`
statement (Statement::LoadData), matching the DuckDbDialect casing convention.
@lustefaniak
feat(hive,databricks): parse LOAD DATA [LOCAL] INPATH ... INTO TABLE
5546b3c
Hive/Spark/Databricks `LOAD DATA [LOCAL] INPATH '<path>' [OVERWRITE] INTO TABLE
<target> [PARTITION ...]` previously errored (the LOAD DATA arm was gated to
BigQuery/Generic and expected `LOAD DATA [OVERWRITE|INTO] <target>`). Widen the
gate to Hive/Databricks and parse the INPATH form, capturing the source path
(`inpath`) and the loaded `target` table for lineage; the partition list and
remaining options stay opaque. LoadData gains `local` and `inpath` fields.
Fixes 2 corpus test failures (Databricks).
@lustefaniak
feat(redshift): record bare-table FROM source in CREATE MODEL
7dce607
Redshift ML `CREATE MODEL <name> FROM <table> ...` previously dropped the
training-input table (only `AS <query>` and `FROM (<query>)` forms recorded an
input). CreateModel now carries an additional `from_table` field that captures
the bare-table source (with its span), so the `training_source -> model`
lineage edge is recorded for that form too. `query` and `from_table` are
mutually exclusive.
@lustefaniaklustefaniak changed the title Capture lineage for previously-dropped CREATE statementsCapture lineage for previously-dropped and opaque statementsJun 2, 2026
@lustefaniak
docs: capture parser learnings (opaque/unsupported nodes, dummy spans…
4391c6e
… in tests, AST naming, corpus regression check)
@lustefaniaklustefaniak changed the title Capture lineage for previously-dropped and opaque statementsCapture lineage for previously-dropped and opaque statements (CREATE/DROP/ALTER/LOAD/DO)Jun 2, 2026
@lustefaniak
lustefaniak enabled auto-merge (rebase) June 2, 2026 16:23
@lustefaniak
lustefaniak merged commit 6478bc3 into mainJun 2, 2026
3 checks passed
@lustefaniak
lustefaniak deleted the lukasz-create-statement-lineage branch June 2, 2026 16:23
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zdenalzdenalzdenal approved these changes
@grasskodegrasskodeAwaiting requested review from grasskode
@iamjasinskiiamjasinskiAwaiting requested review from iamjasinski
@LiHRaMLiHRaMAwaiting requested review from LiHRaM

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@lustefaniak@zdenal