Skip to content

Schema reads do not apply ROW_CAP and follow result chunks (#703) - #737

Merged
WaylandYang merged 1 commit into
deeplethe:devfrom
rollroyces:fix/databricks-schema-row-limit-and-chunk-following
Sep 17, 2026
Merged

WaylandYang merged 1 commit into
deeplethe:devfrom
rollroyces:fix/databricks-schema-row-limit-and-chunk-following

Conversation

@rollroyces

Copy link
Copy Markdown
Contributor

Closes #703.

A Databricks schema read on a wide catalog was cut at 201 rows silently. The same SQL Statement Execution API call went through DatabricksEngine::run with row_limit = ROW_CAP + 1, intended as the chat-query safety valve. information_schema.columns returns one row per column, so any catalog with more than ~201 columns stopped partway through a table and nothing said so. Snowflake has the same shape: the response includes partitionInfo with per-partition URLs following the first inline chunk, and the engine read only that first chunk.

Fix

Two fixes per engine, scoped to schema reads:

  • Schema reads do not send row_limit. fetch_schema calls run with None; execute keeps Some(ROW_CAP + 1). The Databricks run signature is now run(sql: &str, row_limit: Option<usize>). Snowflake has no request-body row limit to begin with — the existing wrap_limit path applies the LIMIT clause to execute only — so this is a Databricks-only behavioural change.
  • Both engines follow result chunks until exhausted. Databricks follows next_chunk_internal_link via repeated GET until the link is absent, then verifies rows.len() == manifest.total_row_count when no row_limit was set. Snowflake iterates partitionInfo from index 1 (the first partition is the inline data already in the response), GETs each url, and verifies the partition rowCounts sum to numRows. Any mismatch bails loudly instead of returning a partial schema.

manifest.truncated on the Databricks side is now an explicit fail-fast: a chat query that actually gets truncated by the server returns an error rather than presenting fewer rows as a complete answer.

Tests

All wiremock, no live cluster needed:

  • a_schema_read_follows_chunks_until_exhausted — three chunks, all rows collected in order.
  • a_chat_query_bails_loudly_when_the_service_says_truncated — chat query with manifest.truncated=true errors.
  • a_schema_read_fails_when_total_row_count_does_not_match — schema read with a mismatch between manifest.total_row_count and the actual rows errors.
  • a_schema_read_follows_partitions_until_exhausted — three partitions on the Snowflake side, all rows collected in order.

utopia-server 304 passed (was 295 before — 4 new tests added); clippy -D warnings clean; cargo fmt clean.

What still needs a real cluster

The two fetch_schema integration tests still need a live Databricks and Snowflake warehouse, per #241 and #242. Those should keep the existing structure: run the engine against a real catalog with >201 columns (Databricks) and >1 partition (Snowflake), confirm the schema document reports every column.

…#703)

A Databricks schema read on a wide catalog was cut at 201 rows
silently. The same SQL Statement Execution API call goes through
DatabricksEngine::run with row_limit = ROW_CAP + 1, intended as the
chat-query safety valve. information_schema.columns returns one row per
column, so any catalog with more than ~201 columns stops partway through
a table and nothing says so. Snowflake has the same shape: the response
includes partitionInfo with per-partition URLs that follow the first
inline chunk, and the engine reads only that first chunk.

Two fixes per engine, scoped to schema reads:

- Schema reads do not send row_limit. fetch_schema calls run with
  None; chat-query execute() keeps Some(ROW_CAP + 1). Snowflake has no
  request-body row limit to begin with — it follows the existing
  LIMIT-clause path through wrap_limit — so this is a Databricks-only
  behavioural change. The row_limit is now an Option<usize> on run.

- Both engines follow result chunks until exhausted. Databricks
  follows next_chunk_internal_link via repeated GET until the link is
  absent, then verifies rows.len() == manifest.total_row_count when
  no row_limit was set. Snowflake iterates partitionInfo from index 1
  (the first partition is the inline data already in the response),
  GET each url, and verifies partition row counts sum to numRows. Any
  mismatch bails loudly instead of returning a partial schema.

manifest.truncated on the Databricks side is now an explicit
fail-fast: a chat query that actually gets truncated by the server
returns an error rather than presenting fewer rows as a complete
answer. The previous code returned the truncated set with no
warning.

Tests (all in wiremock, no live cluster needed):

- a_schema_read_follows_chunks_until_exhausted — three chunks, all
  rows collected in order.
- a_chat_query_bails_loudly_when_the_service_says_truncated — chat
  query with manifest.truncated=true errors instead of returning
  fewer rows.
- a_schema_read_fails_when_total_row_count_does_not_match — schema
  read with a mismatch between manifest.total_row_count and the
  actual rows received errors.
- a_schema_read_follows_partitions_until_exhausted — three
  partitions on the Snowflake side, all rows collected in order.

Both new tests pass alongside the existing 9 (Databricks) and 3
(Snowflake) tests. utopia-server 304 passed, clippy -D warnings
clean, cargo fmt clean. Integration test on a real Databricks /
Snowflake warehouse is still required for end-to-end confirmation.

Refs deeplethe#703

Signed-off-by: rollroyces <rollroyces@users.noreply.github.com>
@rollroyces
rollroyces force-pushed the fix/databricks-schema-row-limit-and-chunk-following branch 2 times, most recently from 2c52139 to 4ac7445 Compare September 16, 2026 15:37
@WaylandYang
WaylandYang merged commit 4137f7d into deeplethe:dev Sep 17, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A Databricks schema read stops at 201 columns

2 participants