Fix Databricks schema read fallback and error handling

[zhongqishang]

Fixes the first two things a real Databricks workspace broke (#241). This engine had only ever been replayed against wiremock.

What went wrong

Refreshing the schema of a mounted source on a live workspace returned 500 Internal server error — and nothing else. The reason Databricks actually gave, [TABLE_OR_VIEW_NOT_FOUND] The table or view \main`.`information_schema`.`columns` cannot be found, existed only in the server log. The statement was the other half: fetch_schemahad exactly one form,.information_schema.columns`.

Changes

• fetch_schema tries the catalog-level information_schema.columns, then falls back to system.information_schema.columns WHERE table_catalog = '<catalog>'. When both fail, the error names the forms tried and carries the engine's own words.
• sync_schema answers 422 schema_sync_failed with the engine's message in detail instead of a bare 500; the code's wording lives in both i18n files. mount is untouched — it already answers 200 with schema_error set, because the source really is mounted.
• Two wiremock tests: the fallback is taken when the catalog-level read is missing, and two dead ends name both forms.

Verification

• cargo fmt --all --check clean; the release build and tsc --noEmit behind the deployed image pass.
• The new tests run in CI — this machine has no Rust toolchain, so they were never executed locally.
• The failure was reproduced end to end against the workspace that produced the log.

Open question

Why main.information_schema is unreadable there is not settled: either the token's identity cannot see the catalog (Unity Catalog reports invisible objects as missing), or that catalog is not Unity Catalog managed. The fallback covers the first, and the returned detail now tells the two apart on sight.

After review (504f248)

• Fallback conditions: the fallback is taken only when the engine reports a missing table or catalog. A fallback that reads 0 rows is a failure: the token cannot see the catalog, or the catalog isn't in Unity Catalog. Before this change it silently replaced the schema document with an empty one. Both engine messages are kept in the error.
• sync_schema responses: schema_sync_failed (422) covers only a failure to read the engine's schema; our own failures stay 500. The route logs a warning and caps the detail at 600 characters.
• Tests: two new ones, and the dead-ends test now checks both messages.
• End to end: a scripted statements API, a real server and a mounted source. Refresh with the catalog missing and system empty answers 422 and keeps the document. Refresh on PERMISSION_DENIED answers 422 after one statement.
• Follow-up: schema reads are also capped at 201 rows, tracked in A Databricks schema read stops at 201 columns #703.

[zhongqishang]

[WaylandYang]

@zhongqishang thanks, and thanks for taking this to a real workspace; the error half is a clear improvement. Before merging I pushed one commit to your branch (504f248) after compiling and running it. Three things changed in the fallback:

• An empty fallback is now a failure. Both information_schema views list only the objects the token can see, and the system one excludes hive_metastore. So in both explanations you gave (the token can't see main, or main isn't in Unity Catalog), system.information_schema.columns WHERE table_catalog = 'main' succeeds with 0 rows. On your head that returned Ok, and Refresh schema replaced the existing schema document with an empty one: I reproduced this against a fake statements API, where the document went from 193 to 141 characters and the response said schema_tables: 0. It now answers 422, saying the catalog can't be seen or isn't Unity Catalog managed, and the old document stays.
• Only a missing table or catalog falls back (TABLE_OR_VIEW_NOT_FOUND, NO_SUCH_CATALOG_EXCEPTION, CATALOG_NOT_FOUND, SCHEMA_NOT_FOUND). A bad token, a stopped warehouse or PERMISSION_DENIED gives the same answer on the second statement, and the retry doubled the wait on a synchronous button (up to 30 s polling each).
• Both engine messages are kept. Only the last error survived before, so a TABLE_OR_VIEW_NOT_FOUND followed by a different second error lost the first.

In sync_schema, only a failure to read the engine's schema maps to 422 schema_sync_failed. Our own failures (database, ingesting the document) stay 500, because their i18n text about checking the catalog would mislead. The route logs a warning, since Invalid errors aren't logged, and the detail is capped at 600 characters.

Tests: two_dead_ends_say_which_ones_were_tried now returns different messages per statement. New tests are an_error_that_is_not_a_missing_table_is_not_retried and a_fallback_that_reads_no_columns_is_a_failure. All three fail on your head. utopia-server: 282 passed; clippy -D warnings is clean.

The next thing a real catalog will hit is separate and existed before your change: schema reads are capped at 201 rows by the chat query limit. That's #703.

🤖 Generated with Claude Code