Skip to content

Fix tabular Analyze truncation and combined-run schema lock - #1240

Merged
Paul Lizer (paullizer) merged 2 commits into
microsoft:Developmentfrom
paullizer:fix/tabular-analyze-truncation-schema-deferral
Aug 13, 2026
Merged

Fix tabular Analyze truncation and combined-run schema lock#1240
Paul Lizer (paullizer) merged 2 commits into
microsoft:Developmentfrom
paullizer:fix/tabular-analyze-truncation-schema-deferral

Conversation

@paullizer

@paullizerPaul Lizer (paullizer) commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Refs #1233

This directly fixes Post-Merge Audit Finding #1 from #1233 ("Production Search and Analyze adapters do not yet build a transformation specification from user rules... schema-only validation unless server-side hints are injected") for the default/no-hints case: instead of locking a broken schema-only validation, the run now defers to batch-1 discovery. Using Refs rather than Fixes/Closes because #1233 is a larger multi-phase epic (7A-7D) with other unchecked items (semantic field verification/repair, recoverable artifact-set publication, full 100k-row/live-semantic integration evidence) that this PR does not complete.

Summary

Fixes production reports of Chat/Analyze truncating exhaustive row-by-row tabular requests (answering ~1.5 rows then reporting the rest as "unprocessed"/"truncated" evidence), plus a stuck/failing combined Analyze CSV+analysis export.

Root causes and fixes

1. Durable-preflight parity was off by default with no admin UI toggle
tabular_request_planner_mode, enable_tabular_search_shared_preflight, and enable_tabular_analyze_durable_preflight defaulted to off/False, and were backend-only settings with no way to enable them from Admin Settings. Every deployment therefore always fell back to the legacy bounded foreground synthesis path, which is exactly the truncation symptom. These now default to active/True. Since there's no legitimate reason to prefer the bounded/truncated behavior, there's intentionally no admin UI toggle; SIMPLECHAT_DISABLE_TABULAR_PARITY_DURABLE_PREFLIGHT=true is the emergency rollback env var for incidents.

2. Combined (Analyze) runs locked output_schema before batch 1 ran
queue_tabular_generated_output_run set the new run's output_schema to the lineage-only internal_checkpoint_schema whenever a deliverable contract had no known public_output_schema yet (the common case for prose-described "Required Output" columns). This locked every batch — including batch 1 — into validating against a schema with none of the model's real output columns, so every batch failed with Generated output schema mismatch ... unexpected=[<the correct columns>]; missing=[], and the run never completed. It now defers output_schema to batch-1 discovery when no public schema is known yet, matching the already-working Search/structured_export behavior.

3. No visibility into the background scheduler
The tabular generated-output scheduler (and file sync / data management schedulers) only logged on exceptions, and the scan-result trace was suppressed on empty scans. Added tick-level debug_print/log_event visibility (lease acquired vs. skipped, runs processed) so a stuck run vs. an idle-but-alive scheduler can be told apart from logs alone. This visibility is what let us pinpoint fix#2 from real production logs.

Testing

  • functional_tests/test_tabular_analyze_search_parity_default_activation.py (new) — settings defaults + SIMPLECHAT_DISABLE_TABULAR_PARITY_DURABLE_PREFLIGHT kill switch.
  • functional_tests/test_tabular_combined_output_schema_deferral_fix.py (new) — reproduces the real (unmocked) deliverable-contract precondition from production logs.
  • functional_tests/test_tabular_queue_run_output_schema_end_to_end.py (new) — calls the real queue_tabular_generated_output_run end-to-end (AST-extracted with only Cosmos/blob/telemetry stubbed) and asserts the persisted run document.
  • Re-ran existing tabular parity/background-export/deliverable-contract suites (test_tabular_shared_request_planner.py, test_tabular_analyze_shared_preflight_adapter.py, test_tabular_search_shared_preflight_adapter.py, test_analyze_artifact_phase7_rollout_rollback.py, test_tabular_phase8_ui_telemetry_rollout.py, test_tabular_execution_settings_sanitization.py, test_tabular_background_generated_exports.py, test_tabular_row_orchestration_scale.py, test_tabular_phase7b_production_correctness.py, test_analyze_deliverable_contract.py, test_tabular_phase3_public_schema_projection.py, test_tabular_phase5_artifact_set_lifecycle.py, test_tabular_transformations_phase4.py, test_data_management_migration_recovery.py) — all pass unchanged.
  • py_compile on all changed Python files, no diagnostics, whitespace-clean diff.
  • Confirmed fixed against the real deployed app by the reporter after redeploying this branch.

Version

config.pyVERSION bumped to 0.250.189.

Docs

  • docs/reference/admin_configuration.md and docs/explanation/features/TABULAR_ANALYZE_SEARCH_PARITY_ROLLOUT.md updated for the new defaults and the env kill switch.
  • docs/explanation/fixes/TABULAR_ANALYZE_SEARCH_PARITY_DEFAULT_ACTIVATION_FIX.md added.

- Default tabular Analyze/Search durable-preflight parity to active (tabular_request_planner_mode, enable_tabular_search_shared_preflight, enable_tabular_analyze_durable_preflight); previously off-by-default with no admin UI toggle, so exhaustive row-by-row requests silently fell back to bounded foreground synthesis and truncated. Add SIMPLECHAT_DISABLE_TABULAR_PARITY_DURABLE_PREFLIGHT env var as the emergency rollback path instead of a UI toggle.
- Add scheduler tick/skip visibility logging (debug_print) to the tabular generated-output, file sync, and data management scheduler loops, and stop suppressing the tabular scheduler scan-result log on empty scans, to diagnose stuck background exports.
- Fix queue_tabular_generated_output_run locking output_schema to the lineage-only internal checkpoint schema before batch 1 runs whenever no public output schema is known yet (e.g. combined Analyze requests with prose-described columns). This made every batch, including batch 1, fail schema validation against a schema with none of the model's real output columns. Now defers to batch-1 discovery when public_output_schema is empty, matching the working Search/structured_export behavior.
- Add regression tests: settings defaults + env kill switch, a real (unmocked) deliverable-contract reproduction of the bug precondition, and a full end-to-end AST-extracted invocation of queue_tabular_generated_output_run itself.
Fixes truncated exhaustive tabular Analyze answers and stuck/failing combined CSV exports reported in production.
… metadata
Root cause of combined Analyze+CSV runs freezing at 'Complete' with no download button: _normalize_tabular_run_planner_metadata() rebuilt the persisted deliverable_contract from an explicit field whitelist that never included requested_artifacts. Every run sanitized through it persisted an empty expected-artifact list, so validate_analysis_artifact_set() rejected both the real Markdown and CSV artifacts as extra_artifact, permanently locking artifact_set.lifecycle_state below 'completed' with no self-heal path.
Confirmed via the 0.250.190 diagnostic log_event additions firing in production for two customer test runs (reason_codes: ['extra_artifact'], expected_artifact_ids: []).
Fix: add a bounded, sanitized requested_artifacts list to the whitelist. Verified end-to-end with a real, unmocked deliverable contract routed through the real sanitizer.
Tests updated to route real contracts through the real sanitizer (closing the gap that let this slip through originally), plus a new direct regression guard. Full related test suite re-verified with no regressions.
Version 0.250.190 -> 0.250.191.
@paullizer
Paul Lizer (paullizer) merged commit 8d86405 into microsoft:DevelopmentAug 13, 2026
11 of 12 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@paullizer