Skip to content

direct: fix persistent drift on model_serving_endpoints - #5708

Merged
pietern merged 11 commits into
mainfrom
investigate-model-serving
Jun 29, 2026
Merged

direct: fix persistent drift on model_serving_endpoints#5708
pietern merged 11 commits into
mainfrom
investigate-model-serving

Conversation

@pietern

@pieternpietern commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Changes

Restore suppression of the backend-defaulted config.traffic_config (as a backend_default) and budget_policy_id (ignore_remote_changes) on model_serving_endpoints, and teach the testserver to default traffic_config like the backend so the drift is covered locally. Builds on #5764, which makes Changes.HasChange skip-aware (merged into this branch).

Why

The backend always returns a defaulted config.traffic_config (100% to the served entity) even when the user specifies none. #5693 dropped the ignore_remote_changes entries for traffic_config and budget_policy_id, assuming the new "skip fields missing from RemoteType" auto-skip covered them — but both fields are present in RemoteType, so the auto-skip never fires and the direct engine reported a permanent update after every clean deploy.

traffic_config is a true backend default the user may also set, so backend_default (skip only when the local value is empty; real drift still detected when set) is the right tool rather than ignore_remote_changes. budget_policy_id is never echoed by GET, so it stays under ignore_remote_changes.

Making the testserver faithful surfaced a latent over-update: HasChange counted skipped changes, so the suppressed traffic_config triggered a spurious PUT /config on unrelated updates. That fix lives in #5764 (skip-aware HasChange), which is merged into this branch — please merge #5764 first.

Tests

running-endpoint now reproduces the drift locally (fails without the fix). Passes on both engines on real cloud (aws-prod-ucws). Full ./task test / fmt / lint / checks pass.

This PR and its description were written by Isaac.

The backend always returns a defaulted config.traffic_config (100% to the
served entity) even when the user does not specify one. #5693 dropped the
suppression for it (and for budget_policy_id) assuming the new
missing-in-remote auto-skip covered them, but both are present in RemoteType
so the auto-skip never fires, and the direct engine reported a permanent
update after every clean deploy.
Suppress config.traffic_config as a backend_default (skip only when the user
leaves it empty; real drift is still detected if it is set) and keep
budget_policy_id under ignore_remote_changes (GET never echoes it).
Make Changes.HasChange skip-aware so a suppressed-only change under a section
no longer triggers a spurious section update (e.g. PUT /config on an
unrelated email/tags update).
Teach the testserver to default traffic_config like the backend so the
running-endpoint acceptance test reproduces the drift locally.
Co-authored-by: Isaac
Co-authored-by: Isaac
@pietern
pieterntemporarily deployed to test-trigger-is June 24, 2026 15:15 — with GitHub Actions Inactive
@pietern
pieterntemporarily deployed to test-trigger-is June 24, 2026 15:15 — with GitHub Actions Inactive
@denik

Copy link
Copy Markdown
Contributor

how did you find this?

Comment threadbundle/deployplan/plan.go
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Jun 24, 2026

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: 2f4d909

Run: 28363573721

Env🟨​KNOWN🔄​flaky💚​RECOVERED🙈​SKIP✅​pass🙈​skipTime
🟨​aws linux711323510355:10
🟨​aws windows711323710337:24
💚​aws-ucws linux8133199535:01
🔄​aws-ucws windows68133159516:04
💚​azure linux21523510344:08
🔄​azure windows121523610323:59
💚​azure-ucws linux2153219505:22
🔄​azure-ucws windows22153219484:31
💚​gcp linux21523410363:41
💚​gcp windows21523610343:49
30 interesting tests: 13 SKIP, 9 flaky, 7 KNOWN, 1 RECOVERED
Test Nameaws linuxaws windowsaws-ucws linuxaws-ucws windowsazure linuxazure windowsazure-ucws linuxazure-ucws windowsgcp linuxgcp windows
🟨​TestAccept🟨​K🟨​K💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R
🙈​TestAccept/bundle/invariant/no_drift🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/permissions🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions🟨​K🟨​K💚​R💚​R🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct🟨​K🟨​K💚​R💚​R
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform🟨​K🟨​K💚​R💚​R
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions🟨​K🟨​K💚​R💚​R🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct🟨​K🟨​K💚​R💚​R
🟨​TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform🟨​K🟨​K💚​R💚​R
🙈​TestAccept/bundle/resources/postgres_branches/basic🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/postgres_branches/recreate🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/postgres_branches/replace_existing🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/postgres_branches/update_protected🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/postgres_branches/without_branch_id🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/postgres_endpoints/basic🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/postgres_projects/update_display_name🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/synced_database_tables/basic🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🙈​TestAccept/ssh/connection🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S🙈​S
🔄​TestFsCpDirToDirWithOverwriteFlag✅​p✅​p✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p
🔄​TestFsCpDirToDirWithOverwriteFlag/dbfs_to_uc-volumes🙈​s🙈​s✅​p🔄​f🙈​s🙈​s✅​p✅​p🙈​s🙈​s
🔄​TestFsCpDirToDirWithOverwriteFlag/uc-volumes_to_dbfs🙈​s🙈​s✅​p🔄​f🙈​s🙈​s✅​p✅​p🙈​s🙈​s
🔄​TestFsCpDirToDirWithOverwriteFlag/uc-volumes_to_local🙈​s🙈​s✅​p✅​p🙈​s🙈​s✅​p🔄​f🙈​s🙈​s
🔄​TestFsCpDirToDirWithOverwriteFlag/uc-volumes_to_uc-volumes🙈​s🙈​s✅​p🔄​f🙈​s🙈​s✅​p✅​p🙈​s🙈​s
🔄​TestFsCpFileToFileWithOverwriteFlag✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p✅​p✅​p
🔄​TestFsCpFileToFileWithOverwriteFlag/dbfs_to_dbfs✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p✅​p✅​p
🔄​TestSyncIncrementalFileOverwritesFolder✅​p✅​p✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p
🔄​TestImportFileFormatAuto✅​p✅​p✅​p🔄​f✅​p✅​p✅​p✅​p✅​p✅​p
💚​TestFetchRepositoryInfoAPI_FromRepo💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R💚​R
Top 4 slowest tests (at least 2 minutes):
durationenvtestname
3:09azure-ucws windowsTestAccept
2:49gcp windowsTestAccept
2:24aws-ucws windowsTestAccept
2:22azure windowsTestAccept

HasChange counted suppressed (Action == Skip) changes, so a section whose
only change was suppressed (backend_default, ignore_remote_changes,
remote_already_set, etc.) still reported a change and triggered a spurious
sub-section update API call in DoUpdate.
Skip suppressed changes, matching HasChangeExcept. All six call sites
(model_serving_endpoint, vector_search_endpoint) gate a per-section update
on HasChange, so ignoring suppressed changes is strictly more correct.
Split out of #5708 with a dedicated regression test.
Co-authored-by: Isaac
Comment threadNEXT_CHANGELOG.md Outdated
@pietern
pieterntemporarily deployed to test-trigger-is June 29, 2026 09:30 — with GitHub Actions Inactive
@pietern
pieterntemporarily deployed to test-trigger-is June 29, 2026 09:30 — with GitHub Actions Inactive
@pietern
pieterntemporarily deployed to test-trigger-is June 29, 2026 09:31 — with GitHub Actions Inactive
@pietern
pieterntemporarily deployed to test-trigger-is June 29, 2026 09:31 — with GitHub Actions Inactive
@pietern
pieterntemporarily deployed to test-trigger-is June 29, 2026 09:36 — with GitHub Actions Inactive
@pietern
pieterntemporarily deployed to test-trigger-is June 29, 2026 09:36 — with GitHub Actions Inactive
@pietern
pieterntemporarily deployed to test-trigger-is June 29, 2026 09:37 — with GitHub Actions Inactive
@pietern
pieterntemporarily deployed to test-trigger-is June 29, 2026 09:37 — with GitHub Actions Inactive
@pietern
pieterntemporarily deployed to test-trigger-is June 29, 2026 09:38 — with GitHub Actions Inactive
@pietern
pieterntemporarily deployed to test-trigger-is June 29, 2026 09:38 — with GitHub Actions Inactive
@pietern
pieterntemporarily deployed to test-trigger-is June 29, 2026 09:39 — with GitHub Actions Inactive
@pietern
pieterntemporarily deployed to test-trigger-is June 29, 2026 09:39 — with GitHub Actions Inactive
@pietern
pieterntemporarily deployed to test-trigger-is June 29, 2026 09:51 — with GitHub Actions Inactive
@pietern
pieterntemporarily deployed to test-trigger-is June 29, 2026 09:51 — with GitHub Actions Inactive
@pietern
pietern enabled auto-merge June 29, 2026 09:52
@pietern
pietern added this pull request to the merge queueJun 29, 2026
Merged via the queue into main with commit 8afce48Jun 29, 2026
23 checks passed
@pietern
pietern deleted the investigate-model-serving branch June 29, 2026 10:39
bradleyjamrozik-origindigital pushed a commit to Origin-Digital-LLC/databricks-cli that referenced this pull request Jun 29, 2026
Split out of databricks#5708 per review feedback, with a dedicated regression
test.
`Changes.HasChange` counted suppressed changes (`Action == Skip`), so a
section whose only change was suppressed — `backend_default`,
`ignore_remote_changes`, `remote_already_set`, `empty`,
`missing_in_remote`, etc. — still reported a change. In
`model_serving_endpoint` and `vector_search_endpoint` `DoUpdate`, this
triggered a spurious per-section update API call (e.g. a `PUT /config`
for a `traffic_config` default the user never set).
`HasChange` now skips suppressed changes, matching the already
skip-aware `HasChangeExcept`.
All six `HasChange` call sites gate a per-section update in `DoUpdate`:
```
bundle/direct/dresources/model_serving_endpoint.go (tags, ai_gateway, config, email_notifications)
bundle/direct/dresources/vector_search_endpoint.go (budget_policy_id, target_qps)
```
In every case the call exists to push a real change to that section, so
ignoring suppressed changes is strictly more correct. The other call
sites denik listed (`bind.go`, `app.go`, `cluster.go`,
`sql_warehouse.go`) use `HasChangeExcept` or the `Action`-based
`HasChanges` field, both of which already filter `Skip`.
Added `TestHasChange` covering the skip-aware behavior and path-boundary
matching.
This pull request and its description were written by Isaac.
deco-sdk-taggingBot added a commit that referenced this pull request Jul 2, 2026
## Release v1.6.0
### CLI
* `ssh connect` now accepts a `--base-environment` flag to run a serverless session on a custom base environment. It takes an `env.yaml` path, a `workspace-base-environments/...` resource ID, or a base environment display name, and is rejected together with `--environment-version` or `--cluster` ([#5706](#5706)).
* `databricks aitools install` is now plugin-first: it installs the Databricks plugin through each agent's own CLI (Claude Code, Codex, GitHub Copilot) instead of copying raw skill files. Agents without a plugin (OpenCode, Antigravity) still get skill files, and Cursor prints the `/add-plugin databricks` step. Use `--skills-only` to force raw skill files for every agent, or `--path <dir>` to write skills to a directory ([#5738](#5738)).
* `databricks labs list` now only shows projects that can be installed ([#5560](#5560)).
### Bundles
* direct: Fixed persistent drift on `model_serving_endpoints` caused by the `traffic_config` field ([#5708](#5708)).
* direct: Fix spurious update when `apply_policy_default_values: true` is set on job task, for-each-task, or job cluster new_cluster ([#5731](#5731)). Also fix spurious updates for for-each-task clusters due to missing backend defaults for `data_security_mode`, `node_type_id`, `driver_node_type_id`, `driver_instance_pool_id`, `enable_elastic_disk`, and `enable_local_disk_encryption`.
* direct: Cluster resize now falls back to regular update if resize fails due to `INVALID_STATE` ([#5716](#5716)).
* `bundle generate dashboard` now honors the `--key` flag when naming the generated resource, and rejects combining `--existing-path`, `--existing-id`, and `--resource` instead of silently ignoring all but one ([#5492](#5492)).
* Fixed `bundle deployment migrate` failing on `model_serving_endpoints`/`database_instances` with permissions (regression since v1.5.0) ([#5775](#5775)).
* After a terraform deploy, the CLI now dry-runs a migration to the direct engine (writing nothing locally or remotely) and reports the outcome via telemetry, warning if the migration could not be completed ([#5797](#5797)).
### Dependency updates
* Bump `github.com/databricks/databricks-sdk-go` from v0.147.0 to v0.152.0 ([#5773](#5773)).
* Bump Terraform provider from v1.118.0 to v1.120.0 ([#5792](#5792)).
radakam added a commit that referenced this pull request Jul 6, 2026
Drop the #5708 and #5716 NEXT_CHANGELOG.md entries that were
re-introduced by a stale merge; they were already rotated out of main
by a release. Leaves only this PR's #5603 entry.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@pietern@denik@eng-dev-ecosystem-bot