[Breaking Change] Remove deprecated path fallback mechanism for jobs and pipelines by andrewnester · Pull Request #3225 · databricks/cli

andrewnester · 2025-07-10T10:31:40Z

Changes

Removed deprecated path fallback mechanism for jobs and pipelines

Effectively, it removes an old deprecated behaviour and only relies on the new one introduced here #1273

What's changing?

Assuming there are the following 2 bundle configuration files in your bundle

./resources/my_job.yml - this is where the job is defined
./override.yml - this is where the notebook path of the job task is overridden

./resources/my_job.yml
resources:
jobs:
my_job:
name: "my job"
tasks:
- task_key: notebook_example
job_cluster_key: default
notebook_task:
# if not overridden, this resolves to ./resources/original_notebook.py
notebook_path: ./original_notebook.py

./override.yml
targets:
development:
resources:
jobs:
my_job:
tasks:
- task_key: notebook_example
notebook_task:
# before this PR: this resolves to <BUNDLE_ROOT>/../src/notebook.py first and if that cannot be done, resolves to <BUNDLE_ROOT>/src/notebook.py
# with this PR: this resolves only to <BUNDLE_ROOT>/../src/notebook.py
notebook_path: ../src/notebook.py

Before this change, on bundle validate / deploy the notebook_path for notebook_example would resolve to ./src/notebook.py relative to the bundle root because the relative path was relative to where the job is defined (/resources/my_job.yml)

After this change, the notebook_path for notebook_example would resolve to ../src/notebook.py relative to the bundle root because now the path is calculated relative to where it's defined (./override.yml)

Action items

If you are affected by this change, you can do one of the following:

(recommended) Make necessary configuration changes to have relative paths defined relative to configuration files they are in, or
Pin the CLI version which you use to 0.261.0 or earlier

Tests

Covered by existing acceptance tests

eng-dev-ecosystem-bot · 2025-07-10T10:45:48Z

Run: 17207257792

	Env	✅‌pass	🙈‌skip
✅‌	aws linux	310	486
✅‌	aws windows	311	485
✅‌	aws-ucws linux	424	384
✅‌	aws-ucws windows	425	383
✅‌	azure linux	310	485
✅‌	azure windows	311	484
✅‌	azure-ucws linux	426	381
✅‌	azure-ucws windows	427	380
✅‌	gcp linux	309	487
✅‌	gcp windows	310	486

shreyas-goenka

This needs an entry in NEXT_CHANGELOG

shreyas-goenka · 2025-07-25T12:55:55Z

 ### CLI

 ### Bundles
+* [Breaking Change] Convert warning about using fallback paths to error ([#3225](https://github.com/databricks/cli/pull/3225))


Should we combine the two changelog entries into one? Someone without context might assume that there were two breaking changes.

I don't mind both but I prefer to keep PRs links with titles like we normally do but call out what's notable additionally

pietern · 2025-08-27T09:16:17Z

+pipeline definition location instead of the configuration file location. Previously, the CLI would show a
+warning and fallback to resolving the path relative to the resource location. Users must update their bundle
+configurations to define all relative paths relative to the configuration file where the path is specified.
+See more details here: ([#3225](https://github.com/databricks/cli/pull/3225))


This sounds scary, but it only affects users with a configuration that already displays warnings. It removes behavior that was already considered deprecated. Is it possible to make it sound less scary?

## Release v0.266.0 ### Notable Changes * Breaking change: DABs now return an error when paths are incorrectly defined relative to the job or pipeline definition location instead of the configuration file location. Previously, the CLI would show a warning and fallback to resolving the path relative to the resource location. Users must update their bundle configurations to define all relative paths relative to the configuration file where the path is specified. See more details here: ([#3225](#3225)) * Add support volumes in Python support ([#3383])(#3383)) ### Bundles * [Breaking Change] Remove deprecated path fallback mechanism for jobs and pipelines ([#3225](#3225)) * Add support for Lakebase synced database tables in DABs ([#3467](#3467)) * Rename Delta Live Tables to Lakeflow Declarative Pipelines in the default-python template ([#3476](#3476)). * Fixed bundle init not working on Standard tier ([#3496](#3496))

…and pipelines (#3225) ## Changes Removed deprecated path fallback mechanism for jobs and pipelines Effectively, it removes an old deprecated behaviour and only relies on the new one introduced here #1273 ## What's changing? Assuming there are the following 2 bundle configuration files in your bundle - `./resources/my_job.yml` - this is where the job is defined - `./override.yml` - this is where the notebook path of the job task is overridden ``` ./resources/my_job.yml resources: jobs: my_job: name: "my job" tasks: - task_key: notebook_example job_cluster_key: default notebook_task: # if not overridden, this resolves to ./resources/original_notebook.py notebook_path: ./original_notebook.py ``` ``` ./override.yml targets: development: resources: jobs: my_job: tasks: - task_key: notebook_example notebook_task: # before this PR: this resolves to <BUNDLE_ROOT>/../src/notebook.py first and if that cannot be done, resolves to <BUNDLE_ROOT>/src/notebook.py # with this PR: this resolves only to <BUNDLE_ROOT>/../src/notebook.py notebook_path: ../src/notebook.py ``` **Before** this change, on `bundle validate / deploy` the notebook_path for notebook_example would resolve to `./src/notebook.py` relative to the bundle root because the relative path was relative to where the job is defined (`/resources/my_job.yml`) **After** this change, the notebook_path for notebook_example would resolve to `../src/notebook.py` relative to the bundle root because now the path is calculated relative to where it's defined (`./override.yml`) ## Action items If you are affected by this change, you can do one of the following: 1. **(recommended)** Make necessary configuration changes to have relative paths defined relative to configuration files they are in, or 2. Pin the CLI version which you use to 0.261.0 or earlier ## Tests Covered by existing acceptance tests

## Release v0.266.0 ### Notable Changes * Breaking change: DABs now return an error when paths are incorrectly defined relative to the job or pipeline definition location instead of the configuration file location. Previously, the CLI would show a warning and fallback to resolving the path relative to the resource location. Users must update their bundle configurations to define all relative paths relative to the configuration file where the path is specified. See more details here: ([#3225](#3225)) * Add support volumes in Python support ([#3383])(#3383)) ### Bundles * [Breaking Change] Remove deprecated path fallback mechanism for jobs and pipelines ([#3225](#3225)) * Add support for Lakebase synced database tables in DABs ([#3467](#3467)) * Rename Delta Live Tables to Lakeflow Declarative Pipelines in the default-python template ([#3476](#3476)). * Fixed bundle init not working on Standard tier ([#3496](#3496))

andrewnester requested review from anton-107, denik, pietern and shreyas-goenka as code owners July 10, 2025 10:31

andrewnester temporarily deployed to test-trigger-is July 10, 2025 10:31 — with GitHub Actions Inactive

andrewnester added the do-not-merge label Jul 10, 2025

andrewnester changed the title ~~Convert warning about using fallback paths to error~~[Breaking Change] Convert warning about using fallback paths to errorJul 10, 2025

shreyas-goenka reviewed Jul 10, 2025
View reviewed changes

Comment threadbundle/config/mutator/translate_paths_jobs.go Outdated
Comment threadbundle/config/mutator/translate_paths_pipelines.go Outdated

shreyas-goenka reviewed Jul 10, 2025
View reviewed changes

Comment threadbundle/config/mutator/translate_paths_pipelines.go

andrewnester force-pushed the fix/fallback-path-error branch from 0db4578 to 3461407Compare July 25, 2025 11:21

andrewnester requested a review from shreyas-goenka July 25, 2025 11:21

andrewnester temporarily deployed to test-trigger-is July 25, 2025 11:21 — with GitHub Actions Inactive

andrewnester removed the do-not-merge label Jul 25, 2025

shreyas-goenka approved these changes Jul 25, 2025
View reviewed changes

andrewnester temporarily deployed to test-trigger-is July 25, 2025 13:49 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is July 28, 2025 08:45 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is July 28, 2025 08:46 — with GitHub Actions Inactive

andrewnester changed the title ~~[Breaking Change] Convert warning about using fallback paths to error~~[Breaking Change] Remove deprecated path fallback mechanism for jobs and pipelinesJul 28, 2025

andrewnester temporarily deployed to test-trigger-is July 28, 2025 11:00 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is July 28, 2025 11:25 — with GitHub Actions Inactive

andrewnester temporarily deployed to test-trigger-is July 28, 2025 11:27 — with GitHub Actions Inactive

andrewnesterand others added 8 commits August 25, 2025 13:09

Convert warning about using fallback paths to error
69c3603

fixes
0b3a312

changelog
a96fd9d

removed trailing slashes
aa3ad3d

Update NEXT_CHANGELOG.md
02daa25

include option to use earlier version
04fd8af

also for pipelines
de6f8b1

merge changelog
ce5a4ca

andrewnester force-pushed the fix/fallback-path-error branch from 4581d2b to ce5a4caCompare August 25, 2025 11:15

andrewnester temporarily deployed to test-trigger-is August 25, 2025 11:15 — with GitHub Actions Inactive

-
dff6707

andrewnester temporarily deployed to test-trigger-is August 25, 2025 11:16 — with GitHub Actions Inactive

-
8c509cf

andrewnester temporarily deployed to test-trigger-is August 25, 2025 11:16 — with GitHub Actions Inactive

denik approved these changes Aug 25, 2025
View reviewed changes

andrewnester enabled auto-merge August 25, 2025 11:23

andrewnester added this pull request to the merge queue Aug 25, 2025

Merged via the queue into main with commit 30c24ffAug 25, 2025
13 checks passed

andrewnester deleted the fix/fallback-path-error branch August 25, 2025 11:59

pietern reviewed Aug 27, 2025
View reviewed changes

Navigation Menu

Uh oh!

[Breaking Change] Remove deprecated path fallback mechanism for jobs and pipelines - #3225

[Breaking Change] Remove deprecated path fallback mechanism for jobs and pipelines#3225
andrewnester merged 10 commits into
mainfrom
fix/fallback-path-error

andrewnester commented Jul 10, 2025•
edited by denik
Loading

Uh oh!

eng-dev-ecosystem-bot commented Jul 10, 2025•
edited
Loading

Uh oh!

Uh oh!

Uh oh!

shreyas-goenka left a comment

Uh oh!

Uh oh!

shreyas-goenkaJul 25, 2025

Uh oh!

andrewnesterJul 25, 2025

Uh oh!

Uh oh!

Uh oh!

pieternAug 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

andrewnester commented Jul 10, 2025• edited by denik Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

What's changing?

Action items

Tests

Uh oh!

eng-dev-ecosystem-bot commented Jul 10, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shreyas-goenka left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shreyas-goenkaJul 25, 2025

Choose a reason for hiding this comment

Uh oh!

andrewnesterJul 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pieternAug 27, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

andrewnester commented Jul 10, 2025•
edited by denik
Loading

eng-dev-ecosystem-bot commented Jul 10, 2025•
edited
Loading