Skip to content

Added support for job environments - #1379

Merged
andrewnester merged 8 commits into
mainfrom
feature/environment-key
Apr 22, 2024
Merged

Added support for job environments#1379
andrewnester merged 8 commits into
mainfrom
feature/environment-key

Conversation

@andrewnester

@andrewnesterandrewnester commented Apr 19, 2024

Copy link
Copy Markdown
Contributor

Changes

The main changes are:

  1. Don't link artifacts to libraries anymore and instead just iterate over all jobs and tasks when uploading artifacts and update local path to remote
  2. Iterating over jobs.environments to check if there are any local libraries and checking that they exist locally
  3. Added tests to check environments are handled correctly

End-to-end test will follow up

Tests

Added regression test, existing tests (including integration one) pass

pietern
pietern reviewed Apr 19, 2024
View reviewed changes
Comment threadbundle/artifacts/artifacts.go Outdated
Comment threadbundle/artifacts/artifacts.go Outdated
Comment threadbundle/libraries/libraries.go Outdated
Comment threadbundle/libraries/libraries.go Outdated
Comment threadbundle/libraries/match.go
Comment threadbundle/tests/environment_key/databricks.yml
Comment threadbundle/tests/python_wheel_test.go
pietern
pietern reviewed Apr 22, 2024
View reviewed changes
Comment threadbundle/artifacts/artifacts.go
}

// libPath can be a glob pattern, so do the match first
matches, err := filepath.Glob(libPath)

@pieternpieternApr 22, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't the globs have been expanded already at this point?

@andrewnesterandrewnesterApr 22, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pietern I don't think we do this in translate paths now but we should but can be a follow up

Comment threadbundle/libraries/local_path.go
Comment threadbundle/libraries/local_path.go
pietern
pietern reviewed Apr 22, 2024
View reviewed changes
Comment threadbundle/artifacts/artifacts.go
if lib.Whl != "" {
lib.Whl = remotePath
continue
// TODO: confirm if we still need to update the remote path to start with /Workspace

@pieternpieternApr 22, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you file an issue for this and link it here? Worth checking indeed (can be async), but we should track it.

@andrewnesterandrewnesterApr 22, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added here #1388

@andrewnester

andrewnester commented Apr 22, 2024

Copy link
Copy Markdown
ContributorAuthor

Integration tests are green

@andrewnester
andrewnester added this pull request to the merge queueApr 22, 2024
Merged via the queue into main with commit 1872aa1Apr 22, 2024
@andrewnester
andrewnester deleted the feature/environment-key branch April 22, 2024 11:51
@pieternpietern mentioned this pull request Apr 23, 2024
Merged
pietern added a commit that referenced this pull request Apr 23, 2024
@pietern
Release v0.218.0
29425f5
This release marks the general availability of Databricks Asset Bundles.
CLI:
* Publish Docker images ([#1353](#1353)).
* Add support for multi-arch Docker images ([#1362](#1362)).
* Do not prefill https:// in prompt for Databricks Host ([#1364](#1364)).
* Add better documentation for the `auth login` command ([#1366](#1366)).
* Add URLs for authentication documentation to the auth command help ([#1365](#1365)).
Bundles:
* Fix compute override for foreach tasks ([#1357](#1357)).
* Transform artifact files source patterns in build not upload stage ([#1359](#1359)).
* Convert between integer and float in normalization ([#1371](#1371)).
* Disable locking for development mode ([#1302](#1302)).
* Resolve variable references inside variable lookup fields ([#1368](#1368)).
* Added validate mutator to surface additional bundle warnings ([#1352](#1352)).
* Upgrade terraform-provider-databricks to 1.40.0 ([#1376](#1376)).
* Print host in `bundle validate` when passed via profile or environment variables ([#1378](#1378)).
* Cleanup remote file path on bundle destroy ([#1374](#1374)).
* Add docs URL for `run_as` in error message ([#1381](#1381)).
* Enable job queueing by default ([#1385](#1385)).
* Added support for job environments ([#1379](#1379)).
* Processing and completion of positional args to bundle run ([#1120](#1120)).
* Add legacy option for `run_as` ([#1384](#1384)).
API Changes:
* Changed `databricks lakehouse-monitors cancel-refresh` command with new required argument order.
* Changed `databricks lakehouse-monitors create` command with new required argument order.
* Changed `databricks lakehouse-monitors delete` command with new required argument order.
* Changed `databricks lakehouse-monitors get` command with new required argument order.
* Changed `databricks lakehouse-monitors get-refresh` command with new required argument order.
* Changed `databricks lakehouse-monitors list-refreshes` command with new required argument order.
* Changed `databricks lakehouse-monitors run-refresh` command with new required argument order.
* Changed `databricks lakehouse-monitors update` command with new required argument order.
* Changed `databricks account workspace-assignment update` command to return response.
OpenAPI commit 94684175b8bd65f8701f89729351f8069e8309c9 (2024-04-11)
Dependency updates:
* Bump github.com/databricks/databricks-sdk-go from 0.37.0 to 0.38.0 ([#1361](#1361)).
* Bump golang.org/x/net from 0.22.0 to 0.23.0 ([#1380](#1380)).
github-merge-queueBot pushed a commit that referenced this pull request Apr 23, 2024
@pietern
Release v0.218.0 (#1391)
27d35d5
This release marks the general availability of Databricks Asset Bundles.
CLI:
* Publish Docker images
([#1353](#1353)).
* Add support for multi-arch Docker images
([#1362](#1362)).
* Do not prefill https:// in prompt for Databricks Host
([#1364](#1364)).
* Add better documentation for the `auth login` command
([#1366](#1366)).
* Add URLs for authentication documentation to the auth command help
([#1365](#1365)).
Bundles:
* Fix compute override for foreach tasks
([#1357](#1357)).
* Transform artifact files source patterns in build not upload stage
([#1359](#1359)).
* Convert between integer and float in normalization
([#1371](#1371)).
* Disable locking for development mode
([#1302](#1302)).
* Resolve variable references inside variable lookup fields
([#1368](#1368)).
* Added validate mutator to surface additional bundle warnings
([#1352](#1352)).
* Upgrade terraform-provider-databricks to 1.40.0
([#1376](#1376)).
* Print host in `bundle validate` when passed via profile or environment
variables ([#1378](#1378)).
* Cleanup remote file path on bundle destroy
([#1374](#1374)).
* Add docs URL for `run_as` in error message
([#1381](#1381)).
* Enable job queueing by default
([#1385](#1385)).
* Added support for job environments
([#1379](#1379)).
* Processing and completion of positional args to bundle run
([#1120](#1120)).
* Add legacy option for `run_as`
([#1384](#1384)).
API Changes:
* Changed `databricks lakehouse-monitors cancel-refresh` command with
new required argument order.
* Changed `databricks lakehouse-monitors create` command with new
required argument order.
* Changed `databricks lakehouse-monitors delete` command with new
required argument order.
* Changed `databricks lakehouse-monitors get` command with new required
argument order.
* Changed `databricks lakehouse-monitors get-refresh` command with new
required argument order.
* Changed `databricks lakehouse-monitors list-refreshes` command with
new required argument order.
* Changed `databricks lakehouse-monitors run-refresh` command with new
required argument order.
* Changed `databricks lakehouse-monitors update` command with new
required argument order.
* Changed `databricks account workspace-assignment update` command to
return response.
OpenAPI commit 94684175b8bd65f8701f89729351f8069e8309c9 (2024-04-11)
Dependency updates:
* Bump github.com/databricks/databricks-sdk-go from 0.37.0 to 0.38.0
([#1361](#1361)).
* Bump golang.org/x/net from 0.22.0 to 0.23.0
([#1380](#1380)).
denik pushed a commit that referenced this pull request May 20, 2026
@andrewnester
Added support for job environments (#1379)
bef7401
## Changes
The main changes are:
1. Don't link artifacts to libraries anymore and instead just iterate
over all jobs and tasks when uploading artifacts and update local path
to remote
2. Iterating over `jobs.environments` to check if there are any local
libraries and checking that they exist locally
3. Added tests to check environments are handled correctly
End-to-end test will follow up
## Tests
Added regression test, existing tests (including integration one) pass
denik pushed a commit that referenced this pull request May 20, 2026
@pietern
Release v0.218.0 (#1391)
e1042af
This release marks the general availability of Databricks Asset Bundles.
CLI:
* Publish Docker images
([#1353](#1353)).
* Add support for multi-arch Docker images
([#1362](#1362)).
* Do not prefill https:// in prompt for Databricks Host
([#1364](#1364)).
* Add better documentation for the `auth login` command
([#1366](#1366)).
* Add URLs for authentication documentation to the auth command help
([#1365](#1365)).
Bundles:
* Fix compute override for foreach tasks
([#1357](#1357)).
* Transform artifact files source patterns in build not upload stage
([#1359](#1359)).
* Convert between integer and float in normalization
([#1371](#1371)).
* Disable locking for development mode
([#1302](#1302)).
* Resolve variable references inside variable lookup fields
([#1368](#1368)).
* Added validate mutator to surface additional bundle warnings
([#1352](#1352)).
* Upgrade terraform-provider-databricks to 1.40.0
([#1376](#1376)).
* Print host in `bundle validate` when passed via profile or environment
variables ([#1378](#1378)).
* Cleanup remote file path on bundle destroy
([#1374](#1374)).
* Add docs URL for `run_as` in error message
([#1381](#1381)).
* Enable job queueing by default
([#1385](#1385)).
* Added support for job environments
([#1379](#1379)).
* Processing and completion of positional args to bundle run
([#1120](#1120)).
* Add legacy option for `run_as`
([#1384](#1384)).
API Changes:
* Changed `databricks lakehouse-monitors cancel-refresh` command with
new required argument order.
* Changed `databricks lakehouse-monitors create` command with new
required argument order.
* Changed `databricks lakehouse-monitors delete` command with new
required argument order.
* Changed `databricks lakehouse-monitors get` command with new required
argument order.
* Changed `databricks lakehouse-monitors get-refresh` command with new
required argument order.
* Changed `databricks lakehouse-monitors list-refreshes` command with
new required argument order.
* Changed `databricks lakehouse-monitors run-refresh` command with new
required argument order.
* Changed `databricks lakehouse-monitors update` command with new
required argument order.
* Changed `databricks account workspace-assignment update` command to
return response.
OpenAPI commit 94684175b8bd65f8701f89729351f8069e8309c9 (2024-04-11)
Dependency updates:
* Bump github.com/databricks/databricks-sdk-go from 0.37.0 to 0.38.0
([#1361](#1361)).
* Bump golang.org/x/net from 0.22.0 to 0.23.0
([#1380](#1380)).
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pieternpieternpietern approved these changes
@shreyas-goenkashreyas-goenkaAwaiting requested review from shreyas-goenka

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@andrewnester@pietern