Skip to content

Add support for Lakehouse monitoring in bundles - #1307

Merged
pietern merged 22 commits into
databricks:mainfrom
aravind-segu:lakehouseMonitoring
May 31, 2024
Merged

Add support for Lakehouse monitoring in bundles#1307
pietern merged 22 commits into
databricks:mainfrom
aravind-segu:lakehouseMonitoring

Conversation

@aravind-segu

@aravind-seguaravind-segu commented Mar 24, 2024•
edited by pietern
Loading

Copy link
Copy Markdown
Contributor

Changes

This change adds support for Lakehouse monitoring in bundles.

The associated resource type name is "quality monitor".

Testing

Unit tests.

aravind-segu
aravind-segu commented Mar 24, 2024
View reviewed changes
Comment threadlibs/dyn/convert/normalize.go Outdated
aravind-segu
aravind-segu commented Mar 24, 2024
View reviewed changes
Comment threadlibs/dyn/convert/struct_info.go Outdated
@pieternpietern changed the title [MLOPS-651] Support Lakehouse monitoring in BundlesSupport Lakehouse monitoring in bundlesMar 25, 2024
pietern
pietern reviewed Mar 25, 2024
View reviewed changes
Comment threadbundle/internal/tf/schema/resource_lakehouse_monitor.go Outdated
Comment threadbundle/config/resources.go Outdated
Comment threadbundle/deploy/terraform/tfdyn/convert_lakehouse_monitor_test.go Outdated
Comment threadbundle/schema/openapi.go Outdated
Comment threadlibs/dyn/convert/normalize.go Outdated
Comment threadlibs/dyn/convert/normalize.go Outdated
Comment threadlibs/dyn/convert/struct_info.go Outdated
pietern
pietern reviewed Mar 25, 2024
View reviewed changes

@pieternpietern left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the patch. Please reach out directly if you want to chat about the approach.

andrewnester
andrewnester reviewed Mar 25, 2024
View reviewed changes
Comment threadbundle/config/mutator/process_target_mode_test.go Outdated
@aravind-segu

aravind-segu commented Apr 15, 2024

Copy link
Copy Markdown
ContributorAuthor

Regarding the hacky approach This looks pretty hacky. Can't we use the url tag of the field?. The url field is empty as well. CreateMonitor takes in the table name as part of the query parameters and it is not passed in json body.

@aravind-segu

aravind-segu commented Apr 22, 2024

Copy link
Copy Markdown
ContributorAuthor

Tests are failing because of the change to not avoid json - parameters in struct_info. How can we solve this? Is it ok to modify the tests?

andrewnester
andrewnester requested changes Apr 23, 2024
View reviewed changes

@andrewnesterandrewnester left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FullName needs to be renamed to TableName because it was renamed in a new GoSDK version being used

Comment threadbundle/deploy/terraform/convert_test.go Outdated
Comment threadbundle/deploy/terraform/convert_test.go Outdated
Comment threadbundle/tests/lakehouse_monitor_test.go Outdated
Comment threadlibs/dyn/convert/struct_info.go Outdated
Comment threadbundle/deploy/terraform/tfdyn/convert_lakehouse_monitor.go Outdated
@codecov-commenter

codecov-commenter commented May 6, 2024•
edited
Loading

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 51.25000% with 39 lines in your changes are missing coverage. Please review.

Project coverage is 53.58%. Comparing base (e22dd8a) to head (a2384c1).
Report is 124 commits behind head on main.

FilesPatch %Lines
bundle/config/resources.go0.00%15 Missing and 3 partials ⚠️
libs/dyn/convert/struct_info.go0.00%6 Missing and 1 partial ⚠️
bundle/deploy/terraform/convert.go70.00%5 Missing and 1 partial ⚠️
.../deploy/terraform/tfdyn/convert_quality_monitor.go64.70%4 Missing and 2 partials ⚠️
bundle/deploy/terraform/interpolate.go0.00%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## main #1307 +/- ##
==========================================
+ Coverage 52.25% 53.58% +1.33% 
==========================================
Files 317 351 +34 Lines 18004 20268 +2264 ==========================================
+ Hits 9408 10861 +1453 - Misses 7903 8609 +706 - Partials 693 798 +105

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@arpitjasa-db
arpitjasa-db requested a review from pieternMay 6, 2024 20:33
pietern
pietern reviewed May 15, 2024
View reviewed changes
Comment threadbundle/config/mutator/process_target_mode_test.go Outdated
Comment threadbundle/config/mutator/run_as_test.go
Comment threadbundle/deploy/terraform/convert_test.go Outdated
Comment threadbundle/schema/openapi.go Outdated
Comment threadbundle/tests/monitor/databricks.yml
Comment threadlibs/dyn/convert/struct_info.go Outdated
pietern
pietern reviewed May 31, 2024
View reviewed changes
Comment threadbundle/config/mutator/process_target_mode_test.go Outdated
Comment threadbundle/config/resources/quality_monitor.go Outdated
Comment threadbundle/config/resources/quality_monitor.go Outdated
Comment threadbundle/deploy/terraform/convert_test.go
Comment threadbundle/schema/openapi.go Outdated
Comment threadlibs/dyn/convert/struct_info.go Outdated
@pietern

pietern commented May 31, 2024

Copy link
Copy Markdown
Contributor

The TF bump is happening in #1460.

@pieternpietern changed the title Support Lakehouse monitoring in bundlesAdd support for Lakehouse monitoring in bundlesMay 31, 2024
andrewnester
andrewnester approved these changes May 31, 2024
View reviewed changes

@andrewnesterandrewnester left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved with a minor comment

Comment threadlibs/dyn/convert/struct_info.go Outdated
@pietern
pietern enabled auto-merge May 31, 2024 09:37
@pietern
pietern disabled auto-merge May 31, 2024 09:37
@pietern
pietern enabled auto-merge May 31, 2024 09:40
@pietern
pietern added this pull request to the merge queueMay 31, 2024
Merged via the queue into databricks:main with commit a33d0c8May 31, 2024
@arpitjasa-db
arpitjasa-db deleted the lakehouseMonitoring branch May 31, 2024 18:37
pietern added a commit that referenced this pull request Jun 4, 2024
@pietern
Release v0.221.0
4e249ce
CLI:
* Update OpenAPI spec ([#1466](#1466)).
Bundles:
* Upgrade TF provider to 1.46.0 ([#1460](#1460)).
* Add support for Lakehouse monitoring ([#1307](#1307)).
* Make dbt-sql and default-sql templates public ([#1463](#1463)).
Internal:
* Abstract over filesystem interaction with libs/vfs ([#1452](#1452)).
* Add `filer.Filer` to read notebooks from WSFS without omitting their extension ([#1457](#1457)).
* Fix listing notebooks in a subdirectory ([#1468](#1468)).
API Changes:
* Changed `databricks account storage-credentials list` command to return .
* Added `databricks consumer-listings batch-get` command.
* Added `databricks consumer-providers batch-get` command.
* Removed `databricks apps create-deployment` command.
* Added `databricks apps deploy` command.
OpenAPI commit 37b925eba37dfb3d7e05b6ba2d458454ce62d3a0 (2024-06-03)
Dependency updates:
* Bump github.com/hashicorp/go-version from 1.6.0 to 1.7.0 ([#1454](#1454)).
* Bump github.com/hashicorp/hc-install from 0.6.4 to 0.7.0 ([#1453](#1453)).
@pieternpietern mentioned this pull request Jun 4, 2024
Merged
github-merge-queueBot pushed a commit that referenced this pull request Jun 4, 2024
@pietern
Release v0.221.0 (#1470)
f8b2cb8
CLI:
* Update OpenAPI spec
([#1466](#1466)).
Bundles:
* Upgrade TF provider to 1.46.0
([#1460](#1460)).
* Add support for Lakehouse monitoring
([#1307](#1307)).
* Make dbt-sql and default-sql templates public
([#1463](#1463)).
Internal:
* Abstract over filesystem interaction with libs/vfs
([#1452](#1452)).
* Add `filer.Filer` to read notebooks from WSFS without omitting their
extension ([#1457](#1457)).
* Fix listing notebooks in a subdirectory
([#1468](#1468)).
API Changes:
* Changed `databricks account storage-credentials list` command to
return .
* Added `databricks consumer-listings batch-get` command.
* Added `databricks consumer-providers batch-get` command.
* Removed `databricks apps create-deployment` command.
* Added `databricks apps deploy` command.
OpenAPI commit 37b925eba37dfb3d7e05b6ba2d458454ce62d3a0 (2024-06-03)
Dependency updates:
* Bump github.com/hashicorp/go-version from 1.6.0 to 1.7.0
([#1454](#1454)).
* Bump github.com/hashicorp/hc-install from 0.6.4 to 0.7.0
([#1453](#1453)).
@pieternpietern mentioned this pull request Nov 14, 2024
Merged
github-merge-queueBot pushed a commit that referenced this pull request Nov 14, 2024
@pietern
MakeTableNamefield part of quality monitor schema (#1903)
1db3840
## Changes
This field was special-cased in #1307 because it's not part of the JSON
payload in the SDK struct.
This approach, while pragmatic, meant it didn't show up in the JSON
schema. While debugging an issue with quality monitors in #1900, I
couldn't figure out why I was getting schema errors on this field, or
how it was passed through to the TF representation. This commit removes
the special case and makes it behave like everything else.
## Tests
* Unit tests pass.
* Confirmed that the updated schema failed validation before this
change.
denik pushed a commit that referenced this pull request May 20, 2026
@aravind-segu@pietern@arpitjasa-db
Add support for Lakehouse monitoring in bundles (#1307)
f130b8c
## Changes
This change adds support for Lakehouse monitoring in bundles.
The associated resource type name is "quality monitor".
## Testing
Unit tests.
---------
Co-authored-by: Pieter Noordhuis <pcnoordhuis@gmail.com>
Co-authored-by: Pieter Noordhuis <pieter.noordhuis@databricks.com>
Co-authored-by: Arpit Jasapara <87999496+arpitjasa-db@users.noreply.github.com>
denik pushed a commit that referenced this pull request May 20, 2026
@pietern
Release v0.221.0 (#1470)
877046b
CLI:
* Update OpenAPI spec
([#1466](#1466)).
Bundles:
* Upgrade TF provider to 1.46.0
([#1460](#1460)).
* Add support for Lakehouse monitoring
([#1307](#1307)).
* Make dbt-sql and default-sql templates public
([#1463](#1463)).
Internal:
* Abstract over filesystem interaction with libs/vfs
([#1452](#1452)).
* Add `filer.Filer` to read notebooks from WSFS without omitting their
extension ([#1457](#1457)).
* Fix listing notebooks in a subdirectory
([#1468](#1468)).
API Changes:
* Changed `databricks account storage-credentials list` command to
return .
* Added `databricks consumer-listings batch-get` command.
* Added `databricks consumer-providers batch-get` command.
* Removed `databricks apps create-deployment` command.
* Added `databricks apps deploy` command.
OpenAPI commit 37b925eba37dfb3d7e05b6ba2d458454ce62d3a0 (2024-06-03)
Dependency updates:
* Bump github.com/hashicorp/go-version from 1.6.0 to 1.7.0
([#1454](#1454)).
* Bump github.com/hashicorp/hc-install from 0.6.4 to 0.7.0
([#1453](#1453)).
denik pushed a commit that referenced this pull request May 20, 2026
@pietern
MakeTableNamefield part of quality monitor schema (#1903)
2092be8
## Changes
This field was special-cased in #1307 because it's not part of the JSON
payload in the SDK struct.
This approach, while pragmatic, meant it didn't show up in the JSON
schema. While debugging an issue with quality monitors in #1900, I
couldn't figure out why I was getting schema errors on this field, or
how it was passed through to the TF representation. This commit removes
the special case and makes it behave like everything else.
## Tests
* Unit tests pass.
* Confirmed that the updated schema failed validation before this
change.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pieternpieternpietern approved these changes
@andrewnesterandrewnesterandrewnester approved these changes
@arpitjasa-dbarpitjasa-dbAwaiting requested review from arpitjasa-db

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@aravind-segu@codecov-commenter@pietern@andrewnester@arpitjasa-db