Update Go SDK to 0.23.0 and use custom marshaller by hectorcast-db · Pull Request #772 · databricks/cli

hectorcast-db · 2023-09-13T11:07:21Z

Changes

Update Go SDK to 0.23.0 and use custom marshaller.

Tests

Run unit tests
Run nightly
Manual test:

./cli jobs create --json @myjob.json

with

{
"name": "my-job-marshal-test-go",
"tasks": [{
"task_key": "testgomarshaltask",
"new_cluster": {
"num_workers": 0,
"spark_version": "10.4.x-scala2.12",
"node_type_id": "Standard_DS3_v2"
},
"libraries": [
{
"jar": "dbfs:/max/jars/exampleJarTask.jar"
}
],
"spark_jar_task": {
"main_class_name": "com.databricks.quickstart.exampleTask"
}
}]
}

Main branch:

Error: Cluster validation error: Missing required field: settings.cluster_spec.new_cluster.size

This branch:

{
"job_id":<jobid>
}

hectorcast-db · 2023-09-13T11:51:37Z

Integration tests
https://github.com/databricks/eng-dev-ecosystem/actions/runs/6171774958/job/16750686211

mgyucht

LGTM with one nit, will give official stamp once we remove go.mod changes.

mgyucht

one micronit, otherwise LGTM! Feel free to address and click the merge when ready button.

Note: this release includes a fix for the issue where zero values (for example `num_workers: 0`) were not included in the request body. CLI: * Use already instantiated WorkspaceClient in sync command ([#867](#867)). Bundles: * Support Unity Catalog Registered Models in bundles ([#846](#846)). * Fixed merging task libraries from targets ([#868](#868)). * Add alias for mlops-stack template URL ([#869](#869)). API Changes: * Changed `databricks account billable-usage download` command to start returning output. * Changed `databricks account storage-credentials delete` command with new required argument order. * Changed `databricks account storage-credentials get` command with new required argument order. * Changed `databricks account storage-credentials update` command with new required argument order. * Added `databricks workspace-bindings get-bindings` command. * Added `databricks workspace-bindings update-bindings` command. * Removed `databricks account network-policy` command group. * Changed `databricks ip-access-lists list` command to return output. OpenAPI commit 493a76554afd3afdd15dc858773d01643f80352a (2023-10-12) Dependency updates: * Update Go SDK to 0.23.0 and use custom marshaller ([#772](#772)). * Bump Terraform provider to v1.28.0 ([#871](#871)). * Bump golang.org/x/net from 0.16.0 to 0.17.0 ([#863](#863)). * Bump github.com/hashicorp/hc-install from 0.6.0 to 0.6.1 ([#870](#870)).

The update to the Go SDK v0.23.0 in #772 included a change to make the billable usage API return its streaming response. This still did not make the command print out the CSV returned by the API, however. To do so, we call `cmdio.RenderReader` in case the response is a byte stream. Fixes#574.

## Changes The update to the Go SDK v0.23.0 in #772 included a change to make the billable usage API return its streaming response. This still did not make the command print out the CSV returned by the API, however. To do so, we call `cmdio.RenderReader` in case the response is a byte stream. Note: there is an opportunity to parse the CSV and return JSON if requested, but that is out of scope for this PR (it is a rather big customization of the command). Fixes#574. ## Tests Manually confirmed that `databricks account billable-usage download` now returns CSV.

## Changes Update Go SDK to 0.23.0 and use custom marshaller. ## Tests * Run unit tests * Run nightly * Manual test: ``` ./cli jobs create --json @myjob.json ``` with ``` { "name": "my-job-marshal-test-go", "tasks": [{ "task_key": "testgomarshaltask", "new_cluster": { "num_workers": 0, "spark_version": "10.4.x-scala2.12", "node_type_id": "Standard_DS3_v2" }, "libraries": [ { "jar": "dbfs:/max/jars/exampleJarTask.jar" } ], "spark_jar_task": { "main_class_name": "com.databricks.quickstart.exampleTask" } }] } ``` Main branch: ``` Error: Cluster validation error: Missing required field: settings.cluster_spec.new_cluster.size ``` This branch: ``` { "job_id":<jobid> } ``` --------- Co-authored-by: Miles Yucht <miles@databricks.com>

Note: this release includes a fix for the issue where zero values (for example `num_workers: 0`) were not included in the request body. CLI: * Use already instantiated WorkspaceClient in sync command ([#867](#867)). Bundles: * Support Unity Catalog Registered Models in bundles ([#846](#846)). * Fixed merging task libraries from targets ([#868](#868)). * Add alias for mlops-stack template URL ([#869](#869)). API Changes: * Changed `databricks account billable-usage download` command to start returning output. * Changed `databricks account storage-credentials delete` command with new required argument order. * Changed `databricks account storage-credentials get` command with new required argument order. * Changed `databricks account storage-credentials update` command with new required argument order. * Added `databricks workspace-bindings get-bindings` command. * Added `databricks workspace-bindings update-bindings` command. * Removed `databricks account network-policy` command group. * Changed `databricks ip-access-lists list` command to return output. OpenAPI commit 493a76554afd3afdd15dc858773d01643f80352a (2023-10-12) Dependency updates: * Update Go SDK to 0.23.0 and use custom marshaller ([#772](#772)). * Bump Terraform provider to v1.28.0 ([#871](#871)). * Bump golang.org/x/net from 0.16.0 to 0.17.0 ([#863](#863)). * Bump github.com/hashicorp/hc-install from 0.6.0 to 0.6.1 ([#870](#870)).

## Changes The update to the Go SDK v0.23.0 in #772 included a change to make the billable usage API return its streaming response. This still did not make the command print out the CSV returned by the API, however. To do so, we call `cmdio.RenderReader` in case the response is a byte stream. Note: there is an opportunity to parse the CSV and return JSON if requested, but that is out of scope for this PR (it is a rather big customization of the command). Fixes#574. ## Tests Manually confirmed that `databricks account billable-usage download` now returns CSV.

hectorcast-db added the do-not-merge label Sep 13, 2023

andrewnester reviewed Sep 14, 2023
View reviewed changes

Comment threadbundle/config/resources/job.go Outdated
Comment threadinternal/jobs_test.go Outdated

hectorcast-db requested a review from mgyucht September 20, 2023 11:39

mgyucht reviewed Sep 22, 2023
View reviewed changes

Comment threadinternal/jobs_test.go Outdated

hectorcast-db marked this pull request as draft October 4, 2023 11:10

andrewnester reviewed Oct 4, 2023
View reviewed changes

Comment threadgo.sum Outdated

hectorcast-db removed the do-not-merge label Oct 13, 2023

hectorcast-db marked this pull request as ready for review October 13, 2023 08:54

hectorcast-db requested a review from pietern October 13, 2023 08:56

hectorcast-db changed the title ~~Use custom marshaller~~Update SDK-Go to 0.23.0 and use custom marshallerOct 13, 2023

mgyucht mentioned this pull request Oct 13, 2023
Use latest version of the Go SDK #865
Closed

pietern changed the title ~~Update SDK-Go to 0.23.0 and use custom marshaller~~Update Go SDK to 0.23.0 and use custom marshallerOct 13, 2023

pietern reviewed Oct 13, 2023
View reviewed changes

Comment threadinternal/jobs_test.go Outdated
Comment threadinternal/jobs_test.go Outdated

pietern approved these changes Oct 13, 2023
View reviewed changes

hectorcast-db added the do-not-merge label Oct 13, 2023

hectorcast-db force-pushed the custom-marshaller-test branch 2 times, most recently from ddbbcdc to af8ffabCompare October 13, 2023 12:42

andrewnester approved these changes Oct 13, 2023
View reviewed changes

hectorcast-dband others added 7 commits October 13, 2023 14:47

Use custom marshaller
06d9100

Update dependency
e0e8bfc

Add integration test
9c377f5

Address PR comments
316bea5

Delete job after test
25d5040

Use latest version of the Go SDK
7711a9b

Improve test
90e48c8

hectorcast-db force-pushed the custom-marshaller-test branch from af8ffab to 90e48c8Compare October 13, 2023 12:56

hectorcast-db removed the do-not-merge label Oct 13, 2023

mgyucht approved these changes Oct 13, 2023
View reviewed changes

Comment threadinternal/jobs_test.go Outdated

Rename test file
98df23d

hectorcast-db added this pull request to the merge queue Oct 16, 2023

Merged via the queue into main with commit 36f30c8Oct 16, 2023

hectorcast-db deleted the custom-marshaller-test branch October 16, 2023 07:02

pietern mentioned this pull request Oct 16, 2023
Release v0.208.0 #875
Merged

pietern mentioned this pull request Oct 16, 2023
Fix rendering of streaming response #876
Merged

Navigation Menu

Uh oh!

Update Go SDK to 0.23.0 and use custom marshaller - #772

Update Go SDK to 0.23.0 and use custom marshaller#772
hectorcast-db merged 8 commits into
mainfrom
custom-marshaller-test

hectorcast-db commented Sep 13, 2023•
edited by pietern
Loading

Uh oh!

hectorcast-db commented Sep 13, 2023

Uh oh!

Uh oh!

Uh oh!

mgyucht left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mgyucht left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

hectorcast-db commented Sep 13, 2023• edited by pietern Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Tests

Uh oh!

hectorcast-db commented Sep 13, 2023

Uh oh!

Uh oh!

Uh oh!

mgyucht left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mgyucht left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

hectorcast-db commented Sep 13, 2023•
edited by pietern
Loading