Add DABs support for Unity Catalog volumes

[shreyas-goenka]

Changes

This PR adds support for UC volumes to DABs.

Can I use a UC volume managed by DABs in artifact_path?

Yes, but we require the volume to exist before being referenced in artifact_path. Otherwise you'll see an error that the volume does not exist. For this case, this PR also adds a warning if we detect that the UC volume is defined in the DAB itself, which informs the user to deploy the UC volume in a separate deployment first before using it in artifact_path.

We cannot create the UC volume and then upload the artifacts to it in the same bundle deploy because bundle deploy always uploads the artifacts to artifact_path before materializing any resources defined in the bundle. Supporting this in a single deployment requires us to migrate away from our dependency on the Databricks Terraform provider to manage the CRUD lifecycle of DABs resources.

Why do we not support preset.name_prefix for UC volumes?

UC volumes will not have a dev_shreyas_goenka prefix added in mode: development. Configuring presets.name_prefix will be a no-op for UC volumes. We have decided not to support prefixing for UC resources. This is because:

1. UC provides its own namespace hierarchy that is independent of DABs.
2. Users can always manually use ${workspace.current_user.short_name} to configure the prefixes manually.

Customers often manually set up a UC hierarchy for dev and prod, including a schema or catalog per developer. Thus, it's often unnecessary for us to add prefixing in mode: development by default for UC resources.

In retrospect, supporting prefixing for UC schemas and registered models was a mistake and will be removed in a future release of DABs.

Tests

Unit, integration test, and manually.

Manual Testing cases:

1. UC volume does not exist:

➜ bundle-playground git:(master) ✗ cli bundle deploy
Error: failed to fetch metadata for the UC volume /Volumes/main/caps/my_volume that is configured in the artifact_path: Not Found

2. UC Volume does not exist, but is defined in the DAB

➜ bundle-playground git:(master) ✗ cli bundle deploy
Error: failed to fetch metadata for the UC volume /Volumes/main/caps/managed_by_dab that is configured in the artifact_path: Not Found
Warning: You might be using a UC volume in your artifact_path that is managed by this bundle but which has not been deployed yet. Please deploy the UC volume in a separate bundle deploy before using it in the artifact_path.
at resources.volumes.bar
in databricks.yml:24:7

[shreyas-goenka]

This PR introduces yet again interactive flow to DABs. At this point, I think it's worth it to invest some time in adding proper end to end tests for this. I'd like to pick it up in the next ~month.

[shreyas-goenka]

Let's pick this next quarter as in scope for terminal UX improvements.

[shreyas-goenka]

Nightlies passed once. Triggered another round after some minor updates.

[pietern]

Thanks! Remaining comments on error/warning text.

[pietern]

One more on consistency in messages.

Approving to unblock merge once that is addressed.

[pietern]

This results in:

Error: volume /Volumes/my/artifact/path does not exist: Not Found

The suffix is superfluous.

[pietern]

Discussed this on DM: error conditions need more work but will be done in a separate PR.

[github-actions]

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

• PR number: 1762
• Commit SHA: d460bd68591f016748d1b06f635e648b82e077e0

Checks will be approved automatically on success.

[eng-dev-ecosystem-bot]

Test Details: go/deco-tests/12122596769