Patch references to UC schemas to capture dependencies automatically

[shreyas-goenka]

Changes

Fixes#1977.

This PR modifies the bundle configuration to capture the dependency that a UC Volume or a DLT pipeline might have on a UC schema at deployment time. It does so by replacing the schema name with a reference of the form ${resources.schemas.foo.name}.

For example:
The following UC Volume definition depends on the UC schema with the name schema_name. This mutator converts this configuration

from:

resources:
volumes:
bar:
catalog_name: catalog_name
name: volume_name
schema_name: schema_name
schemas:
foo:
catalog_name: catalog_name
name: schema_name

to:

resources:
volumes:
bar:
catalog_name: catalog_name
name: volume_name
schema_name: ${resources.schemas.foo.name}`
schemas:
foo:
catalog_name: catalog_name
name: schema_name

Tests

Unit tests and manually.

[eng-dev-ecosystem-bot]

Test Details: go/deco-tests/12239986056

[shreyas-goenka]

The API allows creating a pipeline without a schema/target specified as long as you are not using UC. It's not a forward-looking use case anyway, so I did not look deeper into this.

[denik]

It would be painful to update this test if the message changes, but it's not nice to assert on the message.

I have added a "golden files" test utilities here #2025 , it's a good fit for this cases.

Full output is stored in the file and can be updated by running tests with TESTS_OUTPUT=OVERWRITE

[denik]

Note, that PR only targets integration tests for now, so not ready for your use case yet.

[shreyas-goenka]

I think it's fine in this case since its unit tests that are colocated with the mutator that implements the functionality. It'll be a simple search and replace when updating.

Are you proposing to extend the golden file approach to mutators or unit tests? Possibly by serializing the diagnostics and persisting them in a file?

[pietern]

I understand that we can ask users to capture this dependency through variable interpolation, but could we do this ourselves instead? I.e. we know if someone wants to use a schema for a volume, so we can inject a "depends on" relationship transparently.

[github-actions]

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

• PR number: 1989
• Commit SHA: aa52b1dab3354c56886cae7740b2cb73fb5ade69

Checks will be approved automatically on success.