DABs cleanup of dist and .internal causing locked wheel versions to fail

[magnusasperud]

Describe the issue

We build a wheel with setup.py, incrementing version as we go. We prefer to lock our jobs to specific versions of the wheel to prevent errors down the road.

If we create a job locked to version 0.1.0 and bumping it to 0.2.0, the deployement will fail. Running databricks bundle deploy cleans up the dist directory, causing the deployment to fail with error

Error: no files match pattern: dist/core-0.1.0*.whl
 at resources.jobs.job_locked_version.tasks[1].libraries[0].whl
 in resources\jobs\job_locked_version.yml:21:15

If the deployment were to go through, running the job would also fail due to the missing wheel in .internal.

Configuration

Job locked to a specific version:

resources:
 jobs:
 job_locked_version:
 name: job_locked_version

 tasks:
 - task_key: some_task
 job_cluster_key: job_cluster
 notebook_task:
 notebook_path: ../../src/integration/nb_integration_example.ipynb
 libraries:
 - whl: ../../dist/core-0.1.0*.whl

 job_clusters:
 - job_cluster_key: job_cluster
 new_cluster:
 spark_version: 15.4.x-scala2.12
 azure_attributes:
 availability: ON_DEMAND_AZURE
 node_type_id: Standard_D3_v2
 enable_elastic_disk: true
 num_workers: 1
 data_security_mode: SINGLE_USER 

Steps to reproduce the behavior

1. Prep the bundle so it builds a wheel.
2. Create a job locked to a specific version of the wheel.
3. Deploy to verify success.
4. Bump the version in the wheel.
5. Redeploy

Expected Behavior

• Ability to disable DABs validation of wheel existence in dist.
• Previous wheel versions are retained in .internal.

Actual Behavior

• Deploy fails due to missing wheel in dist.
• Job runs fail due to missing wheel in .internal.

OS and CLI version

• Databricks CLI v0.252.0
• Windows 11 Enterprise

Is this a regression?

Yes for the problem related to .internal, changed in #2526.

[andrewnester]

If the deployment were to go through, running the job would also fail due to the missing wheel in .internal.

Why would it fail if the job will use a new version? If you mean existing active runs will be changed, then cleaning up .internal folder indeed will impact it. See #2958
Instead, you can use --fail-on-active-runs which will stop the deployment if there is any active job runs in progress. Changing the assets while the jobs are running can lead to unexpected behaviour anyway.

For cleaning up dist locally, @denik can you take a look?

[magnusasperud]

Why would it fail if the job will use a new version? If you mean existing active runs will be changed, then cleaning up .internal folder indeed will impact it. See #2958 Instead, you can use --fail-on-active-runs which will stop the deployment if there is any active job runs in progress. Changing the assets while the jobs are running can lead to unexpected behaviour anyway.

@andrewnester If a job references a .whl with a specific version and .internal cleans up on deployments, the old referenced version won't exist in .internal

libraries:
 - whl: ../../dist/core-0.1.0*.whl

This file won't exist if we bump the version.

This might be by design, but it would be very helpful to lock jobs to specific versions of a wheel. Breaking changes will happen down the road, and with the current design this will be difficult to handle inside DABs.

[andrewnester]

@magnusasperud if you want to have multiple versions of wheels being used, you need to keep them local as well. For this you can store old built wheels locally in a separate folder and make the jobs to use it instead. Is it something that works for you?

[magnusasperud]

skip_artifacts_cleanup would be very nice - thanks for the attention to this issue!

[andrewnester]

The fix for keeping .internal folder was just released in 0.254.0
I'll keep this open for discussion on keeping local dist folder

[magnusasperud]

Great, experimental.skip_artifact_cleanup worked as expected.

After reconsidering, I'm not sure keeping the local dist folder is the best approach. Would you consider making DABs skip validation of the path in resources.jobs.[job_key].[task_key].libraries when experimental.skip_artifact_cleanup is enabled?

[andrewnester]

We still have to keep validation of local libraries, but instead for older wheels you can use a full remote path to the library, so it will look like

libraries:
 - whl: ${workspace.artifact_path}/.internal/core-0.1.0*.whl

[magnusasperud]

I assumed validation was only for local files - I wasn’t aware you could reference remote files as well. Thanks for clarifying!

From my side, the issue is resolved. Thanks again.