Libraries requirements not correctly replaced by Workspace path

[ovidiu-eremia]

Describe the issue

We use DABs to deploy workflow jobs that run python_wheel_tasks from the package loaded using the tasks libraries as a whl artifact with a requirements txt file. Both the whl and the requirements txt files are loaded providing the local relative path to them from the workflow yml where they are referenced, to the repository local folder where they are located.

Configuration

# yaml-language-server: $schema=../../bundle_config_schema.json

resources:
 jobs:
 ingest:
 name: ingest

 tasks:
 - task_key: ingest
 job_cluster_key: default
 libraries:
 - requirements: ../../deps/requirements.txt
 - whl: ../../dist/*.whl
 python_wheel_task:
 package_name: my_package
 entry_point: my_package.entry_point
 parameters:
 - "--execution-time"
 - "{{ job.trigger.time.iso_datetime }}"
 - "--environment"
 - "${bundle.target}"

 job_clusters:
 - job_cluster_key: default
 new_cluster:
 spark_version: 15.4.x-scala2.12
 node_type_id: Standard_DS3_v2
 num_workers: 0
 spark_conf:
 spark.master: local[*, 4]
 spark.databricks.cluster.profile: singleNode
 custom_tags:
 ResourceClass: SingleNode
 azure_attributes:
 first_on_demand: 1
 availability: SPOT_WITH_FALLBACK_AZURE
 spot_bid_max_price: -1

Steps to reproduce the behavior

Please list the steps required to reproduce the issue, for example:

1. Run databricks bundle deploy ...
2. Run databricks bundle run ...
3. See error from the cluster driver:

run failed with error message [LibraryInstallationError] 
 Library installation failed for library due to user error. Error messages:
Requirements file must be stored in UC Volumes, dbfs, s3, adls, gs or as a workspace file/local file. Make sure the URI begins with 'dbfs:', 'file:', 's3:', 'abfss:', 'gs:', 'wasbs:', '/Volumes', or '/Workspace'but the URI is '/Shared/.bundle/target/ingest/files/deps/requirements.txt').

Expected Behavior

The dependent libraries requirements file path should be similar to the one of the deployed library whl artifact:

/Workspace/Shared/.bundle/target/ingest/artifacts/.internal/ingest-0.0.1-py3-none-any.whl

like this:

/Workspace/Shared/.bundle/target/ingest/files/deps/requirements.txt

Actual Behavior

The path to the requirements file is in fact:

/Shared/.bundle/target/ingest/files/deps/requirements.txt

OS and CLI version

Databricks CLI: v0.227.1
OS: Debian - Ubuntu 22.04

Is this a regression?

I think this was supposed to be fixed in this PR:
#1543

[andrewnester]

Yes, indeed that's the case at the moment. This is going to be fix with this PR #1724

[frankivo]

Workaround available here: #1517 (comment)