DABs name_prefix manipulates strings in undesirable ways

[kyle-kent]

Describe the issue

Setting a customname_prefix for DAB targets gets manipulated in an undesirable way. It truncates multiple _ down to one, and removes them entirely from the end of the string.

With a schema named qc, something as simple as name_prefix: ${workspace.current_user.short_name}_ (trailing underscore) gets manipulated to user_nameqc. No trailing underscore.

Configuration

# schemas.yaml
resources:
 schemas:
 qc:
 name: qc
 catalog_name: my_catalog

# databricks.yaml
targets:
 dev:
 mode: development
 presets:
 name_prefix: ${workspace.current_user.short_name}_

Steps to reproduce the behavior

Please list the steps required to reproduce the issue, for example:

1. databricks bundle deploy -t dev

Expected Behavior

If user name is my_user then output should be my_catalog.my_user_qc.

If you allow custom inputs don't change them. We also use multiple underscores frequently. It for some reason convers __ to _.

Actual Behavior

If user name is my_user then output is my_catalog.my_userqc.

OS and CLI version

Databricks CLI v0.239.1
macos sequoia 15.3

Is this a regression?

Did this work in a previous version of the CLI? If so, which versions did you try?
Idk

Debug Logs

Output logs if you run the command with debug logs enabled. Example: databricks bundle deploy --log-level=debug. Redact if needed

[LeoTheGriff]

If you allow custom inputs don't change them. We also use multiple underscores frequently. It for some reason convers __ to _.

Have you tried upgrading to v0.240.0?: Added support for double underscore variable references (#2203

I don't know about the trailing underscore.

[kyle-kent]

@LeoTheGriff upgrading to that version did not allow for double underscores

[LeoTheGriff]

OK, good to know. I'm just a customer so can't help further, sorry.

[cornerman]

Afaik, it acutally only happens to schemas created in DAB. Other resources like jobs get the proper name_prefix including any trailing underscores.

Would like to see this fixed as well - though this will now be difficult without breaking existing deployments.

[shreyas-goenka]

@kyle-kent Thanks for reporting this issue.

@cornerman This issue would indeed be difficult to patch without breaking existing deployments. Meanwhile to workaround this, the normalization rules include conversion of ] to _, so a prefix of '[dev shreyas] is automatically normalized to dev_shreyas_ for schemas.

This should help mitigate this issue.

[github-actions]

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[oliverbeagley-pgg]

While this is marked as not planned, this also affects model_serving_endpoints and registered_models.

[sbachstein]

We are currently stumbling across the same issue.

Maybe a bit of context: Generally, we are pretty happy with the default prefix coming with mode: development. However, there is no way for us to get the actual (schema) prefix as a variable to pass to our Python jobs. Instead we have to pass every single (prefixed) schema name by referencing ${resources.schemas.my_schema.name}, which results in a long list of parameters that have to be passed. This is why we are forced to set the name_prefix manually.

Coming back to the actual issue: This seems impossible because a manually set name_prefix, which of course has some kind of trailing separator like an underscore (otherwise the schema name would be barely readable), gets trimmed away.

I consider this a serious flaw which makes the feature practically unusable whenever you want to actually reference the resources by prefix.

@andrewnester Is the "Not planned" decision final and do you have a proposal in mind on how to work around this issue?

[majidfn]

We are facing the same issue.

Based on the CLI's code, I don't think this is an expected behavior.

The function normalizePrefix is called in here and defined here.

There the comment it says:

// If the prefix ends with a ']', we add an underscore to the end.
// This makes sure that we get names like "dev_user_endpoint" instead of "dev_userendpoint"

Then the next line immediately calls textutil.NormalizeString (defined here) which removes all the trailing _s.
It is contradicting the previous comment and intention.