[FEATURE] DAB: Support for YAML Dictionary variables

[stevenayers-bge]

When writing Databricks Asset Bundles, it would be great if we could define things like job cluster configuration as one config block:

 targets:
 dev:
 variables:
 spark_version: 14.3.x-scala2.12
 pool_id: 12334
 cluster_config:
 spark_version: ${var.spark_version}
 instance_pool_id: ${var.pool_id}
 data_security_mode: SINGLE_USER
 runtime_engine: PHOTON
 spark_conf:
 spark.master: local[*, 4]
 spark.databricks.cluster.profile: singleNode
 custom_tags:
 ResourceClass: SingleNode
 spark_env_vars:
 PYSPARK_PYTHON: /databricks/python3/bin/python3
 num_workers: 0
 prod:
 variables:
 spark_version: 14.3.x-scala2.12
 pool_id: aaabbbbcccc
 cluster_config:
 spark_version: ${var.spark_version}
 instance_pool_id: ${var.pool_id}
 data_security_mode: SINGLE_USER
 runtime_engine: PHOTON
 autoscale:
 min_workers: 10
 max_workers: 999999
 spark_env_vars:
 PYSPARK_PYTHON: /databricks/python3/bin/python3

resources:
 jobs:
 job1:
 name: job1
 git_source:
 git_url: ${var.git_url}
 git_provider: ${var.git_provider}
 git_branch: ${var.git_branch}
 tasks:
 - task_key: task1
 spark_python_task:
 python_file: task1.py
 source: GIT
 parameters:
 - --environment
 - ${bundle.target}
 job_cluster_key: cluster
 job_clusters:
 - job_cluster_key: cluster
 new_cluster: ${var.cluster_config}

It would also be nice to be able to group variables using this method:

variables:
 instance_pools:
 description: The pools the use for jobs
 default:
 driver:
 large: id here
 small: id here
 worker:
 large: id here
 small: id here
 
 cluster_config:
 spark_version: ${var.spark_version}
 driver_instance_pool_id: ${var.instance_pools.driver.large}
 instance_pool_id: ${var.instance_pools.worker.large}
 data_security_mode: SINGLE_USER
 runtime_engine: PHOTON
 autoscale:
 min_workers: 10
 max_workers: 999999
 spark_env_vars:
 PYSPARK_PYTHON: /databricks/python3/bin/python3

Please let me know if this is in progress, and either way I'll see if I can help contribute :)

[andrewnester]

@stevenayers-bge is YAML anchors helpful in your case?

[stevenayers-bge]

@stevenayers-bge is YAML anchors helpful in your case?

Thanks for the reply 🙂 so it does, but it also feels messy to be honest. Maintaining your basic type variables in one way and your complex types in another.

My understanding of YAML anchors is that you also can't access nested properties either.

[stevenayers-bge]

@andrewnester it's also worth mentioning that YAML anchors are only valid in the file they are defined in, so you can't define them where you define your variables, and then re-use them elsewhere.

[andrewnester]

Indeed, you're correct on the points above. YAML anchors can be a solution in certain cases. Other than that, we've received multiple similar requests and planning to introduce some functionality which should improve the experience

[stevenayers-bge]

Indeed, you're correct on the points above. YAML anchors can be a solution in certain cases. Other than that, we've received multiple similar requests and planning to introduce some functionality which should improve the experience

@andrewnester ok thanks that would be great. Is there a PR for this work already? If not I can look at kicking it off?

[andrewnester]

There's no PR for this yet, we still need to figure out certain details about how we can surface this functionality in DABs configuration

[kenmyers-8451]

@stevenayers-bge is YAML anchors helpful in your case?

@andrewnester it's also worth mentioning that YAML anchors are only valid in the file they are defined in, so you can't define them where you define your variables, and then re-use them elsewhere.

@andrewnester I saw that you said you've gotten similar requests but just wanted to add one more data point to it. I was actually just going to make a similar feature request regarding job cluster definition when I found this thread. My teams running into the exact problem @stevenayers-bge mentions with anchors. We have been using a single bundle file with anchors for defining the job cluster definitions which we then reference in each of the jobs. However, since the file is getting really long — with 5 jobs so far, each with many tasks, and more to come — we wanted to try breaking the jobs out into their own files, but this throws an error right away because the anchor pointers don't have the anchor definitions in these sub-files, and it doesn't make sense to copy the anchor definitions to each sub-file as then we have to make updates to all of those files anytime we change the cluster.

So first idea that popped into my head was what if the job clusters were defined as a resource or similar top level item. I also like the dictionary idea suggested here.

[stevenayers-bge]

@kenmyers-8451 👍 we have 9 jobs at the moment, and at the moment we're defining the YAML anchors in each our environment yml files and then defining them at target level (target > resources > jobs > job_name) but it's already starting to look really messy.

@andrewnester I'm keen to help contribute this feature if I get time. Do you know in which submodules there would need to be changes? Is it all in bundle/config/variable and bundle/config/mutator?