Example of a job with tasks that run other jobs

[pietern]

Note: this requires databricks/cli#1219.

[bit0x43]

I encountered multiple errors while testing this approach. It appears that variable substitutions do not work with this level of nesting. Do you have any suggestions or workarounds to fix this issue?

Traceback

❯ databricksbundledeployBuildingdefault...
Uploadingsample-1.80.2-py3-none-any.whl...
Uploadingbundlefilesto/Users/myuser/.bundle/sample_core/local/files...
Deployingresources...
Updatingdeploymentstate...
Error: terraformapply: exitstatus1Error: cannotcreatejob: Missingrequiredfield: settings.tasks.task_key.
withdatabricks_job.sample_mlops_main_stage,
onbundle.tf.jsonline56, inresource.databricks_job.sample_mlops_main_stage:
56: },
Error: cannotupdatejob: Missingrequiredfield: new_settings.webhook_notifications.on_failure.idwithdatabricks_job.sample_mlops_stage_1__models_training_and_serving,
onbundle.tf.jsonline1362, inresource.databricks_job.sample_mlops_stage_1__models_training_and_serving:
1362: },

Config files

databricks.yml

...
variables:
webhook_notifications_id:
description: The ID of the webhook notification to use for the job.default: 'xzy-xyz-uxz'training_and_serving_job1_job_id:
default: ''

sample_job.yml

...
resources:
jobs:
sample_mlops_stage_1__models_training_and_serving:
name: 'sample_mlops_stage_1__models_training_and_serving'tasks:
- task_key: job1_idlibraries:
- whl: ../dist/*.whlrun_if: ALL_SUCCESSemail_notifications: {}run_job_task:
job_id: ${var.job1_id}
...
webhook_notifications:
on_failure:
- id: ${var.webhook_notifications_id}

[pietern]

@cristian-rincon Which version of the CLI are you using? Support for this was released in v0.214.0.

[bit0x43]

@cristian-rincon Which version of the CLI are you using? Support for this was released in v0.214.0.

I'm using Databricks CLI v0.213.0, i will test with v0.214.0

[bit0x43]

@cristian-rincon Which version of the CLI are you using? Support for this was released in v0.214.0.

I'm using Databricks CLI v0.213.0, i will test with v0.214.0

@pietern
I have upgraded the CLI and now i got this error:

Traceback

❯ databricksbundledeployBuildingdefault...
Uploadingsample-1.80.2-py3-none-any.whl...
Uploadingbundlefilesto/Users/myuser/.bundle/sample_core/local/files...
Deployingresources...
Updatingdeploymentstate...
Error: terraformapply: exitstatus1Error: Missingrequiredargumentonbundle.tf.jsonline51, inresource.databricks_job.sample_mlops_main_stage.task[0].run_job_task:
51: "run_job_task": {},
Theargument"job_id"isrequired, butnodefinitionwasfound.

main.yml

resources:
jobs:
sample_mlops_main_stage:
name: 'sample_mlops_main_stage'tasks:
- task_key: abtrun_job_task:
# job_id: 234902340127014job_id: ${resources.jobs.sample_mlops_stage_0__job1.id}email_notifications: {}libraries:
- whl: ../dist/*.whlrun_if: ALL_SUCCESS

[pietern]

Could you post this as an issue on the CLI repository, and include the relevant bits of your bundle validate output.

The YAML looks valid if it is used verbatim. Note that a libraries section is not relevant for a run_job_task.

[bit0x43]

Could you post this as an issue on the CLI repository, and include the relevant bits of your bundle validate output.

The YAML looks valid if it is used verbatim. Note that a libraries section is not relevant for a run_job_task.

@pietern I changed the job name by removing the double underscore, and it fixed the issue.
for ex:

• Non working solution : sample_mlops_stage_0__job1
• Working solution: sample_mlops_stage_0_job1