Uh oh!
There was an error while loading. Please reload this page.
Add Python decorator example - #84
Conversation
| # Python decorator | ||
| This example shows how to package a **Python decorator–based job** so you can | ||
| call regular Python functions from a Databricks job with minimal boilerplate. |
There was a problem hiding this comment.
Can you make it more clear this is about Python support for DABs?
E.g. include a doc link in this paragraph.
| raise ValueError("Only keyword arguments are supported") | ||
| func_full_name = f"{self.func.__module__}:{self.func.__name__}" | ||
| parameters: list = [func_full_name] + [f"--{k}={v}" for k, v in kwargs.items()] |
There was a problem hiding this comment.
With string conversion + argparse turning it from a string back into the type that the function accepts, does this limit the function argument types in a particular way?
| from my_project.python_wheel_task import python_wheel_task | ||
| @python_wheel_task |
There was a problem hiding this comment.
For this mini-framework, should this not just be called @task? In a more extensive version, we could always decide to use subtypes like @task.sql or move to top-level task types like @sql_file_task.
| # Makes the value available to downstream tasks as | ||
| # '{{ tasks.<task_key>.values.message }}' | ||
| dbutils.jobs.taskValues.set("message", "Hello World") |
There was a problem hiding this comment.
Could you make this example do something Sparky, like spark.read.table("samples.nyctaxi.trips")?
Add an example of using Python decorators in Databricks jobs, see README.md for more information.