Skip to content

Add DatabricksNotebookOperator - #39178

Merged
Lee-W merged 9 commits into
apache:mainfrom
astronomer:contribute-databricks-notebook-operator
Apr 26, 2024
Merged

Add DatabricksNotebookOperator#39178
Lee-W merged 9 commits into
apache:mainfrom
astronomer:contribute-databricks-notebook-operator

Conversation

@pankajkoti

@pankajkotipankajkoti commented Apr 22, 2024•
edited
Loading

Copy link
Copy Markdown
Member

This pull request introduces the DatabricksNotebookOperator
to the Airflow Databricks provider from the astro-provider-databricks
repository.
It marks one of the initial pull requests aimed at contributing
operators and features from that repository into the Airflow
Databricks provider.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@pankajkoti
pankajkotiforce-pushed the contribute-databricks-notebook-operator branch 2 times, most recently from a98d220 to d5fb847CompareApril 22, 2024 19:46
tatiana
tatiana reviewed Apr 23, 2024
View reviewed changes

@tatianatatiana left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's looking very good, @pankajkoti , thanks for contributing this!

Comment threadairflow/providers/databricks/operators/databricks.py Outdated
Comment threadairflow/providers/databricks/operators/databricks.py Outdated
Comment threadairflow/providers/databricks/operators/databricks.py Outdated
Comment threadairflow/providers/databricks/operators/databricks.py Outdated
Comment threadairflow/providers/databricks/operators/databricks.py Outdated
Comment threadairflow/providers/databricks/operators/databricks.py Outdated
@pankajkotipankajkoti changed the title Contribute databricks notebook operatorContribute DatabricksNotebookOperator from astro-provider-databricks repoApr 23, 2024
pankajkoti
pankajkoti commented Apr 23, 2024
View reviewed changes
Comment threadairflow/providers/databricks/operators/databricks.py Outdated
@pankajkoti
pankajkoti requested a review from tatianaApril 23, 2024 11:42
@pankajkoti
pankajkotiforce-pushed the contribute-databricks-notebook-operator branch from fb7b1a3 to bd08d9dCompareApril 23, 2024 11:43
@Lee-W
Lee-W requested review from Lee-W and removed request for tatianaApril 23, 2024 13:23
@pankajkoti
pankajkoti requested a review from tatianaApril 23, 2024 13:57
@pankajkoti
pankajkoti marked this pull request as ready for review April 24, 2024 06:44
@eladkaleladkal changed the title Contribute DatabricksNotebookOperator from astro-provider-databricks repoAdd DatabricksNotebookOperatorApr 24, 2024
Lee-W
Lee-W reviewed Apr 25, 2024
View reviewed changes
Comment threadairflow/providers/databricks/operators/databricks.py Outdated
Comment threadairflow/providers/databricks/operators/databricks.py Outdated
Comment threadairflow/providers/databricks/operators/databricks.py Outdated
Comment threadairflow/providers/databricks/operators/databricks.py Outdated
Comment threadairflow/providers/databricks/operators/databricks.py Outdated
Comment threadairflow/providers/databricks/operators/databricks.py Outdated
Comment threadairflow/providers/databricks/operators/databricks.py Outdated
Comment threadtests/providers/databricks/operators/test_databricks.py Outdated
Comment threadtests/providers/databricks/operators/test_databricks.py Outdated
Comment threadtests/providers/databricks/operators/test_databricks.py Outdated
@pankajkoti
pankajkotiforce-pushed the contribute-databricks-notebook-operator branch from be84ec0 to 20dacc7CompareApril 25, 2024 13:05
@pankajkoti
pankajkoti requested a review from Lee-WApril 25, 2024 13:06
@Lee-W
Lee-W merged commit 7683344 into apache:mainApr 26, 2024
@Lee-W
Lee-W deleted the contribute-databricks-notebook-operator branch April 26, 2024 08:41
@rawwarrawwar mentioned this pull request Apr 28, 2024
Merged
@eladkaleladkal mentioned this pull request May 1, 2024
Closed
This was referenced May 14, 2024
Open
Open
pankajkoti pushed a commit that referenced this pull request May 14, 2024
@rawwar
add deferrable support toDatabricksNotebookOperator(#39295)
1e4663f
related: #39178
This PR intends to make DatabricksNotebookOperator deferrable
@eladkaleladkal mentioned this pull request May 26, 2024
Closed
@pankajkotipankajkoti mentioned this pull request May 28, 2024
Merged
pankajkoti added a commit that referenced this pull request May 30, 2024
@pankajkoti@dimberman
@tatiana@Lee-W
Add DatabricksWorkflowTaskGroup (#39771)
2ecf7fa
This pull request introduces the [DatabricksWorkflowTaskGroup](https://github.com/astronomer/astro-provider-databricks/blob/main/src/astro_databricks/operators/workflow.py#L226)
to the Airflow Databricks provider from the [astro-provider-databricks](https://github.com/astronomer/astro-provider-databricks/tree/main)
repository. It marks another pull request aimed at contributing operators and features from that repository into the Airflow Databricks provider, the previous PR being #39178.
The task group launches a [Databricks Workflow](https://docs.databricks.com/en/workflows/index.html) and runs the notebook jobs from within it, resulting in a [75% cost reduction](https://www.databricks.com/product/pricing) ($0.40/DBU for all-purpose compute, $0.07/DBU for Jobs compute) when compared to executing ``DatabricksNotebookOperator`` outside of ``DatabricksWorkflowTaskGroup``.
---------
Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>
Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
fdemiane pushed a commit to fdemiane/airflow that referenced this pull request Jun 6, 2024
@pankajkoti@dimberman
@tatiana@Lee-W@fdemiane
Add DatabricksWorkflowTaskGroup (apache#39771)
cba9c45
This pull request introduces the [DatabricksWorkflowTaskGroup](https://github.com/astronomer/astro-provider-databricks/blob/main/src/astro_databricks/operators/workflow.py#L226)
to the Airflow Databricks provider from the [astro-provider-databricks](https://github.com/astronomer/astro-provider-databricks/tree/main)
repository. It marks another pull request aimed at contributing operators and features from that repository into the Airflow Databricks provider, the previous PR being apache#39178.
The task group launches a [Databricks Workflow](https://docs.databricks.com/en/workflows/index.html) and runs the notebook jobs from within it, resulting in a [75% cost reduction](https://www.databricks.com/product/pricing) ($0.40/DBU for all-purpose compute, $0.07/DBU for Jobs compute) when compared to executing ``DatabricksNotebookOperator`` outside of ``DatabricksWorkflowTaskGroup``.
---------
Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>
Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Jul 26, 2024
@rawwar@romsharon98
add deferrable support toDatabricksNotebookOperator(apache#39295)
2230e52
related: apache#39178
This PR intends to make DatabricksNotebookOperator deferrable
romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Jul 26, 2024
@pankajkoti@dimberman
@tatiana@Lee-W@romsharon98
Add DatabricksWorkflowTaskGroup (apache#39771)
25cea3c
This pull request introduces the [DatabricksWorkflowTaskGroup](https://github.com/astronomer/astro-provider-databricks/blob/main/src/astro_databricks/operators/workflow.py#L226)
to the Airflow Databricks provider from the [astro-provider-databricks](https://github.com/astronomer/astro-provider-databricks/tree/main)
repository. It marks another pull request aimed at contributing operators and features from that repository into the Airflow Databricks provider, the previous PR being apache#39178.
The task group launches a [Databricks Workflow](https://docs.databricks.com/en/workflows/index.html) and runs the notebook jobs from within it, resulting in a [75% cost reduction](https://www.databricks.com/product/pricing) ($0.40/DBU for all-purpose compute, $0.07/DBU for Jobs compute) when compared to executing ``DatabricksNotebookOperator`` outside of ``DatabricksWorkflowTaskGroup``.
---------
Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>
Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
@pankajkotipankajkoti mentioned this pull request Aug 5, 2024
Merged
pankajkoti added a commit to astronomer/astro-provider-databricks that referenced this pull request Aug 8, 2024
@pankajkoti
Deprecate the provider and proxy to upstream Airflow provider (#84)
3c858a4
As part of Astronomer's internal plans and decisions, we've decided to contribute the existing functionality provided by the operators and plugins in this repository to the official Apache Airflow Databricks provider. To achieve this, we submitted the following PRs to the Airflow provider:
1. apache/airflow#39178
2. apache/airflow#39771
3. apache/airflow#40013
4. apache/airflow#40724
5. apache/airflow#39295
All functionality has now been contributed to the Airflow Databricks provider, and ongoing support will be maintained there. As a result, we're deprecating the operators and plugins in this repository. Users are encouraged to transition to the official Apache Airflow Databricks provider as soon as possible. The migration process is straightforward—simply update the import path to point to the Airflow provider and ensure that you install `apache-airflow-providers-databricks>=6.8.0`, which includes all the contributions mentioned above.
closes: astronomer/issues-airflow#715
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Nov 9, 2024
@pankajkoti@dimberman
@tatiana@Lee-W
Add DatabricksWorkflowTaskGroup (#39771)
922a7e7
This pull request introduces the [DatabricksWorkflowTaskGroup](https://github.com/astronomer/astro-provider-databricks/blob/main/src/astro_databricks/operators/workflow.py#L226)
to the Airflow Databricks provider from the [astro-provider-databricks](https://github.com/astronomer/astro-provider-databricks/tree/main)
repository.
It marks another pull request aimed at contributing
operators and features from that repository into the Airflow
Databricks provider, the previous PR being apache/airflow#39178.
The task group launches a [Databricks Workflow](https://docs.databricks.com/en/workflows/index.html)
and runs the notebook jobs from within it, resulting in a
[75% cost reduction](https://www.databricks.com/product/pricing) ($0.40/DBU for all-purpose compute,
$0.07/DBU for Jobs compute) when compared to executing
``DatabricksNotebookOperator`` outside of ``DatabricksWorkflowTaskGroup``.
---------
Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>
Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
GitOrigin-RevId: 2ecf7fa07d6d681c73ae4831801f9d98db298d89
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request May 6, 2025
@pankajkoti@dimberman
@tatiana@Lee-W
Add DatabricksWorkflowTaskGroup (#39771)
3e61405
This pull request introduces the [DatabricksWorkflowTaskGroup](https://github.com/astronomer/astro-provider-databricks/blob/main/src/astro_databricks/operators/workflow.py#L226)
to the Airflow Databricks provider from the [astro-provider-databricks](https://github.com/astronomer/astro-provider-databricks/tree/main)
repository.
It marks another pull request aimed at contributing
operators and features from that repository into the Airflow
Databricks provider, the previous PR being apache/airflow#39178.
The task group launches a [Databricks Workflow](https://docs.databricks.com/en/workflows/index.html)
and runs the notebook jobs from within it, resulting in a
[75% cost reduction](https://www.databricks.com/product/pricing) ($0.40/DBU for all-purpose compute,
$0.07/DBU for Jobs compute) when compared to executing
``DatabricksNotebookOperator`` outside of ``DatabricksWorkflowTaskGroup``.
---------
Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>
Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
GitOrigin-RevId: 2ecf7fa07d6d681c73ae4831801f9d98db298d89
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request May 26, 2025
@pankajkoti@dimberman
@tatiana@Lee-W
Add DatabricksWorkflowTaskGroup (#39771)
7672901
This pull request introduces the [DatabricksWorkflowTaskGroup](https://github.com/astronomer/astro-provider-databricks/blob/main/src/astro_databricks/operators/workflow.py#L226)
to the Airflow Databricks provider from the [astro-provider-databricks](https://github.com/astronomer/astro-provider-databricks/tree/main)
repository.
It marks another pull request aimed at contributing
operators and features from that repository into the Airflow
Databricks provider, the previous PR being apache/airflow#39178.
The task group launches a [Databricks Workflow](https://docs.databricks.com/en/workflows/index.html)
and runs the notebook jobs from within it, resulting in a
[75% cost reduction](https://www.databricks.com/product/pricing) ($0.40/DBU for all-purpose compute,
$0.07/DBU for Jobs compute) when compared to executing
``DatabricksNotebookOperator`` outside of ``DatabricksWorkflowTaskGroup``.
---------
Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>
Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
GitOrigin-RevId: 2ecf7fa07d6d681c73ae4831801f9d98db298d89
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Sep 22, 2025
@pankajkoti@dimberman
@tatiana@Lee-W
Add DatabricksWorkflowTaskGroup (#39771)
d02d677
This pull request introduces the [DatabricksWorkflowTaskGroup](https://github.com/astronomer/astro-provider-databricks/blob/main/src/astro_databricks/operators/workflow.py#L226)
to the Airflow Databricks provider from the [astro-provider-databricks](https://github.com/astronomer/astro-provider-databricks/tree/main)
repository.
It marks another pull request aimed at contributing
operators and features from that repository into the Airflow
Databricks provider, the previous PR being apache/airflow#39178.
The task group launches a [Databricks Workflow](https://docs.databricks.com/en/workflows/index.html)
and runs the notebook jobs from within it, resulting in a
[75% cost reduction](https://www.databricks.com/product/pricing) ($0.40/DBU for all-purpose compute,
$0.07/DBU for Jobs compute) when compared to executing
``DatabricksNotebookOperator`` outside of ``DatabricksWorkflowTaskGroup``.
---------
Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>
Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
GitOrigin-RevId: 2ecf7fa07d6d681c73ae4831801f9d98db298d89
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Oct 19, 2025
@pankajkoti@dimberman
@tatiana@Lee-W
Add DatabricksWorkflowTaskGroup (#39771)
7dc64e1
This pull request introduces the [DatabricksWorkflowTaskGroup](https://github.com/astronomer/astro-provider-databricks/blob/main/src/astro_databricks/operators/workflow.py#L226)
to the Airflow Databricks provider from the [astro-provider-databricks](https://github.com/astronomer/astro-provider-databricks/tree/main)
repository.
It marks another pull request aimed at contributing
operators and features from that repository into the Airflow
Databricks provider, the previous PR being apache/airflow#39178.
The task group launches a [Databricks Workflow](https://docs.databricks.com/en/workflows/index.html)
and runs the notebook jobs from within it, resulting in a
[75% cost reduction](https://www.databricks.com/product/pricing) ($0.40/DBU for all-purpose compute,
$0.07/DBU for Jobs compute) when compared to executing
``DatabricksNotebookOperator`` outside of ``DatabricksWorkflowTaskGroup``.
---------
Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>
Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
GitOrigin-RevId: 2ecf7fa07d6d681c73ae4831801f9d98db298d89
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Feb 25, 2026
@pankajkoti@dimberman
@tatiana@Lee-W
Add DatabricksWorkflowTaskGroup (#39771)
e5576ae
This pull request introduces the [DatabricksWorkflowTaskGroup](https://github.com/astronomer/astro-provider-databricks/blob/main/src/astro_databricks/operators/workflow.py#L226)
to the Airflow Databricks provider from the [astro-provider-databricks](https://github.com/astronomer/astro-provider-databricks/tree/main)
repository.
It marks another pull request aimed at contributing
operators and features from that repository into the Airflow
Databricks provider, the previous PR being apache/airflow#39178.
The task group launches a [Databricks Workflow](https://docs.databricks.com/en/workflows/index.html)
and runs the notebook jobs from within it, resulting in a
[75% cost reduction](https://www.databricks.com/product/pricing) ($0.40/DBU for all-purpose compute,
$0.07/DBU for Jobs compute) when compared to executing
``DatabricksNotebookOperator`` outside of ``DatabricksWorkflowTaskGroup``.
---------
Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>
Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
GitOrigin-RevId: 2ecf7fa07d6d681c73ae4831801f9d98db298d89
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request May 10, 2026
@pankajkoti@dimberman
@tatiana@Lee-W
Add DatabricksWorkflowTaskGroup (#39771)
30fe5ab
This pull request introduces the [DatabricksWorkflowTaskGroup](https://github.com/astronomer/astro-provider-databricks/blob/main/src/astro_databricks/operators/workflow.py#L226)
to the Airflow Databricks provider from the [astro-provider-databricks](https://github.com/astronomer/astro-provider-databricks/tree/main)
repository.
It marks another pull request aimed at contributing
operators and features from that repository into the Airflow
Databricks provider, the previous PR being apache/airflow#39178.
The task group launches a [Databricks Workflow](https://docs.databricks.com/en/workflows/index.html)
and runs the notebook jobs from within it, resulting in a
[75% cost reduction](https://www.databricks.com/product/pricing) ($0.40/DBU for all-purpose compute,
$0.07/DBU for Jobs compute) when compared to executing
``DatabricksNotebookOperator`` outside of ``DatabricksWorkflowTaskGroup``.
---------
Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>
Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
GitOrigin-RevId: 2ecf7fa07d6d681c73ae4831801f9d98db298d89
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Jul 26, 2026
@pankajkoti@dimberman
@tatiana@Lee-W
Add DatabricksWorkflowTaskGroup (#39771)
ec3c2c1
This pull request introduces the [DatabricksWorkflowTaskGroup](https://github.com/astronomer/astro-provider-databricks/blob/main/src/astro_databricks/operators/workflow.py#L226)
to the Airflow Databricks provider from the [astro-provider-databricks](https://github.com/astronomer/astro-provider-databricks/tree/main)
repository.
It marks another pull request aimed at contributing
operators and features from that repository into the Airflow
Databricks provider, the previous PR being apache/airflow#39178.
The task group launches a [Databricks Workflow](https://docs.databricks.com/en/workflows/index.html)
and runs the notebook jobs from within it, resulting in a
[75% cost reduction](https://www.databricks.com/product/pricing) ($0.40/DBU for all-purpose compute,
$0.07/DBU for Jobs compute) when compared to executing
``DatabricksNotebookOperator`` outside of ``DatabricksWorkflowTaskGroup``.
---------
Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>
Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
GitOrigin-RevId: 2ecf7fa07d6d681c73ae4831801f9d98db298d89
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Jul 27, 2026
@pankajkoti@dimberman
@tatiana@Lee-W
Add DatabricksWorkflowTaskGroup (#39771)
48daa64
This pull request introduces the [DatabricksWorkflowTaskGroup](https://github.com/astronomer/astro-provider-databricks/blob/main/src/astro_databricks/operators/workflow.py#L226)
to the Airflow Databricks provider from the [astro-provider-databricks](https://github.com/astronomer/astro-provider-databricks/tree/main)
repository.
It marks another pull request aimed at contributing
operators and features from that repository into the Airflow
Databricks provider, the previous PR being apache/airflow#39178.
The task group launches a [Databricks Workflow](https://docs.databricks.com/en/workflows/index.html)
and runs the notebook jobs from within it, resulting in a
[75% cost reduction](https://www.databricks.com/product/pricing) ($0.40/DBU for all-purpose compute,
$0.07/DBU for Jobs compute) when compared to executing
``DatabricksNotebookOperator`` outside of ``DatabricksWorkflowTaskGroup``.
---------
Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>
Co-authored-by: Tatiana Al-Chueyr <tatiana.alchueyr@gmail.com>
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
GitOrigin-RevId: 2ecf7fa07d6d681c73ae4831801f9d98db298d89
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Lee-WLee-WLee-W approved these changes
@cmarteepantscmarteepantsAwaiting requested review from cmarteepants
@phanikumvphanikumvAwaiting requested review from phanikumv
@pankajastropankajastroAwaiting requested review from pankajastro
+2 more reviewers
@tatianatatianatatiana approved these changes
@sunank200sunank200sunank200 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area:providersarea:system-testskind:documentationprovider:databricks

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@pankajkoti@tatiana@Lee-W@sunank200