Skip to content

check status before DatabricksSubmitRunOperator & DatabricksSubmitRunOperator executes in deferrable mode - #36862

Merged
pankajastro merged 3 commits into
apache:mainfrom
astronomer:check-status-before-DatabricksSubmitRunOperator-execute-in-deferrable-mode
Jan 23, 2024
Merged

check status before DatabricksSubmitRunOperator & DatabricksSubmitRunOperator executes in deferrable mode#36862
pankajastro merged 3 commits into
apache:mainfrom
astronomer:check-status-before-DatabricksSubmitRunOperator-execute-in-deferrable-mode

Conversation

@vatsrahul1001

Copy link
Copy Markdown
Contributor

This PR addresses checking the status before deferring the task to the triggerer.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@Lee-WLee-W left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some nitpick. other looks good to me 🙂

Comment threadtests/providers/databricks/operators/test_databricks.py Outdated
Comment threadairflow/providers/databricks/operators/databricks.py Outdated
@vatsrahul1001
vatsrahul1001 marked this pull request as ready for review January 18, 2024 10:17
@vatsrahul1001
vatsrahul1001force-pushed the check-status-before-DatabricksSubmitRunOperator-execute-in-deferrable-mode branch 2 times, most recently from 9dea7c3 to 5a0f3b7CompareJanuary 19, 2024 05:36
@vatsrahul1001
vatsrahul1001force-pushed the check-status-before-DatabricksSubmitRunOperator-execute-in-deferrable-mode branch 5 times, most recently from c36e4d4 to 3578772CompareJanuary 22, 2024 16:09
@Lee-W

Copy link
Copy Markdown
Member

It seems there are some failed tests. Should we rebase and see whether it fixed the issue. If not, I think we'll need to take a deeper look

@vatsrahul1001
vatsrahul1001force-pushed the check-status-before-DatabricksSubmitRunOperator-execute-in-deferrable-mode branch from 3578772 to 87c2151CompareJanuary 23, 2024 03:08
@phanikumv
phanikumvforce-pushed the check-status-before-DatabricksSubmitRunOperator-execute-in-deferrable-mode branch from 87c2151 to 702f539CompareJanuary 23, 2024 06:18

@pankajastropankajastro left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pankajastro
pankajastro merged commit 3473739 into apache:mainJan 23, 2024
@pankajastro
pankajastro deleted the check-status-before-DatabricksSubmitRunOperator-execute-in-deferrable-mode branch January 23, 2024 07:17
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@vatsrahul1001@Lee-W@pankajastro