Add DatabricksWorkflowPlugin

[pankajkoti]

The DatabricksWorkflowPlugin provides with links in the Airflow
UI for tasks that allow us to see the Databricks job run in the
Databricks workspace, additionally it also provides link to
repair task(s) in the workflow.

Databricks does not allow repairing jobs with single tasks launched
outside the workflow, hence we just provide the link for the job run.

Within the workflow, for each of the task, we provide links to the
job run and repair link for the single task

And at the workflow level, for the job launch task, we provide a
link to repair all failed tasks along with the link for job run in
the Databricks workspace that can be used to monitor the job
in the Databricks account.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[pankajkoti]

The failing 1 test Provider checks / Compat 2.8.4:P3.8 provider check is unrelated and coming from the Amazon provider tests

=========================== short test summary info ============================
FAILED tests/providers/amazon/aws/auth_manager/test_aws_auth_manager.py::TestAwsAuthManager::test_aws_auth_manager_index - TypeError: is_authorized_custom_view() got an unexpected keyword argument 'fab_action_name'
============ 1 failed, 3059 passed, 5 skipped in 358.28s (0:05:58) =============

[tatiana]

Great work, @pankajkoti !

This feature looks better than the original implementation, thanks for fixing the behaviour of monitoring the last attempt to run the task and also when we show the repair button (not for standalone NotebookOperator.

Please add documentation with screenshots - this will help end-users.

What do you think about adding a follow-up task to support repair when Airflow attempts to retry a failed task?

[phanikumv]

LGTM, except missing documentation about when/why users should utilize this

[eladkal]

temporary block to avoid merge till #40153 (comment) is resolved.

[potiuk]

@eladkal ?

[eladkal]

@eladkal ?

I will get to it only in few days.
Feel free to dismiss my request changes if the concern I raised was handled

[potiuk]

I think there is a side-effect of registering plugin that should be removed (so all tests that register plugins should deregister them in setup/teardown - thugh not sure how to do it :)

dismiss

[pankajkoti]

The failing tests report

FAILED tests/providers/microsoft/azure/operators/test_container_instances.py::TestACIOperator::test_execute_with_spot_discount - AttributeError: 'ContainerGroup' object has no attribute 'priority'

and

FAILED tests/providers/amazon/aws/auth_manager/test_aws_auth_manager.py::TestAwsAuthManager::test_aws_auth_manager_index - TypeError: is_authorized_custom_view() got an unexpected keyword argument 'fab_action_name'

Tried re-running them, but they still fail. They are unrelated to this PR and hence I'm going ahead with merging the PR.