Skip to content

Provider Migration: Update azure for Airflow 3.0 compatibility - #52618

Merged
kaxil merged 5 commits into
apache:mainfrom
kyungjunleeme:airflow3-compat/azure-provider-data-factory-operator-updates
Jul 1, 2025
Merged

Provider Migration: Update azure for Airflow 3.0 compatibility#52618
kaxil merged 5 commits into
apache:mainfrom
kyungjunleeme:airflow3-compat/azure-provider-data-factory-operator-updates

Conversation

@kyungjunleeme

@kyungjunleemekyungjunleeme commented Jul 1, 2025

Copy link
Copy Markdown
Contributor

Follow-up of #52292. Part of #52378


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@kyungjunleeme
kyungjunleemeforce-pushed the airflow3-compat/azure-provider-data-factory-operator-updates branch from 60c467b to 6a46418CompareJuly 1, 2025 06:51
@kaxil

kaxil commented Jul 1, 2025

Copy link
Copy Markdown
Member

Test failures:

https://github.com/apache/airflow/actions/runs/15992030386/job/45107520283?pr=52618#step:6:1117

FAILED providers/microsoft/azure/tests/unit/microsoft/azure/operators/test_msgraph.py::TestMSGraphAsyncOperator::test_execute_with_old_result_processor_signature - KeyError: 'task_instance'
FAILED providers/microsoft/azure/tests/unit/microsoft/azure/operators/test_msgraph.py::TestMSGraphAsyncOperator::test_execute_with_new_result_processor_signature - KeyError: 'task_instance'
FAILED providers/microsoft/azure/tests/unit/microsoft/azure/operators/test_msgraph.py::TestMSGraphAsyncOperator::test_execute_with_old_paginate_function_signature - KeyError: 'task_instance'
FAILED providers/microsoft/azure/tests/unit/microsoft/azure/operators/test_powerbi.py::TestPowerBIDatasetRefreshOperator::test_powerbi_operator_async_get_refresh_status_success - KeyError: 'task_instance'
FAILED providers/microsoft/azure/tests/unit/microsoft/azure/operators/test_powerbi.py::TestPowerBIDatasetRefreshOperator::test_powerbi_operator_async_execute_complete_success - KeyError: 'task_instance'
FAILED providers/microsoft/azure/tests/unit/microsoft/azure/operators/test_powerbi.py::TestPowerBIDatasetRefreshOperator::test_powerbi_operator_async_execute_complete_fail - KeyError: 'task_instance'
FAILED providers/microsoft/azure/tests/unit/microsoft/azure/operators/test_powerbi.py::TestPowerBIDatasetRefreshOperator::test_powerbi_operator_refresh_fail - KeyError: 'task_instance'
FAILED providers/microsoft/azure/tests/unit/microsoft/azure/operators/test_powerbi_list.py::TestPowerBIDatasetListOperator::test_powerbi_operator_async_execute_complete_success - KeyError: 'task_instance'
FAILED providers/microsoft/azure/tests/unit/microsoft/azure/operators/test_powerbi_list.py::TestPowerBIDatasetListOperator::test_powerbi_operator_async_execute_complete_fail - KeyError: 'task_instance'
FAILED providers/microsoft/azure/tests/unit/microsoft/azure/operators/test_powerbi_list.py::TestPowerBIDatasetListOperator::test_powerbi_operator_dataset_list_fail - KeyError: 'task_instance'
FAILED providers/microsoft/azure/tests/unit/microsoft/azure/operators/test_powerbi_list.py::TestPowerBIWorkspaceListOperator::test_powerbi_operator_async_execute_complete_success - KeyError: 'task_instance'
FAILED providers/microsoft/azure/tests/unit/microsoft/azure/operators/test_powerbi_list.py::TestPowerBIWorkspaceListOperator::test_powerbi_operator_async_execute_complete_fail - KeyError: 'task_instance'
FAILED providers/microsoft/azure/tests/unit/microsoft/azure/operators/test_powerbi_list.py::TestPowerBIWorkspaceListOperator::test_powerbi_operator_workspace_list_fail - KeyError: 'task_instance'

@kyungjunleeme

kyungjunleeme commented Jul 1, 2025

Copy link
Copy Markdown
ContributorAuthor

@kaxil yeah, okay. Thank you. I will add testcode to pr.


I have a question about task.
In this file
providers/microsoft/azure/src/airflow/providers/microsoft/azure/transfers/sftp_to_wasb.py

I can not change from airflow.models import BaseOperator to from airflow.providers.microsoft.azure.version_compat import BaseOperator

In this case, I have to change test code?

classSFTPToWasbOperator(BaseOperator):
......
defdry_run(self) ->None:
super().dry_run()
sftp_files: list[SftpFile] =self.get_sftp_files_map()
forfileinsftp_files:
self.log.info(
"Process will upload file from (SFTP) %s to wasb://%s as %s",
file.sftp_file_path,
self.container_name,
file.blob_name,
)
ifself.move_object:
self.log.info("Executing delete of %s", file)

image
I have a problem with this file.

I was trying to fix the test code.

@kaxil

kaxil commented Jul 1, 2025

Copy link
Copy Markdown
Member

@kaxil yeah, okay. Thank you. I will add testcode to pr.

I have a question about task. In this file providers/microsoft/azure/src/airflow/providers/microsoft/azure/transfers/sftp_to_wasb.py

I can not change from airflow.models import BaseOperator to from airflow.providers.microsoft.azure.version_compat import BaseOperator

In this case, I have to change test code?

image I have a problem with this file.

imo, we can remove the dry_run method in providers/microsoft/azure/src/airflow/providers/microsoft/azure/transfers/sftp_to_wasb.py.

Or just add something like:

defdry_run(self) ->None:
ifnotAIRFLOW_V_3_0_PLUS:
raiseNotImplementedError("Not implemented for Airflow 3.")
super().dry_run()
sftp_files: list[SftpFile] =self.get_sftp_files_map()
forfileinsftp_files:
self.log.info(
"Process will upload file from (SFTP) %s to wasb://%s as %s",
file.sftp_file_path,
self.container_name,
file.blob_name,
)
ifself.move_object:
self.log.info("Executing delete of %s", file)

fixup! fixup! ADD: type: ignore[no-redef]
@kaxil
kaxil marked this pull request as ready for review July 1, 2025 20:19

@kaxilkaxil left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed a fix for the tests: 790bbcb

@kaxil
kaxil merged commit 7adb432 into apache:mainJul 1, 2025
@kyungjunleeme

Copy link
Copy Markdown
ContributorAuthor

Thank you,
I’m currently getting used to contributing to open source projects.
Next time, I’ll try to be more thorough before opening a PR, keeping in mind that the first reviewer is always myself.
Thank you for your help!

@potiuk

Copy link
Copy Markdown
Member

Thank you, I’m currently getting used to contributing to open source projects. Next time, I’ll try to be more thorough before opening a PR, keeping in mind that the first reviewer is always myself. Thank you for your help!

Good job! no worries - we all started one day !

@kaxil

kaxil commented Jul 2, 2025

Copy link
Copy Markdown
Member

Thank you, I’m currently getting used to contributing to open source projects. Next time, I’ll try to be more thorough before opening a PR, keeping in mind that the first reviewer is always myself. Thank you for your help!

Yeah, we all still learn new things everyday. Keep contributing.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@kyungjunleeme@kaxil@potiuk