Migrate ADLSListOperator from ADLS Gen1 to Gen2

[aaron-y-chen]

closes: #44228

Why

The older ADLSListOperator uses AzureDataLakeHook, which uses Gen 1 SDK is already retired.

airflow/providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/data_lake.py

Line 133 in 44d3678

self._conn=core.AzureDLFileSystem(credential, store_name=self.account_name)

How

Replace it with AzureDataLakeStorageV2Hook, which uses Gen 2 SDK.

Given Gen1 is retired, the impact should be limited, but this is a breaking change.

What

I created an object (blob) in an Azure Storage account.

And I used this DAG to test whether I could fetch it.

fromdatetimeimportdatetimefromairflowimportDAGfromairflow.providers.microsoft.azure.operators.adlsimportADLSListOperatorfromairflow.providers.standard.operators.pythonimportPythonOperatorwithDAG(
dag_id="test_adls_issue_44228_fixed",
start_date=datetime(2026, 1, 1),
schedule=None,
catchup=False,
) asdag:
list_files=ADLSListOperator(
task_id="list_adls_files",
file_system_name="testcontainer",
path="",
azure_data_lake_conn_id="adls_gen2_default",
)
defprint_files(ti):
files=ti.xcom_pull(task_ids="list_adls_files")
print("="*50)
print(f"Files found: {files}")
print(f"Total count: {len(files) iffileselse0}")
print("="*50)
print_result=PythonOperator(
task_id="print_result",
python_callable=print_files,
)
list_files>>print_result

It works pretty well.

Discussion

It seems AzureDataLakeHook uses the Gen 1 SDK. Perhaps we need to add @deprecated(...) to it?

airflow/providers/microsoft/azure/src/airflow/providers/microsoft/azure/hooks/data_lake.py

Line 46 in 44d3678

classAzureDataLakeHook(BaseHook):

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)
  Claude Opus 4.5

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[aaron-y-chen]

Hmmm… it looks like ADLSToGCSOperator inherits from ADLSListOperator. Let me take a look.

[aaron-y-chen]

Okay, fixed! I added file_system_name to ADLSToGCSOperator and updated the related tests. If there's anything else I can improve, please don't hesitate to let me know :)

[shahar1]

Discussion

It seems AzureDataLakeHook uses the Gen 1 SDK. Perhaps we need to add @deprecated(...) to it?

If Gen 1 SDK is already retired - there's no point in retrospectively deprecating it, as it doesn't serve a practical purpose (in general, it's better to take care of a proper deprecation much before retiring - but someone needs to track these changes on time 🙂).
However, as file_system_name is now mandatory - maybe instead of unexplained TypeError when missing, we should reflect its "sudden" introduction better to the user (use case - people who had used this operator in the past and don't understand why it is now required):

def__init__(
self,
*,
src_adls: str,
dest_gcs: str,
azure_data_lake_conn_id: str,
gcp_conn_id: str="google_cloud_default",
replace: bool=False,
gzip: bool=False,
google_impersonation_chain: str|Sequence[str] |None=None,
**kwargs,
) ->None:
file_system_name=kwargs.get('file_system_name')
ifnotfile_system_name:
raiseTypeError(
"The 'file_system_name' parameter is required. ""ADLSListOperator has been migrated from Azure Data Lake Storage Gen1 (retired) ""to Gen2, which requires specifying a file system name. ""Please add file_system_name='your-container-name' to your operator instantiation."
)

WDYT?

CC: @VladaZakharova@MaksYermak (for the GCP transfer operator)

[aaron-y-chen]

Totally agree, that's much clearer. Let me add it!

[shahar1]

LGTM, at least in the GCP aspects :)
@dabla could you please review the changes in the Azure provider?

[shahar1]

LGTM, at least in the GCP aspects :) @dabla could you please review the changes in the Azure provider?

@nailo2c Almost missed this one, my bad - we need to add # use next version to the Azure provider in Google provider's dependencies, otherwise it might break:

https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDERS.md#update-versions-of-dependent-providers-to-the-next-version

[aaron-y-chen]

Wow, I didn't know that, learned something new today! Thanks 😄

[shahar1]

@VladaZakharova@MaksYermak
As both CI and system tests passed, if no objections are made, this PR will be merged by the upcoming release on Tuesday.