Return destination GCS URIs from ADLSToGCSOperator by Abhishekmishra2808 · Pull Request #61463 · apache/airflow

Abhishekmishra2808 · 2026-02-04T18:39:36Z

Related: #11323

Return value: execute() now returns a list[str] of destination GCS URIs (gs://bucket/object) for uploaded files.
The return type remains list[str]; only the returned values are updated to reflect the destination in GCS.

SameerMesiah97

Looks good overall. But this would benefit from some more polish.

shahar1

Please resolve comment made by @SameerMesiah97 (great review!),
and I think that it will be good to go.
I recall that we should run the system tests before merging this PR to ensure that they don't break, please let us know if you're able to do it - otherwise Google team or someone else will have to do it.

Abhishekmishra2808 · 2026-02-06T16:18:23Z

@shahar1 @SameerMesiah97 Could you please check now, and tell if something is missing on my end.

shahar1

Looks good!
We need to run the system tests to ensure that nothing breaks (+attach screenshots).

@yuseok89 - if you could help with it, it would be great.

CC: @VladaZakharova @MaksYermak

yuseok89 · 2026-02-12T14:07:41Z

@shahar1
While running the system tests, it looks like this should be using ADLS Gen2 rather than the older Gen1 path.
If I’m mistaken or missing context here, please correct me.

Also, it seems this change may already be reflected in main, so my guess is that resolving the conflict (merge main) and re-running the system tests could be the next step.

Abhishekmishra2808 · 2026-02-26T06:33:37Z

@shahar1 Your views on this ?

shahar1 · 2026-02-26T06:36:43Z

@shahar1 Your views on this ?

First try to resolve the conflict and see what's going on. Then make sure that your changes are aligned with the appropriate version.

Abhishekmishra2808 · 2026-02-26T10:23:40Z

ignore the commits as of now, before merging i will change this to one commit only. but before that @shahar1 please review it

shahar1 · 2026-02-26T10:36:23Z

ignore the commits as of now, before merging i will change this to one commit only. but before that @shahar1 please review it

Something doesn't look right - try to avoid reverting any changes introduced by
#61188
(e.g, materializing file_system_name into a property instead of getting it from kwargs), as it was intended for clearing up that gen 1 was retired.

Abhishekmishra2808 · 2026-02-26T12:04:39Z

@shahar1 Could please provide more detailed review?
improved upon it by making file_system_name an explicit parameter instead of hiding it in kwargs

yuseok89

@shahar1
I ran the system test for ADLSToGCSOperator.
It seems to be working now.

Abhishekmishra2808 · 2026-02-27T10:47:44Z

@yuseok89 Thanks for testing !

shahar1 · 2026-02-27T11:16:19Z

@shahar1 Could please provide more detailed review?
improved upon it by making file_system_name an explicit parameter instead of hiding it in kwargs

Hiding it in the kwargs and raising the TypeError exception (which you deleted in this PR) were deliberate decisions to make transition to Gen 2 clearer and explicit to the end-user.
Please read comments on PR #61188 for more context, and revert changes that were introduced by that PR. To make this PR good to go, at least from my perspective, it should be focused only returning the destination uris.

- Migrate from AzureDataLakeHook (Gen1) to AzureDataLakeStorageV2Hook (Gen2) - Update execute() to return list[str] of destination GCS URIs (gs://bucket/object) - Add file_system_name parameter validation via **kwargs with clear TypeError - Update unit tests to verify Gen2 hooks and return values - Add new test for file_system_name TypeError validation - Remove ADLSToGCSOperator from MISSING_EXAMPLES_FOR_CLASSES This change aligns with Azure's retirement of ADLS Gen1 and ensures operators return destination URIs for downstream task dependencies. The file_system_name parameter is required but passed via **kwargs to maintain explicit Gen1-to-Gen2 migration messaging for users.

Add destination URI tracking and return list of GCS URIs instead of file paths. Changes: - Update execute() to return list[str] of destination GCS URIs (gs://bucket/object) - Add destination_uris list initialization - Append GCS URI after each successful upload - Update docstring to document return value - Update unit tests to assert returned GCS URIs This enables downstream tasks to reference the uploaded GCS objects directly.

Abhishekmishra2808 · 2026-02-27T13:28:31Z

@shahar1 could you please review this now?

shahar1

LGTM, well done!

CC: @VladaZakharova @MaksYermak

Copilot

Pull request overview

This PR updates ADLSToGCSOperator to return a list of destination GCS URIs (gs://bucket/object) instead of source ADLS paths, aligning with the standardization effort across all ToGCS operators (issue #11323). The return type remains list[str], but the values now represent where files were uploaded rather than where they came from.

Changes:

Updated execute() method to return destination GCS URIs in the format gs://bucket/object
Moved _parse_gcs_url() call outside the file upload loop for efficiency
Changed from os.path.join to posixpath.join for cross-platform compatibility
Updated tests to verify destination URIs instead of source paths
Added return value documentation to the class docstring

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
providers/google/src/airflow/providers/google/cloud/transfers/adls_to_gcs.py	Modified execute() to build and return destination GCS URIs; optimized by moving _parse_gcs_url outside loop; switched to posixpath.join for GCS path construction
providers/google/tests/unit/google/cloud/transfers/test_adls_to_gcs.py	Updated test assertions to verify destination GCS URIs match expected format

shahar1 · 2026-03-03T20:51:51Z

@Abhishekmishra2808 I wanted to merge, but realized that we're missing the flag return_gcs_uris to ensure backward compatibility with current returned value, as well as aa warning that the default behavior will change in the future (please see other PRs under the same GitHub issue for reference).
Could you please handle it?

Missing a flag for backward compat.

…ator - Add return_gcs_uris: bool | None = None parameter to __init__ - When None (default), keep legacy behavior (return ADLS file paths) and emit FutureWarning that default will change to GCS URIs - When True, return list of GCS URIs (gs://bucket/path) - When False, return legacy ADLS file paths with no warning - Add tests for return_gcs_uris=False and default (None) warning behavior

Abhishekmishra2808 · 2026-03-06T06:05:44Z

@shahar1 I fixed the backward compatibility issue, please review this!

shahar1 · 2026-03-06T11:42:32Z

Great job Abhishek!

Abhishekmishra2808 requested a review from shahar1 as a code owner February 4, 2026 18:39

boring-cyborgBot added area:providers provider:google Google (including GCP) related issues labels Feb 4, 2026

SameerMesiah97 reviewed Feb 5, 2026
View reviewed changes

shahar1 mentioned this pull request Feb 6, 2026
Return files destination uris in all ToGCS operators #11323
Closed
14 tasks

shahar1 reviewed Feb 6, 2026
View reviewed changes

Comment threadproviders/google/src/airflow/providers/google/cloud/transfers/adls_to_gcs.py Outdated

Abhishekmishra2808 force-pushed the fix/adls-to-gcs-return-destination-uris branch from a6488b4 to 3583457Compare February 6, 2026 13:28

Abhishekmishra2808 requested review from amoghrajesh, ashb and kaxil as code owners February 6, 2026 13:28

Abhishekmishra2808 force-pushed the fix/adls-to-gcs-return-destination-uris branch from 3583457 to aef396eCompare February 6, 2026 13:31

shahar1 reviewed Feb 10, 2026
View reviewed changes

Abhishekmishra2808 force-pushed the fix/adls-to-gcs-return-destination-uris branch from 037e823 to b251621Compare February 26, 2026 11:43

yuseok89 reviewed Feb 27, 2026
View reviewed changes

Abhishekmishra2808 force-pushed the fix/adls-to-gcs-return-destination-uris branch from 6df52e2 to 22844cdCompare February 27, 2026 11:59

Abhishekmishra2808 force-pushed the fix/adls-to-gcs-return-destination-uris branch from 22844cd to a50b9f3Compare February 27, 2026 12:02

Abhishekmishra2808 force-pushed the fix/adls-to-gcs-return-destination-uris branch from c795d27 to 8653cc1Compare February 27, 2026 12:41

shahar1 previously approved these changes Feb 27, 2026
View reviewed changes

kaxil requested a review from Copilot February 27, 2026 14:58

Copilotstarted reviewing on behalf of kaxil February 27, 2026 14:58 View session

CopilotAI reviewed Feb 27, 2026
View reviewed changes

MaksYermak approved these changes Mar 2, 2026
View reviewed changes

shahar1 approved these changes Mar 6, 2026
View reviewed changes

shahar1 merged commit f9ba988 into apache:mainMar 6, 2026
91 checks passed

vincbeck mentioned this pull request Mar 10, 2026
Status of testing Providers that were prepared on March 10, 2026 #63279
Closed
81 tasks

dominikhei pushed a commit to dominikhei/airflow that referenced this pull request Mar 11, 2026
Return destination GCS URIs from ADLSToGCSOperator (apache#61463)
84d4f80

Abhishekmishra2808 deleted the fix/adls-to-gcs-return-destination-uris branch March 24, 2026 21:03

jscheffl mentioned this pull request Mar 24, 2026
Status of testing Providers that were prepared on March 24, 2026 #64195
Closed

Abhishekmishra2808 mentioned this pull request Mar 25, 2026
[Discussion] Configurable Multi-User & POSIX File Permission Support in Apache NuttX apache/nuttx#18458
Open

Uh oh!

Conversation

Abhishekmishra2808 commented Feb 4, 2026

Uh oh!

SameerMesiah97 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shahar1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Abhishekmishra2808 commented Feb 6, 2026

Uh oh!

shahar1 left a comment

Choose a reason for hiding this comment

Uh oh!

yuseok89 commented Feb 12, 2026

Uh oh!

Abhishekmishra2808 commented Feb 26, 2026

Uh oh!

shahar1 commented Feb 26, 2026

Uh oh!

Abhishekmishra2808 commented Feb 26, 2026

Uh oh!

shahar1 commented Feb 26, 2026

Uh oh!

Abhishekmishra2808 commented Feb 26, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yuseok89 left a comment

Choose a reason for hiding this comment

Uh oh!

Abhishekmishra2808 commented Feb 27, 2026

Uh oh!

shahar1 commented Feb 27, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Abhishekmishra2808 commented Feb 27, 2026

Uh oh!

shahar1 left a comment

Choose a reason for hiding this comment

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

shahar1 commented Mar 3, 2026

Uh oh!

Abhishekmishra2808 commented Mar 6, 2026

Uh oh!

shahar1 commented Mar 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Abhishekmishra2808 commented Feb 26, 2026•
edited
Loading

shahar1 commented Feb 27, 2026•
edited
Loading