Skip to content

Deferrable mode for S3ToGCSOperator - #29462

Merged
potiuk merged 1 commit into
apache:mainfrom
lwyszomi:s3_to_gcs_deferrable_mode
Jul 11, 2023
Merged

Deferrable mode for S3ToGCSOperator#29462
potiuk merged 1 commit into
apache:mainfrom
lwyszomi:s3_to_gcs_deferrable_mode

Conversation

@moiseenkov

@moiseenkovmoiseenkov commented Feb 10, 2023

Copy link
Copy Markdown
Contributor

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@moiseenkov
moiseenkovforce-pushed the s3_to_gcs_deferrable_mode branch 4 times, most recently from 6925d4e to 3469f13CompareFebruary 13, 2023 10:37
kaxil
kaxil reviewed Feb 13, 2023
View reviewed changes
Comment threadgenerated/provider_dependencies.json Outdated
kaxil
kaxil reviewed Feb 13, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py Outdated
@moiseenkov
moiseenkovforce-pushed the s3_to_gcs_deferrable_mode branch 9 times, most recently from c6b8ac5 to 74f43c0CompareFebruary 20, 2023 08:24
@moiseenkov
moiseenkovforce-pushed the s3_to_gcs_deferrable_mode branch from 74f43c0 to d682908CompareFebruary 23, 2023 09:57
@moiseenkov

moiseenkov commented Feb 23, 2023

Copy link
Copy Markdown
ContributorAuthor

@kaxil , hi,
Could you review my fixes please?

kaxil
kaxil reviewed Feb 23, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py Outdated
kaxil
kaxil reviewed Feb 23, 2023
View reviewed changes
Comment threaddocs/apache-airflow-providers-google/operators/transfer/s3_to_gcs.rst Outdated
kaxil
kaxil reviewed Feb 23, 2023
View reviewed changes

@kaxilkaxil left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @pankajastro Could you review this too please

pankajastro
pankajastro reviewed Feb 24, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py Outdated
Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py Outdated
Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py Outdated
Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py Outdated
Comment threadtests/providers/google/cloud/hooks/test_cloud_storage_transfer_service_async.py Outdated
Comment threadtests/providers/google/cloud/hooks/test_cloud_storage_transfer_service_async.py Outdated
Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py Outdated
Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py Outdated
Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py Outdated
Comment threadtests/providers/google/cloud/hooks/test_cloud_storage_transfer_service_async.py Outdated
@moiseenkov
moiseenkovforce-pushed the s3_to_gcs_deferrable_mode branch 3 times, most recently from 955d134 to ef075b4CompareFebruary 24, 2023 15:07
@github-actions

github-actionsBot commented May 24, 2023

Copy link
Copy Markdown
Contributor

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actionsgithub-actionsBot added the stale Stale PRs per the .github/workflows/stale.yml policy file label May 24, 2023
@potiuk
potiukforce-pushed the s3_to_gcs_deferrable_mode branch from d409b8e to 9170b73CompareJune 4, 2023 23:15
@potiuk

potiuk commented Jun 4, 2023

Copy link
Copy Markdown
Member

Rebased it after some flaky mssql tools addressing in main, let's see if it helps

@potiuk

potiuk commented Jun 4, 2023

Copy link
Copy Markdown
Member

cc: @kosteev@bjankie1

@potiukpotiuk closed this Jun 4, 2023
@potiukpotiuk reopened this Jun 4, 2023
@moiseenkov
moiseenkovforce-pushed the s3_to_gcs_deferrable_mode branch 3 times, most recently from d7ae276 to c93d669CompareJune 6, 2023 08:32
@Lee-WLee-W mentioned this pull request Jun 6, 2023
Merged
@moiseenkov
moiseenkovforce-pushed the s3_to_gcs_deferrable_mode branch 4 times, most recently from 766895b to c4e6051CompareJune 7, 2023 06:37
pankajastro
pankajastro reviewed Jun 28, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py Outdated
Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py Outdated
Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py Outdated
Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py Outdated
ahidalgob
ahidalgob reviewed Jun 28, 2023
View reviewed changes
Comment threaddocs/apache-airflow-providers-google/operators/transfer/s3_to_gcs.rst Outdated
@VladaZakharova

VladaZakharova commented Jul 5, 2023

Copy link
Copy Markdown
Contributor

Hi @potiuk@pankajastro !
Could you please take a look on this PR? Thanks :)

potiuk
potiuk requested changes Jul 5, 2023
View reviewed changes

@potiukpotiuk left a comment •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We had a recent change on how we treat default deferrable setting and we need to adjust this one too: #31712

One more case in point @Lee-W that we need to add a pre-commit checking the defferrable convention, otherwise we will end up will all kind of mess where some deferrable parameters will have the default values following the convention and some not.

@moiseenkov - can you please update the change to follow this patterm?
@Lee-W -> will it be possible that you add such pre-commit (#32355 (comment)) ?

@Lee-W

Lee-W commented Jul 5, 2023

Copy link
Copy Markdown
Member

@potiuk Sure! Sorry for the late reply. I'm already working on it.

@potiuk

potiuk commented Jul 5, 2023

Copy link
Copy Markdown
Member

@potiuk Sure! Sorry for the late reply. I'm already working on it.

Cool! :)

@moiseenkov

moiseenkov commented Jul 5, 2023

Copy link
Copy Markdown
ContributorAuthor

We had a recent change on how we treat default deferrable setting and we need to adjust this one too: #31712

One more case in point @Lee-W that we need to add a pre-commit checking the defferrable convention, otherwise we will end up will all kind of mess where some deferrable parameters will have the default values following the convention and some not.

@VladaZakharova - can you please update the change to follow this patterm? @Lee-W -> will it be possible that you add such pre-commit (#32355 (comment)) ?

Thanks for the update! I will update PR and ping you afterwards

@moiseenkov

moiseenkov commented Jul 5, 2023

Copy link
Copy Markdown
ContributorAuthor

Hi, @potiuk , @Lee-W ,

I updated the PR:
https://github.com/apache/airflow/pull/29462/files#diff-3e4eedf131305f589cec193af2b406b57a5eab1a3db181272996c7e4d03e6cc6R156

@Lee-WLee-W mentioned this pull request Jul 5, 2023
Merged
Lee-W
Lee-W reviewed Jul 6, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py Outdated
Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py Outdated
Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py Outdated
Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py Outdated
Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py Outdated
Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py Outdated
Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py Outdated
Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py Outdated
Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py Outdated
Comment threadtests/system/providers/google/cloud/gcs/example_s3_to_gcs_async.py Outdated
@moiseenkov
Combine 8 into 1
a2e4755
Implement deferrable mode for S3ToGCSOperator
Update airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/transfers/s3_to_gcs.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/transfers/s3_to_gcs.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/transfers/s3_to_gcs.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Address code review comments
@VladaZakharova

VladaZakharova commented Jul 11, 2023

Copy link
Copy Markdown
Contributor

@potiuk
Hi there!
Could we recheck this PR? Thanks!

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@potiukpotiukpotiuk approved these changes
@pankajastropankajastropankajastro approved these changes
@kaxilkaxilAwaiting requested review from kaxil
@ahidalgobahidalgobAwaiting requested review from ahidalgob
@Lee-WLee-WAwaiting requested review from Lee-W

Assignees

No one assigned

Labels

area:providersarea:system-testskind:documentationprovider:googleGoogle (including GCP) related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@moiseenkov@potiuk@VladaZakharova@Lee-W@kaxil@ahidalgob@pankajastro
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Deferrable mode for S3ToGCSOperator by moiseenkov · Pull Request #29462 · apache/airflow · GitHub
Skip to content
Deferrable mode for S3ToGCSOperator - #29462
Merged
potiuk merged 1 commit into
apache:mainfrom 
lwyszomi:s3_to_gcs_deferrable_mode
Jul 11, 2023
Merged
Deferrable mode for S3ToGCSOperator#29462
potiuk merged 1 commit into
apache:mainfrom 
lwyszomi:s3_to_gcs_deferrable_mode
Conversation
@moiseenkov
@moiseenkovmoiseenkov
commented
Feb 10, 2023

Copy link


Copy Markdown

Contributor
^ Add meaningful description above
Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.
@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
4 times, most recently
from
6925d4e to
3469f13CompareFebruary 13, 2023 10:37
kaxil
kaxil
reviewed
Feb 13, 2023
View reviewed changes
Comment threadgenerated/provider_dependencies.json
Outdated

kaxil
kaxil
reviewed
Feb 13, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
9 times, most recently
from
c6b8ac5 to
74f43c0CompareFebruary 20, 2023 08:24
@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
from
74f43c0 to
d682908CompareFebruary 23, 2023 09:57
@moiseenkov
moiseenkov
commented
Feb 23, 2023

Copy link


Copy Markdown

ContributorAuthor
@kaxil , hi,

Could you review my fixes please?
kaxil
kaxil
reviewed
Feb 23, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

kaxil
kaxil
reviewed
Feb 23, 2023
View reviewed changes
Comment threaddocs/apache-airflow-providers-google/operators/transfer/s3_to_gcs.rst
Outdated

kaxil
kaxil
reviewed
Feb 23, 2023
View reviewed changes
@kaxilkaxil
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @pankajastro Could you review this too please
pankajastro
pankajastro
reviewed
Feb 24, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadtests/providers/google/cloud/hooks/test_cloud_storage_transfer_service_async.py
Outdated

Comment threadtests/providers/google/cloud/hooks/test_cloud_storage_transfer_service_async.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadtests/providers/google/cloud/hooks/test_cloud_storage_transfer_service_async.py
Outdated

@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
3 times, most recently
from
955d134 to
ef075b4CompareFebruary 24, 2023 15:07
@github-actions
github-actionsBot
commented
May 24, 2023

Copy link


Copy Markdown

Contributor
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
@github-actionsgithub-actionsBot
added
the 
stale
Stale PRs per the .github/workflows/stale.yml policy file
label
May 24, 2023
@potiuk
potiukforce-pushed
the
s3_to_gcs_deferrable_mode
branch
from
d409b8e to
9170b73CompareJune 4, 2023 23:15
@potiuk
potiuk
commented
Jun 4, 2023

Copy link


Copy Markdown

Member
Rebased it after some flaky mssql tools addressing in main, let's see if it helps
@potiuk
potiuk
commented
Jun 4, 2023

Copy link


Copy Markdown

Member
cc: @kosteev@bjankie1
@potiukpotiuk
closed this
Jun 4, 2023
@potiukpotiuk
reopened this
Jun 4, 2023
@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
3 times, most recently
from
d7ae276 to
c93d669CompareJune 6, 2023 08:32
@Lee-WLee-W
mentioned this pull request
Jun 6, 2023
 Merged

@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
4 times, most recently
from
766895b to
c4e6051CompareJune 7, 2023 06:37
pankajastro
pankajastro
reviewed
Jun 28, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

ahidalgob
ahidalgob
reviewed
Jun 28, 2023
View reviewed changes
Comment threaddocs/apache-airflow-providers-google/operators/transfer/s3_to_gcs.rst
Outdated

@VladaZakharova
VladaZakharova
commented
Jul 5, 2023

Copy link


Copy Markdown

Contributor
Hi @potiuk@pankajastro !

Could you please take a look on this PR? Thanks :)
potiuk
potiuk
requested changes
Jul 5, 2023
View reviewed changes
@potiukpotiuk
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We had a recent change on how we treat default deferrable setting and we need to adjust this one too: #31712
One more case in point @Lee-W that we need to add a pre-commit checking the defferrable convention, otherwise we will end up will all kind of mess where some deferrable parameters will have the default values following the convention and some not.
@moiseenkov - can you please update the change to follow this patterm?
@Lee-W -> will it be possible that you add such pre-commit (#32355 (comment)) ?
@Lee-W
Lee-W
commented
Jul 5, 2023

Copy link


Copy Markdown

Member
@potiuk Sure! Sorry for the late reply. I'm already working on it.
@potiuk
potiuk
commented
Jul 5, 2023

Copy link


Copy Markdown

Member
@potiuk Sure! Sorry for the late reply. I'm already working on it.
Cool! :)
@moiseenkov
moiseenkov
commented
Jul 5, 2023

Copy link


Copy Markdown

ContributorAuthor
We had a recent change on how we treat default deferrable setting and we need to adjust this one too: #31712
One more case in point @Lee-W that we need to add a pre-commit checking the defferrable convention, otherwise we will end up will all kind of mess where some deferrable parameters will have the default values following the convention and some not.
@VladaZakharova - can you please update the change to follow this patterm? @Lee-W -> will it be possible that you add such pre-commit (#32355 (comment)) ?
Thanks for the update! I will update PR and ping you afterwards
@moiseenkov
moiseenkov
commented
Jul 5, 2023

Copy link


Copy Markdown

ContributorAuthor
Hi, @potiuk , @Lee-W ,
I updated the PR:
https://github.com/apache/airflow/pull/29462/files#diff-3e4eedf131305f589cec193af2b406b57a5eab1a3db181272996c7e4d03e6cc6R156
@Lee-WLee-W
mentioned this pull request
Jul 5, 2023
 Merged

Lee-W
Lee-W
reviewed
Jul 6, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadtests/system/providers/google/cloud/gcs/example_s3_to_gcs_async.py
Outdated

@moiseenkov
Combine 8 into 1
a2e4755
Implement deferrable mode for S3ToGCSOperator
Update airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/transfers/s3_to_gcs.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/transfers/s3_to_gcs.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/transfers/s3_to_gcs.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Address code review comments
@VladaZakharova
VladaZakharova
commented
Jul 11, 2023

Copy link


Copy Markdown

Contributor
@potiuk

Hi there!

Could we recheck this PR? Thanks!
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@potiukpotiukpotiuk approved these changes
@pankajastropankajastropankajastro approved these changes
@kaxilkaxilAwaiting requested review from kaxil
@ahidalgobahidalgobAwaiting requested review from ahidalgob
@Lee-WLee-WAwaiting requested review from Lee-W

Assignees


No one assigned


Labels

area:providersarea:system-testskind:documentationprovider:googleGoogle (including GCP) related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

7 participants

@moiseenkov@potiuk@VladaZakharova@Lee-W@kaxil@ahidalgob@pankajastro

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Deferrable mode for S3ToGCSOperator by moiseenkov · Pull Request #29462 · apache/airflow · GitHub
Skip to content
Deferrable mode for S3ToGCSOperator - #29462
Merged
potiuk merged 1 commit into
apache:mainfrom 
lwyszomi:s3_to_gcs_deferrable_mode
Jul 11, 2023
Merged
Deferrable mode for S3ToGCSOperator#29462
potiuk merged 1 commit into
apache:mainfrom 
lwyszomi:s3_to_gcs_deferrable_mode
Conversation
@moiseenkov
@moiseenkovmoiseenkov
commented
Feb 10, 2023

Copy link


Copy Markdown

Contributor
^ Add meaningful description above
Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.
@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
4 times, most recently
from
6925d4e to
3469f13CompareFebruary 13, 2023 10:37
kaxil
kaxil
reviewed
Feb 13, 2023
View reviewed changes
Comment threadgenerated/provider_dependencies.json
Outdated

kaxil
kaxil
reviewed
Feb 13, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
9 times, most recently
from
c6b8ac5 to
74f43c0CompareFebruary 20, 2023 08:24
@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
from
74f43c0 to
d682908CompareFebruary 23, 2023 09:57
@moiseenkov
moiseenkov
commented
Feb 23, 2023

Copy link


Copy Markdown

ContributorAuthor
@kaxil , hi,

Could you review my fixes please?
kaxil
kaxil
reviewed
Feb 23, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

kaxil
kaxil
reviewed
Feb 23, 2023
View reviewed changes
Comment threaddocs/apache-airflow-providers-google/operators/transfer/s3_to_gcs.rst
Outdated

kaxil
kaxil
reviewed
Feb 23, 2023
View reviewed changes
@kaxilkaxil
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @pankajastro Could you review this too please
pankajastro
pankajastro
reviewed
Feb 24, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadtests/providers/google/cloud/hooks/test_cloud_storage_transfer_service_async.py
Outdated

Comment threadtests/providers/google/cloud/hooks/test_cloud_storage_transfer_service_async.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadtests/providers/google/cloud/hooks/test_cloud_storage_transfer_service_async.py
Outdated

@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
3 times, most recently
from
955d134 to
ef075b4CompareFebruary 24, 2023 15:07
@github-actions
github-actionsBot
commented
May 24, 2023

Copy link


Copy Markdown

Contributor
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
@github-actionsgithub-actionsBot
added
the 
stale
Stale PRs per the .github/workflows/stale.yml policy file
label
May 24, 2023
@potiuk
potiukforce-pushed
the
s3_to_gcs_deferrable_mode
branch
from
d409b8e to
9170b73CompareJune 4, 2023 23:15
@potiuk
potiuk
commented
Jun 4, 2023

Copy link


Copy Markdown

Member
Rebased it after some flaky mssql tools addressing in main, let's see if it helps
@potiuk
potiuk
commented
Jun 4, 2023

Copy link


Copy Markdown

Member
cc: @kosteev@bjankie1
@potiukpotiuk
closed this
Jun 4, 2023
@potiukpotiuk
reopened this
Jun 4, 2023
@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
3 times, most recently
from
d7ae276 to
c93d669CompareJune 6, 2023 08:32
@Lee-WLee-W
mentioned this pull request
Jun 6, 2023
 Merged

@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
4 times, most recently
from
766895b to
c4e6051CompareJune 7, 2023 06:37
pankajastro
pankajastro
reviewed
Jun 28, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

ahidalgob
ahidalgob
reviewed
Jun 28, 2023
View reviewed changes
Comment threaddocs/apache-airflow-providers-google/operators/transfer/s3_to_gcs.rst
Outdated

@VladaZakharova
VladaZakharova
commented
Jul 5, 2023

Copy link


Copy Markdown

Contributor
Hi @potiuk@pankajastro !

Could you please take a look on this PR? Thanks :)
potiuk
potiuk
requested changes
Jul 5, 2023
View reviewed changes
@potiukpotiuk
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We had a recent change on how we treat default deferrable setting and we need to adjust this one too: #31712
One more case in point @Lee-W that we need to add a pre-commit checking the defferrable convention, otherwise we will end up will all kind of mess where some deferrable parameters will have the default values following the convention and some not.
@moiseenkov - can you please update the change to follow this patterm?
@Lee-W -> will it be possible that you add such pre-commit (#32355 (comment)) ?
@Lee-W
Lee-W
commented
Jul 5, 2023

Copy link


Copy Markdown

Member
@potiuk Sure! Sorry for the late reply. I'm already working on it.
@potiuk
potiuk
commented
Jul 5, 2023

Copy link


Copy Markdown

Member
@potiuk Sure! Sorry for the late reply. I'm already working on it.
Cool! :)
@moiseenkov
moiseenkov
commented
Jul 5, 2023

Copy link


Copy Markdown

ContributorAuthor
We had a recent change on how we treat default deferrable setting and we need to adjust this one too: #31712
One more case in point @Lee-W that we need to add a pre-commit checking the defferrable convention, otherwise we will end up will all kind of mess where some deferrable parameters will have the default values following the convention and some not.
@VladaZakharova - can you please update the change to follow this patterm? @Lee-W -> will it be possible that you add such pre-commit (#32355 (comment)) ?
Thanks for the update! I will update PR and ping you afterwards
@moiseenkov
moiseenkov
commented
Jul 5, 2023

Copy link


Copy Markdown

ContributorAuthor
Hi, @potiuk , @Lee-W ,
I updated the PR:
https://github.com/apache/airflow/pull/29462/files#diff-3e4eedf131305f589cec193af2b406b57a5eab1a3db181272996c7e4d03e6cc6R156
@Lee-WLee-W
mentioned this pull request
Jul 5, 2023
 Merged

Lee-W
Lee-W
reviewed
Jul 6, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadtests/system/providers/google/cloud/gcs/example_s3_to_gcs_async.py
Outdated

@moiseenkov
Combine 8 into 1
a2e4755
Implement deferrable mode for S3ToGCSOperator
Update airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/transfers/s3_to_gcs.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/transfers/s3_to_gcs.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/transfers/s3_to_gcs.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Address code review comments
@VladaZakharova
VladaZakharova
commented
Jul 11, 2023

Copy link


Copy Markdown

Contributor
@potiuk

Hi there!

Could we recheck this PR? Thanks!
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@potiukpotiukpotiuk approved these changes
@pankajastropankajastropankajastro approved these changes
@kaxilkaxilAwaiting requested review from kaxil
@ahidalgobahidalgobAwaiting requested review from ahidalgob
@Lee-WLee-WAwaiting requested review from Lee-W

Assignees


No one assigned


Labels

area:providersarea:system-testskind:documentationprovider:googleGoogle (including GCP) related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

7 participants

@moiseenkov@potiuk@VladaZakharova@Lee-W@kaxil@ahidalgob@pankajastro

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Deferrable mode for S3ToGCSOperator by moiseenkov · Pull Request #29462 · apache/airflow · GitHub
Skip to content
Deferrable mode for S3ToGCSOperator - #29462
Merged
potiuk merged 1 commit into
apache:mainfrom 
lwyszomi:s3_to_gcs_deferrable_mode
Jul 11, 2023
Merged
Deferrable mode for S3ToGCSOperator#29462
potiuk merged 1 commit into
apache:mainfrom 
lwyszomi:s3_to_gcs_deferrable_mode
Conversation
@moiseenkov
@moiseenkovmoiseenkov
commented
Feb 10, 2023

Copy link


Copy Markdown

Contributor
^ Add meaningful description above
Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.
@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
4 times, most recently
from
6925d4e to
3469f13CompareFebruary 13, 2023 10:37
kaxil
kaxil
reviewed
Feb 13, 2023
View reviewed changes
Comment threadgenerated/provider_dependencies.json
Outdated

kaxil
kaxil
reviewed
Feb 13, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
9 times, most recently
from
c6b8ac5 to
74f43c0CompareFebruary 20, 2023 08:24
@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
from
74f43c0 to
d682908CompareFebruary 23, 2023 09:57
@moiseenkov
moiseenkov
commented
Feb 23, 2023

Copy link


Copy Markdown

ContributorAuthor
@kaxil , hi,

Could you review my fixes please?
kaxil
kaxil
reviewed
Feb 23, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

kaxil
kaxil
reviewed
Feb 23, 2023
View reviewed changes
Comment threaddocs/apache-airflow-providers-google/operators/transfer/s3_to_gcs.rst
Outdated

kaxil
kaxil
reviewed
Feb 23, 2023
View reviewed changes
@kaxilkaxil
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @pankajastro Could you review this too please
pankajastro
pankajastro
reviewed
Feb 24, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadtests/providers/google/cloud/hooks/test_cloud_storage_transfer_service_async.py
Outdated

Comment threadtests/providers/google/cloud/hooks/test_cloud_storage_transfer_service_async.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadtests/providers/google/cloud/hooks/test_cloud_storage_transfer_service_async.py
Outdated

@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
3 times, most recently
from
955d134 to
ef075b4CompareFebruary 24, 2023 15:07
@github-actions
github-actionsBot
commented
May 24, 2023

Copy link


Copy Markdown

Contributor
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
@github-actionsgithub-actionsBot
added
the 
stale
Stale PRs per the .github/workflows/stale.yml policy file
label
May 24, 2023
@potiuk
potiukforce-pushed
the
s3_to_gcs_deferrable_mode
branch
from
d409b8e to
9170b73CompareJune 4, 2023 23:15
@potiuk
potiuk
commented
Jun 4, 2023

Copy link


Copy Markdown

Member
Rebased it after some flaky mssql tools addressing in main, let's see if it helps
@potiuk
potiuk
commented
Jun 4, 2023

Copy link


Copy Markdown

Member
cc: @kosteev@bjankie1
@potiukpotiuk
closed this
Jun 4, 2023
@potiukpotiuk
reopened this
Jun 4, 2023
@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
3 times, most recently
from
d7ae276 to
c93d669CompareJune 6, 2023 08:32
@Lee-WLee-W
mentioned this pull request
Jun 6, 2023
 Merged

@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
4 times, most recently
from
766895b to
c4e6051CompareJune 7, 2023 06:37
pankajastro
pankajastro
reviewed
Jun 28, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

ahidalgob
ahidalgob
reviewed
Jun 28, 2023
View reviewed changes
Comment threaddocs/apache-airflow-providers-google/operators/transfer/s3_to_gcs.rst
Outdated

@VladaZakharova
VladaZakharova
commented
Jul 5, 2023

Copy link


Copy Markdown

Contributor
Hi @potiuk@pankajastro !

Could you please take a look on this PR? Thanks :)
potiuk
potiuk
requested changes
Jul 5, 2023
View reviewed changes
@potiukpotiuk
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We had a recent change on how we treat default deferrable setting and we need to adjust this one too: #31712
One more case in point @Lee-W that we need to add a pre-commit checking the defferrable convention, otherwise we will end up will all kind of mess where some deferrable parameters will have the default values following the convention and some not.
@moiseenkov - can you please update the change to follow this patterm?
@Lee-W -> will it be possible that you add such pre-commit (#32355 (comment)) ?
@Lee-W
Lee-W
commented
Jul 5, 2023

Copy link


Copy Markdown

Member
@potiuk Sure! Sorry for the late reply. I'm already working on it.
@potiuk
potiuk
commented
Jul 5, 2023

Copy link


Copy Markdown

Member
@potiuk Sure! Sorry for the late reply. I'm already working on it.
Cool! :)
@moiseenkov
moiseenkov
commented
Jul 5, 2023

Copy link


Copy Markdown

ContributorAuthor
We had a recent change on how we treat default deferrable setting and we need to adjust this one too: #31712
One more case in point @Lee-W that we need to add a pre-commit checking the defferrable convention, otherwise we will end up will all kind of mess where some deferrable parameters will have the default values following the convention and some not.
@VladaZakharova - can you please update the change to follow this patterm? @Lee-W -> will it be possible that you add such pre-commit (#32355 (comment)) ?
Thanks for the update! I will update PR and ping you afterwards
@moiseenkov
moiseenkov
commented
Jul 5, 2023

Copy link


Copy Markdown

ContributorAuthor
Hi, @potiuk , @Lee-W ,
I updated the PR:
https://github.com/apache/airflow/pull/29462/files#diff-3e4eedf131305f589cec193af2b406b57a5eab1a3db181272996c7e4d03e6cc6R156
@Lee-WLee-W
mentioned this pull request
Jul 5, 2023
 Merged

Lee-W
Lee-W
reviewed
Jul 6, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadtests/system/providers/google/cloud/gcs/example_s3_to_gcs_async.py
Outdated

@moiseenkov
Combine 8 into 1
a2e4755
Implement deferrable mode for S3ToGCSOperator
Update airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/transfers/s3_to_gcs.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/transfers/s3_to_gcs.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/transfers/s3_to_gcs.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Address code review comments
@VladaZakharova
VladaZakharova
commented
Jul 11, 2023

Copy link


Copy Markdown

Contributor
@potiuk

Hi there!

Could we recheck this PR? Thanks!
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@potiukpotiukpotiuk approved these changes
@pankajastropankajastropankajastro approved these changes
@kaxilkaxilAwaiting requested review from kaxil
@ahidalgobahidalgobAwaiting requested review from ahidalgob
@Lee-WLee-WAwaiting requested review from Lee-W

Assignees


No one assigned


Labels

area:providersarea:system-testskind:documentationprovider:googleGoogle (including GCP) related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

7 participants

@moiseenkov@potiuk@VladaZakharova@Lee-W@kaxil@ahidalgob@pankajastro

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
Deferrable mode for S3ToGCSOperator by moiseenkov · Pull Request #29462 · apache/airflow · GitHub
Skip to content
Deferrable mode for S3ToGCSOperator - #29462
Merged
potiuk merged 1 commit into
apache:mainfrom 
lwyszomi:s3_to_gcs_deferrable_mode
Jul 11, 2023
Merged
Deferrable mode for S3ToGCSOperator#29462
potiuk merged 1 commit into
apache:mainfrom 
lwyszomi:s3_to_gcs_deferrable_mode
Conversation
@moiseenkov
@moiseenkovmoiseenkov
commented
Feb 10, 2023

Copy link


Copy Markdown

Contributor
^ Add meaningful description above
Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.
@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
4 times, most recently
from
6925d4e to
3469f13CompareFebruary 13, 2023 10:37
kaxil
kaxil
reviewed
Feb 13, 2023
View reviewed changes
Comment threadgenerated/provider_dependencies.json
Outdated

kaxil
kaxil
reviewed
Feb 13, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
9 times, most recently
from
c6b8ac5 to
74f43c0CompareFebruary 20, 2023 08:24
@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
from
74f43c0 to
d682908CompareFebruary 23, 2023 09:57
@moiseenkov
moiseenkov
commented
Feb 23, 2023

Copy link


Copy Markdown

ContributorAuthor
@kaxil , hi,

Could you review my fixes please?
kaxil
kaxil
reviewed
Feb 23, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

kaxil
kaxil
reviewed
Feb 23, 2023
View reviewed changes
Comment threaddocs/apache-airflow-providers-google/operators/transfer/s3_to_gcs.rst
Outdated

kaxil
kaxil
reviewed
Feb 23, 2023
View reviewed changes
@kaxilkaxil
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @pankajastro Could you review this too please
pankajastro
pankajastro
reviewed
Feb 24, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadtests/providers/google/cloud/hooks/test_cloud_storage_transfer_service_async.py
Outdated

Comment threadtests/providers/google/cloud/hooks/test_cloud_storage_transfer_service_async.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadtests/providers/google/cloud/hooks/test_cloud_storage_transfer_service_async.py
Outdated

@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
3 times, most recently
from
955d134 to
ef075b4CompareFebruary 24, 2023 15:07
@github-actions
github-actionsBot
commented
May 24, 2023

Copy link


Copy Markdown

Contributor
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
@github-actionsgithub-actionsBot
added
the 
stale
Stale PRs per the .github/workflows/stale.yml policy file
label
May 24, 2023
@potiuk
potiukforce-pushed
the
s3_to_gcs_deferrable_mode
branch
from
d409b8e to
9170b73CompareJune 4, 2023 23:15
@potiuk
potiuk
commented
Jun 4, 2023

Copy link


Copy Markdown

Member
Rebased it after some flaky mssql tools addressing in main, let's see if it helps
@potiuk
potiuk
commented
Jun 4, 2023

Copy link


Copy Markdown

Member
cc: @kosteev@bjankie1
@potiukpotiuk
closed this
Jun 4, 2023
@potiukpotiuk
reopened this
Jun 4, 2023
@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
3 times, most recently
from
d7ae276 to
c93d669CompareJune 6, 2023 08:32
@Lee-WLee-W
mentioned this pull request
Jun 6, 2023
 Merged

@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
4 times, most recently
from
766895b to
c4e6051CompareJune 7, 2023 06:37
pankajastro
pankajastro
reviewed
Jun 28, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

ahidalgob
ahidalgob
reviewed
Jun 28, 2023
View reviewed changes
Comment threaddocs/apache-airflow-providers-google/operators/transfer/s3_to_gcs.rst
Outdated

@VladaZakharova
VladaZakharova
commented
Jul 5, 2023

Copy link


Copy Markdown

Contributor
Hi @potiuk@pankajastro !

Could you please take a look on this PR? Thanks :)
potiuk
potiuk
requested changes
Jul 5, 2023
View reviewed changes
@potiukpotiuk
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We had a recent change on how we treat default deferrable setting and we need to adjust this one too: #31712
One more case in point @Lee-W that we need to add a pre-commit checking the defferrable convention, otherwise we will end up will all kind of mess where some deferrable parameters will have the default values following the convention and some not.
@moiseenkov - can you please update the change to follow this patterm?
@Lee-W -> will it be possible that you add such pre-commit (#32355 (comment)) ?
@Lee-W
Lee-W
commented
Jul 5, 2023

Copy link


Copy Markdown

Member
@potiuk Sure! Sorry for the late reply. I'm already working on it.
@potiuk
potiuk
commented
Jul 5, 2023

Copy link


Copy Markdown

Member
@potiuk Sure! Sorry for the late reply. I'm already working on it.
Cool! :)
@moiseenkov
moiseenkov
commented
Jul 5, 2023

Copy link


Copy Markdown

ContributorAuthor
We had a recent change on how we treat default deferrable setting and we need to adjust this one too: #31712
One more case in point @Lee-W that we need to add a pre-commit checking the defferrable convention, otherwise we will end up will all kind of mess where some deferrable parameters will have the default values following the convention and some not.
@VladaZakharova - can you please update the change to follow this patterm? @Lee-W -> will it be possible that you add such pre-commit (#32355 (comment)) ?
Thanks for the update! I will update PR and ping you afterwards
@moiseenkov
moiseenkov
commented
Jul 5, 2023

Copy link


Copy Markdown

ContributorAuthor
Hi, @potiuk , @Lee-W ,
I updated the PR:
https://github.com/apache/airflow/pull/29462/files#diff-3e4eedf131305f589cec193af2b406b57a5eab1a3db181272996c7e4d03e6cc6R156
@Lee-WLee-W
mentioned this pull request
Jul 5, 2023
 Merged

Lee-W
Lee-W
reviewed
Jul 6, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadtests/system/providers/google/cloud/gcs/example_s3_to_gcs_async.py
Outdated

@moiseenkov
Combine 8 into 1
a2e4755
Implement deferrable mode for S3ToGCSOperator
Update airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/transfers/s3_to_gcs.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/transfers/s3_to_gcs.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/transfers/s3_to_gcs.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Address code review comments
@VladaZakharova
VladaZakharova
commented
Jul 11, 2023

Copy link


Copy Markdown

Contributor
@potiuk

Hi there!

Could we recheck this PR? Thanks!
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@potiukpotiukpotiuk approved these changes
@pankajastropankajastropankajastro approved these changes
@kaxilkaxilAwaiting requested review from kaxil
@ahidalgobahidalgobAwaiting requested review from ahidalgob
@Lee-WLee-WAwaiting requested review from Lee-W

Assignees


No one assigned


Labels

area:providersarea:system-testskind:documentationprovider:googleGoogle (including GCP) related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

7 participants

@moiseenkov@potiuk@VladaZakharova@Lee-W@kaxil@ahidalgob@pankajastro

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Deferrable mode for S3ToGCSOperator by moiseenkov · Pull Request #29462 · apache/airflow · GitHub
Skip to content
Deferrable mode for S3ToGCSOperator - #29462
Merged
potiuk merged 1 commit into
apache:mainfrom 
lwyszomi:s3_to_gcs_deferrable_mode
Jul 11, 2023
Merged
Deferrable mode for S3ToGCSOperator#29462
potiuk merged 1 commit into
apache:mainfrom 
lwyszomi:s3_to_gcs_deferrable_mode
Conversation
@moiseenkov
@moiseenkovmoiseenkov
commented
Feb 10, 2023

Copy link


Copy Markdown

Contributor
^ Add meaningful description above
Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.
@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
4 times, most recently
from
6925d4e to
3469f13CompareFebruary 13, 2023 10:37
kaxil
kaxil
reviewed
Feb 13, 2023
View reviewed changes
Comment threadgenerated/provider_dependencies.json
Outdated

kaxil
kaxil
reviewed
Feb 13, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
9 times, most recently
from
c6b8ac5 to
74f43c0CompareFebruary 20, 2023 08:24
@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
from
74f43c0 to
d682908CompareFebruary 23, 2023 09:57
@moiseenkov
moiseenkov
commented
Feb 23, 2023

Copy link


Copy Markdown

ContributorAuthor
@kaxil , hi,

Could you review my fixes please?
kaxil
kaxil
reviewed
Feb 23, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

kaxil
kaxil
reviewed
Feb 23, 2023
View reviewed changes
Comment threaddocs/apache-airflow-providers-google/operators/transfer/s3_to_gcs.rst
Outdated

kaxil
kaxil
reviewed
Feb 23, 2023
View reviewed changes
@kaxilkaxil
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @pankajastro Could you review this too please
pankajastro
pankajastro
reviewed
Feb 24, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadtests/providers/google/cloud/hooks/test_cloud_storage_transfer_service_async.py
Outdated

Comment threadtests/providers/google/cloud/hooks/test_cloud_storage_transfer_service_async.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadtests/providers/google/cloud/hooks/test_cloud_storage_transfer_service_async.py
Outdated

@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
3 times, most recently
from
955d134 to
ef075b4CompareFebruary 24, 2023 15:07
@github-actions
github-actionsBot
commented
May 24, 2023

Copy link


Copy Markdown

Contributor
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
@github-actionsgithub-actionsBot
added
the 
stale
Stale PRs per the .github/workflows/stale.yml policy file
label
May 24, 2023
@potiuk
potiukforce-pushed
the
s3_to_gcs_deferrable_mode
branch
from
d409b8e to
9170b73CompareJune 4, 2023 23:15
@potiuk
potiuk
commented
Jun 4, 2023

Copy link


Copy Markdown

Member
Rebased it after some flaky mssql tools addressing in main, let's see if it helps
@potiuk
potiuk
commented
Jun 4, 2023

Copy link


Copy Markdown

Member
cc: @kosteev@bjankie1
@potiukpotiuk
closed this
Jun 4, 2023
@potiukpotiuk
reopened this
Jun 4, 2023
@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
3 times, most recently
from
d7ae276 to
c93d669CompareJune 6, 2023 08:32
@Lee-WLee-W
mentioned this pull request
Jun 6, 2023
 Merged

@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
4 times, most recently
from
766895b to
c4e6051CompareJune 7, 2023 06:37
pankajastro
pankajastro
reviewed
Jun 28, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

ahidalgob
ahidalgob
reviewed
Jun 28, 2023
View reviewed changes
Comment threaddocs/apache-airflow-providers-google/operators/transfer/s3_to_gcs.rst
Outdated

@VladaZakharova
VladaZakharova
commented
Jul 5, 2023

Copy link


Copy Markdown

Contributor
Hi @potiuk@pankajastro !

Could you please take a look on this PR? Thanks :)
potiuk
potiuk
requested changes
Jul 5, 2023
View reviewed changes
@potiukpotiuk
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We had a recent change on how we treat default deferrable setting and we need to adjust this one too: #31712
One more case in point @Lee-W that we need to add a pre-commit checking the defferrable convention, otherwise we will end up will all kind of mess where some deferrable parameters will have the default values following the convention and some not.
@moiseenkov - can you please update the change to follow this patterm?
@Lee-W -> will it be possible that you add such pre-commit (#32355 (comment)) ?
@Lee-W
Lee-W
commented
Jul 5, 2023

Copy link


Copy Markdown

Member
@potiuk Sure! Sorry for the late reply. I'm already working on it.
@potiuk
potiuk
commented
Jul 5, 2023

Copy link


Copy Markdown

Member
@potiuk Sure! Sorry for the late reply. I'm already working on it.
Cool! :)
@moiseenkov
moiseenkov
commented
Jul 5, 2023

Copy link


Copy Markdown

ContributorAuthor
We had a recent change on how we treat default deferrable setting and we need to adjust this one too: #31712
One more case in point @Lee-W that we need to add a pre-commit checking the defferrable convention, otherwise we will end up will all kind of mess where some deferrable parameters will have the default values following the convention and some not.
@VladaZakharova - can you please update the change to follow this patterm? @Lee-W -> will it be possible that you add such pre-commit (#32355 (comment)) ?
Thanks for the update! I will update PR and ping you afterwards
@moiseenkov
moiseenkov
commented
Jul 5, 2023

Copy link


Copy Markdown

ContributorAuthor
Hi, @potiuk , @Lee-W ,
I updated the PR:
https://github.com/apache/airflow/pull/29462/files#diff-3e4eedf131305f589cec193af2b406b57a5eab1a3db181272996c7e4d03e6cc6R156
@Lee-WLee-W
mentioned this pull request
Jul 5, 2023
 Merged

Lee-W
Lee-W
reviewed
Jul 6, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadtests/system/providers/google/cloud/gcs/example_s3_to_gcs_async.py
Outdated

@moiseenkov
Combine 8 into 1
a2e4755
Implement deferrable mode for S3ToGCSOperator
Update airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/transfers/s3_to_gcs.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/transfers/s3_to_gcs.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/transfers/s3_to_gcs.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Address code review comments
@VladaZakharova
VladaZakharova
commented
Jul 11, 2023

Copy link


Copy Markdown

Contributor
@potiuk

Hi there!

Could we recheck this PR? Thanks!
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@potiukpotiukpotiuk approved these changes
@pankajastropankajastropankajastro approved these changes
@kaxilkaxilAwaiting requested review from kaxil
@ahidalgobahidalgobAwaiting requested review from ahidalgob
@Lee-WLee-WAwaiting requested review from Lee-W

Assignees


No one assigned


Labels

area:providersarea:system-testskind:documentationprovider:googleGoogle (including GCP) related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

7 participants

@moiseenkov@potiuk@VladaZakharova@Lee-W@kaxil@ahidalgob@pankajastro

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Deferrable mode for S3ToGCSOperator by moiseenkov · Pull Request #29462 · apache/airflow · GitHub
Skip to content
Deferrable mode for S3ToGCSOperator - #29462
Merged
potiuk merged 1 commit into
apache:mainfrom 
lwyszomi:s3_to_gcs_deferrable_mode
Jul 11, 2023
Merged
Deferrable mode for S3ToGCSOperator#29462
potiuk merged 1 commit into
apache:mainfrom 
lwyszomi:s3_to_gcs_deferrable_mode
Conversation
@moiseenkov
@moiseenkovmoiseenkov
commented
Feb 10, 2023

Copy link


Copy Markdown

Contributor
^ Add meaningful description above
Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.
@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
4 times, most recently
from
6925d4e to
3469f13CompareFebruary 13, 2023 10:37
kaxil
kaxil
reviewed
Feb 13, 2023
View reviewed changes
Comment threadgenerated/provider_dependencies.json
Outdated

kaxil
kaxil
reviewed
Feb 13, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
9 times, most recently
from
c6b8ac5 to
74f43c0CompareFebruary 20, 2023 08:24
@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
from
74f43c0 to
d682908CompareFebruary 23, 2023 09:57
@moiseenkov
moiseenkov
commented
Feb 23, 2023

Copy link


Copy Markdown

ContributorAuthor
@kaxil , hi,

Could you review my fixes please?
kaxil
kaxil
reviewed
Feb 23, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

kaxil
kaxil
reviewed
Feb 23, 2023
View reviewed changes
Comment threaddocs/apache-airflow-providers-google/operators/transfer/s3_to_gcs.rst
Outdated

kaxil
kaxil
reviewed
Feb 23, 2023
View reviewed changes
@kaxilkaxil
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @pankajastro Could you review this too please
pankajastro
pankajastro
reviewed
Feb 24, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadtests/providers/google/cloud/hooks/test_cloud_storage_transfer_service_async.py
Outdated

Comment threadtests/providers/google/cloud/hooks/test_cloud_storage_transfer_service_async.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadtests/providers/google/cloud/hooks/test_cloud_storage_transfer_service_async.py
Outdated

@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
3 times, most recently
from
955d134 to
ef075b4CompareFebruary 24, 2023 15:07
@github-actions
github-actionsBot
commented
May 24, 2023

Copy link


Copy Markdown

Contributor
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
@github-actionsgithub-actionsBot
added
the 
stale
Stale PRs per the .github/workflows/stale.yml policy file
label
May 24, 2023
@potiuk
potiukforce-pushed
the
s3_to_gcs_deferrable_mode
branch
from
d409b8e to
9170b73CompareJune 4, 2023 23:15
@potiuk
potiuk
commented
Jun 4, 2023

Copy link


Copy Markdown

Member
Rebased it after some flaky mssql tools addressing in main, let's see if it helps
@potiuk
potiuk
commented
Jun 4, 2023

Copy link


Copy Markdown

Member
cc: @kosteev@bjankie1
@potiukpotiuk
closed this
Jun 4, 2023
@potiukpotiuk
reopened this
Jun 4, 2023
@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
3 times, most recently
from
d7ae276 to
c93d669CompareJune 6, 2023 08:32
@Lee-WLee-W
mentioned this pull request
Jun 6, 2023
 Merged

@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
4 times, most recently
from
766895b to
c4e6051CompareJune 7, 2023 06:37
pankajastro
pankajastro
reviewed
Jun 28, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

ahidalgob
ahidalgob
reviewed
Jun 28, 2023
View reviewed changes
Comment threaddocs/apache-airflow-providers-google/operators/transfer/s3_to_gcs.rst
Outdated

@VladaZakharova
VladaZakharova
commented
Jul 5, 2023

Copy link


Copy Markdown

Contributor
Hi @potiuk@pankajastro !

Could you please take a look on this PR? Thanks :)
potiuk
potiuk
requested changes
Jul 5, 2023
View reviewed changes
@potiukpotiuk
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We had a recent change on how we treat default deferrable setting and we need to adjust this one too: #31712
One more case in point @Lee-W that we need to add a pre-commit checking the defferrable convention, otherwise we will end up will all kind of mess where some deferrable parameters will have the default values following the convention and some not.
@moiseenkov - can you please update the change to follow this patterm?
@Lee-W -> will it be possible that you add such pre-commit (#32355 (comment)) ?
@Lee-W
Lee-W
commented
Jul 5, 2023

Copy link


Copy Markdown

Member
@potiuk Sure! Sorry for the late reply. I'm already working on it.
@potiuk
potiuk
commented
Jul 5, 2023

Copy link


Copy Markdown

Member
@potiuk Sure! Sorry for the late reply. I'm already working on it.
Cool! :)
@moiseenkov
moiseenkov
commented
Jul 5, 2023

Copy link


Copy Markdown

ContributorAuthor
We had a recent change on how we treat default deferrable setting and we need to adjust this one too: #31712
One more case in point @Lee-W that we need to add a pre-commit checking the defferrable convention, otherwise we will end up will all kind of mess where some deferrable parameters will have the default values following the convention and some not.
@VladaZakharova - can you please update the change to follow this patterm? @Lee-W -> will it be possible that you add such pre-commit (#32355 (comment)) ?
Thanks for the update! I will update PR and ping you afterwards
@moiseenkov
moiseenkov
commented
Jul 5, 2023

Copy link


Copy Markdown

ContributorAuthor
Hi, @potiuk , @Lee-W ,
I updated the PR:
https://github.com/apache/airflow/pull/29462/files#diff-3e4eedf131305f589cec193af2b406b57a5eab1a3db181272996c7e4d03e6cc6R156
@Lee-WLee-W
mentioned this pull request
Jul 5, 2023
 Merged

Lee-W
Lee-W
reviewed
Jul 6, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadtests/system/providers/google/cloud/gcs/example_s3_to_gcs_async.py
Outdated

@moiseenkov
Combine 8 into 1
a2e4755
Implement deferrable mode for S3ToGCSOperator
Update airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/transfers/s3_to_gcs.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/transfers/s3_to_gcs.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/transfers/s3_to_gcs.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Address code review comments
@VladaZakharova
VladaZakharova
commented
Jul 11, 2023

Copy link


Copy Markdown

Contributor
@potiuk

Hi there!

Could we recheck this PR? Thanks!
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@potiukpotiukpotiuk approved these changes
@pankajastropankajastropankajastro approved these changes
@kaxilkaxilAwaiting requested review from kaxil
@ahidalgobahidalgobAwaiting requested review from ahidalgob
@Lee-WLee-WAwaiting requested review from Lee-W

Assignees


No one assigned


Labels

area:providersarea:system-testskind:documentationprovider:googleGoogle (including GCP) related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

7 participants

@moiseenkov@potiuk@VladaZakharova@Lee-W@kaxil@ahidalgob@pankajastro

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
Deferrable mode for S3ToGCSOperator by moiseenkov · Pull Request #29462 · apache/airflow · GitHub
Skip to content
Deferrable mode for S3ToGCSOperator - #29462
Merged
potiuk merged 1 commit into
apache:mainfrom 
lwyszomi:s3_to_gcs_deferrable_mode
Jul 11, 2023
Merged
Deferrable mode for S3ToGCSOperator#29462
potiuk merged 1 commit into
apache:mainfrom 
lwyszomi:s3_to_gcs_deferrable_mode
Conversation
@moiseenkov
@moiseenkovmoiseenkov
commented
Feb 10, 2023

Copy link


Copy Markdown

Contributor
^ Add meaningful description above
Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.
@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
4 times, most recently
from
6925d4e to
3469f13CompareFebruary 13, 2023 10:37
kaxil
kaxil
reviewed
Feb 13, 2023
View reviewed changes
Comment threadgenerated/provider_dependencies.json
Outdated

kaxil
kaxil
reviewed
Feb 13, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
9 times, most recently
from
c6b8ac5 to
74f43c0CompareFebruary 20, 2023 08:24
@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
from
74f43c0 to
d682908CompareFebruary 23, 2023 09:57
@moiseenkov
moiseenkov
commented
Feb 23, 2023

Copy link


Copy Markdown

ContributorAuthor
@kaxil , hi,

Could you review my fixes please?
kaxil
kaxil
reviewed
Feb 23, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

kaxil
kaxil
reviewed
Feb 23, 2023
View reviewed changes
Comment threaddocs/apache-airflow-providers-google/operators/transfer/s3_to_gcs.rst
Outdated

kaxil
kaxil
reviewed
Feb 23, 2023
View reviewed changes
@kaxilkaxil
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @pankajastro Could you review this too please
pankajastro
pankajastro
reviewed
Feb 24, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadtests/providers/google/cloud/hooks/test_cloud_storage_transfer_service_async.py
Outdated

Comment threadtests/providers/google/cloud/hooks/test_cloud_storage_transfer_service_async.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadtests/providers/google/cloud/hooks/test_cloud_storage_transfer_service_async.py
Outdated

@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
3 times, most recently
from
955d134 to
ef075b4CompareFebruary 24, 2023 15:07
@github-actions
github-actionsBot
commented
May 24, 2023

Copy link


Copy Markdown

Contributor
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
@github-actionsgithub-actionsBot
added
the 
stale
Stale PRs per the .github/workflows/stale.yml policy file
label
May 24, 2023
@potiuk
potiukforce-pushed
the
s3_to_gcs_deferrable_mode
branch
from
d409b8e to
9170b73CompareJune 4, 2023 23:15
@potiuk
potiuk
commented
Jun 4, 2023

Copy link


Copy Markdown

Member
Rebased it after some flaky mssql tools addressing in main, let's see if it helps
@potiuk
potiuk
commented
Jun 4, 2023

Copy link


Copy Markdown

Member
cc: @kosteev@bjankie1
@potiukpotiuk
closed this
Jun 4, 2023
@potiukpotiuk
reopened this
Jun 4, 2023
@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
3 times, most recently
from
d7ae276 to
c93d669CompareJune 6, 2023 08:32
@Lee-WLee-W
mentioned this pull request
Jun 6, 2023
 Merged

@moiseenkov
moiseenkovforce-pushed
the
s3_to_gcs_deferrable_mode
branch
4 times, most recently
from
766895b to
c4e6051CompareJune 7, 2023 06:37
pankajastro
pankajastro
reviewed
Jun 28, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

ahidalgob
ahidalgob
reviewed
Jun 28, 2023
View reviewed changes
Comment threaddocs/apache-airflow-providers-google/operators/transfer/s3_to_gcs.rst
Outdated

@VladaZakharova
VladaZakharova
commented
Jul 5, 2023

Copy link


Copy Markdown

Contributor
Hi @potiuk@pankajastro !

Could you please take a look on this PR? Thanks :)
potiuk
potiuk
requested changes
Jul 5, 2023
View reviewed changes
@potiukpotiuk
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We had a recent change on how we treat default deferrable setting and we need to adjust this one too: #31712
One more case in point @Lee-W that we need to add a pre-commit checking the defferrable convention, otherwise we will end up will all kind of mess where some deferrable parameters will have the default values following the convention and some not.
@moiseenkov - can you please update the change to follow this patterm?
@Lee-W -> will it be possible that you add such pre-commit (#32355 (comment)) ?
@Lee-W
Lee-W
commented
Jul 5, 2023

Copy link


Copy Markdown

Member
@potiuk Sure! Sorry for the late reply. I'm already working on it.
@potiuk
potiuk
commented
Jul 5, 2023

Copy link


Copy Markdown

Member
@potiuk Sure! Sorry for the late reply. I'm already working on it.
Cool! :)
@moiseenkov
moiseenkov
commented
Jul 5, 2023

Copy link


Copy Markdown

ContributorAuthor
We had a recent change on how we treat default deferrable setting and we need to adjust this one too: #31712
One more case in point @Lee-W that we need to add a pre-commit checking the defferrable convention, otherwise we will end up will all kind of mess where some deferrable parameters will have the default values following the convention and some not.
@VladaZakharova - can you please update the change to follow this patterm? @Lee-W -> will it be possible that you add such pre-commit (#32355 (comment)) ?
Thanks for the update! I will update PR and ping you afterwards
@moiseenkov
moiseenkov
commented
Jul 5, 2023

Copy link


Copy Markdown

ContributorAuthor
Hi, @potiuk , @Lee-W ,
I updated the PR:
https://github.com/apache/airflow/pull/29462/files#diff-3e4eedf131305f589cec193af2b406b57a5eab1a3db181272996c7e4d03e6cc6R156
@Lee-WLee-W
mentioned this pull request
Jul 5, 2023
 Merged

Lee-W
Lee-W
reviewed
Jul 6, 2023
View reviewed changes
Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/transfers/s3_to_gcs.py
Outdated

Comment threadairflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Outdated

Comment threadtests/system/providers/google/cloud/gcs/example_s3_to_gcs_async.py
Outdated

@moiseenkov
Combine 8 into 1
a2e4755
Implement deferrable mode for S3ToGCSOperator
Update airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/hooks/cloud_storage_transfer_service.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/transfers/s3_to_gcs.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/transfers/s3_to_gcs.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/transfers/s3_to_gcs.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Update airflow/providers/google/cloud/triggers/cloud_storage_transfer_service.py
Co-authored-by: Wei Lee <weilee.rx@gmail.com>
Address code review comments
@VladaZakharova
VladaZakharova
commented
Jul 11, 2023

Copy link


Copy Markdown

Contributor
@potiuk

Hi there!

Could we recheck this PR? Thanks!
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@potiukpotiukpotiuk approved these changes
@pankajastropankajastropankajastro approved these changes
@kaxilkaxilAwaiting requested review from kaxil
@ahidalgobahidalgobAwaiting requested review from ahidalgob
@Lee-WLee-WAwaiting requested review from Lee-W

Assignees


No one assigned


Labels

area:providersarea:system-testskind:documentationprovider:googleGoogle (including GCP) related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

7 participants

@moiseenkov@potiuk@VladaZakharova@Lee-W@kaxil@ahidalgob@pankajastro