Skip to content

Add deferrable mode for S3KeySensor - #31018

Merged
ephraimbuddy merged 37 commits into
apache:mainfrom
sunank200:s3keysensor
Jun 8, 2023
Merged

Add deferrable mode for S3KeySensor#31018
ephraimbuddy merged 37 commits into
apache:mainfrom
sunank200:s3keysensor

Conversation

@sunank200

@sunank200sunank200 commented May 2, 2023•
edited
Loading

Copy link
Copy Markdown
Collaborator

This PR donates the following S3KeySensor with a deferrable mode, waiting for one or multiple keys (a file-like instance on S3) to be in an S3 bucket. This was developed in astronomer-providers repository to apache-airflow.

cc: @phanikumv

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborgboring-cyborgBot added area:providers provider:amazon AWS/Amazon - related issues labels May 2, 2023
pankajastro
pankajastro reviewed May 2, 2023
View reviewed changes
Comment threadairflow/providers/amazon/provider.yaml Outdated
@potiukpotiuk mentioned this pull request May 2, 2023
Merged
@sunank200
sunank200 marked this pull request as ready for review May 10, 2023 02:22
@potiuk

potiuk commented May 17, 2023

Copy link
Copy Markdown
Member

@ferruzzi@o-nikolas@vincbeck -> your feedback is welcome :)

@o-nikolas

o-nikolas commented May 17, 2023

Copy link
Copy Markdown
Contributor

CC @syedahsn for when you're back

vincbeck
vincbeck requested changes May 18, 2023
View reviewed changes
Comment threadairflow/providers/amazon/aws/hooks/s3.py Outdated
@sunank200
sunank200 requested a review from vincbeckMay 29, 2023 14:58
vincbeck
vincbeck requested changes May 29, 2023
View reviewed changes
Comment threadtests/providers/amazon/aws/compat.py Outdated
Comment threadairflow/providers/amazon/aws/hooks/s3.py
pankajastro
pankajastro reviewed May 29, 2023
View reviewed changes
Comment threadairflow/providers/amazon/provider.yaml Outdated
@sunank200sunank200 mentioned this pull request Jun 7, 2023
Closed
ephraimbuddy
ephraimbuddy reviewed Jun 7, 2023
View reviewed changes
Comment threadairflow/providers/amazon/aws/hooks/s3.py Outdated
ephraimbuddy
ephraimbuddy reviewed Jun 7, 2023
View reviewed changes
Comment threadairflow/providers/amazon/aws/hooks/s3.py Outdated
vandonr-amz
vandonr-amz reviewed Jun 7, 2023
View reviewed changes
Comment threadairflow/providers/amazon/aws/sensors/s3.py
Comment threadairflow/providers/amazon/aws/sensors/s3.py Outdated
Comment threadairflow/providers/amazon/aws/sensors/s3.py Outdated
Comment threadairflow/providers/amazon/aws/triggers/s3.py
vandonr-amz
vandonr-amz reviewed Jun 7, 2023
View reviewed changes
Comment threadairflow/providers/amazon/aws/sensors/s3.py Outdated
jedcunningham
jedcunningham approved these changes Jun 8, 2023
View reviewed changes
Comment threadtests/providers/amazon/aws/triggers/test_s3.py
Comment on lines +24 to +26
from airflow.providers.amazon.aws.triggers.s3 import (
S3KeyTrigger,
)

@jedcunninghamjedcunninghamJun 8, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
fromairflow.providers.amazon.aws.triggers.s3import (
S3KeyTrigger,
)
fromairflow.providers.amazon.aws.triggers.s3importS3KeyTrigger

nit

@ephraimbuddyephraimbuddyJun 8, 2023•
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So sorry. Didn't see this. Was on mobile

@ephraimbuddy
ephraimbuddy merged commit b9b8da7 into apache:mainJun 8, 2023
@eladkaleladkal mentioned this pull request Jun 20, 2023
Closed
86 tasks
@patgarzpatgarz mentioned this pull request Aug 3, 2023
Closed
2 tasks
@TaragolisTaragolis mentioned this pull request Apr 15, 2024
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ephraimbuddyephraimbuddyephraimbuddy approved these changes
@jedcunninghamjedcunninghamjedcunningham approved these changes
@phanikumvphanikumvphanikumv approved these changes
@eladkaleladkalAwaiting requested review from eladkal
@o-nikolaso-nikolasAwaiting requested review from o-nikolas
@potiukpotiukAwaiting requested review from potiuk
@pankajastropankajastroAwaiting requested review from pankajastro
@syedahsnsyedahsnAwaiting requested review from syedahsn
@vincbeckvincbeckAwaiting requested review from vincbeck
@vandonr-amzvandonr-amzAwaiting requested review from vandonr-amz

Assignees

No one assigned

Labels

area:providersprovider:amazonAWS/Amazon - related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

10 participants

@sunank200@potiuk@o-nikolas@syedahsn@ephraimbuddy@vandonr-amz@vincbeck@jedcunningham@phanikumv@pankajastro