S3KeySensor to use S3Hook url parser by dstandish · Pull Request #21500 · apache/airflow

dstandish · 2022-02-10T17:47:28Z

S3KeySensor resolves the bucket_name and key in the poke method.

Pulling this "resolution" logic out into a _resolve_bucket_and_key method makes it a little clearer what's happening there.

Unfortunately we can't just do it in __init__ because we have to wait for templates to render first.

I took the opportunity here to remove redundant s3 uri parsing logic, which already exists in a static method on the hook.

And since we use that hook method now, I had to update the tests to not mock the entire hook class but only the check methods.

Finally, since while I was at it I strengthened the test slightly, replacing assert op.poke(None) with assert op.poke(None) is True etc, since mocks can evaluate to "truthy"

Rather than relying on the poke method to parse the attrs we should either do it in `__init__` or in a cached property. Since it's a pretty insignifigant computation let's just do it in `__init__`. And anyway, if the params are bad then best to get a warning before deploying your code.

josh-fell · 2022-02-10T19:00:53Z

Parsing bucket_key in the __init__() won't work if the value is an XComArg or a Jinja string. We've seen several issues, namely AttributeErrors, when some sort of string parsing is used on templated fields in __init__() since the value doesn't get rendered until the execute() method. Check out #14682 as an example.

josh-fell · 2022-02-10T19:07:33Z

FWIW, datatype checks on templated fields within __init__() have their issues as well. See #20347.

dstandish · 2022-02-10T19:11:18Z

Parsing bucket_key in the __init__() won't work if the value is an XComArg or a Jinja string. We've seen several issues, namely AttributeErrors, when some sort of string parsing is used on templated fields in __init__() since the value doesn't get rendered until the execute() method. Check out #14682 as an example.

darnit, good catch... maybe i'll go cached prop route

josh-fell

Trivial nit + a musing but LGTM

josh-fell · 2022-02-11T15:26:06Z

- ' should be relative path from root'
- ' level, rather than a full s3:// url'
- )
+ raise AirflowException('If bucket_name provided, bucket_key must be relative path, not URI.')


Suggested change
raiseAirflowException('If bucket_name provided, bucket_key must be relative path, not URI.')
raiseAirflowException('If bucket_name is provided, bucket_key must be a relative path, not a URI.')

i was trying to keep it on one line :) lemme see if that works

josh-fell · 2022-02-11T15:28:51Z

- self.bucket_key = parsed_url.path.lstrip('/')
+ self.bucket_name, self.bucket_key = S3Hook.parse_s3_url(self.bucket_key)
 else:
 parsed_url = urlparse(self.bucket_key)


Doesn't have to happen in this PR but I wonder if S3Hook.parse_s3_url() could be augmented with some sort of configurable check between validating a URI vs. a relative path? Centralize some of this logic there without have urlparse() is different places. Anyway, just a thought.

dstandish · 2022-02-10T17:47:28Z

S3KeySensor resolves the bucket_name and key in the poke method.

Pulling this "resolution" logic out into a _resolve_bucket_and_key method makes it a little clearer what's happening there.

Unfortunately we can't just do it in __init__ because we have to wait for templates to render first.

I took the opportunity here to remove redundant s3 uri parsing logic, which already exists in a static method on the hook.

And since we use that hook method now, I had to update the tests to not mock the entire hook class but only the check methods.

Finally, since while I was at it I strengthened the test slightly, replacing assert op.poke(None) with assert op.poke(None) is True etc, since mocks can evaluate to "truthy"

Rather than relying on the poke method to parse the attrs we should either do it in `__init__` or in a cached property. Since it's a pretty insignifigant computation let's just do it in `__init__`. And anyway, if the params are bad then best to get a warning before deploying your code.

josh-fell · 2022-02-10T19:00:53Z

Parsing bucket_key in the __init__() won't work if the value is an XComArg or a Jinja string. We've seen several issues, namely AttributeErrors, when some sort of string parsing is used on templated fields in __init__() since the value doesn't get rendered until the execute() method. Check out #14682 as an example.

josh-fell · 2022-02-10T19:07:33Z

FWIW, datatype checks on templated fields within __init__() have their issues as well. See #20347.

dstandish · 2022-02-10T19:11:18Z

Parsing bucket_key in the __init__() won't work if the value is an XComArg or a Jinja string. We've seen several issues, namely AttributeErrors, when some sort of string parsing is used on templated fields in __init__() since the value doesn't get rendered until the execute() method. Check out #14682 as an example.

darnit, good catch... maybe i'll go cached prop route

josh-fell

Trivial nit + a musing but LGTM

josh-fell · 2022-02-11T15:26:06Z

- ' should be relative path from root'
- ' level, rather than a full s3:// url'
- )
+ raise AirflowException('If bucket_name provided, bucket_key must be relative path, not URI.')


Suggested change
raiseAirflowException('If bucket_name provided, bucket_key must be relative path, not URI.')
raiseAirflowException('If bucket_name is provided, bucket_key must be a relative path, not a URI.')

i was trying to keep it on one line :) lemme see if that works

josh-fell · 2022-02-11T15:28:51Z

- self.bucket_key = parsed_url.path.lstrip('/')
+ self.bucket_name, self.bucket_key = S3Hook.parse_s3_url(self.bucket_key)
 else:
 parsed_url = urlparse(self.bucket_key)


Doesn't have to happen in this PR but I wonder if S3Hook.parse_s3_url() could be augmented with some sort of configurable check between validating a URI vs. a relative path? Centralize some of this logic there without have urlparse() is different places. Anyway, just a thought.

dstandish · 2022-02-10T17:47:28Z

S3KeySensor resolves the bucket_name and key in the poke method.

Pulling this "resolution" logic out into a _resolve_bucket_and_key method makes it a little clearer what's happening there.

Unfortunately we can't just do it in __init__ because we have to wait for templates to render first.

I took the opportunity here to remove redundant s3 uri parsing logic, which already exists in a static method on the hook.

And since we use that hook method now, I had to update the tests to not mock the entire hook class but only the check methods.

Finally, since while I was at it I strengthened the test slightly, replacing assert op.poke(None) with assert op.poke(None) is True etc, since mocks can evaluate to "truthy"

Rather than relying on the poke method to parse the attrs we should either do it in `__init__` or in a cached property. Since it's a pretty insignifigant computation let's just do it in `__init__`. And anyway, if the params are bad then best to get a warning before deploying your code.

josh-fell · 2022-02-10T19:00:53Z

Parsing bucket_key in the __init__() won't work if the value is an XComArg or a Jinja string. We've seen several issues, namely AttributeErrors, when some sort of string parsing is used on templated fields in __init__() since the value doesn't get rendered until the execute() method. Check out #14682 as an example.

josh-fell · 2022-02-10T19:07:33Z

FWIW, datatype checks on templated fields within __init__() have their issues as well. See #20347.

dstandish · 2022-02-10T19:11:18Z

Parsing bucket_key in the __init__() won't work if the value is an XComArg or a Jinja string. We've seen several issues, namely AttributeErrors, when some sort of string parsing is used on templated fields in __init__() since the value doesn't get rendered until the execute() method. Check out #14682 as an example.

darnit, good catch... maybe i'll go cached prop route

josh-fell

Trivial nit + a musing but LGTM

josh-fell · 2022-02-11T15:26:06Z

- ' should be relative path from root'
- ' level, rather than a full s3:// url'
- )
+ raise AirflowException('If bucket_name provided, bucket_key must be relative path, not URI.')


Suggested change
raiseAirflowException('If bucket_name provided, bucket_key must be relative path, not URI.')
raiseAirflowException('If bucket_name is provided, bucket_key must be a relative path, not a URI.')

i was trying to keep it on one line :) lemme see if that works

josh-fell · 2022-02-11T15:28:51Z

- self.bucket_key = parsed_url.path.lstrip('/')
+ self.bucket_name, self.bucket_key = S3Hook.parse_s3_url(self.bucket_key)
 else:
 parsed_url = urlparse(self.bucket_key)


Doesn't have to happen in this PR but I wonder if S3Hook.parse_s3_url() could be augmented with some sort of configurable check between validating a URI vs. a relative path? Centralize some of this logic there without have urlparse() is different places. Anyway, just a thought.

dstandish · 2022-02-10T17:47:28Z

S3KeySensor resolves the bucket_name and key in the poke method.

Pulling this "resolution" logic out into a _resolve_bucket_and_key method makes it a little clearer what's happening there.

Unfortunately we can't just do it in __init__ because we have to wait for templates to render first.

I took the opportunity here to remove redundant s3 uri parsing logic, which already exists in a static method on the hook.

And since we use that hook method now, I had to update the tests to not mock the entire hook class but only the check methods.

Finally, since while I was at it I strengthened the test slightly, replacing assert op.poke(None) with assert op.poke(None) is True etc, since mocks can evaluate to "truthy"

Rather than relying on the poke method to parse the attrs we should either do it in `__init__` or in a cached property. Since it's a pretty insignifigant computation let's just do it in `__init__`. And anyway, if the params are bad then best to get a warning before deploying your code.

josh-fell · 2022-02-10T19:00:53Z

Parsing bucket_key in the __init__() won't work if the value is an XComArg or a Jinja string. We've seen several issues, namely AttributeErrors, when some sort of string parsing is used on templated fields in __init__() since the value doesn't get rendered until the execute() method. Check out #14682 as an example.

josh-fell · 2022-02-10T19:07:33Z

FWIW, datatype checks on templated fields within __init__() have their issues as well. See #20347.

dstandish · 2022-02-10T19:11:18Z

Parsing bucket_key in the __init__() won't work if the value is an XComArg or a Jinja string. We've seen several issues, namely AttributeErrors, when some sort of string parsing is used on templated fields in __init__() since the value doesn't get rendered until the execute() method. Check out #14682 as an example.

darnit, good catch... maybe i'll go cached prop route

josh-fell

Trivial nit + a musing but LGTM

josh-fell · 2022-02-11T15:26:06Z

- ' should be relative path from root'
- ' level, rather than a full s3:// url'
- )
+ raise AirflowException('If bucket_name provided, bucket_key must be relative path, not URI.')


Suggested change
raiseAirflowException('If bucket_name provided, bucket_key must be relative path, not URI.')
raiseAirflowException('If bucket_name is provided, bucket_key must be a relative path, not a URI.')

i was trying to keep it on one line :) lemme see if that works

josh-fell · 2022-02-11T15:28:51Z

- self.bucket_key = parsed_url.path.lstrip('/')
+ self.bucket_name, self.bucket_key = S3Hook.parse_s3_url(self.bucket_key)
 else:
 parsed_url = urlparse(self.bucket_key)


Doesn't have to happen in this PR but I wonder if S3Hook.parse_s3_url() could be augmented with some sort of configurable check between validating a URI vs. a relative path? Centralize some of this logic there without have urlparse() is different places. Anyway, just a thought.

dstandish · 2022-02-10T17:47:28Z

S3KeySensor resolves the bucket_name and key in the poke method.

Pulling this "resolution" logic out into a _resolve_bucket_and_key method makes it a little clearer what's happening there.

Unfortunately we can't just do it in __init__ because we have to wait for templates to render first.

I took the opportunity here to remove redundant s3 uri parsing logic, which already exists in a static method on the hook.

And since we use that hook method now, I had to update the tests to not mock the entire hook class but only the check methods.

Finally, since while I was at it I strengthened the test slightly, replacing assert op.poke(None) with assert op.poke(None) is True etc, since mocks can evaluate to "truthy"

Rather than relying on the poke method to parse the attrs we should either do it in `__init__` or in a cached property. Since it's a pretty insignifigant computation let's just do it in `__init__`. And anyway, if the params are bad then best to get a warning before deploying your code.

josh-fell · 2022-02-10T19:00:53Z

Parsing bucket_key in the __init__() won't work if the value is an XComArg or a Jinja string. We've seen several issues, namely AttributeErrors, when some sort of string parsing is used on templated fields in __init__() since the value doesn't get rendered until the execute() method. Check out #14682 as an example.

josh-fell · 2022-02-10T19:07:33Z

FWIW, datatype checks on templated fields within __init__() have their issues as well. See #20347.

dstandish · 2022-02-10T19:11:18Z

Parsing bucket_key in the __init__() won't work if the value is an XComArg or a Jinja string. We've seen several issues, namely AttributeErrors, when some sort of string parsing is used on templated fields in __init__() since the value doesn't get rendered until the execute() method. Check out #14682 as an example.

darnit, good catch... maybe i'll go cached prop route

josh-fell

Trivial nit + a musing but LGTM

josh-fell · 2022-02-11T15:26:06Z

- ' should be relative path from root'
- ' level, rather than a full s3:// url'
- )
+ raise AirflowException('If bucket_name provided, bucket_key must be relative path, not URI.')


Suggested change
raiseAirflowException('If bucket_name provided, bucket_key must be relative path, not URI.')
raiseAirflowException('If bucket_name is provided, bucket_key must be a relative path, not a URI.')

i was trying to keep it on one line :) lemme see if that works

josh-fell · 2022-02-11T15:28:51Z

- self.bucket_key = parsed_url.path.lstrip('/')
+ self.bucket_name, self.bucket_key = S3Hook.parse_s3_url(self.bucket_key)
 else:
 parsed_url = urlparse(self.bucket_key)


Doesn't have to happen in this PR but I wonder if S3Hook.parse_s3_url() could be augmented with some sort of configurable check between validating a URI vs. a relative path? Centralize some of this logic there without have urlparse() is different places. Anyway, just a thought.

dstandish · 2022-02-10T17:47:28Z

S3KeySensor resolves the bucket_name and key in the poke method.

Pulling this "resolution" logic out into a _resolve_bucket_and_key method makes it a little clearer what's happening there.

Unfortunately we can't just do it in __init__ because we have to wait for templates to render first.

I took the opportunity here to remove redundant s3 uri parsing logic, which already exists in a static method on the hook.

And since we use that hook method now, I had to update the tests to not mock the entire hook class but only the check methods.

Finally, since while I was at it I strengthened the test slightly, replacing assert op.poke(None) with assert op.poke(None) is True etc, since mocks can evaluate to "truthy"

Rather than relying on the poke method to parse the attrs we should either do it in `__init__` or in a cached property. Since it's a pretty insignifigant computation let's just do it in `__init__`. And anyway, if the params are bad then best to get a warning before deploying your code.

josh-fell · 2022-02-10T19:00:53Z

Parsing bucket_key in the __init__() won't work if the value is an XComArg or a Jinja string. We've seen several issues, namely AttributeErrors, when some sort of string parsing is used on templated fields in __init__() since the value doesn't get rendered until the execute() method. Check out #14682 as an example.

josh-fell · 2022-02-10T19:07:33Z

FWIW, datatype checks on templated fields within __init__() have their issues as well. See #20347.

dstandish · 2022-02-10T19:11:18Z

Parsing bucket_key in the __init__() won't work if the value is an XComArg or a Jinja string. We've seen several issues, namely AttributeErrors, when some sort of string parsing is used on templated fields in __init__() since the value doesn't get rendered until the execute() method. Check out #14682 as an example.

darnit, good catch... maybe i'll go cached prop route

josh-fell

Trivial nit + a musing but LGTM

josh-fell · 2022-02-11T15:26:06Z

- ' should be relative path from root'
- ' level, rather than a full s3:// url'
- )
+ raise AirflowException('If bucket_name provided, bucket_key must be relative path, not URI.')


Suggested change
raiseAirflowException('If bucket_name provided, bucket_key must be relative path, not URI.')
raiseAirflowException('If bucket_name is provided, bucket_key must be a relative path, not a URI.')

i was trying to keep it on one line :) lemme see if that works

josh-fell · 2022-02-11T15:28:51Z

- self.bucket_key = parsed_url.path.lstrip('/')
+ self.bucket_name, self.bucket_key = S3Hook.parse_s3_url(self.bucket_key)
 else:
 parsed_url = urlparse(self.bucket_key)


Doesn't have to happen in this PR but I wonder if S3Hook.parse_s3_url() could be augmented with some sort of configurable check between validating a URI vs. a relative path? Centralize some of this logic there without have urlparse() is different places. Anyway, just a thought.

dstandish · 2022-02-10T17:47:28Z

S3KeySensor resolves the bucket_name and key in the poke method.

Pulling this "resolution" logic out into a _resolve_bucket_and_key method makes it a little clearer what's happening there.

Unfortunately we can't just do it in __init__ because we have to wait for templates to render first.

I took the opportunity here to remove redundant s3 uri parsing logic, which already exists in a static method on the hook.

And since we use that hook method now, I had to update the tests to not mock the entire hook class but only the check methods.

Finally, since while I was at it I strengthened the test slightly, replacing assert op.poke(None) with assert op.poke(None) is True etc, since mocks can evaluate to "truthy"

Rather than relying on the poke method to parse the attrs we should either do it in `__init__` or in a cached property. Since it's a pretty insignifigant computation let's just do it in `__init__`. And anyway, if the params are bad then best to get a warning before deploying your code.

josh-fell · 2022-02-10T19:00:53Z

Parsing bucket_key in the __init__() won't work if the value is an XComArg or a Jinja string. We've seen several issues, namely AttributeErrors, when some sort of string parsing is used on templated fields in __init__() since the value doesn't get rendered until the execute() method. Check out #14682 as an example.

josh-fell · 2022-02-10T19:07:33Z

FWIW, datatype checks on templated fields within __init__() have their issues as well. See #20347.

dstandish · 2022-02-10T19:11:18Z

Parsing bucket_key in the __init__() won't work if the value is an XComArg or a Jinja string. We've seen several issues, namely AttributeErrors, when some sort of string parsing is used on templated fields in __init__() since the value doesn't get rendered until the execute() method. Check out #14682 as an example.

darnit, good catch... maybe i'll go cached prop route

josh-fell

Trivial nit + a musing but LGTM

josh-fell · 2022-02-11T15:26:06Z

- ' should be relative path from root'
- ' level, rather than a full s3:// url'
- )
+ raise AirflowException('If bucket_name provided, bucket_key must be relative path, not URI.')


Suggested change
raiseAirflowException('If bucket_name provided, bucket_key must be relative path, not URI.')
raiseAirflowException('If bucket_name is provided, bucket_key must be a relative path, not a URI.')

i was trying to keep it on one line :) lemme see if that works

josh-fell · 2022-02-11T15:28:51Z

- self.bucket_key = parsed_url.path.lstrip('/')
+ self.bucket_name, self.bucket_key = S3Hook.parse_s3_url(self.bucket_key)
 else:
 parsed_url = urlparse(self.bucket_key)


Doesn't have to happen in this PR but I wonder if S3Hook.parse_s3_url() could be augmented with some sort of configurable check between validating a URI vs. a relative path? Centralize some of this logic there without have urlparse() is different places. Anyway, just a thought.

dstandish · 2022-02-10T17:47:28Z

S3KeySensor resolves the bucket_name and key in the poke method.

Pulling this "resolution" logic out into a _resolve_bucket_and_key method makes it a little clearer what's happening there.

Unfortunately we can't just do it in __init__ because we have to wait for templates to render first.

I took the opportunity here to remove redundant s3 uri parsing logic, which already exists in a static method on the hook.

And since we use that hook method now, I had to update the tests to not mock the entire hook class but only the check methods.

Finally, since while I was at it I strengthened the test slightly, replacing assert op.poke(None) with assert op.poke(None) is True etc, since mocks can evaluate to "truthy"

Rather than relying on the poke method to parse the attrs we should either do it in `__init__` or in a cached property. Since it's a pretty insignifigant computation let's just do it in `__init__`. And anyway, if the params are bad then best to get a warning before deploying your code.

josh-fell · 2022-02-10T19:00:53Z

Parsing bucket_key in the __init__() won't work if the value is an XComArg or a Jinja string. We've seen several issues, namely AttributeErrors, when some sort of string parsing is used on templated fields in __init__() since the value doesn't get rendered until the execute() method. Check out #14682 as an example.

josh-fell · 2022-02-10T19:07:33Z

FWIW, datatype checks on templated fields within __init__() have their issues as well. See #20347.

dstandish · 2022-02-10T19:11:18Z

Parsing bucket_key in the __init__() won't work if the value is an XComArg or a Jinja string. We've seen several issues, namely AttributeErrors, when some sort of string parsing is used on templated fields in __init__() since the value doesn't get rendered until the execute() method. Check out #14682 as an example.

darnit, good catch... maybe i'll go cached prop route

josh-fell

Trivial nit + a musing but LGTM

josh-fell · 2022-02-11T15:26:06Z

- ' should be relative path from root'
- ' level, rather than a full s3:// url'
- )
+ raise AirflowException('If bucket_name provided, bucket_key must be relative path, not URI.')


Suggested change
raiseAirflowException('If bucket_name provided, bucket_key must be relative path, not URI.')
raiseAirflowException('If bucket_name is provided, bucket_key must be a relative path, not a URI.')

i was trying to keep it on one line :) lemme see if that works

josh-fell · 2022-02-11T15:28:51Z

- self.bucket_key = parsed_url.path.lstrip('/')
+ self.bucket_name, self.bucket_key = S3Hook.parse_s3_url(self.bucket_key)
 else:
 parsed_url = urlparse(self.bucket_key)


Doesn't have to happen in this PR but I wonder if S3Hook.parse_s3_url() could be augmented with some sort of configurable check between validating a URI vs. a relative path? Centralize some of this logic there without have urlparse() is different places. Anyway, just a thought.

boring-cyborgBot added area:providers provider:amazon AWS/Amazon - related issues labels Feb 10, 2022

dstandish mentioned this pull request Feb 10, 2022
Implement Async S3KeySensorastronomer/astronomer-providers#45
Merged

dstandish marked this pull request as draft February 10, 2022 19:11

dstandish added 3 commits February 10, 2022 15:35

must resolve only after template rendering
ba4f318

fixup! must resolve only after template rendering
ce061ac

economize
1e40c03

dstandish changed the title ~~S3KeySensor should parse bucket and key outside of poke~~S3KeySensor to use S3Hook url parserFeb 10, 2022

fixup! economize
923ed05

dstandish marked this pull request as ready for review February 11, 2022 06:37

josh-fell reviewed Feb 11, 2022
View reviewed changes

josh-fell approved these changes Feb 11, 2022
View reviewed changes

potiuk approved these changes Feb 15, 2022
View reviewed changes

potiuk merged commit fc44836 into apache:mainFeb 15, 2022

potiuk mentioned this pull request Mar 7, 2022
Status of testing Providers that were prepared on March 07, 2022 #22063
Closed
84 tasks

potiuk mentioned this pull request Mar 15, 2022
Status of testing Providers that were prepared on March 15, 2022 #22264
Closed
41 tasks

boring-cyborgBot added area:providers provider:amazon AWS/Amazon - related issues labels Feb 10, 2022

dstandish mentioned this pull request Feb 10, 2022
Implement Async S3KeySensorastronomer/astronomer-providers#45
Merged

dstandish marked this pull request as draft February 10, 2022 19:11

dstandish added 3 commits February 10, 2022 15:35

must resolve only after template rendering
ba4f318

fixup! must resolve only after template rendering
ce061ac

economize
1e40c03

dstandish changed the title ~~S3KeySensor should parse bucket and key outside of poke~~S3KeySensor to use S3Hook url parserFeb 10, 2022

fixup! economize
923ed05

dstandish marked this pull request as ready for review February 11, 2022 06:37

josh-fell reviewed Feb 11, 2022
View reviewed changes

josh-fell approved these changes Feb 11, 2022
View reviewed changes

potiuk approved these changes Feb 15, 2022
View reviewed changes

potiuk merged commit fc44836 into apache:mainFeb 15, 2022

potiuk mentioned this pull request Mar 7, 2022
Status of testing Providers that were prepared on March 07, 2022 #22063
Closed
84 tasks

potiuk mentioned this pull request Mar 15, 2022
Status of testing Providers that were prepared on March 15, 2022 #22264
Closed
41 tasks

boring-cyborgBot added area:providers provider:amazon AWS/Amazon - related issues labels Feb 10, 2022

dstandish mentioned this pull request Feb 10, 2022
Implement Async S3KeySensorastronomer/astronomer-providers#45
Merged

dstandish marked this pull request as draft February 10, 2022 19:11

dstandish added 3 commits February 10, 2022 15:35

must resolve only after template rendering
ba4f318

fixup! must resolve only after template rendering
ce061ac

economize
1e40c03

dstandish changed the title ~~S3KeySensor should parse bucket and key outside of poke~~S3KeySensor to use S3Hook url parserFeb 10, 2022

fixup! economize
923ed05

dstandish marked this pull request as ready for review February 11, 2022 06:37

josh-fell reviewed Feb 11, 2022
View reviewed changes

josh-fell approved these changes Feb 11, 2022
View reviewed changes

potiuk approved these changes Feb 15, 2022
View reviewed changes

potiuk merged commit fc44836 into apache:mainFeb 15, 2022

potiuk mentioned this pull request Mar 7, 2022
Status of testing Providers that were prepared on March 07, 2022 #22063
Closed
84 tasks

potiuk mentioned this pull request Mar 15, 2022
Status of testing Providers that were prepared on March 15, 2022 #22264
Closed
41 tasks

boring-cyborgBot added area:providers provider:amazon AWS/Amazon - related issues labels Feb 10, 2022

dstandish mentioned this pull request Feb 10, 2022
Implement Async S3KeySensorastronomer/astronomer-providers#45
Merged

dstandish marked this pull request as draft February 10, 2022 19:11

dstandish added 3 commits February 10, 2022 15:35

must resolve only after template rendering
ba4f318

fixup! must resolve only after template rendering
ce061ac

economize
1e40c03

dstandish changed the title ~~S3KeySensor should parse bucket and key outside of poke~~S3KeySensor to use S3Hook url parserFeb 10, 2022

fixup! economize
923ed05

dstandish marked this pull request as ready for review February 11, 2022 06:37

josh-fell reviewed Feb 11, 2022
View reviewed changes

josh-fell approved these changes Feb 11, 2022
View reviewed changes

potiuk approved these changes Feb 15, 2022
View reviewed changes

potiuk merged commit fc44836 into apache:mainFeb 15, 2022

potiuk mentioned this pull request Mar 7, 2022
Status of testing Providers that were prepared on March 07, 2022 #22063
Closed
84 tasks

potiuk mentioned this pull request Mar 15, 2022
Status of testing Providers that were prepared on March 15, 2022 #22264
Closed
41 tasks

boring-cyborgBot added area:providers provider:amazon AWS/Amazon - related issues labels Feb 10, 2022

dstandish mentioned this pull request Feb 10, 2022
Implement Async S3KeySensorastronomer/astronomer-providers#45
Merged

dstandish marked this pull request as draft February 10, 2022 19:11

dstandish added 3 commits February 10, 2022 15:35

must resolve only after template rendering
ba4f318

fixup! must resolve only after template rendering
ce061ac

economize
1e40c03

dstandish changed the title ~~S3KeySensor should parse bucket and key outside of poke~~S3KeySensor to use S3Hook url parserFeb 10, 2022

fixup! economize
923ed05

dstandish marked this pull request as ready for review February 11, 2022 06:37

josh-fell reviewed Feb 11, 2022
View reviewed changes

josh-fell approved these changes Feb 11, 2022
View reviewed changes

potiuk approved these changes Feb 15, 2022
View reviewed changes

potiuk merged commit fc44836 into apache:mainFeb 15, 2022

potiuk mentioned this pull request Mar 7, 2022
Status of testing Providers that were prepared on March 07, 2022 #22063
Closed
84 tasks

potiuk mentioned this pull request Mar 15, 2022
Status of testing Providers that were prepared on March 15, 2022 #22264
Closed
41 tasks

boring-cyborgBot added area:providers provider:amazon AWS/Amazon - related issues labels Feb 10, 2022

dstandish mentioned this pull request Feb 10, 2022
Implement Async S3KeySensorastronomer/astronomer-providers#45
Merged

dstandish marked this pull request as draft February 10, 2022 19:11

dstandish added 3 commits February 10, 2022 15:35

must resolve only after template rendering
ba4f318

fixup! must resolve only after template rendering
ce061ac

economize
1e40c03

dstandish changed the title ~~S3KeySensor should parse bucket and key outside of poke~~S3KeySensor to use S3Hook url parserFeb 10, 2022

fixup! economize
923ed05

dstandish marked this pull request as ready for review February 11, 2022 06:37

josh-fell reviewed Feb 11, 2022
View reviewed changes

josh-fell approved these changes Feb 11, 2022
View reviewed changes

potiuk approved these changes Feb 15, 2022
View reviewed changes

potiuk merged commit fc44836 into apache:mainFeb 15, 2022

potiuk mentioned this pull request Mar 7, 2022
Status of testing Providers that were prepared on March 07, 2022 #22063
Closed
84 tasks

potiuk mentioned this pull request Mar 15, 2022
Status of testing Providers that were prepared on March 15, 2022 #22264
Closed
41 tasks

boring-cyborgBot added area:providers provider:amazon AWS/Amazon - related issues labels Feb 10, 2022

dstandish mentioned this pull request Feb 10, 2022
Implement Async S3KeySensorastronomer/astronomer-providers#45
Merged

dstandish marked this pull request as draft February 10, 2022 19:11

dstandish added 3 commits February 10, 2022 15:35

must resolve only after template rendering
ba4f318

fixup! must resolve only after template rendering
ce061ac

economize
1e40c03

dstandish changed the title ~~S3KeySensor should parse bucket and key outside of poke~~S3KeySensor to use S3Hook url parserFeb 10, 2022

fixup! economize
923ed05

dstandish marked this pull request as ready for review February 11, 2022 06:37

josh-fell reviewed Feb 11, 2022
View reviewed changes

josh-fell approved these changes Feb 11, 2022
View reviewed changes

potiuk approved these changes Feb 15, 2022
View reviewed changes

potiuk merged commit fc44836 into apache:mainFeb 15, 2022

potiuk mentioned this pull request Mar 7, 2022
Status of testing Providers that were prepared on March 07, 2022 #22063
Closed
84 tasks

potiuk mentioned this pull request Mar 15, 2022
Status of testing Providers that were prepared on March 15, 2022 #22264
Closed
41 tasks

boring-cyborgBot added area:providers provider:amazon AWS/Amazon - related issues labels Feb 10, 2022

dstandish mentioned this pull request Feb 10, 2022
Implement Async S3KeySensorastronomer/astronomer-providers#45
Merged

dstandish marked this pull request as draft February 10, 2022 19:11

dstandish added 3 commits February 10, 2022 15:35

must resolve only after template rendering
ba4f318

fixup! must resolve only after template rendering
ce061ac

economize
1e40c03

dstandish changed the title ~~S3KeySensor should parse bucket and key outside of poke~~S3KeySensor to use S3Hook url parserFeb 10, 2022

fixup! economize
923ed05

dstandish marked this pull request as ready for review February 11, 2022 06:37

josh-fell reviewed Feb 11, 2022
View reviewed changes

josh-fell approved these changes Feb 11, 2022
View reviewed changes

potiuk approved these changes Feb 15, 2022
View reviewed changes

potiuk merged commit fc44836 into apache:mainFeb 15, 2022

potiuk mentioned this pull request Mar 7, 2022
Status of testing Providers that were prepared on March 07, 2022 #22063
Closed
84 tasks

potiuk mentioned this pull request Mar 15, 2022
Status of testing Providers that were prepared on March 15, 2022 #22264
Closed
41 tasks

	raiseAirflowException('If bucket_name provided, bucket_key must be relative path, not URI.')
	raiseAirflowException('If bucket_name is provided, bucket_key must be a relative path, not a URI.')

Uh oh!

Conversation

dstandish commented Feb 10, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

josh-fell commented Feb 10, 2022

Uh oh!

josh-fell commented Feb 10, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dstandish commented Feb 10, 2022

Uh oh!

josh-fell left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

dstandish commented Feb 10, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

josh-fell commented Feb 10, 2022

Uh oh!

josh-fell commented Feb 10, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dstandish commented Feb 10, 2022

Uh oh!

josh-fell left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

dstandish commented Feb 10, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

josh-fell commented Feb 10, 2022

Uh oh!

josh-fell commented Feb 10, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dstandish commented Feb 10, 2022

Uh oh!

josh-fell left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

dstandish commented Feb 10, 2022•
edited
Loading

josh-fell commented Feb 10, 2022•
edited
Loading

dstandish commented Feb 10, 2022•
edited
Loading

josh-fell commented Feb 10, 2022•
edited
Loading

dstandish commented Feb 10, 2022•
edited
Loading

josh-fell commented Feb 10, 2022•
edited
Loading

dstandish commented Feb 10, 2022•
edited
Loading

josh-fell commented Feb 10, 2022•
edited
Loading