Add AWS operators to create and delete RDS Database - #24099

Merged
eladkal merged 14 commits into
apache:mainfrom
eskarimov:23822-add-aws-operator-to-create-rds
Jun 28, 2022
Merged

Add AWS operators to create and delete RDS Database#24099
eladkal merged 14 commits into
apache:mainfrom
eskarimov:23822-add-aws-operator-to-create-rds

Conversation

@eskarimov

@eskarimoveskarimov commented Jun 2, 2022

Copy link
Copy Markdown
Contributor

This PR intends to close#23822 by adding new AWS operators for create and delete RDS database.

@eskarimov
eskarimov requested a review from mik-laj as a code ownerJune 2, 2022 06:59
eladkal
eladkal reviewed Jun 2, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/rds.py Outdated
@eskarimov
eskarimovforce-pushed the 23822-add-aws-operator-to-create-rds branch from c7e8c93 to 627c80bCompareJune 2, 2022 12:31
potiuk
potiuk reviewed Jun 6, 2022
View reviewed changes

@potiukpotiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please implement examples in AIP-47 compatible way? You will find a lot of AIP-47 as examples.

josh-fell
josh-fell reviewed Jun 6, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/rds.py Outdated
@eskarimov
eskarimovforce-pushed the 23822-add-aws-operator-to-create-rds branch from 627c80b to bbd3156CompareJune 8, 2022 06:49
@eskarimov

eskarimov commented Jun 8, 2022

Copy link
Copy Markdown
ContributorAuthor

Can you please implement examples in AIP-47 compatible way? You will find a lot of AIP-47 as examples.

Sure, I should have checked this epic 👍

I've just pushed a commit with according changes.

eladkal
eladkal reviewed Jun 8, 2022
View reviewed changes
Comment threadtests/providers/amazon/aws/operators/test_rds.py Outdated
Comment on lines 476 to 478

@eladkaleladkalJun 8, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we have db_instance_identifier why do we need DBName in rds_kwargs?

@eskarimoveskarimovJun 8, 2022•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case with Postgres it's different parameters - db_instance_identifier identifies the DB instance, while DBName is the name of the database to create when the DB instance is created.

We can actually omit DBName, then a database named postgres is created in the DB instance.

Comment threadairflow/providers/amazon/aws/operators/rds.py Outdated
Comment threadairflow/providers/amazon/aws/operators/rds.py Outdated
@eladkal

eladkal commented Jun 8, 2022

Copy link
Copy Markdown
Contributor

Would be nice also to fix dms example dag to use the operators added by this PRas you can see from the review in #23681 (comment)
This will help also to make sure we have everything we need covered by these operators (the feature request was created from #23681 )

vincbeck
vincbeck requested changes Jun 8, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/rds.py Outdated
Comment threadairflow/providers/amazon/aws/operators/rds.py Outdated
Comment threadtests/system/providers/amazon/aws/rds/example_rds_db_instance.py Outdated
Comment threadtests/system/providers/amazon/aws/rds/example_rds_db_instance.py Outdated
@potiuk

potiuk commented Jun 11, 2022

Copy link
Copy Markdown
Member

Comments to @vincbeck comments @eskarimov ?

@eskarimov

eskarimov commented Jun 13, 2022

Copy link
Copy Markdown
ContributorAuthor

Would be nice also to fix dms example dag to use the operators added by this PRas you can see from the review in #23681 (comment) This will help also to make sure we have everything we need covered by these operators (the feature request was created from #23681 )

I agree it'd be nice, however, maybe it'd be better to make in a separate PR together with DAG refactoring making it AIP-47 compatible, in order to keep the scope of this PR not too broad?

Comments to @vincbeck comments @eskarimov ?

Just replied, very nice feedback, I'm going to fix mentioned things in the upcoming days.

@eladkal

eladkal commented Jun 13, 2022

Copy link
Copy Markdown
Contributor

in order to keep the scope of this PR not too broad?

I think that it's not too broad.
The reason that I'm interested in changing dms example dag here as well is to see if we have everything we need covered. We might find that we are missing some parameter or we have something to discuss. It should be relatively simple.
Please try it - if it will cause difficulties then just report back and we can learn from it as well.

@eskarimov
eskarimovforce-pushed the 23822-add-aws-operator-to-create-rds branch from ee5b7a8 to a11b12dCompareJune 17, 2022 10:21
@eskarimov

eskarimov commented Jun 17, 2022

Copy link
Copy Markdown
ContributorAuthor

@eladkal could you review refactored example_dms DAG, please?
I've tried to split it into more granular tasks, i.e. instead of one set_up task having create_db_instance >> create_sample_table() -> create_dms_assets() (and according deletion tasks).

josh-fell
josh-fell reviewed Jun 17, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/example_dags/example_dms.py Outdated
Comment threadairflow/providers/amazon/aws/operators/rds.py Outdated
Comment threadairflow/providers/amazon/aws/operators/rds.py Outdated
Comment threadairflow/providers/amazon/aws/operators/rds.py Outdated
@eladkal
eladkal self-requested a review June 17, 2022 15:56
eladkal
eladkal approved these changes Jun 26, 2022
View reviewed changes

@eladkaleladkal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
@eskarimov can you please address josh comments above?

@github-actionsgithub-actionsBot added the full tests needed We need to run full set of tests for this PR to merge label Jun 26, 2022
@github-actions

github-actionsBot commented Jun 26, 2022

Copy link
Copy Markdown
Contributor

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

@eskarimov

eskarimov commented Jun 27, 2022•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

LGTM @eskarimov can you please address josh comments above?

Sure, pushing updated and rebased branch. Thank you for the reviews, @josh-fell I'll mark the suggestions above as resolved then.

@eskarimov
eskarimovforce-pushed the 23822-add-aws-operator-to-create-rds branch from a11b12d to 206e884CompareJune 27, 2022 10:48
@eladkal
eladkal merged commit bf72752 into apache:mainJun 28, 2022
@eskarimov
eskarimov deleted the 23822-add-aws-operator-to-create-rds branch June 29, 2022 06:39
@potiukpotiuk mentioned this pull request Jul 13, 2022
Closed
95 tasks
@hankehlyhankehly mentioned this pull request Aug 27, 2022
Merged
2 tasks
kaxil pushed a commit that referenced this pull request Sep 5, 2022
@hankehly@ferruzzi
Set template_fields on RDS operators (#26005)
751cb9a
Related: #24099#25952
### Summary
This PR sets the `template_fields` property on `RdsCreateDbInstanceOperator` and `RdsDeleteDbInstanceOperator` to allow users to programmatically change the database id, instance size, allocated storage, etc..
It also replaces use of `@task` decorated functions with their appropriate operator in system tests.
Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@potiukpotiukpotiuk left review comments
@josh-felljosh-felljosh-fell left review comments
@eladkaleladkaleladkal approved these changes
@vincbeckvincbeckvincbeck approved these changes
@mik-lajmik-lajAwaiting requested review from mik-laj

Assignees

No one assigned

Labels

area:providersfull tests neededWe need to run full set of tests for this PR to mergekind:documentationprovider:amazonAWS/Amazon - related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add an AWS operator for Create RDS Database

5 participants

@eskarimov@eladkal@potiuk@josh-fell@vincbeck
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Add AWS operators to create and delete RDS Database - #24099
Merged
eladkal merged 14 commits into
apache:mainfrom 
eskarimov:23822-add-aws-operator-to-create-rds
Jun 28, 2022
Merged
Add AWS operators to create and delete RDS Database#24099
eladkal merged 14 commits into
apache:mainfrom 
eskarimov:23822-add-aws-operator-to-create-rds
Conversation
@eskarimov
@eskarimoveskarimov
commented
Jun 2, 2022

Copy link


Copy Markdown

Contributor
This PR intends to close#23822 by adding new AWS operators for create and delete RDS database.
@eskarimov
eskarimov
requested a review
from mik-laj
as a code ownerJune 2, 2022 06:59
eladkal
eladkal
reviewed
Jun 2, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
c7e8c93 to
627c80bCompareJune 2, 2022 12:31
potiuk
potiuk
reviewed
Jun 6, 2022
View reviewed changes
@potiukpotiuk
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please implement examples in AIP-47 compatible way? You will find a lot of AIP-47 as examples.
josh-fell
josh-fell
reviewed
Jun 6, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
627c80b to
bbd3156CompareJune 8, 2022 06:49
@eskarimov
eskarimov
commented
Jun 8, 2022

Copy link


Copy Markdown

ContributorAuthor
Can you please implement examples in AIP-47 compatible way? You will find a lot of AIP-47 as examples.
Sure, I should have checked this epic 👍
I've just pushed a commit with according changes.
eladkal
eladkal
reviewed
Jun 8, 2022
View reviewed changes
Comment threadtests/providers/amazon/aws/operators/test_rds.py
Outdated


Comment on lines

476
 to 
478

@eladkaleladkalJun 8, 2022

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we have db_instance_identifier why do we need DBName in rds_kwargs?
@eskarimoveskarimovJun 8, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case with Postgres it's different parameters - db_instance_identifier identifies the DB instance, while DBName is the name of the database to create when the DB instance is created.
We can actually omit DBName, then a database named postgres is created in the DB instance.
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eladkal
eladkal
commented
Jun 8, 2022

Copy link


Copy Markdown

Contributor
Would be nice also to fix dms example dag to use the operators added by this PRas you can see from the review in #23681 (comment)

This will help also to make sure we have everything we need covered by these operators (the feature request was created from #23681 )
vincbeck
vincbeck
requested changes
Jun 8, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadtests/system/providers/amazon/aws/rds/example_rds_db_instance.py
Outdated

Comment threadtests/system/providers/amazon/aws/rds/example_rds_db_instance.py
Outdated

@potiuk
potiuk
commented
Jun 11, 2022

Copy link


Copy Markdown

Member
Comments to @vincbeck comments @eskarimov ?
@eskarimov
eskarimov
commented
Jun 13, 2022

Copy link


Copy Markdown

ContributorAuthor
Would be nice also to fix dms example dag to use the operators added by this PRas you can see from the review in #23681 (comment) This will help also to make sure we have everything we need covered by these operators (the feature request was created from #23681 )
I agree it'd be nice, however, maybe it'd be better to make in a separate PR together with DAG refactoring making it AIP-47 compatible, in order to keep the scope of this PR not too broad?
Comments to @vincbeck comments @eskarimov ?
Just replied, very nice feedback, I'm going to fix mentioned things in the upcoming days.
@eladkal
eladkal
commented
Jun 13, 2022

Copy link


Copy Markdown

Contributor
in order to keep the scope of this PR not too broad?
I think that it's not too broad.

The reason that I'm interested in changing dms example dag here as well is to see if we have everything we need covered. We might find that we are missing some parameter or we have something to discuss. It should be relatively simple.

Please try it - if it will cause difficulties then just report back and we can learn from it as well.
@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
ee5b7a8 to
a11b12dCompareJune 17, 2022 10:21
@eskarimov
eskarimov
commented
Jun 17, 2022

Copy link


Copy Markdown

ContributorAuthor
@eladkal could you review refactored example_dms DAG, please?

I've tried to split it into more granular tasks, i.e. instead of one set_up task having create_db_instance >> create_sample_table() -> create_dms_assets() (and according deletion tasks).
josh-fell
josh-fell
reviewed
Jun 17, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/example_dags/example_dms.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eladkal
eladkal
self-requested a review
June 17, 2022 15:56
eladkal
eladkal
approved these changes
Jun 26, 2022
View reviewed changes
@eladkaleladkal
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
@eskarimov can you please address josh comments above?
@github-actionsgithub-actionsBot
added
the 
full tests needed
We need to run full set of tests for this PR to merge
label
Jun 26, 2022
@github-actions
github-actionsBot
commented
Jun 26, 2022

Copy link


Copy Markdown

Contributor
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
@eskarimov
eskarimov
commented
Jun 27, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
LGTM @eskarimov can you please address josh comments above?
Sure, pushing updated and rebased branch. Thank you for the reviews, @josh-fell I'll mark the suggestions above as resolved then.
@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
a11b12d to
206e884CompareJune 27, 2022 10:48
@eladkal
eladkal
merged commit bf72752
into
apache:mainJun 28, 2022
@eskarimov
eskarimov
deleted the

23822-add-aws-operator-to-create-rds

branch
June 29, 2022 06:39
@potiukpotiuk
mentioned this pull request
Jul 13, 2022

Closed

95 tasks
@hankehlyhankehly
mentioned this pull request
Aug 27, 2022
 Merged

2 tasks
kaxil pushed a commit
that referenced
this pull request
Sep 5, 2022
@hankehly@ferruzzi
Set template_fields on RDS operators (#26005)
751cb9a
Related: #24099#25952
### Summary
This PR sets the `template_fields` property on `RdsCreateDbInstanceOperator` and `RdsDeleteDbInstanceOperator` to allow users to programmatically change the database id, instance size, allocated storage, etc..
It also replaces use of `@task` decorated functions with their appropriate operator in system tests.
Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@potiukpotiukpotiuk left review comments
@josh-felljosh-felljosh-fell left review comments
@eladkaleladkaleladkal approved these changes
@vincbeckvincbeckvincbeck approved these changes
@mik-lajmik-lajAwaiting requested review from mik-laj

Assignees


No one assigned


Labels

area:providersfull tests neededWe need to run full set of tests for this PR to mergekind:documentationprovider:amazonAWS/Amazon - related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Add an AWS operator for Create RDS Database


5 participants

@eskarimov@eladkal@potiuk@josh-fell@vincbeck

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add AWS operators to create and delete RDS Database - #24099
Merged
eladkal merged 14 commits into
apache:mainfrom 
eskarimov:23822-add-aws-operator-to-create-rds
Jun 28, 2022
Merged
Add AWS operators to create and delete RDS Database#24099
eladkal merged 14 commits into
apache:mainfrom 
eskarimov:23822-add-aws-operator-to-create-rds
Conversation
@eskarimov
@eskarimoveskarimov
commented
Jun 2, 2022

Copy link


Copy Markdown

Contributor
This PR intends to close#23822 by adding new AWS operators for create and delete RDS database.
@eskarimov
eskarimov
requested a review
from mik-laj
as a code ownerJune 2, 2022 06:59
eladkal
eladkal
reviewed
Jun 2, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
c7e8c93 to
627c80bCompareJune 2, 2022 12:31
potiuk
potiuk
reviewed
Jun 6, 2022
View reviewed changes
@potiukpotiuk
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please implement examples in AIP-47 compatible way? You will find a lot of AIP-47 as examples.
josh-fell
josh-fell
reviewed
Jun 6, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
627c80b to
bbd3156CompareJune 8, 2022 06:49
@eskarimov
eskarimov
commented
Jun 8, 2022

Copy link


Copy Markdown

ContributorAuthor
Can you please implement examples in AIP-47 compatible way? You will find a lot of AIP-47 as examples.
Sure, I should have checked this epic 👍
I've just pushed a commit with according changes.
eladkal
eladkal
reviewed
Jun 8, 2022
View reviewed changes
Comment threadtests/providers/amazon/aws/operators/test_rds.py
Outdated


Comment on lines

476
 to 
478

@eladkaleladkalJun 8, 2022

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we have db_instance_identifier why do we need DBName in rds_kwargs?
@eskarimoveskarimovJun 8, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case with Postgres it's different parameters - db_instance_identifier identifies the DB instance, while DBName is the name of the database to create when the DB instance is created.
We can actually omit DBName, then a database named postgres is created in the DB instance.
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eladkal
eladkal
commented
Jun 8, 2022

Copy link


Copy Markdown

Contributor
Would be nice also to fix dms example dag to use the operators added by this PRas you can see from the review in #23681 (comment)

This will help also to make sure we have everything we need covered by these operators (the feature request was created from #23681 )
vincbeck
vincbeck
requested changes
Jun 8, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadtests/system/providers/amazon/aws/rds/example_rds_db_instance.py
Outdated

Comment threadtests/system/providers/amazon/aws/rds/example_rds_db_instance.py
Outdated

@potiuk
potiuk
commented
Jun 11, 2022

Copy link


Copy Markdown

Member
Comments to @vincbeck comments @eskarimov ?
@eskarimov
eskarimov
commented
Jun 13, 2022

Copy link


Copy Markdown

ContributorAuthor
Would be nice also to fix dms example dag to use the operators added by this PRas you can see from the review in #23681 (comment) This will help also to make sure we have everything we need covered by these operators (the feature request was created from #23681 )
I agree it'd be nice, however, maybe it'd be better to make in a separate PR together with DAG refactoring making it AIP-47 compatible, in order to keep the scope of this PR not too broad?
Comments to @vincbeck comments @eskarimov ?
Just replied, very nice feedback, I'm going to fix mentioned things in the upcoming days.
@eladkal
eladkal
commented
Jun 13, 2022

Copy link


Copy Markdown

Contributor
in order to keep the scope of this PR not too broad?
I think that it's not too broad.

The reason that I'm interested in changing dms example dag here as well is to see if we have everything we need covered. We might find that we are missing some parameter or we have something to discuss. It should be relatively simple.

Please try it - if it will cause difficulties then just report back and we can learn from it as well.
@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
ee5b7a8 to
a11b12dCompareJune 17, 2022 10:21
@eskarimov
eskarimov
commented
Jun 17, 2022

Copy link


Copy Markdown

ContributorAuthor
@eladkal could you review refactored example_dms DAG, please?

I've tried to split it into more granular tasks, i.e. instead of one set_up task having create_db_instance >> create_sample_table() -> create_dms_assets() (and according deletion tasks).
josh-fell
josh-fell
reviewed
Jun 17, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/example_dags/example_dms.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eladkal
eladkal
self-requested a review
June 17, 2022 15:56
eladkal
eladkal
approved these changes
Jun 26, 2022
View reviewed changes
@eladkaleladkal
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
@eskarimov can you please address josh comments above?
@github-actionsgithub-actionsBot
added
the 
full tests needed
We need to run full set of tests for this PR to merge
label
Jun 26, 2022
@github-actions
github-actionsBot
commented
Jun 26, 2022

Copy link


Copy Markdown

Contributor
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
@eskarimov
eskarimov
commented
Jun 27, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
LGTM @eskarimov can you please address josh comments above?
Sure, pushing updated and rebased branch. Thank you for the reviews, @josh-fell I'll mark the suggestions above as resolved then.
@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
a11b12d to
206e884CompareJune 27, 2022 10:48
@eladkal
eladkal
merged commit bf72752
into
apache:mainJun 28, 2022
@eskarimov
eskarimov
deleted the

23822-add-aws-operator-to-create-rds

branch
June 29, 2022 06:39
@potiukpotiuk
mentioned this pull request
Jul 13, 2022

Closed

95 tasks
@hankehlyhankehly
mentioned this pull request
Aug 27, 2022
 Merged

2 tasks
kaxil pushed a commit
that referenced
this pull request
Sep 5, 2022
@hankehly@ferruzzi
Set template_fields on RDS operators (#26005)
751cb9a
Related: #24099#25952
### Summary
This PR sets the `template_fields` property on `RdsCreateDbInstanceOperator` and `RdsDeleteDbInstanceOperator` to allow users to programmatically change the database id, instance size, allocated storage, etc..
It also replaces use of `@task` decorated functions with their appropriate operator in system tests.
Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@potiukpotiukpotiuk left review comments
@josh-felljosh-felljosh-fell left review comments
@eladkaleladkaleladkal approved these changes
@vincbeckvincbeckvincbeck approved these changes
@mik-lajmik-lajAwaiting requested review from mik-laj

Assignees


No one assigned


Labels

area:providersfull tests neededWe need to run full set of tests for this PR to mergekind:documentationprovider:amazonAWS/Amazon - related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Add an AWS operator for Create RDS Database


5 participants

@eskarimov@eladkal@potiuk@josh-fell@vincbeck

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add AWS operators to create and delete RDS Database - #24099
Merged
eladkal merged 14 commits into
apache:mainfrom 
eskarimov:23822-add-aws-operator-to-create-rds
Jun 28, 2022
Merged
Add AWS operators to create and delete RDS Database#24099
eladkal merged 14 commits into
apache:mainfrom 
eskarimov:23822-add-aws-operator-to-create-rds
Conversation
@eskarimov
@eskarimoveskarimov
commented
Jun 2, 2022

Copy link


Copy Markdown

Contributor
This PR intends to close#23822 by adding new AWS operators for create and delete RDS database.
@eskarimov
eskarimov
requested a review
from mik-laj
as a code ownerJune 2, 2022 06:59
eladkal
eladkal
reviewed
Jun 2, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
c7e8c93 to
627c80bCompareJune 2, 2022 12:31
potiuk
potiuk
reviewed
Jun 6, 2022
View reviewed changes
@potiukpotiuk
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please implement examples in AIP-47 compatible way? You will find a lot of AIP-47 as examples.
josh-fell
josh-fell
reviewed
Jun 6, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
627c80b to
bbd3156CompareJune 8, 2022 06:49
@eskarimov
eskarimov
commented
Jun 8, 2022

Copy link


Copy Markdown

ContributorAuthor
Can you please implement examples in AIP-47 compatible way? You will find a lot of AIP-47 as examples.
Sure, I should have checked this epic 👍
I've just pushed a commit with according changes.
eladkal
eladkal
reviewed
Jun 8, 2022
View reviewed changes
Comment threadtests/providers/amazon/aws/operators/test_rds.py
Outdated


Comment on lines

476
 to 
478

@eladkaleladkalJun 8, 2022

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we have db_instance_identifier why do we need DBName in rds_kwargs?
@eskarimoveskarimovJun 8, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case with Postgres it's different parameters - db_instance_identifier identifies the DB instance, while DBName is the name of the database to create when the DB instance is created.
We can actually omit DBName, then a database named postgres is created in the DB instance.
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eladkal
eladkal
commented
Jun 8, 2022

Copy link


Copy Markdown

Contributor
Would be nice also to fix dms example dag to use the operators added by this PRas you can see from the review in #23681 (comment)

This will help also to make sure we have everything we need covered by these operators (the feature request was created from #23681 )
vincbeck
vincbeck
requested changes
Jun 8, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadtests/system/providers/amazon/aws/rds/example_rds_db_instance.py
Outdated

Comment threadtests/system/providers/amazon/aws/rds/example_rds_db_instance.py
Outdated

@potiuk
potiuk
commented
Jun 11, 2022

Copy link


Copy Markdown

Member
Comments to @vincbeck comments @eskarimov ?
@eskarimov
eskarimov
commented
Jun 13, 2022

Copy link


Copy Markdown

ContributorAuthor
Would be nice also to fix dms example dag to use the operators added by this PRas you can see from the review in #23681 (comment) This will help also to make sure we have everything we need covered by these operators (the feature request was created from #23681 )
I agree it'd be nice, however, maybe it'd be better to make in a separate PR together with DAG refactoring making it AIP-47 compatible, in order to keep the scope of this PR not too broad?
Comments to @vincbeck comments @eskarimov ?
Just replied, very nice feedback, I'm going to fix mentioned things in the upcoming days.
@eladkal
eladkal
commented
Jun 13, 2022

Copy link


Copy Markdown

Contributor
in order to keep the scope of this PR not too broad?
I think that it's not too broad.

The reason that I'm interested in changing dms example dag here as well is to see if we have everything we need covered. We might find that we are missing some parameter or we have something to discuss. It should be relatively simple.

Please try it - if it will cause difficulties then just report back and we can learn from it as well.
@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
ee5b7a8 to
a11b12dCompareJune 17, 2022 10:21
@eskarimov
eskarimov
commented
Jun 17, 2022

Copy link


Copy Markdown

ContributorAuthor
@eladkal could you review refactored example_dms DAG, please?

I've tried to split it into more granular tasks, i.e. instead of one set_up task having create_db_instance >> create_sample_table() -> create_dms_assets() (and according deletion tasks).
josh-fell
josh-fell
reviewed
Jun 17, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/example_dags/example_dms.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eladkal
eladkal
self-requested a review
June 17, 2022 15:56
eladkal
eladkal
approved these changes
Jun 26, 2022
View reviewed changes
@eladkaleladkal
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
@eskarimov can you please address josh comments above?
@github-actionsgithub-actionsBot
added
the 
full tests needed
We need to run full set of tests for this PR to merge
label
Jun 26, 2022
@github-actions
github-actionsBot
commented
Jun 26, 2022

Copy link


Copy Markdown

Contributor
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
@eskarimov
eskarimov
commented
Jun 27, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
LGTM @eskarimov can you please address josh comments above?
Sure, pushing updated and rebased branch. Thank you for the reviews, @josh-fell I'll mark the suggestions above as resolved then.
@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
a11b12d to
206e884CompareJune 27, 2022 10:48
@eladkal
eladkal
merged commit bf72752
into
apache:mainJun 28, 2022
@eskarimov
eskarimov
deleted the

23822-add-aws-operator-to-create-rds

branch
June 29, 2022 06:39
@potiukpotiuk
mentioned this pull request
Jul 13, 2022

Closed

95 tasks
@hankehlyhankehly
mentioned this pull request
Aug 27, 2022
 Merged

2 tasks
kaxil pushed a commit
that referenced
this pull request
Sep 5, 2022
@hankehly@ferruzzi
Set template_fields on RDS operators (#26005)
751cb9a
Related: #24099#25952
### Summary
This PR sets the `template_fields` property on `RdsCreateDbInstanceOperator` and `RdsDeleteDbInstanceOperator` to allow users to programmatically change the database id, instance size, allocated storage, etc..
It also replaces use of `@task` decorated functions with their appropriate operator in system tests.
Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@potiukpotiukpotiuk left review comments
@josh-felljosh-felljosh-fell left review comments
@eladkaleladkaleladkal approved these changes
@vincbeckvincbeckvincbeck approved these changes
@mik-lajmik-lajAwaiting requested review from mik-laj

Assignees


No one assigned


Labels

area:providersfull tests neededWe need to run full set of tests for this PR to mergekind:documentationprovider:amazonAWS/Amazon - related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Add an AWS operator for Create RDS Database


5 participants

@eskarimov@eladkal@potiuk@josh-fell@vincbeck

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Add AWS operators to create and delete RDS Database - #24099
Merged
eladkal merged 14 commits into
apache:mainfrom 
eskarimov:23822-add-aws-operator-to-create-rds
Jun 28, 2022
Merged
Add AWS operators to create and delete RDS Database#24099
eladkal merged 14 commits into
apache:mainfrom 
eskarimov:23822-add-aws-operator-to-create-rds
Conversation
@eskarimov
@eskarimoveskarimov
commented
Jun 2, 2022

Copy link


Copy Markdown

Contributor
This PR intends to close#23822 by adding new AWS operators for create and delete RDS database.
@eskarimov
eskarimov
requested a review
from mik-laj
as a code ownerJune 2, 2022 06:59
eladkal
eladkal
reviewed
Jun 2, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
c7e8c93 to
627c80bCompareJune 2, 2022 12:31
potiuk
potiuk
reviewed
Jun 6, 2022
View reviewed changes
@potiukpotiuk
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please implement examples in AIP-47 compatible way? You will find a lot of AIP-47 as examples.
josh-fell
josh-fell
reviewed
Jun 6, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
627c80b to
bbd3156CompareJune 8, 2022 06:49
@eskarimov
eskarimov
commented
Jun 8, 2022

Copy link


Copy Markdown

ContributorAuthor
Can you please implement examples in AIP-47 compatible way? You will find a lot of AIP-47 as examples.
Sure, I should have checked this epic 👍
I've just pushed a commit with according changes.
eladkal
eladkal
reviewed
Jun 8, 2022
View reviewed changes
Comment threadtests/providers/amazon/aws/operators/test_rds.py
Outdated


Comment on lines

476
 to 
478

@eladkaleladkalJun 8, 2022

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we have db_instance_identifier why do we need DBName in rds_kwargs?
@eskarimoveskarimovJun 8, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case with Postgres it's different parameters - db_instance_identifier identifies the DB instance, while DBName is the name of the database to create when the DB instance is created.
We can actually omit DBName, then a database named postgres is created in the DB instance.
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eladkal
eladkal
commented
Jun 8, 2022

Copy link


Copy Markdown

Contributor
Would be nice also to fix dms example dag to use the operators added by this PRas you can see from the review in #23681 (comment)

This will help also to make sure we have everything we need covered by these operators (the feature request was created from #23681 )
vincbeck
vincbeck
requested changes
Jun 8, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadtests/system/providers/amazon/aws/rds/example_rds_db_instance.py
Outdated

Comment threadtests/system/providers/amazon/aws/rds/example_rds_db_instance.py
Outdated

@potiuk
potiuk
commented
Jun 11, 2022

Copy link


Copy Markdown

Member
Comments to @vincbeck comments @eskarimov ?
@eskarimov
eskarimov
commented
Jun 13, 2022

Copy link


Copy Markdown

ContributorAuthor
Would be nice also to fix dms example dag to use the operators added by this PRas you can see from the review in #23681 (comment) This will help also to make sure we have everything we need covered by these operators (the feature request was created from #23681 )
I agree it'd be nice, however, maybe it'd be better to make in a separate PR together with DAG refactoring making it AIP-47 compatible, in order to keep the scope of this PR not too broad?
Comments to @vincbeck comments @eskarimov ?
Just replied, very nice feedback, I'm going to fix mentioned things in the upcoming days.
@eladkal
eladkal
commented
Jun 13, 2022

Copy link


Copy Markdown

Contributor
in order to keep the scope of this PR not too broad?
I think that it's not too broad.

The reason that I'm interested in changing dms example dag here as well is to see if we have everything we need covered. We might find that we are missing some parameter or we have something to discuss. It should be relatively simple.

Please try it - if it will cause difficulties then just report back and we can learn from it as well.
@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
ee5b7a8 to
a11b12dCompareJune 17, 2022 10:21
@eskarimov
eskarimov
commented
Jun 17, 2022

Copy link


Copy Markdown

ContributorAuthor
@eladkal could you review refactored example_dms DAG, please?

I've tried to split it into more granular tasks, i.e. instead of one set_up task having create_db_instance >> create_sample_table() -> create_dms_assets() (and according deletion tasks).
josh-fell
josh-fell
reviewed
Jun 17, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/example_dags/example_dms.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eladkal
eladkal
self-requested a review
June 17, 2022 15:56
eladkal
eladkal
approved these changes
Jun 26, 2022
View reviewed changes
@eladkaleladkal
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
@eskarimov can you please address josh comments above?
@github-actionsgithub-actionsBot
added
the 
full tests needed
We need to run full set of tests for this PR to merge
label
Jun 26, 2022
@github-actions
github-actionsBot
commented
Jun 26, 2022

Copy link


Copy Markdown

Contributor
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
@eskarimov
eskarimov
commented
Jun 27, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
LGTM @eskarimov can you please address josh comments above?
Sure, pushing updated and rebased branch. Thank you for the reviews, @josh-fell I'll mark the suggestions above as resolved then.
@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
a11b12d to
206e884CompareJune 27, 2022 10:48
@eladkal
eladkal
merged commit bf72752
into
apache:mainJun 28, 2022
@eskarimov
eskarimov
deleted the

23822-add-aws-operator-to-create-rds

branch
June 29, 2022 06:39
@potiukpotiuk
mentioned this pull request
Jul 13, 2022

Closed

95 tasks
@hankehlyhankehly
mentioned this pull request
Aug 27, 2022
 Merged

2 tasks
kaxil pushed a commit
that referenced
this pull request
Sep 5, 2022
@hankehly@ferruzzi
Set template_fields on RDS operators (#26005)
751cb9a
Related: #24099#25952
### Summary
This PR sets the `template_fields` property on `RdsCreateDbInstanceOperator` and `RdsDeleteDbInstanceOperator` to allow users to programmatically change the database id, instance size, allocated storage, etc..
It also replaces use of `@task` decorated functions with their appropriate operator in system tests.
Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@potiukpotiukpotiuk left review comments
@josh-felljosh-felljosh-fell left review comments
@eladkaleladkaleladkal approved these changes
@vincbeckvincbeckvincbeck approved these changes
@mik-lajmik-lajAwaiting requested review from mik-laj

Assignees


No one assigned


Labels

area:providersfull tests neededWe need to run full set of tests for this PR to mergekind:documentationprovider:amazonAWS/Amazon - related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Add an AWS operator for Create RDS Database


5 participants

@eskarimov@eladkal@potiuk@josh-fell@vincbeck

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add AWS operators to create and delete RDS Database - #24099
Merged
eladkal merged 14 commits into
apache:mainfrom 
eskarimov:23822-add-aws-operator-to-create-rds
Jun 28, 2022
Merged
Add AWS operators to create and delete RDS Database#24099
eladkal merged 14 commits into
apache:mainfrom 
eskarimov:23822-add-aws-operator-to-create-rds
Conversation
@eskarimov
@eskarimoveskarimov
commented
Jun 2, 2022

Copy link


Copy Markdown

Contributor
This PR intends to close#23822 by adding new AWS operators for create and delete RDS database.
@eskarimov
eskarimov
requested a review
from mik-laj
as a code ownerJune 2, 2022 06:59
eladkal
eladkal
reviewed
Jun 2, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
c7e8c93 to
627c80bCompareJune 2, 2022 12:31
potiuk
potiuk
reviewed
Jun 6, 2022
View reviewed changes
@potiukpotiuk
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please implement examples in AIP-47 compatible way? You will find a lot of AIP-47 as examples.
josh-fell
josh-fell
reviewed
Jun 6, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
627c80b to
bbd3156CompareJune 8, 2022 06:49
@eskarimov
eskarimov
commented
Jun 8, 2022

Copy link


Copy Markdown

ContributorAuthor
Can you please implement examples in AIP-47 compatible way? You will find a lot of AIP-47 as examples.
Sure, I should have checked this epic 👍
I've just pushed a commit with according changes.
eladkal
eladkal
reviewed
Jun 8, 2022
View reviewed changes
Comment threadtests/providers/amazon/aws/operators/test_rds.py
Outdated


Comment on lines

476
 to 
478

@eladkaleladkalJun 8, 2022

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we have db_instance_identifier why do we need DBName in rds_kwargs?
@eskarimoveskarimovJun 8, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case with Postgres it's different parameters - db_instance_identifier identifies the DB instance, while DBName is the name of the database to create when the DB instance is created.
We can actually omit DBName, then a database named postgres is created in the DB instance.
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eladkal
eladkal
commented
Jun 8, 2022

Copy link


Copy Markdown

Contributor
Would be nice also to fix dms example dag to use the operators added by this PRas you can see from the review in #23681 (comment)

This will help also to make sure we have everything we need covered by these operators (the feature request was created from #23681 )
vincbeck
vincbeck
requested changes
Jun 8, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadtests/system/providers/amazon/aws/rds/example_rds_db_instance.py
Outdated

Comment threadtests/system/providers/amazon/aws/rds/example_rds_db_instance.py
Outdated

@potiuk
potiuk
commented
Jun 11, 2022

Copy link


Copy Markdown

Member
Comments to @vincbeck comments @eskarimov ?
@eskarimov
eskarimov
commented
Jun 13, 2022

Copy link


Copy Markdown

ContributorAuthor
Would be nice also to fix dms example dag to use the operators added by this PRas you can see from the review in #23681 (comment) This will help also to make sure we have everything we need covered by these operators (the feature request was created from #23681 )
I agree it'd be nice, however, maybe it'd be better to make in a separate PR together with DAG refactoring making it AIP-47 compatible, in order to keep the scope of this PR not too broad?
Comments to @vincbeck comments @eskarimov ?
Just replied, very nice feedback, I'm going to fix mentioned things in the upcoming days.
@eladkal
eladkal
commented
Jun 13, 2022

Copy link


Copy Markdown

Contributor
in order to keep the scope of this PR not too broad?
I think that it's not too broad.

The reason that I'm interested in changing dms example dag here as well is to see if we have everything we need covered. We might find that we are missing some parameter or we have something to discuss. It should be relatively simple.

Please try it - if it will cause difficulties then just report back and we can learn from it as well.
@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
ee5b7a8 to
a11b12dCompareJune 17, 2022 10:21
@eskarimov
eskarimov
commented
Jun 17, 2022

Copy link


Copy Markdown

ContributorAuthor
@eladkal could you review refactored example_dms DAG, please?

I've tried to split it into more granular tasks, i.e. instead of one set_up task having create_db_instance >> create_sample_table() -> create_dms_assets() (and according deletion tasks).
josh-fell
josh-fell
reviewed
Jun 17, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/example_dags/example_dms.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eladkal
eladkal
self-requested a review
June 17, 2022 15:56
eladkal
eladkal
approved these changes
Jun 26, 2022
View reviewed changes
@eladkaleladkal
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
@eskarimov can you please address josh comments above?
@github-actionsgithub-actionsBot
added
the 
full tests needed
We need to run full set of tests for this PR to merge
label
Jun 26, 2022
@github-actions
github-actionsBot
commented
Jun 26, 2022

Copy link


Copy Markdown

Contributor
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
@eskarimov
eskarimov
commented
Jun 27, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
LGTM @eskarimov can you please address josh comments above?
Sure, pushing updated and rebased branch. Thank you for the reviews, @josh-fell I'll mark the suggestions above as resolved then.
@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
a11b12d to
206e884CompareJune 27, 2022 10:48
@eladkal
eladkal
merged commit bf72752
into
apache:mainJun 28, 2022
@eskarimov
eskarimov
deleted the

23822-add-aws-operator-to-create-rds

branch
June 29, 2022 06:39
@potiukpotiuk
mentioned this pull request
Jul 13, 2022

Closed

95 tasks
@hankehlyhankehly
mentioned this pull request
Aug 27, 2022
 Merged

2 tasks
kaxil pushed a commit
that referenced
this pull request
Sep 5, 2022
@hankehly@ferruzzi
Set template_fields on RDS operators (#26005)
751cb9a
Related: #24099#25952
### Summary
This PR sets the `template_fields` property on `RdsCreateDbInstanceOperator` and `RdsDeleteDbInstanceOperator` to allow users to programmatically change the database id, instance size, allocated storage, etc..
It also replaces use of `@task` decorated functions with their appropriate operator in system tests.
Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@potiukpotiukpotiuk left review comments
@josh-felljosh-felljosh-fell left review comments
@eladkaleladkaleladkal approved these changes
@vincbeckvincbeckvincbeck approved these changes
@mik-lajmik-lajAwaiting requested review from mik-laj

Assignees


No one assigned


Labels

area:providersfull tests neededWe need to run full set of tests for this PR to mergekind:documentationprovider:amazonAWS/Amazon - related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Add an AWS operator for Create RDS Database


5 participants

@eskarimov@eladkal@potiuk@josh-fell@vincbeck

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add AWS operators to create and delete RDS Database - #24099
Merged
eladkal merged 14 commits into
apache:mainfrom 
eskarimov:23822-add-aws-operator-to-create-rds
Jun 28, 2022
Merged
Add AWS operators to create and delete RDS Database#24099
eladkal merged 14 commits into
apache:mainfrom 
eskarimov:23822-add-aws-operator-to-create-rds
Conversation
@eskarimov
@eskarimoveskarimov
commented
Jun 2, 2022

Copy link


Copy Markdown

Contributor
This PR intends to close#23822 by adding new AWS operators for create and delete RDS database.
@eskarimov
eskarimov
requested a review
from mik-laj
as a code ownerJune 2, 2022 06:59
eladkal
eladkal
reviewed
Jun 2, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
c7e8c93 to
627c80bCompareJune 2, 2022 12:31
potiuk
potiuk
reviewed
Jun 6, 2022
View reviewed changes
@potiukpotiuk
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please implement examples in AIP-47 compatible way? You will find a lot of AIP-47 as examples.
josh-fell
josh-fell
reviewed
Jun 6, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
627c80b to
bbd3156CompareJune 8, 2022 06:49
@eskarimov
eskarimov
commented
Jun 8, 2022

Copy link


Copy Markdown

ContributorAuthor
Can you please implement examples in AIP-47 compatible way? You will find a lot of AIP-47 as examples.
Sure, I should have checked this epic 👍
I've just pushed a commit with according changes.
eladkal
eladkal
reviewed
Jun 8, 2022
View reviewed changes
Comment threadtests/providers/amazon/aws/operators/test_rds.py
Outdated


Comment on lines

476
 to 
478

@eladkaleladkalJun 8, 2022

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we have db_instance_identifier why do we need DBName in rds_kwargs?
@eskarimoveskarimovJun 8, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case with Postgres it's different parameters - db_instance_identifier identifies the DB instance, while DBName is the name of the database to create when the DB instance is created.
We can actually omit DBName, then a database named postgres is created in the DB instance.
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eladkal
eladkal
commented
Jun 8, 2022

Copy link


Copy Markdown

Contributor
Would be nice also to fix dms example dag to use the operators added by this PRas you can see from the review in #23681 (comment)

This will help also to make sure we have everything we need covered by these operators (the feature request was created from #23681 )
vincbeck
vincbeck
requested changes
Jun 8, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadtests/system/providers/amazon/aws/rds/example_rds_db_instance.py
Outdated

Comment threadtests/system/providers/amazon/aws/rds/example_rds_db_instance.py
Outdated

@potiuk
potiuk
commented
Jun 11, 2022

Copy link


Copy Markdown

Member
Comments to @vincbeck comments @eskarimov ?
@eskarimov
eskarimov
commented
Jun 13, 2022

Copy link


Copy Markdown

ContributorAuthor
Would be nice also to fix dms example dag to use the operators added by this PRas you can see from the review in #23681 (comment) This will help also to make sure we have everything we need covered by these operators (the feature request was created from #23681 )
I agree it'd be nice, however, maybe it'd be better to make in a separate PR together with DAG refactoring making it AIP-47 compatible, in order to keep the scope of this PR not too broad?
Comments to @vincbeck comments @eskarimov ?
Just replied, very nice feedback, I'm going to fix mentioned things in the upcoming days.
@eladkal
eladkal
commented
Jun 13, 2022

Copy link


Copy Markdown

Contributor
in order to keep the scope of this PR not too broad?
I think that it's not too broad.

The reason that I'm interested in changing dms example dag here as well is to see if we have everything we need covered. We might find that we are missing some parameter or we have something to discuss. It should be relatively simple.

Please try it - if it will cause difficulties then just report back and we can learn from it as well.
@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
ee5b7a8 to
a11b12dCompareJune 17, 2022 10:21
@eskarimov
eskarimov
commented
Jun 17, 2022

Copy link


Copy Markdown

ContributorAuthor
@eladkal could you review refactored example_dms DAG, please?

I've tried to split it into more granular tasks, i.e. instead of one set_up task having create_db_instance >> create_sample_table() -> create_dms_assets() (and according deletion tasks).
josh-fell
josh-fell
reviewed
Jun 17, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/example_dags/example_dms.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eladkal
eladkal
self-requested a review
June 17, 2022 15:56
eladkal
eladkal
approved these changes
Jun 26, 2022
View reviewed changes
@eladkaleladkal
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
@eskarimov can you please address josh comments above?
@github-actionsgithub-actionsBot
added
the 
full tests needed
We need to run full set of tests for this PR to merge
label
Jun 26, 2022
@github-actions
github-actionsBot
commented
Jun 26, 2022

Copy link


Copy Markdown

Contributor
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
@eskarimov
eskarimov
commented
Jun 27, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
LGTM @eskarimov can you please address josh comments above?
Sure, pushing updated and rebased branch. Thank you for the reviews, @josh-fell I'll mark the suggestions above as resolved then.
@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
a11b12d to
206e884CompareJune 27, 2022 10:48
@eladkal
eladkal
merged commit bf72752
into
apache:mainJun 28, 2022
@eskarimov
eskarimov
deleted the

23822-add-aws-operator-to-create-rds

branch
June 29, 2022 06:39
@potiukpotiuk
mentioned this pull request
Jul 13, 2022

Closed

95 tasks
@hankehlyhankehly
mentioned this pull request
Aug 27, 2022
 Merged

2 tasks
kaxil pushed a commit
that referenced
this pull request
Sep 5, 2022
@hankehly@ferruzzi
Set template_fields on RDS operators (#26005)
751cb9a
Related: #24099#25952
### Summary
This PR sets the `template_fields` property on `RdsCreateDbInstanceOperator` and `RdsDeleteDbInstanceOperator` to allow users to programmatically change the database id, instance size, allocated storage, etc..
It also replaces use of `@task` decorated functions with their appropriate operator in system tests.
Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@potiukpotiukpotiuk left review comments
@josh-felljosh-felljosh-fell left review comments
@eladkaleladkaleladkal approved these changes
@vincbeckvincbeckvincbeck approved these changes
@mik-lajmik-lajAwaiting requested review from mik-laj

Assignees


No one assigned


Labels

area:providersfull tests neededWe need to run full set of tests for this PR to mergekind:documentationprovider:amazonAWS/Amazon - related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Add an AWS operator for Create RDS Database


5 participants

@eskarimov@eladkal@potiuk@josh-fell@vincbeck

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Add AWS operators to create and delete RDS Database - #24099
Merged
eladkal merged 14 commits into
apache:mainfrom 
eskarimov:23822-add-aws-operator-to-create-rds
Jun 28, 2022
Merged
Add AWS operators to create and delete RDS Database#24099
eladkal merged 14 commits into
apache:mainfrom 
eskarimov:23822-add-aws-operator-to-create-rds
Conversation
@eskarimov
@eskarimoveskarimov
commented
Jun 2, 2022

Copy link


Copy Markdown

Contributor
This PR intends to close#23822 by adding new AWS operators for create and delete RDS database.
@eskarimov
eskarimov
requested a review
from mik-laj
as a code ownerJune 2, 2022 06:59
eladkal
eladkal
reviewed
Jun 2, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
c7e8c93 to
627c80bCompareJune 2, 2022 12:31
potiuk
potiuk
reviewed
Jun 6, 2022
View reviewed changes
@potiukpotiuk
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please implement examples in AIP-47 compatible way? You will find a lot of AIP-47 as examples.
josh-fell
josh-fell
reviewed
Jun 6, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
627c80b to
bbd3156CompareJune 8, 2022 06:49
@eskarimov
eskarimov
commented
Jun 8, 2022

Copy link


Copy Markdown

ContributorAuthor
Can you please implement examples in AIP-47 compatible way? You will find a lot of AIP-47 as examples.
Sure, I should have checked this epic 👍
I've just pushed a commit with according changes.
eladkal
eladkal
reviewed
Jun 8, 2022
View reviewed changes
Comment threadtests/providers/amazon/aws/operators/test_rds.py
Outdated


Comment on lines

476
 to 
478

@eladkaleladkalJun 8, 2022

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we have db_instance_identifier why do we need DBName in rds_kwargs?
@eskarimoveskarimovJun 8, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In case with Postgres it's different parameters - db_instance_identifier identifies the DB instance, while DBName is the name of the database to create when the DB instance is created.
We can actually omit DBName, then a database named postgres is created in the DB instance.
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eladkal
eladkal
commented
Jun 8, 2022

Copy link


Copy Markdown

Contributor
Would be nice also to fix dms example dag to use the operators added by this PRas you can see from the review in #23681 (comment)

This will help also to make sure we have everything we need covered by these operators (the feature request was created from #23681 )
vincbeck
vincbeck
requested changes
Jun 8, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadtests/system/providers/amazon/aws/rds/example_rds_db_instance.py
Outdated

Comment threadtests/system/providers/amazon/aws/rds/example_rds_db_instance.py
Outdated

@potiuk
potiuk
commented
Jun 11, 2022

Copy link


Copy Markdown

Member
Comments to @vincbeck comments @eskarimov ?
@eskarimov
eskarimov
commented
Jun 13, 2022

Copy link


Copy Markdown

ContributorAuthor
Would be nice also to fix dms example dag to use the operators added by this PRas you can see from the review in #23681 (comment) This will help also to make sure we have everything we need covered by these operators (the feature request was created from #23681 )
I agree it'd be nice, however, maybe it'd be better to make in a separate PR together with DAG refactoring making it AIP-47 compatible, in order to keep the scope of this PR not too broad?
Comments to @vincbeck comments @eskarimov ?
Just replied, very nice feedback, I'm going to fix mentioned things in the upcoming days.
@eladkal
eladkal
commented
Jun 13, 2022

Copy link


Copy Markdown

Contributor
in order to keep the scope of this PR not too broad?
I think that it's not too broad.

The reason that I'm interested in changing dms example dag here as well is to see if we have everything we need covered. We might find that we are missing some parameter or we have something to discuss. It should be relatively simple.

Please try it - if it will cause difficulties then just report back and we can learn from it as well.
@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
ee5b7a8 to
a11b12dCompareJune 17, 2022 10:21
@eskarimov
eskarimov
commented
Jun 17, 2022

Copy link


Copy Markdown

ContributorAuthor
@eladkal could you review refactored example_dms DAG, please?

I've tried to split it into more granular tasks, i.e. instead of one set_up task having create_db_instance >> create_sample_table() -> create_dms_assets() (and according deletion tasks).
josh-fell
josh-fell
reviewed
Jun 17, 2022
View reviewed changes
Comment threadairflow/providers/amazon/aws/example_dags/example_dms.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

Comment threadairflow/providers/amazon/aws/operators/rds.py
Outdated

@eladkal
eladkal
self-requested a review
June 17, 2022 15:56
eladkal
eladkal
approved these changes
Jun 26, 2022
View reviewed changes
@eladkaleladkal
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
@eskarimov can you please address josh comments above?
@github-actionsgithub-actionsBot
added
the 
full tests needed
We need to run full set of tests for this PR to merge
label
Jun 26, 2022
@github-actions
github-actionsBot
commented
Jun 26, 2022

Copy link


Copy Markdown

Contributor
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
@eskarimov
eskarimov
commented
Jun 27, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
LGTM @eskarimov can you please address josh comments above?
Sure, pushing updated and rebased branch. Thank you for the reviews, @josh-fell I'll mark the suggestions above as resolved then.
@eskarimov
eskarimovforce-pushed
the
23822-add-aws-operator-to-create-rds
branch
from
a11b12d to
206e884CompareJune 27, 2022 10:48
@eladkal
eladkal
merged commit bf72752
into
apache:mainJun 28, 2022
@eskarimov
eskarimov
deleted the

23822-add-aws-operator-to-create-rds

branch
June 29, 2022 06:39
@potiukpotiuk
mentioned this pull request
Jul 13, 2022

Closed

95 tasks
@hankehlyhankehly
mentioned this pull request
Aug 27, 2022
 Merged

2 tasks
kaxil pushed a commit
that referenced
this pull request
Sep 5, 2022
@hankehly@ferruzzi
Set template_fields on RDS operators (#26005)
751cb9a
Related: #24099#25952
### Summary
This PR sets the `template_fields` property on `RdsCreateDbInstanceOperator` and `RdsDeleteDbInstanceOperator` to allow users to programmatically change the database id, instance size, allocated storage, etc..
It also replaces use of `@task` decorated functions with their appropriate operator in system tests.
Co-authored-by: D. Ferruzzi <ferruzzi@amazon.com>
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@potiukpotiukpotiuk left review comments
@josh-felljosh-felljosh-fell left review comments
@eladkaleladkaleladkal approved these changes
@vincbeckvincbeckvincbeck approved these changes
@mik-lajmik-lajAwaiting requested review from mik-laj

Assignees


No one assigned


Labels

area:providersfull tests neededWe need to run full set of tests for this PR to mergekind:documentationprovider:amazonAWS/Amazon - related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Add an AWS operator for Create RDS Database


5 participants

@eskarimov@eladkal@potiuk@josh-fell@vincbeck