Upgrade moto library to version 3.0 - #22005

Merged
potiuk merged 2 commits into
apache:mainfrom
kazanzhy:upgrade_moto_lib
Mar 5, 2022
Merged

Upgrade moto library to version 3.0#22005
potiuk merged 2 commits into
apache:mainfrom
kazanzhy:upgrade_moto_lib

Conversation

@kazanzhy

@kazanzhykazanzhy commented Mar 4, 2022

Copy link
Copy Markdown
Contributor

Some of the RDS functionality was added to moto recently.
Therefore, there are needed to upgrade version of the library

@boring-cyborgboring-cyborgBot added area:providers provider:amazon AWS/Amazon - related issues labels Mar 4, 2022
This was referenced Mar 4, 2022
Merged
Merged
@kazanzhykazanzhy changed the title Update moto library to version 3.0Upgrade moto library to version 3.0Mar 4, 2022
@potiuk

potiuk commented Mar 5, 2022

Copy link
Copy Markdown
Member

This is fine - but it requires constraints to get regenerated. The failure there indicates that. I will need to manually update constraints and merge it at about the same time to accomodate !

@github-actionsgithub-actionsBot added the full tests needed We need to run full set of tests for this PR to merge label Mar 5, 2022
@github-actions

github-actionsBot commented Mar 5, 2022

Copy link
Copy Markdown
Contributor

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

@uranusjr

uranusjr commented Mar 5, 2022

Copy link
Copy Markdown
Member

Is it impossible to support both 2.x and 3.0+?

@kazanzhy

kazanzhy commented Mar 5, 2022

Copy link
Copy Markdown
ContributorAuthor

Good question @uranusjr. I think it's not possible.
But as I understand for Airflow 2.2 and less we will have moto==2.x and for Airflow 2.3+ there will be moto==3.0.

@potiuk

potiuk commented Mar 5, 2022

Copy link
Copy Markdown
Member

But as I understand for Airflow 2.2 and less we will have moto==2.x and for Airflow 2.3+ there will be moto==3.0.

I think it does not really matter, moto is really a devel-only dependency - only used in tests, so it is just used for tests only. Once it works and constraints are updated - it will work. But I agree with @uranusjr, it might be a bit easier for us to just remove the limit (and it will get upgraded automatically). According to our newly agreed rules it should not be upper-bound (and this is even more so as this is test dependency only)

I will slightly modify it and see if it will pass (and use moto > 3) - it should.

potiuk
potiuk reviewed Mar 5, 2022
View reviewed changes
Comment threadsetup.py Outdated
@potiuk

potiuk commented Mar 5, 2022

Copy link
Copy Markdown
Member

UPDATE. Ah I see that we shoudl anyway have > 3.0.3 so I will just remove the ~ (and update the constraints as planned)

@kazanzhy

kazanzhy commented Mar 5, 2022

Copy link
Copy Markdown
ContributorAuthor

I had many problems with the "latest version" in my projects

If we are not fixing the version of moto then we should keep in mind that we can have failed tests for all PRs if let's say in version 3.1 something will be changed.

@potiuk

potiuk commented Mar 5, 2022

Copy link
Copy Markdown
Member

I had many problems with the "latest version" in my projects

If we are not fixing the version of moto then we should keep in mind that we can have failed tests for all PRs if let's say in version 3.1 something will be changed.

This is what our constraint mechanism provides. All "regular" PRs if they do not change setup.py/setup.cfg use the constraints to run the PR - i.e. latest "good" version of dependencies. Only in main builds (after merge or on schedule) or in case someone modifies the setup.py/.cfg we run "eager upgrade" of all dependencies (and if the tests succeed, those become new constraints - they are automatically pushed to our repo).

This way:

  1. we have stable set of dependencies for regular constraints

  2. we automatically upgrade the stable set with the newly released versions (with every merge and on nightly schedule) - and only when ALL tests succeed with them (this is great for security!). When we have no frequent flakes, we have 2-3 updates a day https://github.com/apache/airflow/commits/constraints-main (because we have 580 dependencies in total)

  3. when there is a breaking release we detect it (as our main and "dependecy changing" PRs start failing) - and it gives us a chance to fix them without impacting regular PRs

So - getting a failure in main when 3.1 is released with breaking change is GOOD and we are prepared to handle it - this will give us a chance to see it and fix it before most of the contributors even realise it happened.

If want to see more - my talk about it from November: https://youtu.be/_SjMdQLP30s?t=2519

@potiuk

potiuk commented Mar 5, 2022

Copy link
Copy Markdown
Member

And here is our (agreed in the community after recent discussion) approach on how we treat dependencies: https://github.com/apache/airflow#approach-to-dependencies-of-airflow

@potiuk
potiuk merged commit de29eb6 into apache:mainMar 5, 2022
@potiuk

potiuk commented Mar 5, 2022

Copy link
Copy Markdown
Member

I rebuilt the constraints and merged :). All Good!

@kazanzhy

kazanzhy commented Mar 5, 2022

Copy link
Copy Markdown
ContributorAuthor

Thanks for the information.
That's a little bit more complex that I imagined

@potiuk

potiuk commented Mar 5, 2022•
edited
Loading

Copy link
Copy Markdown
Member

Ah yeah. With 580 deps we have ~ 20 new released dep versions every week :). Hard to keep up without the automation :)

@kazanzhy
kazanzhy deleted the upgrade_moto_lib branch March 5, 2022 18:03
@ephraimbuddyephraimbuddy added the type:misc/internal Changelog: Misc changes that should appear in change log label Apr 11, 2022
@ephraimbuddyephraimbuddy added changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) and removed type:misc/internal Changelog: Misc changes that should appear in change log labels Apr 26, 2022
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@potiukpotiukpotiuk approved these changes

Assignees

No one assigned

Labels

area:providerschangelog:skipChanges that should be skipped from the changelog (CI, tests, etc..)full tests neededWe need to run full set of tests for this PR to mergeprovider:amazonAWS/Amazon - related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@kazanzhy@potiuk@uranusjr@ephraimbuddy
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Upgrade moto library to version 3.0 - #22005
Merged
potiuk merged 2 commits into
apache:mainfrom 
kazanzhy:upgrade_moto_lib
Mar 5, 2022
Merged
Upgrade moto library to version 3.0#22005
potiuk merged 2 commits into
apache:mainfrom 
kazanzhy:upgrade_moto_lib
Conversation
@kazanzhy
@kazanzhykazanzhy
commented
Mar 4, 2022

Copy link


Copy Markdown

Contributor
Some of the RDS functionality was added to moto recently.

Therefore, there are needed to upgrade version of the library
@boring-cyborgboring-cyborgBot
added

area:providers

provider:amazon
AWS/Amazon - related issues
labels
Mar 4, 2022

This was referenced Mar 4, 2022
 Merged

 Merged

@kazanzhykazanzhy
changed the title
Update moto library to version 3.0Upgrade moto library to version 3.0Mar 4, 2022
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
This is fine - but it requires constraints to get regenerated. The failure there indicates that. I will need to manually update constraints and merge it at about the same time to accomodate !
@github-actionsgithub-actionsBot
added
the 
full tests needed
We need to run full set of tests for this PR to merge
label
Mar 5, 2022
@github-actions
github-actionsBot
commented
Mar 5, 2022

Copy link


Copy Markdown

Contributor
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
@uranusjr
uranusjr
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
Is it impossible to support both 2.x and 3.0+?
@kazanzhy
kazanzhy
commented
Mar 5, 2022

Copy link


Copy Markdown

ContributorAuthor
Good question @uranusjr. I think it's not possible.

But as I understand for Airflow 2.2 and less we will have moto==2.x and for Airflow 2.3+ there will be moto==3.0.
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
But as I understand for Airflow 2.2 and less we will have moto==2.x and for Airflow 2.3+ there will be moto==3.0.
I think it does not really matter, moto is really a devel-only dependency - only used in tests, so it is just used for tests only. Once it works and constraints are updated - it will work. But I agree with @uranusjr, it might be a bit easier for us to just remove the limit (and it will get upgraded automatically). According to our newly agreed rules it should not be upper-bound (and this is even more so as this is test dependency only)
I will slightly modify it and see if it will pass (and use moto > 3) - it should.
potiuk
potiuk
reviewed
Mar 5, 2022
View reviewed changes
Comment threadsetup.py
Outdated

@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
UPDATE. Ah I see that we shoudl anyway have > 3.0.3 so I will just remove the ~ (and update the constraints as planned)
@kazanzhy
kazanzhy
commented
Mar 5, 2022

Copy link


Copy Markdown

ContributorAuthor
I had many problems with the "latest version" in my projects
If we are not fixing the version of moto then we should keep in mind that we can have failed tests for all PRs if let's say in version 3.1 something will be changed.
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
I had many problems with the "latest version" in my projects
If we are not fixing the version of moto then we should keep in mind that we can have failed tests for all PRs if let's say in version 3.1 something will be changed.
This is what our constraint mechanism provides. All "regular" PRs if they do not change setup.py/setup.cfg use the constraints to run the PR - i.e. latest "good" version of dependencies. Only in main builds (after merge or on schedule) or in case someone modifies the setup.py/.cfg we run "eager upgrade" of all dependencies (and if the tests succeed, those become new constraints - they are automatically pushed to our repo).
This way:
  1. we have stable set of dependencies for regular constraints
  2. we automatically upgrade the stable set with the newly released versions (with every merge and on nightly schedule) - and only when ALL tests succeed with them (this is great for security!). When we have no frequent flakes, we have 2-3 updates a day https://github.com/apache/airflow/commits/constraints-main (because we have 580 dependencies in total)
  3. when there is a breaking release we detect it (as our main and "dependecy changing" PRs start failing) - and it gives us a chance to fix them without impacting regular PRs
So - getting a failure in main when 3.1 is released with breaking change is GOOD and we are prepared to handle it - this will give us a chance to see it and fix it before most of the contributors even realise it happened.
If want to see more - my talk about it from November: https://youtu.be/_SjMdQLP30s?t=2519
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
And here is our (agreed in the community after recent discussion) approach on how we treat dependencies: https://github.com/apache/airflow#approach-to-dependencies-of-airflow
@potiuk
potiuk
merged commit de29eb6
into
apache:mainMar 5, 2022
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
I rebuilt the constraints and merged :). All Good!
@kazanzhy
kazanzhy
commented
Mar 5, 2022

Copy link


Copy Markdown

ContributorAuthor
Thanks for the information.

That's a little bit more complex that I imagined
@potiuk
potiuk
commented
Mar 5, 2022•

edited

Loading

Copy link


Copy Markdown

Member
Ah yeah. With 580 deps we have ~ 20 new released dep versions every week :). Hard to keep up without the automation :)
@kazanzhy
kazanzhy
deleted the

upgrade_moto_lib

branch
March 5, 2022 18:03
@ephraimbuddyephraimbuddy
added
the 
type:misc/internal
Changelog: Misc changes that should appear in change log
label
Apr 11, 2022
@ephraimbuddyephraimbuddy
added

changelog:skip
Changes that should be skipped from the changelog (CI, tests, etc..)
and removed

type:misc/internal
Changelog: Misc changes that should appear in change log
labels
Apr 26, 2022
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@potiukpotiukpotiuk approved these changes

Assignees


No one assigned


Labels

area:providerschangelog:skipChanges that should be skipped from the changelog (CI, tests, etc..)full tests neededWe need to run full set of tests for this PR to mergeprovider:amazonAWS/Amazon - related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@kazanzhy@potiuk@uranusjr@ephraimbuddy

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Upgrade moto library to version 3.0 - #22005
Merged
potiuk merged 2 commits into
apache:mainfrom 
kazanzhy:upgrade_moto_lib
Mar 5, 2022
Merged
Upgrade moto library to version 3.0#22005
potiuk merged 2 commits into
apache:mainfrom 
kazanzhy:upgrade_moto_lib
Conversation
@kazanzhy
@kazanzhykazanzhy
commented
Mar 4, 2022

Copy link


Copy Markdown

Contributor
Some of the RDS functionality was added to moto recently.

Therefore, there are needed to upgrade version of the library
@boring-cyborgboring-cyborgBot
added

area:providers

provider:amazon
AWS/Amazon - related issues
labels
Mar 4, 2022

This was referenced Mar 4, 2022
 Merged

 Merged

@kazanzhykazanzhy
changed the title
Update moto library to version 3.0Upgrade moto library to version 3.0Mar 4, 2022
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
This is fine - but it requires constraints to get regenerated. The failure there indicates that. I will need to manually update constraints and merge it at about the same time to accomodate !
@github-actionsgithub-actionsBot
added
the 
full tests needed
We need to run full set of tests for this PR to merge
label
Mar 5, 2022
@github-actions
github-actionsBot
commented
Mar 5, 2022

Copy link


Copy Markdown

Contributor
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
@uranusjr
uranusjr
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
Is it impossible to support both 2.x and 3.0+?
@kazanzhy
kazanzhy
commented
Mar 5, 2022

Copy link


Copy Markdown

ContributorAuthor
Good question @uranusjr. I think it's not possible.

But as I understand for Airflow 2.2 and less we will have moto==2.x and for Airflow 2.3+ there will be moto==3.0.
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
But as I understand for Airflow 2.2 and less we will have moto==2.x and for Airflow 2.3+ there will be moto==3.0.
I think it does not really matter, moto is really a devel-only dependency - only used in tests, so it is just used for tests only. Once it works and constraints are updated - it will work. But I agree with @uranusjr, it might be a bit easier for us to just remove the limit (and it will get upgraded automatically). According to our newly agreed rules it should not be upper-bound (and this is even more so as this is test dependency only)
I will slightly modify it and see if it will pass (and use moto > 3) - it should.
potiuk
potiuk
reviewed
Mar 5, 2022
View reviewed changes
Comment threadsetup.py
Outdated

@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
UPDATE. Ah I see that we shoudl anyway have > 3.0.3 so I will just remove the ~ (and update the constraints as planned)
@kazanzhy
kazanzhy
commented
Mar 5, 2022

Copy link


Copy Markdown

ContributorAuthor
I had many problems with the "latest version" in my projects
If we are not fixing the version of moto then we should keep in mind that we can have failed tests for all PRs if let's say in version 3.1 something will be changed.
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
I had many problems with the "latest version" in my projects
If we are not fixing the version of moto then we should keep in mind that we can have failed tests for all PRs if let's say in version 3.1 something will be changed.
This is what our constraint mechanism provides. All "regular" PRs if they do not change setup.py/setup.cfg use the constraints to run the PR - i.e. latest "good" version of dependencies. Only in main builds (after merge or on schedule) or in case someone modifies the setup.py/.cfg we run "eager upgrade" of all dependencies (and if the tests succeed, those become new constraints - they are automatically pushed to our repo).
This way:
  1. we have stable set of dependencies for regular constraints
  2. we automatically upgrade the stable set with the newly released versions (with every merge and on nightly schedule) - and only when ALL tests succeed with them (this is great for security!). When we have no frequent flakes, we have 2-3 updates a day https://github.com/apache/airflow/commits/constraints-main (because we have 580 dependencies in total)
  3. when there is a breaking release we detect it (as our main and "dependecy changing" PRs start failing) - and it gives us a chance to fix them without impacting regular PRs
So - getting a failure in main when 3.1 is released with breaking change is GOOD and we are prepared to handle it - this will give us a chance to see it and fix it before most of the contributors even realise it happened.
If want to see more - my talk about it from November: https://youtu.be/_SjMdQLP30s?t=2519
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
And here is our (agreed in the community after recent discussion) approach on how we treat dependencies: https://github.com/apache/airflow#approach-to-dependencies-of-airflow
@potiuk
potiuk
merged commit de29eb6
into
apache:mainMar 5, 2022
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
I rebuilt the constraints and merged :). All Good!
@kazanzhy
kazanzhy
commented
Mar 5, 2022

Copy link


Copy Markdown

ContributorAuthor
Thanks for the information.

That's a little bit more complex that I imagined
@potiuk
potiuk
commented
Mar 5, 2022•

edited

Loading

Copy link


Copy Markdown

Member
Ah yeah. With 580 deps we have ~ 20 new released dep versions every week :). Hard to keep up without the automation :)
@kazanzhy
kazanzhy
deleted the

upgrade_moto_lib

branch
March 5, 2022 18:03
@ephraimbuddyephraimbuddy
added
the 
type:misc/internal
Changelog: Misc changes that should appear in change log
label
Apr 11, 2022
@ephraimbuddyephraimbuddy
added

changelog:skip
Changes that should be skipped from the changelog (CI, tests, etc..)
and removed

type:misc/internal
Changelog: Misc changes that should appear in change log
labels
Apr 26, 2022
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@potiukpotiukpotiuk approved these changes

Assignees


No one assigned


Labels

area:providerschangelog:skipChanges that should be skipped from the changelog (CI, tests, etc..)full tests neededWe need to run full set of tests for this PR to mergeprovider:amazonAWS/Amazon - related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@kazanzhy@potiuk@uranusjr@ephraimbuddy

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Upgrade moto library to version 3.0 - #22005
Merged
potiuk merged 2 commits into
apache:mainfrom 
kazanzhy:upgrade_moto_lib
Mar 5, 2022
Merged
Upgrade moto library to version 3.0#22005
potiuk merged 2 commits into
apache:mainfrom 
kazanzhy:upgrade_moto_lib
Conversation
@kazanzhy
@kazanzhykazanzhy
commented
Mar 4, 2022

Copy link


Copy Markdown

Contributor
Some of the RDS functionality was added to moto recently.

Therefore, there are needed to upgrade version of the library
@boring-cyborgboring-cyborgBot
added

area:providers

provider:amazon
AWS/Amazon - related issues
labels
Mar 4, 2022

This was referenced Mar 4, 2022
 Merged

 Merged

@kazanzhykazanzhy
changed the title
Update moto library to version 3.0Upgrade moto library to version 3.0Mar 4, 2022
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
This is fine - but it requires constraints to get regenerated. The failure there indicates that. I will need to manually update constraints and merge it at about the same time to accomodate !
@github-actionsgithub-actionsBot
added
the 
full tests needed
We need to run full set of tests for this PR to merge
label
Mar 5, 2022
@github-actions
github-actionsBot
commented
Mar 5, 2022

Copy link


Copy Markdown

Contributor
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
@uranusjr
uranusjr
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
Is it impossible to support both 2.x and 3.0+?
@kazanzhy
kazanzhy
commented
Mar 5, 2022

Copy link


Copy Markdown

ContributorAuthor
Good question @uranusjr. I think it's not possible.

But as I understand for Airflow 2.2 and less we will have moto==2.x and for Airflow 2.3+ there will be moto==3.0.
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
But as I understand for Airflow 2.2 and less we will have moto==2.x and for Airflow 2.3+ there will be moto==3.0.
I think it does not really matter, moto is really a devel-only dependency - only used in tests, so it is just used for tests only. Once it works and constraints are updated - it will work. But I agree with @uranusjr, it might be a bit easier for us to just remove the limit (and it will get upgraded automatically). According to our newly agreed rules it should not be upper-bound (and this is even more so as this is test dependency only)
I will slightly modify it and see if it will pass (and use moto > 3) - it should.
potiuk
potiuk
reviewed
Mar 5, 2022
View reviewed changes
Comment threadsetup.py
Outdated

@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
UPDATE. Ah I see that we shoudl anyway have > 3.0.3 so I will just remove the ~ (and update the constraints as planned)
@kazanzhy
kazanzhy
commented
Mar 5, 2022

Copy link


Copy Markdown

ContributorAuthor
I had many problems with the "latest version" in my projects
If we are not fixing the version of moto then we should keep in mind that we can have failed tests for all PRs if let's say in version 3.1 something will be changed.
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
I had many problems with the "latest version" in my projects
If we are not fixing the version of moto then we should keep in mind that we can have failed tests for all PRs if let's say in version 3.1 something will be changed.
This is what our constraint mechanism provides. All "regular" PRs if they do not change setup.py/setup.cfg use the constraints to run the PR - i.e. latest "good" version of dependencies. Only in main builds (after merge or on schedule) or in case someone modifies the setup.py/.cfg we run "eager upgrade" of all dependencies (and if the tests succeed, those become new constraints - they are automatically pushed to our repo).
This way:
  1. we have stable set of dependencies for regular constraints
  2. we automatically upgrade the stable set with the newly released versions (with every merge and on nightly schedule) - and only when ALL tests succeed with them (this is great for security!). When we have no frequent flakes, we have 2-3 updates a day https://github.com/apache/airflow/commits/constraints-main (because we have 580 dependencies in total)
  3. when there is a breaking release we detect it (as our main and "dependecy changing" PRs start failing) - and it gives us a chance to fix them without impacting regular PRs
So - getting a failure in main when 3.1 is released with breaking change is GOOD and we are prepared to handle it - this will give us a chance to see it and fix it before most of the contributors even realise it happened.
If want to see more - my talk about it from November: https://youtu.be/_SjMdQLP30s?t=2519
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
And here is our (agreed in the community after recent discussion) approach on how we treat dependencies: https://github.com/apache/airflow#approach-to-dependencies-of-airflow
@potiuk
potiuk
merged commit de29eb6
into
apache:mainMar 5, 2022
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
I rebuilt the constraints and merged :). All Good!
@kazanzhy
kazanzhy
commented
Mar 5, 2022

Copy link


Copy Markdown

ContributorAuthor
Thanks for the information.

That's a little bit more complex that I imagined
@potiuk
potiuk
commented
Mar 5, 2022•

edited

Loading

Copy link


Copy Markdown

Member
Ah yeah. With 580 deps we have ~ 20 new released dep versions every week :). Hard to keep up without the automation :)
@kazanzhy
kazanzhy
deleted the

upgrade_moto_lib

branch
March 5, 2022 18:03
@ephraimbuddyephraimbuddy
added
the 
type:misc/internal
Changelog: Misc changes that should appear in change log
label
Apr 11, 2022
@ephraimbuddyephraimbuddy
added

changelog:skip
Changes that should be skipped from the changelog (CI, tests, etc..)
and removed

type:misc/internal
Changelog: Misc changes that should appear in change log
labels
Apr 26, 2022
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@potiukpotiukpotiuk approved these changes

Assignees


No one assigned


Labels

area:providerschangelog:skipChanges that should be skipped from the changelog (CI, tests, etc..)full tests neededWe need to run full set of tests for this PR to mergeprovider:amazonAWS/Amazon - related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@kazanzhy@potiuk@uranusjr@ephraimbuddy

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Upgrade moto library to version 3.0 - #22005
Merged
potiuk merged 2 commits into
apache:mainfrom 
kazanzhy:upgrade_moto_lib
Mar 5, 2022
Merged
Upgrade moto library to version 3.0#22005
potiuk merged 2 commits into
apache:mainfrom 
kazanzhy:upgrade_moto_lib
Conversation
@kazanzhy
@kazanzhykazanzhy
commented
Mar 4, 2022

Copy link


Copy Markdown

Contributor
Some of the RDS functionality was added to moto recently.

Therefore, there are needed to upgrade version of the library
@boring-cyborgboring-cyborgBot
added

area:providers

provider:amazon
AWS/Amazon - related issues
labels
Mar 4, 2022

This was referenced Mar 4, 2022
 Merged

 Merged

@kazanzhykazanzhy
changed the title
Update moto library to version 3.0Upgrade moto library to version 3.0Mar 4, 2022
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
This is fine - but it requires constraints to get regenerated. The failure there indicates that. I will need to manually update constraints and merge it at about the same time to accomodate !
@github-actionsgithub-actionsBot
added
the 
full tests needed
We need to run full set of tests for this PR to merge
label
Mar 5, 2022
@github-actions
github-actionsBot
commented
Mar 5, 2022

Copy link


Copy Markdown

Contributor
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
@uranusjr
uranusjr
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
Is it impossible to support both 2.x and 3.0+?
@kazanzhy
kazanzhy
commented
Mar 5, 2022

Copy link


Copy Markdown

ContributorAuthor
Good question @uranusjr. I think it's not possible.

But as I understand for Airflow 2.2 and less we will have moto==2.x and for Airflow 2.3+ there will be moto==3.0.
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
But as I understand for Airflow 2.2 and less we will have moto==2.x and for Airflow 2.3+ there will be moto==3.0.
I think it does not really matter, moto is really a devel-only dependency - only used in tests, so it is just used for tests only. Once it works and constraints are updated - it will work. But I agree with @uranusjr, it might be a bit easier for us to just remove the limit (and it will get upgraded automatically). According to our newly agreed rules it should not be upper-bound (and this is even more so as this is test dependency only)
I will slightly modify it and see if it will pass (and use moto > 3) - it should.
potiuk
potiuk
reviewed
Mar 5, 2022
View reviewed changes
Comment threadsetup.py
Outdated

@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
UPDATE. Ah I see that we shoudl anyway have > 3.0.3 so I will just remove the ~ (and update the constraints as planned)
@kazanzhy
kazanzhy
commented
Mar 5, 2022

Copy link


Copy Markdown

ContributorAuthor
I had many problems with the "latest version" in my projects
If we are not fixing the version of moto then we should keep in mind that we can have failed tests for all PRs if let's say in version 3.1 something will be changed.
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
I had many problems with the "latest version" in my projects
If we are not fixing the version of moto then we should keep in mind that we can have failed tests for all PRs if let's say in version 3.1 something will be changed.
This is what our constraint mechanism provides. All "regular" PRs if they do not change setup.py/setup.cfg use the constraints to run the PR - i.e. latest "good" version of dependencies. Only in main builds (after merge or on schedule) or in case someone modifies the setup.py/.cfg we run "eager upgrade" of all dependencies (and if the tests succeed, those become new constraints - they are automatically pushed to our repo).
This way:
  1. we have stable set of dependencies for regular constraints
  2. we automatically upgrade the stable set with the newly released versions (with every merge and on nightly schedule) - and only when ALL tests succeed with them (this is great for security!). When we have no frequent flakes, we have 2-3 updates a day https://github.com/apache/airflow/commits/constraints-main (because we have 580 dependencies in total)
  3. when there is a breaking release we detect it (as our main and "dependecy changing" PRs start failing) - and it gives us a chance to fix them without impacting regular PRs
So - getting a failure in main when 3.1 is released with breaking change is GOOD and we are prepared to handle it - this will give us a chance to see it and fix it before most of the contributors even realise it happened.
If want to see more - my talk about it from November: https://youtu.be/_SjMdQLP30s?t=2519
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
And here is our (agreed in the community after recent discussion) approach on how we treat dependencies: https://github.com/apache/airflow#approach-to-dependencies-of-airflow
@potiuk
potiuk
merged commit de29eb6
into
apache:mainMar 5, 2022
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
I rebuilt the constraints and merged :). All Good!
@kazanzhy
kazanzhy
commented
Mar 5, 2022

Copy link


Copy Markdown

ContributorAuthor
Thanks for the information.

That's a little bit more complex that I imagined
@potiuk
potiuk
commented
Mar 5, 2022•

edited

Loading

Copy link


Copy Markdown

Member
Ah yeah. With 580 deps we have ~ 20 new released dep versions every week :). Hard to keep up without the automation :)
@kazanzhy
kazanzhy
deleted the

upgrade_moto_lib

branch
March 5, 2022 18:03
@ephraimbuddyephraimbuddy
added
the 
type:misc/internal
Changelog: Misc changes that should appear in change log
label
Apr 11, 2022
@ephraimbuddyephraimbuddy
added

changelog:skip
Changes that should be skipped from the changelog (CI, tests, etc..)
and removed

type:misc/internal
Changelog: Misc changes that should appear in change log
labels
Apr 26, 2022
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@potiukpotiukpotiuk approved these changes

Assignees


No one assigned


Labels

area:providerschangelog:skipChanges that should be skipped from the changelog (CI, tests, etc..)full tests neededWe need to run full set of tests for this PR to mergeprovider:amazonAWS/Amazon - related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@kazanzhy@potiuk@uranusjr@ephraimbuddy

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Upgrade moto library to version 3.0 - #22005
Merged
potiuk merged 2 commits into
apache:mainfrom 
kazanzhy:upgrade_moto_lib
Mar 5, 2022
Merged
Upgrade moto library to version 3.0#22005
potiuk merged 2 commits into
apache:mainfrom 
kazanzhy:upgrade_moto_lib
Conversation
@kazanzhy
@kazanzhykazanzhy
commented
Mar 4, 2022

Copy link


Copy Markdown

Contributor
Some of the RDS functionality was added to moto recently.

Therefore, there are needed to upgrade version of the library
@boring-cyborgboring-cyborgBot
added

area:providers

provider:amazon
AWS/Amazon - related issues
labels
Mar 4, 2022

This was referenced Mar 4, 2022
 Merged

 Merged

@kazanzhykazanzhy
changed the title
Update moto library to version 3.0Upgrade moto library to version 3.0Mar 4, 2022
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
This is fine - but it requires constraints to get regenerated. The failure there indicates that. I will need to manually update constraints and merge it at about the same time to accomodate !
@github-actionsgithub-actionsBot
added
the 
full tests needed
We need to run full set of tests for this PR to merge
label
Mar 5, 2022
@github-actions
github-actionsBot
commented
Mar 5, 2022

Copy link


Copy Markdown

Contributor
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
@uranusjr
uranusjr
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
Is it impossible to support both 2.x and 3.0+?
@kazanzhy
kazanzhy
commented
Mar 5, 2022

Copy link


Copy Markdown

ContributorAuthor
Good question @uranusjr. I think it's not possible.

But as I understand for Airflow 2.2 and less we will have moto==2.x and for Airflow 2.3+ there will be moto==3.0.
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
But as I understand for Airflow 2.2 and less we will have moto==2.x and for Airflow 2.3+ there will be moto==3.0.
I think it does not really matter, moto is really a devel-only dependency - only used in tests, so it is just used for tests only. Once it works and constraints are updated - it will work. But I agree with @uranusjr, it might be a bit easier for us to just remove the limit (and it will get upgraded automatically). According to our newly agreed rules it should not be upper-bound (and this is even more so as this is test dependency only)
I will slightly modify it and see if it will pass (and use moto > 3) - it should.
potiuk
potiuk
reviewed
Mar 5, 2022
View reviewed changes
Comment threadsetup.py
Outdated

@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
UPDATE. Ah I see that we shoudl anyway have > 3.0.3 so I will just remove the ~ (and update the constraints as planned)
@kazanzhy
kazanzhy
commented
Mar 5, 2022

Copy link


Copy Markdown

ContributorAuthor
I had many problems with the "latest version" in my projects
If we are not fixing the version of moto then we should keep in mind that we can have failed tests for all PRs if let's say in version 3.1 something will be changed.
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
I had many problems with the "latest version" in my projects
If we are not fixing the version of moto then we should keep in mind that we can have failed tests for all PRs if let's say in version 3.1 something will be changed.
This is what our constraint mechanism provides. All "regular" PRs if they do not change setup.py/setup.cfg use the constraints to run the PR - i.e. latest "good" version of dependencies. Only in main builds (after merge or on schedule) or in case someone modifies the setup.py/.cfg we run "eager upgrade" of all dependencies (and if the tests succeed, those become new constraints - they are automatically pushed to our repo).
This way:
  1. we have stable set of dependencies for regular constraints
  2. we automatically upgrade the stable set with the newly released versions (with every merge and on nightly schedule) - and only when ALL tests succeed with them (this is great for security!). When we have no frequent flakes, we have 2-3 updates a day https://github.com/apache/airflow/commits/constraints-main (because we have 580 dependencies in total)
  3. when there is a breaking release we detect it (as our main and "dependecy changing" PRs start failing) - and it gives us a chance to fix them without impacting regular PRs
So - getting a failure in main when 3.1 is released with breaking change is GOOD and we are prepared to handle it - this will give us a chance to see it and fix it before most of the contributors even realise it happened.
If want to see more - my talk about it from November: https://youtu.be/_SjMdQLP30s?t=2519
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
And here is our (agreed in the community after recent discussion) approach on how we treat dependencies: https://github.com/apache/airflow#approach-to-dependencies-of-airflow
@potiuk
potiuk
merged commit de29eb6
into
apache:mainMar 5, 2022
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
I rebuilt the constraints and merged :). All Good!
@kazanzhy
kazanzhy
commented
Mar 5, 2022

Copy link


Copy Markdown

ContributorAuthor
Thanks for the information.

That's a little bit more complex that I imagined
@potiuk
potiuk
commented
Mar 5, 2022•

edited

Loading

Copy link


Copy Markdown

Member
Ah yeah. With 580 deps we have ~ 20 new released dep versions every week :). Hard to keep up without the automation :)
@kazanzhy
kazanzhy
deleted the

upgrade_moto_lib

branch
March 5, 2022 18:03
@ephraimbuddyephraimbuddy
added
the 
type:misc/internal
Changelog: Misc changes that should appear in change log
label
Apr 11, 2022
@ephraimbuddyephraimbuddy
added

changelog:skip
Changes that should be skipped from the changelog (CI, tests, etc..)
and removed

type:misc/internal
Changelog: Misc changes that should appear in change log
labels
Apr 26, 2022
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@potiukpotiukpotiuk approved these changes

Assignees


No one assigned


Labels

area:providerschangelog:skipChanges that should be skipped from the changelog (CI, tests, etc..)full tests neededWe need to run full set of tests for this PR to mergeprovider:amazonAWS/Amazon - related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@kazanzhy@potiuk@uranusjr@ephraimbuddy

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Upgrade moto library to version 3.0 - #22005
Merged
potiuk merged 2 commits into
apache:mainfrom 
kazanzhy:upgrade_moto_lib
Mar 5, 2022
Merged
Upgrade moto library to version 3.0#22005
potiuk merged 2 commits into
apache:mainfrom 
kazanzhy:upgrade_moto_lib
Conversation
@kazanzhy
@kazanzhykazanzhy
commented
Mar 4, 2022

Copy link


Copy Markdown

Contributor
Some of the RDS functionality was added to moto recently.

Therefore, there are needed to upgrade version of the library
@boring-cyborgboring-cyborgBot
added

area:providers

provider:amazon
AWS/Amazon - related issues
labels
Mar 4, 2022

This was referenced Mar 4, 2022
 Merged

 Merged

@kazanzhykazanzhy
changed the title
Update moto library to version 3.0Upgrade moto library to version 3.0Mar 4, 2022
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
This is fine - but it requires constraints to get regenerated. The failure there indicates that. I will need to manually update constraints and merge it at about the same time to accomodate !
@github-actionsgithub-actionsBot
added
the 
full tests needed
We need to run full set of tests for this PR to merge
label
Mar 5, 2022
@github-actions
github-actionsBot
commented
Mar 5, 2022

Copy link


Copy Markdown

Contributor
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
@uranusjr
uranusjr
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
Is it impossible to support both 2.x and 3.0+?
@kazanzhy
kazanzhy
commented
Mar 5, 2022

Copy link


Copy Markdown

ContributorAuthor
Good question @uranusjr. I think it's not possible.

But as I understand for Airflow 2.2 and less we will have moto==2.x and for Airflow 2.3+ there will be moto==3.0.
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
But as I understand for Airflow 2.2 and less we will have moto==2.x and for Airflow 2.3+ there will be moto==3.0.
I think it does not really matter, moto is really a devel-only dependency - only used in tests, so it is just used for tests only. Once it works and constraints are updated - it will work. But I agree with @uranusjr, it might be a bit easier for us to just remove the limit (and it will get upgraded automatically). According to our newly agreed rules it should not be upper-bound (and this is even more so as this is test dependency only)
I will slightly modify it and see if it will pass (and use moto > 3) - it should.
potiuk
potiuk
reviewed
Mar 5, 2022
View reviewed changes
Comment threadsetup.py
Outdated

@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
UPDATE. Ah I see that we shoudl anyway have > 3.0.3 so I will just remove the ~ (and update the constraints as planned)
@kazanzhy
kazanzhy
commented
Mar 5, 2022

Copy link


Copy Markdown

ContributorAuthor
I had many problems with the "latest version" in my projects
If we are not fixing the version of moto then we should keep in mind that we can have failed tests for all PRs if let's say in version 3.1 something will be changed.
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
I had many problems with the "latest version" in my projects
If we are not fixing the version of moto then we should keep in mind that we can have failed tests for all PRs if let's say in version 3.1 something will be changed.
This is what our constraint mechanism provides. All "regular" PRs if they do not change setup.py/setup.cfg use the constraints to run the PR - i.e. latest "good" version of dependencies. Only in main builds (after merge or on schedule) or in case someone modifies the setup.py/.cfg we run "eager upgrade" of all dependencies (and if the tests succeed, those become new constraints - they are automatically pushed to our repo).
This way:
  1. we have stable set of dependencies for regular constraints
  2. we automatically upgrade the stable set with the newly released versions (with every merge and on nightly schedule) - and only when ALL tests succeed with them (this is great for security!). When we have no frequent flakes, we have 2-3 updates a day https://github.com/apache/airflow/commits/constraints-main (because we have 580 dependencies in total)
  3. when there is a breaking release we detect it (as our main and "dependecy changing" PRs start failing) - and it gives us a chance to fix them without impacting regular PRs
So - getting a failure in main when 3.1 is released with breaking change is GOOD and we are prepared to handle it - this will give us a chance to see it and fix it before most of the contributors even realise it happened.
If want to see more - my talk about it from November: https://youtu.be/_SjMdQLP30s?t=2519
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
And here is our (agreed in the community after recent discussion) approach on how we treat dependencies: https://github.com/apache/airflow#approach-to-dependencies-of-airflow
@potiuk
potiuk
merged commit de29eb6
into
apache:mainMar 5, 2022
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
I rebuilt the constraints and merged :). All Good!
@kazanzhy
kazanzhy
commented
Mar 5, 2022

Copy link


Copy Markdown

ContributorAuthor
Thanks for the information.

That's a little bit more complex that I imagined
@potiuk
potiuk
commented
Mar 5, 2022•

edited

Loading

Copy link


Copy Markdown

Member
Ah yeah. With 580 deps we have ~ 20 new released dep versions every week :). Hard to keep up without the automation :)
@kazanzhy
kazanzhy
deleted the

upgrade_moto_lib

branch
March 5, 2022 18:03
@ephraimbuddyephraimbuddy
added
the 
type:misc/internal
Changelog: Misc changes that should appear in change log
label
Apr 11, 2022
@ephraimbuddyephraimbuddy
added

changelog:skip
Changes that should be skipped from the changelog (CI, tests, etc..)
and removed

type:misc/internal
Changelog: Misc changes that should appear in change log
labels
Apr 26, 2022
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@potiukpotiukpotiuk approved these changes

Assignees


No one assigned


Labels

area:providerschangelog:skipChanges that should be skipped from the changelog (CI, tests, etc..)full tests neededWe need to run full set of tests for this PR to mergeprovider:amazonAWS/Amazon - related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@kazanzhy@potiuk@uranusjr@ephraimbuddy

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Upgrade moto library to version 3.0 - #22005
Merged
potiuk merged 2 commits into
apache:mainfrom 
kazanzhy:upgrade_moto_lib
Mar 5, 2022
Merged
Upgrade moto library to version 3.0#22005
potiuk merged 2 commits into
apache:mainfrom 
kazanzhy:upgrade_moto_lib
Conversation
@kazanzhy
@kazanzhykazanzhy
commented
Mar 4, 2022

Copy link


Copy Markdown

Contributor
Some of the RDS functionality was added to moto recently.

Therefore, there are needed to upgrade version of the library
@boring-cyborgboring-cyborgBot
added

area:providers

provider:amazon
AWS/Amazon - related issues
labels
Mar 4, 2022

This was referenced Mar 4, 2022
 Merged

 Merged

@kazanzhykazanzhy
changed the title
Update moto library to version 3.0Upgrade moto library to version 3.0Mar 4, 2022
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
This is fine - but it requires constraints to get regenerated. The failure there indicates that. I will need to manually update constraints and merge it at about the same time to accomodate !
@github-actionsgithub-actionsBot
added
the 
full tests needed
We need to run full set of tests for this PR to merge
label
Mar 5, 2022
@github-actions
github-actionsBot
commented
Mar 5, 2022

Copy link


Copy Markdown

Contributor
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.
@uranusjr
uranusjr
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
Is it impossible to support both 2.x and 3.0+?
@kazanzhy
kazanzhy
commented
Mar 5, 2022

Copy link


Copy Markdown

ContributorAuthor
Good question @uranusjr. I think it's not possible.

But as I understand for Airflow 2.2 and less we will have moto==2.x and for Airflow 2.3+ there will be moto==3.0.
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
But as I understand for Airflow 2.2 and less we will have moto==2.x and for Airflow 2.3+ there will be moto==3.0.
I think it does not really matter, moto is really a devel-only dependency - only used in tests, so it is just used for tests only. Once it works and constraints are updated - it will work. But I agree with @uranusjr, it might be a bit easier for us to just remove the limit (and it will get upgraded automatically). According to our newly agreed rules it should not be upper-bound (and this is even more so as this is test dependency only)
I will slightly modify it and see if it will pass (and use moto > 3) - it should.
potiuk
potiuk
reviewed
Mar 5, 2022
View reviewed changes
Comment threadsetup.py
Outdated

@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
UPDATE. Ah I see that we shoudl anyway have > 3.0.3 so I will just remove the ~ (and update the constraints as planned)
@kazanzhy
kazanzhy
commented
Mar 5, 2022

Copy link


Copy Markdown

ContributorAuthor
I had many problems with the "latest version" in my projects
If we are not fixing the version of moto then we should keep in mind that we can have failed tests for all PRs if let's say in version 3.1 something will be changed.
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
I had many problems with the "latest version" in my projects
If we are not fixing the version of moto then we should keep in mind that we can have failed tests for all PRs if let's say in version 3.1 something will be changed.
This is what our constraint mechanism provides. All "regular" PRs if they do not change setup.py/setup.cfg use the constraints to run the PR - i.e. latest "good" version of dependencies. Only in main builds (after merge or on schedule) or in case someone modifies the setup.py/.cfg we run "eager upgrade" of all dependencies (and if the tests succeed, those become new constraints - they are automatically pushed to our repo).
This way:
  1. we have stable set of dependencies for regular constraints
  2. we automatically upgrade the stable set with the newly released versions (with every merge and on nightly schedule) - and only when ALL tests succeed with them (this is great for security!). When we have no frequent flakes, we have 2-3 updates a day https://github.com/apache/airflow/commits/constraints-main (because we have 580 dependencies in total)
  3. when there is a breaking release we detect it (as our main and "dependecy changing" PRs start failing) - and it gives us a chance to fix them without impacting regular PRs
So - getting a failure in main when 3.1 is released with breaking change is GOOD and we are prepared to handle it - this will give us a chance to see it and fix it before most of the contributors even realise it happened.
If want to see more - my talk about it from November: https://youtu.be/_SjMdQLP30s?t=2519
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
And here is our (agreed in the community after recent discussion) approach on how we treat dependencies: https://github.com/apache/airflow#approach-to-dependencies-of-airflow
@potiuk
potiuk
merged commit de29eb6
into
apache:mainMar 5, 2022
@potiuk
potiuk
commented
Mar 5, 2022

Copy link


Copy Markdown

Member
I rebuilt the constraints and merged :). All Good!
@kazanzhy
kazanzhy
commented
Mar 5, 2022

Copy link


Copy Markdown

ContributorAuthor
Thanks for the information.

That's a little bit more complex that I imagined
@potiuk
potiuk
commented
Mar 5, 2022•

edited

Loading

Copy link


Copy Markdown

Member
Ah yeah. With 580 deps we have ~ 20 new released dep versions every week :). Hard to keep up without the automation :)
@kazanzhy
kazanzhy
deleted the

upgrade_moto_lib

branch
March 5, 2022 18:03
@ephraimbuddyephraimbuddy
added
the 
type:misc/internal
Changelog: Misc changes that should appear in change log
label
Apr 11, 2022
@ephraimbuddyephraimbuddy
added

changelog:skip
Changes that should be skipped from the changelog (CI, tests, etc..)
and removed

type:misc/internal
Changelog: Misc changes that should appear in change log
labels
Apr 26, 2022
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@potiukpotiukpotiuk approved these changes

Assignees


No one assigned


Labels

area:providerschangelog:skipChanges that should be skipped from the changelog (CI, tests, etc..)full tests neededWe need to run full set of tests for this PR to mergeprovider:amazonAWS/Amazon - related issues

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@kazanzhy@potiuk@uranusjr@ephraimbuddy