Skip to content

Upgrade botocore/aiobotocore minimum requirements - #33649

Merged
potiuk merged 1 commit into
apache:mainfrom
potiuk:upgrade-botocore-requirements
Aug 23, 2023
Merged

Upgrade botocore/aiobotocore minimum requirements#33649
potiuk merged 1 commit into
apache:mainfrom
potiuk:upgrade-botocore-requirements

Conversation

@potiuk

Copy link
Copy Markdown
Member

Botocore has a very peculiar process of releasing new version every day, which means that it gives pip hard time to figure what will be the non-conflicting set of packages when we have too low of a minium version set as requirement.

Since we had > 1.24 that means that pip had to consider more than 340 versions for botocore, but also for related mypy packages and also a number of aiobotocore packages when resolving eager-upgrade.

We limit all the relevant packages to 1.28 as minimum version now, and we should continue doing that regularly in the future.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiukpotiuk added the full tests needed We need to run full set of tests for this PR to merge label Aug 23, 2023
@potiukpotiuk closed this Aug 23, 2023
@potiukpotiuk reopened this Aug 23, 2023
@potiuk

Copy link
Copy Markdown
MemberAuthor

Applied full tests needed to build for all Python versions. Let's see if the help given to pip by specifying higher versions of boto3-related packages will help.

Comment threadDockerfile.ci Outdated
Comment threadgenerated/provider_dependencies.json Outdated
@potiuk
potiukforce-pushed the upgrade-botocore-requirements branch from 7608926 to 820af9cCompareAugust 23, 2023 08:17
@potiukpotiuk mentioned this pull request Aug 23, 2023
@potiukpotiuk added this to the Airflow 2.7.1 milestone Aug 23, 2023
@potiuk

Copy link
Copy Markdown
MemberAuthor

ok. I think i will need to add a little "trick" for "pypi" providers to also help pip to resolve that (at least until we release newer version of the amazon provider with those limits included

Botocore has a very peculiar process of releasing new version
every day, which means that it gives `pip` hard time to figure
what will be the non-conflicting set of packages when we have
too low of a minium version set as requirement.
Since we had > 1.24 that means that `pip` had to consider
more than 340 versions for botocore, but also for related
mypy packages and also a number of aiobotocore packages when
resolving eager-upgrade.
We limit all the relevant packages to 1.28 as minimum version
now, and we should continue doing that regularly in the future.
@potiuk
potiukforce-pushed the upgrade-botocore-requirements branch from 820af9c to b20bdb0CompareAugust 23, 2023 12:03
@potiuk

Copy link
Copy Markdown
MemberAuthor

All right - so fix for constraints was easy - the root cause was yandex we just resumed. I have to temporary disable it until we release it.

@potiukpotiuk closed this Aug 23, 2023
@potiukpotiuk reopened this Aug 23, 2023
@potiuk
potiuk merged commit 5f504e9 into apache:mainAug 23, 2023
@potiuk
potiuk deleted the upgrade-botocore-requirements branch August 23, 2023 12:04
potiuk added a commit that referenced this pull request Aug 25, 2023
Botocore has a very peculiar process of releasing new version
every day, which means that it gives `pip` hard time to figure
what will be the non-conflicting set of packages when we have
too low of a minium version set as requirement.
Since we had > 1.24 that means that `pip` had to consider
more than 340 versions for botocore, but also for related
mypy packages and also a number of aiobotocore packages when
resolving eager-upgrade.
We limit all the relevant packages to 1.28 as minimum version
now, and we should continue doing that regularly in the future.
(cherry picked from commit 5f504e9)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-toolsarea:providersfull tests neededWe need to run full set of tests for this PR to merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@potiuk@Taragolis@Lee-W@eladkal@pankajastro