Skip to content

Limit parallelism for constraints generation in CI - #41383

Merged
kaxil merged 1 commit into
apache:mainfrom
potiuk:limit-parallelism-for-constraint-generation
Aug 12, 2024
Merged

Limit parallelism for constraints generation in CI#41383
kaxil merged 1 commit into
apache:mainfrom
potiuk:limit-parallelism-for-constraint-generation

Conversation

@potiuk

Copy link
Copy Markdown
Member

Apparently constraints generation in 4 parallel docker containers causes "no space left on device" error for public runners. This one limits parallelism to 3 to limit disk usage


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiukpotiuk added the upgrade to newer dependencies If set, upgrade to newer dependencies is forced label Aug 11, 2024
@potiuk
potiuk requested review from ashb and kaxil as code ownersAugust 11, 2024 19:37
@potiuk

Copy link
Copy Markdown
MemberAuthor

Example error here (in main - likely will affect v2-10-test/stble as well):

https://github.com/apache/airflow/actions/runs/10340160246/job/28620494570#step:10:5663

 src/krb5/_ccache.c:37124:1: fatal error: error writing to /tmp/ccGVRTgE.s: No space left on device

@potiuk
potiukforce-pushed the limit-parallelism-for-constraint-generation branch from c5ef553 to 8b69b97CompareAugust 11, 2024 19:57
Comment threadairflow/providers/microsoft/azure/hooks/wasb.py Outdated
@potiuk

Copy link
Copy Markdown
MemberAuthor

Extracted the MyPy fixes to #41386

@potiuk
potiukforce-pushed the limit-parallelism-for-constraint-generation branch from e64166c to 3838596CompareAugust 11, 2024 21:22
@potiukpotiuk added the all versions If set, the CI build will be forced to use all versions of Python/K8S/DBs label Aug 11, 2024
@potiuk
potiukforce-pushed the limit-parallelism-for-constraint-generation branch from 3838596 to 2341187CompareAugust 11, 2024 21:22
Apparently constraints generation in 4 parallel docker containers causes
"no space left on device" error for public runners. This one limits
parallelism to 3 to limit disk usage
@potiuk
potiukforce-pushed the limit-parallelism-for-constraint-generation branch from 2341187 to b0405b2CompareAugust 11, 2024 21:36
@potiukpotiuk added the canary When set on PR running from apache repo - behave as canary run label Aug 11, 2024
@potiukpotiuk closed this Aug 11, 2024
@potiukpotiuk reopened this Aug 11, 2024
@potiuk

Copy link
Copy Markdown
MemberAuthor

Looks like it works and should bring back main to be green

@kaxil
kaxil merged commit fffd41e into apache:mainAug 12, 2024
romsharon98 pushed a commit to romsharon98/airflow that referenced this pull request Aug 20, 2024
Apparently constraints generation in 4 parallel docker containers causes
"no space left on device" error for public runners. This one limits
parallelism to 3 to limit disk usage
@potiuk
potiuk deleted the limit-parallelism-for-constraint-generation branch April 21, 2026 15:58
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

all versionsIf set, the CI build will be forced to use all versions of Python/K8S/DBsarea:dev-toolscanaryWhen set on PR running from apache repo - behave as canary runupgrade to newer dependenciesIf set, upgrade to newer dependencies is forced

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@potiuk@kaxil