Skip to content

Automate exclusion of providers in lowest-dependency tests - #41991

Merged
potiuk merged 1 commit into
apache:mainfrom
potiuk:fix-individual-pytests-when-ignoring-providers
Sep 4, 2024
Merged

Automate exclusion of providers in lowest-dependency tests#41991
potiuk merged 1 commit into
apache:mainfrom
potiuk:fix-individual-pytests-when-ignoring-providers

Conversation

@potiuk

Copy link
Copy Markdown
Member

The "lowest" dependency tests had "hard-coded" exclusion for those providers that were excluded in specific python versions.

This change automates it:

  • selective checks produce dictionary python -> excluded versions
  • the dictionary (as string) is passed to unit tests in actions
  • the dictionary (as string) is passed to breeze container as EXCLUDED_PROVIDERS variable
  • "is_provider_excluded.py" checks if the provider is excluded and exits with success if so

Related: #41555


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiukpotiuk added full tests needed We need to run full set of tests for this PR to merge all versions If set, the CI build will be forced to use all versions of Python/K8S/DBs labels Sep 4, 2024
@potiukpotiuk closed this Sep 4, 2024
@potiukpotiuk reopened this Sep 4, 2024
@potiuk
potiukforce-pushed the fix-individual-pytests-when-ignoring-providers branch 2 times, most recently from d49157d to 6d5c9c3CompareSeptember 4, 2024 11:16
@potiuk

Copy link
Copy Markdown
MemberAuthor

That one should also speed up "lowest dependencies" test in main - because we - erroneously excluded papermill from Python 3.12 (where Papermill had been added back a while ago) while we did not exclude apache.beam, so apache.beam tests actually (due to the way how the scripts work) would run ALL tests in a single thread.

@topherinternational

Copy link
Copy Markdown
Contributor

I think the followup to #41967 I am writing will also address this problem.

@potiukpotiuk added the include success outputs Whether to include success outputs in test results label Sep 4, 2024
@potiuk

Copy link
Copy Markdown
MemberAuthor

I think the followup to #41967 I am writing will also address this problem.

We can compare and choose better one :)

@potiuk
potiukforce-pushed the fix-individual-pytests-when-ignoring-providers branch 3 times, most recently from 0102e45 to 5ecbaafCompareSeptember 4, 2024 13:01
@potiuk

Copy link
Copy Markdown
MemberAuthor

Actually this one also solves "main" failures of Python 3.12 "Lowest dependencies" tests that I've seen recently!

@topherinternational

Copy link
Copy Markdown
Contributor

I think the followup to #41967 I am writing will also address this problem.

We can compare and choose better one :)

Or we can have defense in depth :D

@potiuk
potiukforce-pushed the fix-individual-pytests-when-ignoring-providers branch 2 times, most recently from 63075e2 to 34917b0CompareSeptember 4, 2024 16:05
The "lowest" dependency tests had "hard-coded" exclusion for
those providers that were excluded in specific python versions.
This change automates it:
* selective checks produce dictionary python -> excluded versions
* the dictionary (as string) is passed to unit tests in actions
* the dictionary (as string) is passed to breeze container
as EXCLUDED_PROVIDERS variable
* "is_provider_excluded.py" checks if the provider is excluded
and exits with success if so
Related: apache#41555
@potiuk
potiukforce-pushed the fix-individual-pytests-when-ignoring-providers branch from 34917b0 to 52227f2CompareSeptember 4, 2024 18:11
@potiuk
potiuk merged commit 28e7213 into apache:mainSep 4, 2024
@potiuk
potiuk deleted the fix-individual-pytests-when-ignoring-providers branch September 4, 2024 20:46
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

all versionsIf set, the CI build will be forced to use all versions of Python/K8S/DBsarea:dev-toolsfull tests neededWe need to run full set of tests for this PR to mergeinclude success outputsWhether to include success outputs in test results

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@potiuk@topherinternational@eladkal