Skip to content

tests: cleanup and ci hardening - #2397

Merged
henryiii merged 11 commits into
pybind:masterfrom
henryiii:tests/cleanup
Aug 16, 2020
Merged

tests: cleanup and ci hardening#2397
henryiii merged 11 commits into
pybind:masterfrom
henryiii:tests/cleanup

Conversation

@henryiii

@henryiiihenryiii commented Aug 15, 2020

Copy link
Copy Markdown
Collaborator

Refactor the tests. Use a consistent failing/skipping scheme.

  • Attempt to allow 3.9-dev to fail (not sure how it works)
  • Fail-fast diagnostic removed, fails fast (once) with the correct error message without a custom wrapper.
  • Using xfail where possible (skip means not-applicable, xfail means it should ideally pass)
  • Use importorskip instead of ugly custom skips
  • PyTest 3.1+ required instead of 3.0 (current is 6.0 for Py3, 4.6.9 for Py2) - better warning reporting
  • Use info.py to simplify skips/xfails/checks
  • Update links to PyPy issues
  • Several (7 or so) skipped PyPy tests are now passing (especially those requiring NumPy) (reported by xfail!)
  • Strings used for the skipif/xfails in most cases so that PyTest reports the condition causing the skip/xfail directly in the report.

Skips/xfails are more consistent, no more pytest.bug_in_pypy, pytest.unsupported_on_pypy_lt_6, etc. (PyPy 7+ required, so ignoring check for old PyPy verisions). No more injection into the pytest module. xfails are reported at the end and most are strictly required. We can use reasons or raises to improve the test reporting.

@henryiii
henryiii marked this pull request as draft August 16, 2020 02:47
@henryiii
henryiiiforce-pushed the tests/cleanup branch 5 times, most recently from 3d37b12 to ec86a98CompareAugust 16, 2020 04:24
Comment threadtests/info.py
Comment threadtests/test_eval.py Outdated
Comment thread.pre-commit-config.yaml Outdated

@bstaleticbstaletic left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like this PR.

Comment threadtests/pytest.ini Outdated
Comment threadtests/info.py
Comment threadtests/test_eval.py
@henryiii
henryiii marked this pull request as ready for review August 16, 2020 16:30
@henryiii

henryiii commented Aug 16, 2020

Copy link
Copy Markdown
CollaboratorAuthor

I believe 7 or so tests are now passing in PyPy that used to fail (and were hidden by skip instead of xfail), mostly due to better NumPy support in recent PyPy's.

Comment threadtests/test_methods_and_attributes.py Outdated
@YannickJadoul

YannickJadoul commented Aug 16, 2020

Copy link
Copy Markdown
Collaborator

I believe 7 or so tests are now passing in PyPy that used to fail (and were hidden by skip instead of xfail), mostly due to better NumPy support in recent PyPy's.

That's great to hear! :-)

@henryiii

Copy link
Copy Markdown
CollaboratorAuthor

If I get two approvals, I'll squash and merge. :)

@henryiiihenryiii changed the title Tests/cleanuptests: cleanup and ci hardeningAug 16, 2020

@bstaleticbstaletic left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great. Thanks for the PR.

@rwgkrwgk left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thanks Henry!

@henryiii
henryiii merged commit 4d9024e into pybind:masterAug 16, 2020
@henryiii
henryiii deleted the tests/cleanup branch August 16, 2020 20:02
YannickJadoul added a commit to YannickJadoul/pybind11 that referenced this pull request Aug 16, 2020
rwgk pushed a commit that referenced this pull request Aug 17, 2020
@henryiiihenryiii added the ci related to the CI system label Aug 18, 2020
@rwgkrwgk mentioned this pull request Feb 10, 2023
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cirelated to the CI system

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@henryiii@YannickJadoul@rwgk@bstaletic