gh-62432: unittest runner: Exit code 5 if no tests were run by stefanor · Pull Request #102051 · python/cpython

stefanor · 2023-02-19T16:31:32Z

As discussed in https://discuss.python.org/t/unittest-fail-if-zero-tests-were-discovered/21498/7

It is common for test runner misconfiguration to fail to find any tests,
this should be an error.

Issue: running a suite with no tests is not an error #62432

As discussed in https://discuss.python.org/t/unittest-fail-if-zero-tests-were-discovered/21498/7 It is common for test runner misconfiguration to fail to find any tests, this should be an error. Fixes: python#62432

terryjreedy · 2023-02-20T03:20:02Z

Tests/macOS is routinely failing; ignore currently.

This is a design bug fix, hence enhancement (feature change), as much as I would like to pretend otherwise.

stefanor · 2023-02-20T11:52:11Z

@stefanor stefanor requested review from hugovk and removed request for gpshead and ezio-melotti

Urgh, that removal wasn't intentional, I just hit the re-request review button.

stefanor · 2023-04-26T21:48:01Z

@gpshead: Ping? (it's been 2 months)

stefanor · 2023-04-27T11:50:41Z

Thank you!

- Either test filtering is used - Sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+. Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12. However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used. This is the existing behavior before Python 3.12. Also test absl-py on Python 3.12. PiperOrigin-RevId: 565190992

- Either test filtering is used - Sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+. Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12. However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12. Also test absl-py on Python 3.12. PiperOrigin-RevId: 565190992

- Either test filtering is used - Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+. Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12. However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12. Also test absl-py on Python 3.12. PiperOrigin-RevId: 565190992

- Either test filtering is used - Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+. Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12. However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12. Also test absl-py on Python 3.12. PiperOrigin-RevId: 565771935

vstinner · 2024-01-08T23:31:47Z

The new Python 3.12 message surprises users when all tests are skipped: see issue #113661.

fishy · 2024-01-08T23:54:30Z

The PR description says:

It is common for test runner misconfiguration to fail to find any tests

Which seems to suggest that it should not fail test if all tests are skipped, as skipped tests are still "found", just intentionally skipped from running.

stefanor · 2024-01-09T00:05:44Z

Which seems to suggest that it should not fail test if all tests are skipped, as skipped tests are still "found", just intentionally skipped from running.

Yeah, I'd agree with that.

to ensure that the CI `colcon test` run works, because with a change to python 3.12 the `unittest` standard library used by default with colcon now exits with an error code of 5 for an empty test suite. See: colcon/colcon-core#678 See: python/cpython#102051

to ensure that the CI `colcon test` run works, because with a change to python 3.12 the `unittest` standard library used by default with colcon now exits with an error code of 5 for an empty test suite. Additionally removed old ros 1 integration tests. See: colcon/colcon-core#678 See: python/cpython#102051

to ensure that the CI `colcon test` run works, because with a change to python 3.12 the `unittest` standard library used by default with colcon now exits with an error code of 5 for an empty test suite. See: colcon/colcon-core#678 See: python/cpython#102051

to ensure that the CI `colcon test` run works, because with a change to python 3.12 the `unittest` standard library used by default with colcon now exits with an error code of 5 for an empty test suite. Additionally removed old ros 1 integration tests. See: colcon/colcon-core#678 See: python/cpython#102051

stefanor added 3 commits February 19, 2023 11:41

Refactor test_program to allow for multiple test suites
0a2007d

Refactor test_program to use assertRaises as a context manager
6532200

unittest: Print an error message if no tests ran
ebba337

bedevere-bot added the awaiting review label Feb 19, 2023

bedevere-bot mentioned this pull request Feb 19, 2023
running a suite with no tests is not an error #62432
Closed

stefanor force-pushed the fail-zero-tests branch from eab4b21 to d8b54a3Compare February 19, 2023 16:37

stefanor added 3 commits February 19, 2023 12:41

unittest runner: Exit code 5 if no tests were run
5e1a88d
As discussed in https://discuss.python.org/t/unittest-fail-if-zero-tests-were-discovered/21498/7 It is common for test runner misconfiguration to fail to find any tests, this should be an error. Fixes: python#62432

Count the test run in our stub test so that the suite succeeds
fb67714

Update ACKs
0687ca8

stefanor force-pushed the fail-zero-tests branch from d8b54a3 to 0687ca8Compare February 19, 2023 16:41

terryjreedy requested review from ezio-melotti and gpshead February 20, 2023 03:13

terryjreedy added type-feature A feature request or enhancement tests Tests in the Lib/test dir labels Feb 20, 2023

hugovk reviewed Feb 20, 2023
View reviewed changes

stefanor added 4 commits February 20, 2023 07:45

Put success first in the if ladder
3ac2e71

NO TESTS RAN, in past tense
7470779

The parameter name was testcase
ecb363c

Drop a comma
7d6c322

stefanor requested review from hugovk and removed request for ezio-melotti and gpshead February 20, 2023 11:51

hugovk requested review from ezio-melotti and gpshead February 20, 2023 12:42

Merge branch 'main' into fail-zero-tests
058e22c

gpshead self-assigned this Feb 20, 2023

Merge branch 'main' into fail-zero-tests
455ca8b

bedevere-bot added the awaiting merge label Apr 27, 2023

gpshead enabled auto-merge (squash) April 27, 2023 01:04

gpshead merged commit 76632b8 into python:mainApr 27, 2023

bedevere-bot removed the awaiting merge label Apr 27, 2023

stefanor deleted the fail-zero-tests branch April 27, 2023 11:49

gpshead added the 3.12 only security fixes label Jun 13, 2023

copybara-serviceBot mentioned this pull request Sep 14, 2023
absltest: do not fail tests on Python 3.12+ when no tests ran and: abseil/abseil-py#253
Merged

zware mentioned this pull request Jan 3, 2024
3.12 starts to return an error when all tests in a test file are skipped #113661
Closed

cottsay mentioned this pull request Nov 25, 2024
NO TESTS RAN failure on Noble for an ament_python package colcon/colcon-core#678
Open

texhnolyze mentioned this pull request Dec 16, 2024
Move to pytest testing bit-bots/udp_bridge#39
Merged
6 tasks

texhnolyze mentioned this pull request Dec 16, 2024
Configure pytest testing for vizualization bit-bots/dynamic_stack_decider#111
Merged
6 tasks

Navigation Menu

Uh oh!

Uh oh!

gh-62432: unittest runner: Exit code 5 if no tests were run - #102051

gh-62432: unittest runner: Exit code 5 if no tests were run#102051
gpshead merged 13 commits into
python:mainfrom
stefanor:fail-zero-tests

stefanor commented Feb 19, 2023•
edited by bedevere-bot
Loading

Uh oh!

terryjreedy commented Feb 20, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stefanor commented Feb 20, 2023

Uh oh!

stefanor commented Apr 26, 2023

Uh oh!

stefanor commented Apr 27, 2023

Uh oh!

vstinner commented Jan 8, 2024

Uh oh!

fishy commented Jan 8, 2024

Uh oh!

stefanor commented Jan 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Uh oh!

Uh oh!

Conversation

stefanor commented Feb 19, 2023• edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

terryjreedy commented Feb 20, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stefanor commented Feb 20, 2023

Uh oh!

stefanor commented Apr 26, 2023

Uh oh!

stefanor commented Apr 27, 2023

Uh oh!

vstinner commented Jan 8, 2024

Uh oh!

fishy commented Jan 8, 2024

Uh oh!

stefanor commented Jan 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

stefanor commented Feb 19, 2023•
edited by bedevere-bot
Loading