regr_test.py: improve interaction between --all and --quiet flags

[JelleZijlstra]

cf. #9348 cc @jab

[JelleZijlstra]

pyright 1.1.278
/home/runner/work/typeshed/typeshed/test_cases/stdlib/typing/check_regression_issue_9296.py
/home/runner/work/typeshed/typeshed/test_cases/stdlib/typing/check_regression_issue_9296.py:16:12 - error: Cannot access member "regixster" for type "Type[MyKeysView[KT@MyKeysView]]"
Member "regixster" is unknown (reportGeneralTypeIssues)
/home/runner/work/typeshed/typeshed/test_cases/stdlib/typing/check_regression_issue_9296.py:16:1 - error: Type of "regixster" is unknown (reportUnknownMemberType)

It works as expected.

[AlexWaygood]

And we have a failing mypy run as well: https://github.com/python/typeshed/actions/runs/3665455929/jobs/6196543252

(Though it looks like I could probably sink a little more work into improving that traceback)

[JelleZijlstra]

Is the reason the failure shows up ~16 times that we run mypy in a matrix (Python version, OS) and report the failure every time?

[AlexWaygood]

Is the reason the failure shows up ~16 times that we run mypy in a matrix (Python version, OS) and report the failure every time?

Yup! Just pushed a commit to this PR branch that hopefully clarifies things a bit?

The issue was that the new --quiet flag didn't really interact very well with the --all flag

[JelleZijlstra]

Nice, feel free to hijack this PR for your change, or just open a new PR.

A more radical change could be to show only the first failure if there are multiple identical ones. After all, in most cases test cases are going to fail in the same way on multiple runs.

[AlexWaygood]

After all, in most cases test cases are going to fail in the same way on multiple runs.

This is true, but it won't always be the case. I feel like this solution is ~good enough for now. I'm gonna commence hijacking ;)