Disable explicit mypy tests?

[shoyer]

What is your issue?

The explicit mypy tests introduced in #10038 seem to be quite fragile, e.g., they are currently broken at head and I know have been broken at head previously at least once. In neither case was there a real regression.

I see at least three issues with them:

1. They seem to be particularly prone to version skew issues where they fail on GitHub but pass locally due to slight differences in environments. I think the current failure is due to mis-matched NumPy versions. (This might be improved when we switch to pixi.)
2. They are really slow. Testing ~40 specific errors takes ~9 minutes on a single GitHub CI worker, which is not a great use of CI resources and clearly not scalable. Currently we use two CI workers just to test typing for the pipe method!
3. They are inherently very fragile, checking for exact messages printed by mypy. Pretty much any package update causes error messages to change, which means these tests break, whether or not there is a real underlying issue.

[shoyer]

Looking into this more carefully, I don't think the version skew issues are particular to these tests -- our standard mypy tests are also failing. But the second and third concerns (speed and fragility) are still quite valid.

[chuckwondo]

2. They are really slow. Testing ~40 specific errors takes ~9 minutes on a single GitHub CI worker, which is not a great use of CI resources and clearly not scalable. Currently we use two CI workers just to test typing for the pipe method!

Wow! Nine minutes!? Really? I know they're not the speediest, but running locally certainly takes nowhere near that much time, or at least I don't recall them being so slow as to question including them when I added them a while ago. Are the CI workers that small?

3. They are inherently very fragile, checking for exact messages printed by mypy. Pretty much any package update causes error messages to change, which means these tests break, whether or not there is a real underlying issue.

Yes, this seems to have always been an issue, and last I checked, there doesn't seem to be a convenient way to make matching more lenient/forgiving. The mypy pytest plugin does support some regex matching, but as far as I could find, it doesn't seem to be flexible enough to use to allow us to avoid such flakiness due to message drift over time.

---

Given the problems we're having with them, I wouldn't be opposed to removing them.

[max-sixty]

we could relegate them to a slow nightly run and then disable them if they repeatedly fail and aren't fixed

if no one wants to do that, then fine to disable from my side

I did think they were good and filled a nice gap in ensuring our typing was good (which used to be terrible, was sometimes totally wrong!)