Script to run all checks locally

[Avasam]

Closes#8686

There's still an issue that needs to be fixed with the mypy test (see #8797), and I would like to be able to seamlessly run the pytype test on windows, if that's even feasible.

Summary output:

Please let me know how to better word the "next steps" in create_baseline_stubs.

Notes of things to do in follow-up PR(s):

• Different return codes for different errors in pyright_test and regr_test (Script to run all checks locally #8798 (comment))
• Run pytype_test using WSL on Windows if a distribution is installed (Script to run all checks locally #8798 (comment))
• Run tests for more than 1 stubs (Script to run all checks locally #8798 (comment))
• Improve windows docs (Script to run all checks locally #8798 (comment))
• Consider adding an optional CLI argument to automatically run or skip third-party stubtest (Script to run all checks locally #8798 (comment))

[AlexWaygood]

Thanks! This looks like a great start.

I have a few questions, a few suggestions, and a few nits ;)

[AlexWaygood]

Here's the output currently if I try running `python scripts/runtests.py redis`.

(.venv) C:\Users\alexw\coding\typeshed>python scripts/runtests.py stubs/redisLooks good! ✨ 🍰 ✨50 files left unchanged.All done! ✨ 🍰 ✨50 files left unchanged.error finding npx; is Node.js installed?*** Testing Python 3.11 on win32Testing third-party packages...Running mypy --python-version 3.11 --show-traceback --warn-incomplete-stub --show-error-codes --no-error-summary --platform win32 --no-site-packages --custom-typeshed-dir C:\Users\alexw\coding\typeshed --no-implicit-optional --disallow-untyped-decorators --disallow-any-generics --strict-equality --enable-error-code ignore-without-code --config-file /tmp/...testing redis (50 files)... success*** Testing Python 3.10 on win32Testing third-party packages...Running mypy --python-version 3.10 --show-traceback --warn-incomplete-stub --show-error-codes --no-error-summary --platform win32 --no-site-packages --custom-typeshed-dir C:\Users\alexw\coding\typeshed --no-implicit-optional --disallow-untyped-decorators --disallow-any-generics --strict-equality --enable-error-code ignore-without-code --config-file /tmp/...testing redis (50 files)... success*** Testing Python 3.9 on win32Testing third-party packages...Running mypy --python-version 3.9 --show-traceback --warn-incomplete-stub --show-error-codes --no-error-summary --platform win32 --no-site-packages --custom-typeshed-dir C:\Users\alexw\coding\typeshed --no-implicit-optional --disallow-untyped-decorators --disallow-any-generics --strict-equality --enable-error-code ignore-without-code --config-file /tmp/...testing redis (50 files)... success*** Testing Python 3.8 on win32Testing third-party packages...Running mypy --python-version 3.8 --show-traceback --warn-incomplete-stub --show-error-codes --no-error-summary --platform win32 --no-site-packages --custom-typeshed-dir C:\Users\alexw\coding\typeshed --no-implicit-optional --disallow-untyped-decorators --disallow-any-generics --strict-equality --enable-error-code ignore-without-code --config-file /tmp/...testing redis (50 files)... success*** Testing Python 3.7 on win32Testing third-party packages...Running mypy --python-version 3.7 --show-traceback --warn-incomplete-stub --show-error-codes --no-error-summary --platform win32 --no-site-packages --custom-typeshed-dir C:\Users\alexw\coding\typeshed --no-implicit-optional --disallow-untyped-decorators --disallow-any-generics --strict-equality --enable-error-code ignore-without-code --config-file /tmp/...testing redis (50 files)... success--- success, 250 files checked ---redis... successSkipping pytype on Windows. You can run the test with WSL.usage: regr_test.py [-h] [--all] [--platform [{linux,darwin,win32} ...]] [-p [{3.11,3.10,3.9,3.8,3.7} ...]] [packages_to_test ...]regr_test.py: error: argument packages_to_test: No test cases found for 'redis'!One or more tests failed. See above for details.flake8: SuccessCheck consistent: SuccessCheck new syntax: Successpyright: Failedmypy: Successstubtest: Successpytype: SkippedRegression test: Success

Some notes:

• It's not obvious currently which tests are running when. pycln and black have very similar "success!!1!1!" messages, for example, and isort doesn't have one at all. It might be nice to add some print() statements before each subprocess.run() call: "Running black...", etc.
• It looks like pyright_test.py is still marked as "Failed" if npx is not available; it should be marked as "skipped".
• regr_test.py is correctly marked as "Skipped", but the error message from argparse is still printed to the terminal, which makes it look like it failed. If we use capture_output=True in the subprocess.run() call, we can stop any output being printed to the screen. We can then manually print the captured output in stdout and stderr, if and only if test cases were found. (We should do the same thing with the subprocess.run() call for pyright_test.py, as well.)

[AlexWaygood]

(Nit): we currently get a bit of an ugly traceback if the script is interrupted halfway through:

(.venv) C:\Users\alexw\coding\typeshed>python scripts/runtests.py stubs/redisRunning pycln...Looks good! ✨ 🍰 ✨50 files left unchanged.Traceback (most recent call last): File "C:\Users\alexw\coding\typeshed\scripts\runtests.py", line 55, in <module> subprocess.run([sys.executable, "-m", "pycln", path, "--all"]) File "C:\Users\alexw\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 503, in run stdout, stderr = process.communicate(input, timeout=timeout) File "C:\Users\alexw\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 1144, in communicate self.wait() File "C:\Users\alexw\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 1207, in wait return self._wait(timeout=timeout) File "C:\Users\alexw\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 1488, in _wait result = _winapi.WaitForSingleObject(self._handle,KeyboardInterrupt^C

It might be nice to wrap the whole if __name__ == "__main__" block inside a main() function, and then put the whole main() function inside a try/except block to catch KeyboardInterrupts, similar to how we do it in some of our other test scripts:

typeshed/tests/regr_test.py

Lines 179 to 185 in 380022c

	if__name__=="__main__":
	try:
	code=main()
	exceptKeyboardInterrupt:
	print_error("Test aborted due to KeyboardInterrupt!")
	code=1
	raiseSystemExit(code)

[Avasam]

(Nit): we currently get a bit of an ugly traceback if the script is interrupted halfway through:
[...]

A traceback can still appear, but it's clearer why.

Unless maybe we start piping the output of all tests, checking for a KeyboardInterrupt and then printing the output. I don't think that's really worth it.
As a user, you did interrupt a script mid-execution after all.

[AlexWaygood]

Unless maybe we start piping the output of all tests, checking for a KeyboardInterrupt and then printing the output. I don't think that's really worth it.

Agreed, it looks good now!

[AlexWaygood]

Some very minor suggestions about some of the terminal output

[AlexWaygood]

Awesome, thank you! Just one last comment, but other than that -- I've run out of things to say!

I'll leave this open for a day or two in case any other maintainers want to take a look, but this looks great to me :)