stubtest: Detect abstract properties mismatches

[sobolevn]

Closes#13646

[AlexWaygood]

It would be good to also add tests for abstract read-write properties, as well as abstract read-only properties. Read-write properties have a different internal representation to read-only properties -- read-only properties are nodes.FuncDef objects, but read-write properties are nodes.OverloadedFuncDef objects

[AlexWaygood]

I just tried to see what impact this would have on typeshed, but this got in the way:

• stubtest --custom-typeshed-dir broken on master and 0.980 #13654

[sobolevn]

@AlexWaygood do you have time to try again? :)

[AlexWaygood]

I will!

[AlexWaygood]

The diff looks pretty good! Here's the new hits on my machine (on Windows, using Python 3.10.6):

Details

diff --git a/old.txt b/new.txt
index ac8937a9..3f60a744 100644
--- a/old.txt+++ b/new.txt@@ -1,9 +1,63 @@+error: importlib.abc.Traversable.name is inconsistent, runtime property is abstract but stub is not+Stub: at line 180 in file stdlib\importlib\abc.pyi+def (self: importlib.abc.Traversable) -> builtins.str+Runtime:+<abc.abstractproperty object at 0x0000021F9CBBE440>+
error: locale names exported from the stub do not correspond to the names exported at runtime. This is probably due to an inaccurate `__all__` in the stub or things being missing from the stub.
Stub: in file stdlib\locale.pyi
Names exported in the stub but not at runtime: ['LC_MESSAGES']
Runtime:
Names exported at runtime but not in the stub: []
+error: typing.IO.closed is inconsistent, runtime property is abstract but stub is not+Stub: at line 647 in file stdlib\typing.pyi+def (self: typing.IO[AnyStr`1]) -> builtins.bool+Runtime:+<property object at 0x0000021F9C981E90>++error: typing.IO.mode is inconsistent, runtime property is abstract but stub is not+Stub: at line 641 in file stdlib\typing.pyi+def (self: typing.IO[AnyStr`1]) -> builtins.str+Runtime:+<property object at 0x0000021F9C981DF0>++error: typing.IO.name is inconsistent, runtime property is abstract but stub is not+Stub: at line 643 in file stdlib\typing.pyi+def (self: typing.IO[AnyStr`1]) -> builtins.str+Runtime:+<property object at 0x0000021F9C981E40>++error: typing.TextIO.buffer is inconsistent, runtime property is abstract but stub is not+Stub: at line 694 in file stdlib\typing.pyi+def (self: typing.TextIO) -> typing.BinaryIO+Runtime:+<property object at 0x0000021F9C9822F0>++error: typing.TextIO.encoding is inconsistent, runtime property is abstract but stub is not+Stub: at line 696 in file stdlib\typing.pyi+def (self: typing.TextIO) -> builtins.str+Runtime:+<property object at 0x0000021F9C982340>++error: typing.TextIO.errors is inconsistent, runtime property is abstract but stub is not+Stub: at line 698 in file stdlib\typing.pyi+def (self: typing.TextIO) -> Union[builtins.str, None]+Runtime:+<property object at 0x0000021F9C982480>++error: typing.TextIO.line_buffering is inconsistent, runtime property is abstract but stub is not+Stub: at line 700 in file stdlib\typing.pyi+def (self: typing.TextIO) -> builtins.int+Runtime:+<property object at 0x0000021F9C9824D0>++error: typing.TextIO.newlines is inconsistent, runtime property is abstract but stub is not+Stub: at line 702 in file stdlib\typing.pyi+def (self: typing.TextIO) -> Any+Runtime:+<property object at 0x0000021F9C982520>+
note: unused allowlist entry sqlite3.dbapi2.Binary.__contains__
note: unused allowlist entry sqlite3.Binary.__contains__
note: unused allowlist entry enum.Enum.name
@@ -33,4 +87,4 @@ note: unused allowlist entry typing.SupportsInt.__init__
note: unused allowlist entry typing.SupportsRound.__init__
note: unused allowlist entry builtins.WindowsError.characters_written
note: unused allowlist entry winreg.error.characters_written
-Found 30 errors (checked 482 modules)+Found 39 errors (checked 482 modules)

[AlexWaygood]

LGTM!

[sobolevn]

Awesome! I will double check the output later tomorrow, send a PR to typeshed with new changes, and merge this :)

Thanks a lot for your time and help!

[sobolevn]

I don't quite like this one:

+error: importlib.abc.Traversable.name is inconsistent, runtime property is abstract but stub is not
+Stub: at line 180 in file stdlib\importlib\abc.pyi
+def (self: importlib.abc.Traversable) -> builtins.str
+Runtime:
+<abc.abstractproperty object at 0x0000021F9CBBE440>

[AlexWaygood]

I don't quite like this one:

+error: importlib.abc.Traversable.name is inconsistent, runtime property is abstract but stub is not
+Stub: at line 180 in file stdlib\importlib\abc.pyi
+def (self: importlib.abc.Traversable) -> builtins.str
+Runtime:
+<abc.abstractproperty object at 0x0000021F9CBBE440>

Arguably that should be fixed in CPython, rather than typeshed or mypy. abc.abstractproperty isn't really supported by mypy, but it's been deprecated since Python 3.3.

[hauntsaninja]

Agreed on fixing in CPython. I made a PR, but no one reviewed it and it eventually got closed for being stale

[sobolevn]

I've made a new one!
I am sometimes a bit sad of how many good PRs are becoming stale due to the fact that there's not enough time to review them 😞

[AlexWaygood]

The diff looks pretty good! Here's the new hits on my machine (on Windows, using Python 3.10.6):

Here's a slightly hacky script to produce these diffs, btw:

Details

importsubprocessimportsysimporttempfilefrompathlibimportPathfromtypingimportAnydefrun(*args: str, check: bool=True, **kwargs: Any) ->str:
result=subprocess.run(list(args), check=check, capture_output=True, text=True, **kwargs)
returnresult.stdoutor""defrun_stubtest(*, typeshed_dir: str) ->str:
"""This function is basically copied from typeshed's stubtest_stdlib.py."""allowlist_dir=Path(typeshed_dir, "tests", "stubtest_allowlists")
version_allowlist=f"py{sys.version_info.major}{sys.version_info.minor}.txt"platform_allowlist=f"{sys.platform}.txt"combined_allowlist=f"{sys.platform}-py{sys.version_info.major}{sys.version_info.minor}.txt"cmd= [
sys.executable,
"-m",
"mypy.stubtest",
"--check-typeshed",
"--custom-typeshed-dir",
typeshed_dir,
"--allowlist",
str(allowlist_dir/"py3_common.txt"),
"--allowlist",
str(allowlist_dir/version_allowlist),
]
if (allowlist_dir/platform_allowlist).exists():
cmd+= ["--allowlist", str(allowlist_dir/platform_allowlist)]
if (allowlist_dir/combined_allowlist).exists():
cmd+= ["--allowlist", str(allowlist_dir/combined_allowlist)]
ifsys.version_info< (3, 10):
# As discussed in https://github.com/python/typeshed/issues/3693, we only aim for# positional-only arg accuracy for the latest Python version.cmd.append("--ignore-positional-only")
returnrun(*cmd, check=False)
defcheckout_and_sync_with_master() ->None:
run("git", "checkout", "master")
run("git", "pull", "upstream", "master")
run("git", "push")
defget_stubtest_diff(*, pr_url: str, typeshed_dir: str) ->str:
print("Checking out and syncing mypy master...")
checkout_and_sync_with_master()
print("Running stubtest using mypy master...")
old_results=run_stubtest(typeshed_dir=typeshed_dir)
print("Checking out and syncing the PR...")
run("gh", "pr", "checkout", pr_url)
print("Running stubtest using the PR...")
new_results=run_stubtest(typeshed_dir=typeshed_dir)
print("Getting the diff...")
withtempfile.TemporaryDirectory() astd:
old, new=Path(td, "old.txt"), Path(td, "new.txt")
forpath, resultin [(old, old_results), (new, new_results)]:
path.write_text(result)
diff_results=run("git", "diff", "--no-index", str(old), str(new), check=False)
print("Cleaning up...")
checkout_and_sync_with_master()
return"\n".join(filter(None, diff_results.splitlines()))
defmain() ->None:
importargparseparser=argparse.ArgumentParser()
parser.add_argument("pr", type=str, help="URL to a stubtest PR to check")
parser.add_argument(
"--typeshed-dir", type=str, required=True, help="Path to a local clone of typeshed"
)
parser.add_argument(
"--output-file", type=str, help="File to write output to (writes to stdout if not set)"
)
args=parser.parse_args()
diff=get_stubtest_diff(pr_url=args.pr, typeshed_dir=args.typeshed_dir)
ifargs.output_file:
withopen(args.output_file, "w") asf:
f.write(diff)
print(f"Success! The diff has been written to {args.output_file!r}")
else:
print(diff)
if__name__=="__main__":
main()

[sobolevn]

Refs python/typeshed#8726

[sobolevn]

Thanks everyone!

[JelleZijlstra]

@sobolevn and @hauntsaninja Feel free to ping me if you need a CPython PR reviewed.