Fail fast if special module like typing not from typeshed

[drakedevel]

mypy generally crashes with an internal error if it winds up trying to handle a file named typing.py, apparently due to confusion with the built-in typing module. Here's the simplest test case I've found (tested on 0.4.3 and master):

(mypyve) vm:/tmp/test1$ cat typing.py 
from collections import namedtuple

x = namedtuple('x', [])
(mypyve) vm:/tmp/test1$ mypy -s typing.py 
Traceback (most recent call last):
 File "/tmp/mypyve/bin/mypy", line 6, in <module>
 main(__file__)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/main.py", line 40, in main
 res = type_check_only(sources, bin_dir, options)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/main.py", line 81, in type_check_only
 options=options)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 177, in build
 dispatch(sources, manager)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 1338, in dispatch
 process_graph(graph, manager)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 1481, in process_graph
 process_stale_scc(graph, scc)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 1554, in process_stale_scc
 graph[id].semantic_analysis()
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 1303, in semantic_analysis
 self.manager.semantic_analyzer.visit_file(self.tree, self.xpath)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 243, in visit_file
 self.accept(d)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 2307, in accept
 node.accept(self)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/nodes.py", line 761, in accept
 return visitor.visit_assignment_stmt(self)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 1076, in visit_assignment_stmt
 self.process_namedtuple_definition(s)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 1416, in process_namedtuple_definition
 named_tuple = self.check_namedtuple(s.rvalue, name)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 1454, in check_namedtuple
 info = self.build_namedtuple_typeinfo(name, items, types)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 1543, in build_namedtuple_typeinfo
 info.mro = [info] + info.tuple_type.fallback.type.mro
TypeError: can only concatenate list (not "NoneType") to list

*** INTERNAL ERROR ***

typing.py:3: error: Internal error -- please report a bug at https://github.com/python/mypy/issues

NOTE: you can use "mypy --pdb ..." to drop into the debugger when this happens.

This is the same error message reported in a comment in #1293. The root cause is that mypy ships a file called typing.py in site-packages, so if site-packages winds up on the path (even implicitly, even with -s), it dies:

(mypyve) vm:/tmp/mypyve/lib/python3.4/site-packages$ mypy -s typing.py 
Traceback (most recent call last):
...
(mypyve) vm:/tmp/mypyve/lib/python3.4/site-packages$ touch hello.py
(mypyve) vm:/tmp/mypyve/lib/python3.4/site-packages$ mypy -s hello.py 
Traceback (most recent call last):
...

This makes it impossible to analyze a package along with even a subset of its dependencies as the implicit addition to MYPYPATH results in an internal error.

[gvanrossum]

We apologize for the crash and will look into producing a more meaningful error message.

Is the file typing.py that you have a copy of the official typing.py file or an unrelated file with an unfortunate name clash?

When analyzing a package and its dependencies it's usually best not to include site-packages -- there's an explicit warning about this in the docs (see the very bottom of http://mypy.readthedocs.io/en/latest/basics.html).

[drakedevel]

The first example simulates the "unfortunate name clash" case, contents of the file displayed. The second example it is the official typing.py taken from git master shortly before I posted this issue.

Note that in neither case did I explicitly set MYPYPATH. Just referencing a file in the site-packages directory (which implicitly adds it to the search path) is sufficient.

I ran into this while setting up mypy for the first time after having added the venv site-packages to the MYPYPATH variable in response to the tremendous number of import errors from the first run for libraries that I knew were installed. I only read mypy --help so I didn't see the warning in the docs until significantly later.

I thought it would be reasonable to "whitelist" site-packages that were known to be analyzable, with an invocation like mypy -s src/ venv/.../site-packages/tiny_lib/. If library authors add types going forward, this seems like a preferable approach to out-of-tree stubs. Unfortunately, today this adds the whole of site-packages to the path implicitly, and then this bug causes an internal error.

[gvanrossum]

Yeah, the feature of automatically adding the parent directory to the import path fails spectacularly for site-packages -- perhaps we should just exclude site-packages from this behavior even if we preserve it for other locations (because it is often truly useful as well!).

We've also discussed the idea of a flag that adds a stdlib module to the packages to be analyzed (without this auto-path-setting behavior).

When you have a tremendous number of failing imports, you may also consider --silent-imports -- see here for the docs: http://mypy.readthedocs.io/en/latest/command_line.html#following-imports-or-not

[drakedevel]

Yeah, I started using -s after I found that page. Our codebase has a lot of dependencies, including sqlalchemy and a few other heavyweights so it looks like there's no other options for now. I was a little surprised to discover that even with -s, even if I specify only a single file to analyze, it will process other files in the path (like typing.py!) It makes sense but the semantics could be more clearly documented.

[gvanrossum]

Actually typing.py is probably the exception. What does -v output for a single file? There's a little cycle of dependencies around builtins (at least builtins, typing, and abc) but typically these are gotten from typeshed stubs, not from .py files.

[drakedevel]

Is this what you wanted?

(mypyve) vm:/tmp/test5$ cat typing.py 
from collections import namedtuple
x = namedtuple('x', [])
(mypyve) vm:/tmp/test5$ mypy -vs typing.py 
0.000:LOG: Using new dependency manager
0.000:LOG: Parsing typing.py (typing)
0.001:LOG: Parsing /tmp/mypyve/lib/mypy/typeshed/stdlib/3/collections/__init__.pyi (collections)
0.010:LOG: Parsing /tmp/mypyve/lib/mypy/typeshed/stdlib/3/builtins.pyi (builtins)
0.089:LOG: Parsing /tmp/mypyve/lib/mypy/typeshed/stdlib/3/abc.pyi (abc)
0.090:LOG: Parsing /tmp/mypyve/lib/mypy/typeshed/stdlib/3/sys.pyi (sys)
0.098:LOG: Parsing /tmp/mypyve/lib/mypy/typeshed/stdlib/3/types.pyi (types)
0.109:LOG: Parsing /tmp/mypyve/lib/mypy/typeshed/stdlib/3/_importlib_modulespec.pyi (_importlib_modulespec)
0.112:LOG: Loaded graph with 7 nodes
0.113:LOG: Found 1 SCCs; largest has 7 nodes
0.113:LOG: Processing SCC of size 7 (abc collections typing _importlib_modulespec types sys builtins) as inherently stale (_importlib_modulespec abc builtins collections sys types typing)
Traceback (most recent call last):
 File "/tmp/mypyve/bin/mypy", line 6, in <module>
 main(__file__)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/main.py", line 40, in main
 res = type_check_only(sources, bin_dir, options)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/main.py", line 81, in type_check_only
 options=options)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 177, in build
 dispatch(sources, manager)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 1338, in dispatch
 process_graph(graph, manager)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 1481, in process_graph
 process_stale_scc(graph, scc)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 1554, in process_stale_scc
 graph[id].semantic_analysis()
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 1303, in semantic_analysis
 self.manager.semantic_analyzer.visit_file(self.tree, self.xpath)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 243, in visit_file
 self.accept(d)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 2307, in accept
 node.accept(self)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/nodes.py", line 761, in accept
 return visitor.visit_assignment_stmt(self)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 1076, in visit_assignment_stmt
 self.process_namedtuple_definition(s)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 1416, in process_namedtuple_definition
 named_tuple = self.check_namedtuple(s.rvalue, name)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 1454, in check_namedtuple
 info = self.build_namedtuple_typeinfo(name, items, types)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 1543, in build_namedtuple_typeinfo
 info.mro = [info] + info.tuple_type.fallback.type.mro
TypeError: can only concatenate list (not "NoneType") to list

*** INTERNAL ERROR ***

typing.py:2: error: Internal error -- please report a bug at https://github.com/python/mypy/issues

NOTE: you can use "mypy --pdb ..." to drop into the debugger when this happens.
0.119:LOG: Build finished with 7 modules, 0 types, and 57 errors

[gvanrossum]

That one's not so surprising because you pass a bogus typing.py on the command line -- this gets substituted for the stub in typeshed so that whatever builtins needs from typing isn't there, and hence the crash. I thought that you were reporting problems when passing foo.py on the command line where there's a bogus typing.py in the same directory when also passing -s. Also, what version of mypy? I can repro this with 0.4.3-dev but not with 0.4.3.

[drakedevel]

Here's the case with that same typing.py alongside an empty hello.py with 0.4.3:

(app) vm:/tmp/test5$ cat typing.py 
from collections import namedtuple
x = namedtuple('x', [])
(app) vm:/tmp/test5$ cat hello.py 
(app) vm:/tmp/test5$ mypy --version
mypy 0.4.3
(app) vm:/tmp/test5$ mypy -sv hello.py 
0.000:LOG: Using new dependency manager
0.000:LOG: Parsing hello.py (hello)
0.001:LOG: Parsing /home/adrake/ve/app/lib/mypy/typeshed/stdlib/3/builtins.pyi (builtins)
0.077:LOG: Parsing /tmp/test5/typing.py (typing)
0.078:LOG: Parsing /home/adrake/ve/app/lib/mypy/typeshed/stdlib/3/abc.pyi (abc)
0.079:LOG: Parsing /home/adrake/ve/app/lib/mypy/typeshed/stdlib/3/collections/__init__.pyi (collections)
0.087:LOG: Parsing /home/adrake/ve/app/lib/mypy/typeshed/stdlib/3/sys.pyi (sys)
0.096:LOG: Parsing /home/adrake/ve/app/lib/mypy/typeshed/stdlib/3/types.pyi (types)
0.106:LOG: Parsing /home/adrake/ve/app/lib/mypy/typeshed/stdlib/3/_importlib_modulespec.pyi (_importlib_modulespec)
0.109:LOG: Loaded graph with 8 nodes
0.109:LOG: Found 2 SCCs; largest has 7 nodes
0.110:LOG: Processing SCC of size 7 (collections abc typing _importlib_modulespec types sys builtins) as inherently stale (_importlib_modulespec abc builtins collections sys types typing)
Traceback (most recent call last):
 File "/home/adrake/ve/app/bin/mypy", line 6, in <module>
 main(__file__)
 File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/main.py", line 40, in main
 res = type_check_only(sources, bin_dir, options)
 File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/main.py", line 81, in type_check_only
 options=options)
 File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/build.py", line 177, in build
 dispatch(sources, manager)
 File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/build.py", line 1323, in dispatch
 process_graph(graph, manager)
 File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/build.py", line 1461, in process_graph
 process_stale_scc(graph, scc)
 File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/build.py", line 1534, in process_stale_scc
 graph[id].semantic_analysis()
 File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/build.py", line 1288, in semantic_analysis
 self.manager.semantic_analyzer.visit_file(self.tree, self.xpath)
 File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/semanal.py", line 243, in visit_file
 self.accept(d)
 File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/semanal.py", line 2306, in accept
 node.accept(self)
 File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/nodes.py", line 761, in accept
 return visitor.visit_assignment_stmt(self)
 File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/semanal.py", line 1075, in visit_assignment_stmt
 self.process_namedtuple_definition(s)
 File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/semanal.py", line 1415, in process_namedtuple_definition
 named_tuple = self.check_namedtuple(s.rvalue, name)
 File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/semanal.py", line 1453, in check_namedtuple
 info = self.build_namedtuple_typeinfo(name, items, types)
 File "/home/adrake/ve/app/lib/python3.4/site-packages/mypy/semanal.py", line 1542, in build_namedtuple_typeinfo
 info.mro = [info] + info.tuple_type.fallback.type.mro
TypeError: can only concatenate list (not "NoneType") to list

*** INTERNAL ERROR ***

/tmp/test5/typing.py:2: error: Internal error -- please report a bug at https://github.com/python/mypy/issues

NOTE: you can use "mypy --pdb ..." to drop into the debugger when this happens.
0.116:LOG: Build finished with 8 modules, 0 types, and 57 errors
(app) vm:/tmp/test5$ 

And here's git master, same file setup:

(mypyve) vm:/tmp/test5$ mypy --version
mypy 0.4.4-dev
(mypyve) vm:/tmp/test5$ mypy -sv hello.py 
0.000:LOG: Using new dependency manager
0.000:LOG: Parsing hello.py (hello)
0.001:LOG: Parsing /tmp/mypyve/lib/mypy/typeshed/stdlib/3/builtins.pyi (builtins)
0.079:LOG: Parsing /tmp/test5/typing.py (typing)
0.080:LOG: Parsing /tmp/mypyve/lib/mypy/typeshed/stdlib/3/abc.pyi (abc)
0.081:LOG: Parsing /tmp/mypyve/lib/mypy/typeshed/stdlib/3/collections/__init__.pyi (collections)
0.089:LOG: Parsing /tmp/mypyve/lib/mypy/typeshed/stdlib/3/sys.pyi (sys)
0.096:LOG: Parsing /tmp/mypyve/lib/mypy/typeshed/stdlib/3/types.pyi (types)
0.107:LOG: Parsing /tmp/mypyve/lib/mypy/typeshed/stdlib/3/_importlib_modulespec.pyi (_importlib_modulespec)
0.110:LOG: Loaded graph with 8 nodes
0.110:LOG: Found 2 SCCs; largest has 7 nodes
0.110:LOG: Processing SCC of size 7 (collections abc typing _importlib_modulespec types sys builtins) as inherently stale (_importlib_modulespec abc builtins collections sys types typing)
Traceback (most recent call last):
 File "/tmp/mypyve/bin/mypy", line 6, in <module>
 main(__file__)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/main.py", line 40, in main
 res = type_check_only(sources, bin_dir, options)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/main.py", line 81, in type_check_only
 options=options)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 177, in build
 dispatch(sources, manager)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 1338, in dispatch
 process_graph(graph, manager)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 1481, in process_graph
 process_stale_scc(graph, scc)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 1554, in process_stale_scc
 graph[id].semantic_analysis()
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/build.py", line 1303, in semantic_analysis
 self.manager.semantic_analyzer.visit_file(self.tree, self.xpath)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 243, in visit_file
 self.accept(d)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 2307, in accept
 node.accept(self)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/nodes.py", line 761, in accept
 return visitor.visit_assignment_stmt(self)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 1076, in visit_assignment_stmt
 self.process_namedtuple_definition(s)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 1416, in process_namedtuple_definition
 named_tuple = self.check_namedtuple(s.rvalue, name)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 1454, in check_namedtuple
 info = self.build_namedtuple_typeinfo(name, items, types)
 File "/tmp/mypyve/lib/python3.4/site-packages/mypy/semanal.py", line 1543, in build_namedtuple_typeinfo
 info.mro = [info] + info.tuple_type.fallback.type.mro
TypeError: can only concatenate list (not "NoneType") to list

*** INTERNAL ERROR ***

/tmp/test5/typing.py:2: error: Internal error -- please report a bug at https://github.com/python/mypy/issues

NOTE: you can use "mypy --pdb ..." to drop into the debugger when this happens.
0.116:LOG: Build finished with 8 modules, 0 types, and 57 errors

[gvanrossum]

Hm, the only reasonable way that I can think to avoid such crashes is to explicitly check that the stub for typing, builtins and everything else in that initial import cycle is actually loaded from typeshed. Hopefully you can work around it now!

[drakedevel]

Was there a workaround for wanting to include a venv dependency in the analysis? Or is that future work

[gvanrossum]

That's future work -- a cumbersome workaround would be to just download and untar the tarball (or unzip the zipfile) for the package's source distro, or clone the repo, and point mypy at the package in the source tree. For most reasonable source distributions that should work.

[gvanrossum]

See PR #1895 -- once that lands anybody can install typeshed additions.

[gvanrossum]

I don't think finding a solution that avoids the initially-reported crash has much priority. When you feed mypy the wrong typing.py (or builtins.py, or whatever) it's going to be unhappy. So I'm moving the milestone up.