Defer all types whos metaclass is not ready

[sobolevn]

Finally! I've spent more than 8 hours on this test.
It illustrates a problem with sympy from #13565

[sobolevn]

I got it right to pass my test, but now we have different problems:

 deferred, incomplete, progress = semantic_analyze_target(
File "/Users/sobolev/Desktop/mypy/mypy/semanal_main.py", line 348, in semantic_analyze_target
infer_decorator_signature_if_simple(node, analyzer)
File "/Users/sobolev/.pyenv/versions/3.8.9/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/Users/sobolev/Desktop/mypy/mypy/build.py", line 2064, in wrap_context
yield
File "/Users/sobolev/Desktop/mypy/mypy/semanal_main.py", line 339, in semantic_analyze_target
analyzer.refresh_partial(
File "/Users/sobolev/Desktop/mypy/mypy/semanal.py", line 575, in refresh_partial
self.refresh_top_level(node)
File "/Users/sobolev/Desktop/mypy/mypy/semanal.py", line 584, in refresh_top_level
self.add_implicit_module_attrs(file_node)
File "/Users/sobolev/Desktop/mypy/mypy/semanal.py", line 616, in add_implicit_module_attrs
assert isinstance(node, TypeInfo)
AssertionError: 

Big 'thank you' goes to @ilevkivskyi for pushing me in the right direction.
Now I will need to find what is causing this new crash.

[sobolevn]

@ilevkivskyi maybe you have any ideas where we can safely call .add_implicit_module_attrs? Because I am out of ideas. I've tried:

1. Different places of semanal_main, like: https://github.com/python/mypy/pull/13579/files#diff-2924f9c93ed3b7e350efc02d9068d570cf76f32dc9485fee43df5ee11d4d9ae8R219
2. SemanticAnalyzer.process_top_level and other methods

Looks like I am stuck: it still fails for some cases 😒

[ilevkivskyi]

As I mentioned yesterday on discord, you will likely need to special-case builtins SCC, i.e. for this SCC add implicit symbols after processing the whole SCC, but for others keep the things as is.

[sobolevn]

Thank you for your guidance! Without it I would spend so much more time.

[ilevkivskyi]

Generally looks good, I have couple more suggestions/ideas.

[ilevkivskyi]

I would instead re-use core_modules (defined in semanal_main.py) here and in other place. Also you may need to use a flag for this method to avoid an import cycle with the latter (like skip_implicit_attrs).

[ilevkivskyi]

The problem is not the cyclic import itself, but that the method assumes that e.g. dict (for __annotations__) would be never marked incomplete (i.e. it may be not defined yet, but as soon as it is, it will be TypeInfo, rather than PlaceholderNode). This makes me think maybe a cleaner solution (if it works), would be just to replace assert with an if (and simply continue if it is a placeholder, like we do if the symbol is None). Can you try this?

Btw curiously this seem to only affect tests for now, it looks in real typeshed currently dict is not an instance of ABCMeta.

[sobolevn]

Can you try this?

Done, looks like it works! Let's wait to see what tests will show us :)

[sobolevn]

mypy_primer fails with:

 Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 940, in main
retcode = inner()
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 933, in inner
retcode = asyncio.run(coro)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
return future.result()
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 753, in primer
result = await result_fut
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
return f.result() # May raise f.exception().
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 414, in primer_result
await self.setup()
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 345, in setup
repo_dir = await ensure_repo_at_revision(
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 168, in ensure_repo_at_revision
await clone(repo_url, cwd, shallow=revision_like is None)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 134, in clone
await run(cmd, cwd=cwd)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 110, in run
raise subprocess.CalledProcessError(proc.returncode, cmd, output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['git', 'clone', '--recurse-submodules', 'https://gitlab.com/dkg/python-sop', '--depth', '1']' returned non-zero exit status 128.

I am sure it is not related, I will try to retrigger it a bit later.

[AlexWaygood]

mypy_primer fails with:

 Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 940, in main
retcode = inner()
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 933, in inner
retcode = asyncio.run(coro)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
return future.result()
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 753, in primer
result = await result_fut
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
return f.result() # May raise f.exception().
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 414, in primer_result
await self.setup()
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 345, in setup
repo_dir = await ensure_repo_at_revision(
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 168, in ensure_repo_at_revision
await clone(repo_url, cwd, shallow=revision_like is None)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 134, in clone
await run(cmd, cwd=cwd)
File "/opt/hostedtoolcache/Python/3.10.6/x64/lib/python3.10/site-packages/mypy_primer.py", line 110, in run
raise subprocess.CalledProcessError(proc.returncode, cmd, output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['git', 'clone', '--recurse-submodules', 'https://gitlab.com/dkg/python-sop', '--depth', '1']' returned non-zero exit status 128.

I am sure it is not related, I will try to retrigger it a bit later.

We have the same problem on all typeshed PRs at the moment:

• mypy-primer is failing on PRs typeshed#8677

[github-actions]

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

[sobolevn]

Thanks everyone! Next is #13565