[1.20 regression] reachability in django-modern-rest

[sobolevn]

Hi!

I am facing 150+ errors in my project https://github.com/wemake-services/django-modern-rest after updating mypy to 1.20 from 1.19.

Here are the most frequest ones:

lru_cache issues

dmr/response.py:236: error: Cannot determine type of "resolve_setting" [has-type]
dmr/response.py:236: note: Revealed type is "Any"
dmr/response.py:237: error: Cannot determine type of "resolve_setting" [has-type]
dmr/routing.py:108: error: Cannot determine type of "resolve_setting" [has-type]
dmr/routing.py:190: error: Cannot determine type of "resolve_setting" [has-type]
dmr/parsers.py:232: error: Cannot determine type of "resolve_setting" [has-type]
dmr/parsers.py:284: error: Cannot determine type of "resolve_setting" [has-type]

For some reason all code that is using resolve_setting fails with this error:

renderers_list = (
 resolve_setting(Settings.renderers) if renderers is None else renderers
)

Here's how it is defined:

@lru_cache(maxsize=MAX_CACHE_SIZE)
def resolve_setting(
 setting_name: Settings,
 *,
 import_string: bool = False,
) -> Any:
 """
 Resolves setting by *setting_name*.

 The result is cached using ``@lru_cache`` for performance.
 When testing with custom settings, you *must* call
 :func:`clear_settings_cache` before and after modifying
 Django settings to ensure the cache is invalidated properly.
 """
 setting = _resolve_defaults().get(
 setting_name,
 _DEFAULTS[setting_name],
 )
 if import_string and isinstance(setting, str):
 return module_loading.import_string(setting)
 return setting # pyright: ignore[reportUnknownVariableType]

Maybe @lru_cache(maxsize=MAX_CACHE_SIZE) is the root cause? If I remove this decorator - all errors are gone.

We also faced a similiar problem in our django-stubs PR: typeddjango/django-stubs#3252 (comment)

Unreachable

I have more than 120+ errors that some statements are unreachable.
Examples:

(.venv) ~/Desktop/django-modern-rest master ✗ 1 ⚠️
» mypy dmr
dmr/settings.py:122: error: Statement is unreachable [unreachable]
dmr/validation/settings.py:41: error: Statement is unreachable [unreachable]

1. https://github.com/wemake-services/django-modern-rest/blob/9062072376962f8b9ba6e195244ca702fe3d15e8/dmr/settings.py#L122
2. https://github.com/wemake-services/django-modern-rest/blob/9062072376962f8b9ba6e195244ca702fe3d15e8/dmr/validation/settings.py#L41

These are two different module-level definitions. Not sure what is the root cause here :(

[ilevkivskyi]

cc @hauntsaninja for the unreachable part.

[ilevkivskyi]

The first error (Cannot determine type) may be caused by (either or both):

• different order of processing files in an import cycle.
• different signature of lru_cache() in typeshed that can't be handled by semanal_infer.py anymore.

There is really not much we can do about the first possibility (and deferring module top-levels is hard) I will look at current signature of lru_cache() to see what can be improved in semanal_infer.py.

[ilevkivskyi]

Hm, the signature of lru_cache() is non-trivial, and didn't change recently. The only recent change is _lru_cache_wrapper class was made covariant. Also at least some of the errors are from functions, not from top-levels, so this must be something subtle.

I will try to take a look later today. In the meantime, @sobolevn is there any chance you can bisect these errors to a specific mypy commit?

[ilevkivskyi]

Oh, I think I see it, there is

assert SettingsDict.__optional_keys__ == set(Settings), ( # noqa: S101
 'Settings enum and its type SettingsDict have different keys'
)

few lines above resolve_setting() definition. So I guess both issues are caused by @hauntsaninja reachability work.

[woodruffw]

I think I have a similar case of this, in pypi/pypi-attestations#178 -- we have pytest assertions like this:

assert TypeAdapter(impl.Publisher).validate_json(json.dumps(gh_raw)) == gh

..which mypy now thinks are always asserted, causing subsequent lines to be flagged as unreachable. In reality these assertions pass and subsequent lines are reachable, but I suspect the mismatch here might happen because mypy sees TypeAdapter as a discrete type and assumes that the __eq__ cannot hold?

[hauntsaninja]

The resolve_setting is a genuine issue (comparing frozenset[str] to set[StrEnum]). I will look further

For the unreachable issues in your tests, a lot of them trace back to a bug in django-stubs. I opened a PR for this: typeddjango/django-stubs#3256

(Note in the limit I expect warn_unreachable to have more false positives in tests. It is currently an opt-in config knob. So different configuration for tests may be part of the story)

In woodruffw's case, mypy and earlier versions of mypy reveal TypeAdapter(impl.Publisher).validate_json(json.dumps(gh_raw)) as Never. In 1.20, this now has an impact on reachability. Looking further on this as well

[hauntsaninja]

Okay.

The first issue is that resolve_setting is a confusing symptom. I opened #21144 which is one way to fix this

The second issue is the underlying reachability implicated by resolve_setting involving set and frozenset. I have opened a new issue for that over here: #21143

The third issue was a bug in django-stubs for which the fix has been merged

The fourth issue is woodruffw's. For that one, I'm currently leaning towards not doing anything. I think it's desirable that warn_unreachable indicates you are narrowing to Never here. It is an opt-in setting, so seems useful to know that TypeAdapter(impl.Publisher).validate_json(json.dumps(gh_raw)) is not doing what we hope it might.

The next question is why isn't that doing what we hope it might, and the issue there is with the typing of pydantic.TypeAdapter. One way to fix this might be when TypeForm support is landed.

λ cat test224.py
from typing import Any, overload
from typing_extensions import TypeForm

class A: ...
class B: ...

class G1[T]:
 def __init__(self, t: type[T]) -> None: ...
 def get(self) -> T: ...

def f1() -> None:
 # E: Argument 1 to "G1" has incompatible type "UnionType | type[A]"; expected "type[A]"
 # N: Revealed type is "test224.A"
 reveal_type(G1(A | B).get())

# current TypeAdapter looks something like this:
class G2[T]:
 @overload
 def __init__(self, t: type[T]) -> None: ...
 @overload
 def __init__(self, t: Any) -> None: ...
 def __init__(self, *a, **kw): ...
 def get(self) -> T: ...

def f2() -> None:
 # N: Revealed type is "Never"
 reveal_type(G2(A | B).get())

class G3[T]:
 def __init__(self, t: TypeForm[T]) -> None: ...
 def get(self) -> T: ...

def f3() -> None:
 # N: Revealed type is "test224.A | test224.B"
 reveal_type(G3(A | B).get())

I think that gets everything currently mentioned here, so will close this issue out once #21144 is merged. If you are stumbling upon this issue, consider opening a new one.