typing: accept buffers in typing.IO.write by JelleZijlstra · Pull Request #9084 · python/typeshed

JelleZijlstra · 2022-11-04T03:02:37Z

JelleZijlstra · 2022-11-04T03:11:45Z

The mypy failures are a bug, python/mypy#14002.

srittau · 2022-11-04T11:31:56Z

This looks quite disruptive to me and I'm not sure whether it's worth it, considering the slightly legacy nature of IO.

JelleZijlstra · 2022-11-04T15:45:15Z

Worth noting that most of the mypy-primer fallout is due to a mypy bug (python/mypy#14002).

hauntsaninja · 2022-11-05T22:33:50Z

This seems useful, but given that I think I have a mypy fix, let's wait until it's released: python/mypy#14017

AlexWaygood

Looks like we can go back to this now, with the latest mypy release!

AlexWaygood · 2023-02-08T08:44:40Z

 def __iter__(self: Self) -> Self: ...
 def write(self, data: str) -> None: ... # type: ignore[override]
- def writelines(self, list: Iterable[str]) -> None: ...
+ def writelines(self, list: Iterable[str]) -> None: ... # type: ignore[override] # https://github.com/python/mypy/issues/14002


We probably don't need this type: ignore anymore (as well as several others), with the latest mypy release.

Cleaned up all comments referencing that issue.

AlexWaygood · 2023-02-08T08:45:24Z

 def parse_headers(fp: io.BufferedIOBase, _class: Callable[[], email.message.Message] = ...) -> HTTPMessage: ...

-class HTTPResponse(io.BufferedIOBase, BinaryIO):
+class HTTPResponse(io.BufferedIOBase, BinaryIO): # type: ignore[misc] # https://github.com/python/mypy/issues/14002


We do still need this type: ignore, but not because of the mypy issue linked (which is now closed) -- we need it because the definitions of writelines() in the two base classes are incompatible. The same goes for all the type: ignores in io.pyi and the one in lzma.pyi.

Updated these comments.

AlexWaygood · 2023-02-08T08:48:14Z

+ @overload
+ def write(self: _TemporaryFileWrapper[str], s: str) -> int: ...
+ @overload
+ def write(self: _TemporaryFileWrapper[bytes], s: ReadableBuffer) -> int: ...
+ @overload
+ def writelines(self: _TemporaryFileWrapper[str], lines: Iterable[str]) -> None: ...
+ @overload
+ def writelines(self: _TemporaryFileWrapper[bytes], lines: Iterable[ReadableBuffer]) -> None: ...


These overloads might not work very well with pyright, which has different handling of self types to mypy (#9591). The same goes for the overloads on lines 401-408, and the overloads in typing.pyi. You might want to instead do something similar to what I did with re.Pattern and re.Match in #9592.

Changed, and added some testcases for IO.write.

AlexWaygood · 2023-03-09T17:26:08Z

Ping @JelleZijlstra, have you had a chance to take a look at my review feedback? :)

JelleZijlstra · 2023-03-09T17:29:07Z

Oops, I think I assumed this was still blocked. I will take a look soon!

JelleZijlstra · 2023-03-10T03:58:23Z

hm, the new way to write the overloads makes subclassing much worse. Let me see if I can fix that.

github-actions · 2023-03-10T04:26:10Z

Diff from mypy_primer, showing the effect of this PR on open source code:

rich (https://github.com/Textualize/rich)
+ rich/progress.py:173: error: Definition of "writelines" in base class "IOBase" is incompatible with definition in base class "IO" [misc]
psycopg (https://github.com/psycopg/psycopg)
+ psycopg/psycopg/copy.py:464: error: Unused "type: ignore" comment
cwltool (https://github.com/common-workflow-language/cwltool)
+ cwltool/main.py:963: error: Unused "type: ignore[arg-type]" comment

JelleZijlstra · 2023-03-10T04:29:58Z

Seems like we have a choice between:

One overload for AnyStr and another for Buffer: pyright accepts IO.write(AnyStr), but mypy hits Incorrect incompatible override report with self type + overload + TypeVar with constraints mypy#14866 and subclasses of IO classes that override the write/writelines method start complaining.
One overload for str and another for bytes + Buffer: mypy is fine with more subclasses, but pyright rejects IO.write(AnyStr).

I'm inclined to go with (1) because mypy's behavior is more clearly buggy than pyright's, but ideally we'd fix the mypy bug first.

AlexWaygood · 2023-03-12T11:55:57Z

Now that python/mypy#14882 has been merged, I guess we have two options:

we can either merge [alt] typing: accept buffers in IO.write #9861 now (and simplify the overloads after mypy 1.2 comes out, perhaps);
or we close [alt] typing: accept buffers in IO.write #9861, wait for mypy 1.2, and then merge this.

I'm easy either way :)

typing: accept buffers in typing.IO.write
2f87ea0
Fixespython#9082

JelleZijlstra mentioned this pull request Nov 4, 2022
Track review of bytes types #9006
Open

type ignore
e940cdf

This comment has been minimized.
Sign in to view

JelleZijlstra added 2 commits November 3, 2022 20:29

need this way
8d82f82

fix arg name
5aacf0a

This comment has been minimized.
Sign in to view

hauntsaninja added the status: deferred Issue or PR deferred until some precondition is fixed label Nov 5, 2022

Merge branch 'main' into typing
9293f5e

This comment has been minimized.
Sign in to view

AlexWaygood added 2 commits February 8, 2023 08:29

Merge branch 'main' into typing
717e864

Update lzma.pyi
2894148

This comment has been minimized.
Sign in to view

AlexWaygood removed the status: deferred Issue or PR deferred until some precondition is fixed label Feb 8, 2023

AlexWaygood reviewed Feb 8, 2023
View reviewed changes

Merge branch 'main' into typing
6532cd4

This comment has been minimized.
Sign in to view

JelleZijlstraand others added 7 commits March 9, 2023 19:10

Merge remote-tracking branch 'upstream/main' into typing
642f843

Remove references to closed mypy issue
641511b

Adjust overloads to work with pyright
590851b

[pre-commit.ci] auto fixes from pre-commit.com hooks
b6174cb

ignore another
92c2606

no picklebuffer in 3.7
7f10fc8

into the uncertain future
684b4ee

This comment has been minimized.
Sign in to view

JelleZijlstra requested a review from AlexWaygood March 10, 2023 03:55

maybe help mypy primer
bd78b5b

This comment has been minimized.
Sign in to view

JelleZijlstra mentioned this pull request Mar 10, 2023
Incorrect incompatible override report with self type + overload + TypeVar with constraints python/mypy#14866
Closed

try the other way
37bb27e

AlexWaygood mentioned this pull request Mar 10, 2023
[alt] typing: accept buffers in IO.write#9861
Merged

JelleZijlstra closed this in #9861 Mar 15, 2023

Uh oh!

Uh oh!

Conversation

JelleZijlstra commented Nov 4, 2022

Uh oh!

JelleZijlstra commented Nov 4, 2022

Uh oh!

This comment has been minimized.

This comment has been minimized.

srittau commented Nov 4, 2022

Uh oh!

JelleZijlstra commented Nov 4, 2022

Uh oh!

hauntsaninja commented Nov 5, 2022

Uh oh!

This comment has been minimized.

This comment has been minimized.

AlexWaygood left a comment

Choose a reason for hiding this comment

Uh oh!

AlexWaygoodFeb 8, 2023

Choose a reason for hiding this comment

Uh oh!

JelleZijlstraMar 10, 2023

Choose a reason for hiding this comment

Uh oh!

AlexWaygoodFeb 8, 2023

Choose a reason for hiding this comment

Uh oh!

JelleZijlstraMar 10, 2023

Choose a reason for hiding this comment

Uh oh!

AlexWaygoodFeb 8, 2023

Choose a reason for hiding this comment

Uh oh!

JelleZijlstraMar 10, 2023

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

AlexWaygood commented Mar 9, 2023

Uh oh!

JelleZijlstra commented Mar 9, 2023

Uh oh!

This comment has been minimized.

JelleZijlstra commented Mar 10, 2023

Uh oh!

This comment has been minimized.

github-actionsBot commented Mar 10, 2023

Uh oh!

JelleZijlstra commented Mar 10, 2023

Uh oh!

AlexWaygood commented Mar 12, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants