Render typed iterators (e.g. Iterator[int]) in docstrings

[sizmailov]

This PR makes pybind11 generate much more informative typed iterators.

this PR
__iter__(self: example.VectorInt) -> Iterator[int] __iter__(self: example.MapTupleToComplex) -> Iterator[Tuple[int, str]] items(self: example.MapTupleToComplex) -> Iterator[Tuple[Tuple[int, str], complex]]

master
__iter__(self: example.VectorInt) -> iterator __iter__(self: example.MapTupleToComplex) -> iterator items(self: example.MapTupleToComplex) -> iterator

Strictly speaking its a potential breaking change but seems like no existing code is affected, see below.

This PR makes py::make_iterator and py::make_key_iterator return py::detail::iterator_state instance instead of py::iterator.
It doesn't affect regular use of those functions (immediate return from __iter__ lambda),
but requires changes in user code with implicit assignments/conversions, e.g.:

// next line is ok now, but will break with this PR// fix on user side: apply py::cast() explicitly
py::iterator it = make_iterator(...); 

Search for code that would break:
To justify intuition about low frequency of "not-immediate-return" make_iterator usage I've turned to github search API for py::make_iterator. Search is limited to 1000 files, so can't tell only for first 1000 found files (~5726 in total [web search]).

In 1000 files there are 229 unique strings with return py:make_iterator.
The only non-comment occurrence of py::make_iterator without preceding return was found in (pybind?) test code here. This should also be no problem since cast should happen automatically.

py::make_map_iterator search find only two unique strings with preceding return in another 1000 files (did a separate search). No other usages. Looks like this function is much less popular.

While strictly speaking a breaking change was introduced it seems like it breaks no existing code.
If you think this argument is not applicable and change to py::make_iterator should go to major pybind release (e.g. 3.0).

compare.cpp

#include<map>
#include<complex>
#include"pybind11/pybind11.h"
#include"pybind11/stl_bind.h"
#include"pybind11/embed.h"namespacepy= pybind11;
using map_tuple_to_complex = std::map<std::tuple<int,std::string>, std::complex<double>>;
PYBIND11_MAKE_OPAQUE(std::vector<int>);
PYBIND11_MAKE_OPAQUE(map_tuple_to_complex);
PYBIND11_EMBEDDED_MODULE(example, m) {
py::bind_vector<std::vector<int>>(m, "VectorInt");
py::bind_map<map_tuple_to_complex>(m, "MapTupleToComplex");
}
intmain() {
py::scoped_interpreter guard{};
py::exec(R"( from example import * print(VectorInt.__iter__.__doc__, end="") print(MapTupleToComplex.__iter__.__doc__, end="") print(MapTupleToComplex.items.__doc__, end=""))");
}

[wjakob]

I'm against merging this -- first, because of the incompatibility, and because it adds a bunch of code for what is a relative niche feature. This is generally a dangerous rabbit hole: we could increase the complexity of pybind11 to generate near-perfect type annotations, but I don't think that is in the interest of the majority of the users of this project. My goal post is: do an okay job in a minimal fashion.

[wjakob]

(Side note: sorry for rejecting it after you've clearly put quite a bit of work into the PR and explanations above).

[sizmailov]

That's sad. Would it make a difference if this PR won't be breaking?
I can add _ng versions of py::make*_iterator and use them in std bindings. py::make*_iterator will be marked as deprecated.