Demo: Testing pybind11-stubgen + mypy

[timohl]

Description

This PR shows how pybind11-stubgen and mypy could be used in pytest to check for correct type hints.
It stems from the discussion at #5663.

This is just a quick demo with one positive and one negative test.

Sources for calling pybind11-stubgen and mypy from Python:

• Integrating mypy into another Python application
• Calling pybind11-stubgen from Python:
  Add args to main pybind11-stubgen#252
  (thx @gentlegiantJGC for this 👍)

TODOs:

• Integrate test cases from Experimenting: Annotated[Any, pybind11.CppType("cpp_namespace::UserType")] #4888
• Maybe use this to generate stubs for all test modules (just to check if pybind11-stubgen and mypy run successful or find errors)
• Pin versions of mypy and pybind11-stubgen (or even use a version matrix in CI).
• Consider other type checkers (most notably Pyright/Pylance since this is the default in vscode).

---

📚 Documentation preview 📚: https://pybind11--5678.org.readthedocs.build/

[timohl]

Some errors I see so far:

• Python 3.14 seems like causing some mypy internal errors
• Some checks do not properly install pybind11-stubgen, although I added it to requirements.txt and pyproject.toml
• mypy report on https://github.com/pybind/pybind11/actions/runs/15149485803/job/42592785722?pr=5678 seems scrambled (maybe some formatting or windows specific stuff)

[rwgk]

The new test dependencies seem "small/lightweight" and "super lightweight":

$ time pip install mypy
Collecting mypy
Using cached mypy-1.15.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_28_x86_64.whl.metadata (2.1 kB)
Requirement already satisfied: typing_extensions>=4.6.0 in ./python/venvs/scratch/cpwheels/lib/python3.12/site-packages (from mypy) (4.13.2)
Requirement already satisfied: mypy_extensions>=1.0.0 in ./python/venvs/scratch/cpwheels/lib/python3.12/site-packages (from mypy) (1.1.0)
Using cached mypy-1.15.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_28_x86_64.whl (12.4 MB)
Installing collected packages: mypy
Successfully installed mypy-1.15.0
real 0m1.098s
user 0m0.936s
sys 0m0.150s

$ time pip install pybind11-stubgen
Collecting pybind11-stubgen
Using cached pybind11_stubgen-2.5.4-py3-none-any.whl.metadata (1.9 kB)
Using cached pybind11_stubgen-2.5.4-py3-none-any.whl (30 kB)
Installing collected packages: pybind11-stubgen
Successfully installed pybind11-stubgen-2.5.4
real 0m0.418s
user 0m0.242s
sys 0m0.048s

I think you're going in a great direction.

Maybe use this to generate stubs for all test modules (just to check if pybind11-stubgen and mypy run successful or find errors)

That sounds like a great idea, as long as you build in an easy way to turn that off during development iterations. I'd only want to turn that on again while I clean up for the next git push.

If you do that, I'd say forget the tests under #4888 (those came mostly from mypy). I believe they'll be 100% redundant.

Pyright/Pylance since this is the default in vscode

I'd look at that is the top priority. — I don't know a lot about typing, but last time I quizzed ChatGPT about it (a few months ago), it told me mypy is a good start, but — liberally reinterpreted by me — what really counts is LSP.

[henryiii]

If you haven't started using uv, you should try it, even if it's just uv venv and uv pip:

$ uv venvUsing CPython 3.13.3 interpreter at: /usr/local/opt/python@3.13/bin/python3.13Creating virtual environment at: .venvActivate with: source .venv/bin/activate.fish________________________________________________________Executed in 28.56 millis fish external usr time 12.44 millis 0.25 millis 12.19 millis sys time 13.04 millis 1.91 millis 11.13 millis
$ time uv pip install mypyResolved 3 packages in 3msInstalled 3 packages in 50ms + mypy==1.15.0 + mypy-extensions==1.1.0 + typing-extensions==4.13.2________________________________________________________Executed in 81.80 millis fish external usr time 18.27 millis 0.29 millis 17.98 millis sys time 54.27 millis 3.55 millis 50.73 millis

I think ty and pyrefly, the two new Python type checkers in Rust, are likely to have a big impact in the future, but for now, most projects work on passing mypy, and the existing type checkers like pylance know that and try to be compatible. I don't think I use anything other than mypy in CI on all my projects, though I use pyright locally in VSCode and via LSP in Vim.

The bug in Python 3.14.0b1 that we've reported also affects mypy, and it's been marked as a release blocker. So mypy will likely work better in beta 2.

[timohl]

I fixed most of the CI issues:

• Added xfail for Python 3.14
• Using --no-color-output fixes parsing the mypy report correctly
• I have changed CI at a couple of places to use tests/requirements.txt instead of manually installing pytest, numpy and so on. Since I added pybind11-stubgen and mypy there, the tests now run successfully (except for mingw because I am not sure how to change this the right was)

@henryiii
The CI is uses a mix of uv, pip, pipx, apt and the github actions setup-python and setup-uv.
Would it make sense to have this more consistently use uv, setup-python and setup-uv where possible (mingw seems to be an exception)?
Also is there a downside to using venv in CI checks?
In https://github.com/pybind/pybind11/actions/runs/15182964725/job/42696579100 uv complained about the system-wide environment being externally managed (while almost all other jobs did not complain).
Adding activate-environment: true to the setup-uv action solved this.

@rwgk

Maybe use this to generate stubs for all test modules (just to check if pybind11-stubgen and mypy run successful or find errors)

That sounds like a great idea, as long as you build in an easy way to turn that off during development iterations. I'd only want to turn that on again while I clean up for the next git push.

Probably best to use pytest markers for that (which could be excluded by default).
Custom markers (pytest docs)
Exclude custom marker by default (stackoverflow)

[timohl]

I have added a test that applies stubgen and mypy to the whole pybind11_tests module.

This is the stubgen error log:

error log

ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.lvalue_nested : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.lvalue_pair : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.lvalue_tuple : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.nodefer_none_void : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.rvalue_nested : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.rvalue_pair : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.rvalue_tuple : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes : Can't find/import 'ConstRefCasted' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_const_ptr : Can't find/import 'ConstRefCasted' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_const_ref : Can't find/import 'ConstRefCasted' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_const_ref_wrap : Can't find/import 'ConstRefCasted' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_move : Can't find/import 'ConstRefCasted' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_ptr : Can't find/import 'ConstRefCasted' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_ref : Can't find/import 'ConstRefCasted' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_ref_wrap : Can't find/import 'ConstRefCasted' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.callbacks : Can't find/import 'pybind11_tests.callbacks.custom_function' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.callbacks : Can't find/import 'pybind11_tests.callbacks.custom_function2' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.callbacks.func_accepting_func_accepting_base : Invalid expression 'test_submodule_callbacks(module_&)::AbstractBase' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.class_.ProtectedB.get_self : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.class_.ProtectedB.void_foo : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.class_ : Invalid identifier '' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.local_bindings.get_mixed_gl : Invalid expression 'LocalBase<5>' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.local_bindings.get_mixed_lg : Invalid expression 'LocalBase<4>' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.local_bindings.load_external1 : Invalid expression 'LocalBase<6>' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.local_bindings.load_external2 : Invalid expression 'LocalBase<7>' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.methods_and_attributes.exercise_is_setter.Field.int_value. : Invalid expression 'pybind11_tests::exercise_is_setter::FieldBase' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.native_enum.pass_some_proto_enum : Invalid expression 'test_native_enum::some_proto_enum' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.native_enum.return_some_proto_enum : Invalid expression 'test_native_enum::some_proto_enum' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_array.index_using_ellipsis : Invalid expression 'detail::accessor' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_array_array : Can't find/import 'ArrayStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_complex_array : Can't find/import 'ComplexStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_enum_array : Can't find/import 'EnumStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_rec_nested : Can't find/import 'NestedStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_rec_partial : Can't find/import 'PartialStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_rec_partial_nested : Can't find/import 'PartialNestedStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_string_array : Can't find/import 'StringStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.f_nested : Can't find/import 'NestedStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_array_array : Can't find/import 'ArrayStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_complex_array : Can't find/import 'ComplexStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_enum_array : Can't find/import 'EnumStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_rec_nested : Can't find/import 'NestedStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_string_array : Can't find/import 'StringStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.opaque_types.get_void_ptr_value : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.opaque_types.return_void_ptr : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.annotate_never : Can't find/import 'typing.Never' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.annotate_type_is : Can't find/import 'typing.TypeIs' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.check_type_is : Can't find/import 'typing.TypeIs' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_destructor : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_destructor_2 : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_destructor_3 : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_explicit_nullptr_dtor : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_name_and_destructor : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_renamed_capsule_with_destructor : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_renamed_capsule_with_destructor_2 : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.test_list_slicing : Invalid expression 'detail::accessor' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.test_list_slicing_default : Invalid expression 'detail::accessor' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.weakref_from_handle : Can't find/import 'weakref' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.weakref_from_handle_and_function : Can't find/import 'weakref' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.weakref_from_object : Can't find/import 'weakref' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.weakref_from_object_and_function : Can't find/import 'weakref' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.smart_ptr.ContainerUsingPrivateESFT.ptr. : Invalid expression 'test_submodule_smart_ptr(module_&)::PrivateESFT' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.smart_ptr.ContainerUsingPrivateESFT.ptr. : Invalid expression 'test_submodule_smart_ptr(module_&)::PrivateESFT' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.smart_ptr.return_std_shared_ptr_example_drvd : Invalid expression 'holder_caster_traits_test::example_drvd' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.smart_ptr.return_std_unique_ptr_example_drvd : Invalid expression 'holder_caster_traits_test::example_drvd' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_lv_array : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_lv_map : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_lv_nested : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_lv_vector : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_ptr_vector : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_rv_array : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_rv_map : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_rv_nested : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_rv_vector : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.func_with_string_or_vector_string_arg_overload : Invalid expression 'std::vector' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.func_with_string_or_vector_string_arg_overload : Invalid expression ':allocator >' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.func_with_string_or_vector_string_arg_overload : Invalid expression ':allocator > > >' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.tpl_constr_optional : Can't find/import 'TplCtorClass' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.tpl_ctor_map : Can't find/import 'TplCtorClass' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.tpl_ctor_set : Can't find/import 'TplCtorClass' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.tpl_ctor_vector : Can't find/import 'TplCtorClass' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stubgen_error.identity_capsule : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.virtual_functions.Adder.__call__ : Invalid expression 'AdderBase::Data' WARNING pybind11_stubgen:error_handlers.py:150 Raw C++ types/values were found in signatures extracted from docstrings. Please check the corresponding sections of pybind11 documentation to avoid common mistakes in binding code: - https://pybind11.readthedocs.io/en/latest/advanced/misc.html#avoiding-cpp-types-in-docstrings - https://pybind11.readthedocs.io/en/latest/advanced/functions.html#default-arguments-revisited

This is the mypy report:

/tmp/pytest-of-timohl/pytest-5/test_stubgen_all0/pybind11_tests/stl.pyi:187: error: Ellipses cannot accompany other argument types in function type signature [syntax]
/tmp/pytest-of-timohl/pytest-5/test_stubgen_all0/pybind11_tests/stl.pyi:187: error: Duplicate argument "std" in function definition
Found 2 errors in 1 file (errors prevented further checking)

I have extracted __doc__ of the function mentioned by mypy to see, what it actually contains:

func_with_string_or_vector_string_arg_overload(*args, **kwargs)
Overloaded function.
1. func_with_string_or_vector_string_arg_overload(arg0: std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >) -> int
2. func_with_string_or_vector_string_arg_overload(arg0: collections.abc.Sequence[str]) -> int
3. func_with_string_or_vector_string_arg_overload(arg0: str) -> int

The function is bound here:

pybind11/tests/test_stl.cpp

Lines 577 to 582 in 98bd78f

	// #1258: pybind11/stl.h converts string to vector<string>
	m.def("func_with_string_or_vector_string_arg_overload",
	[](const std::vector<std::string> &) { return1; });
	m.def("func_with_string_or_vector_string_arg_overload",
	[](const std::list<std::string> &) { return2; });
	m.def("func_with_string_or_vector_string_arg_overload", [](const std::string &) { return3; });

Not sure what is going on here.
I would have assumed that the list_caster would give collections.abc.Sequence:

pybind11/include/pybind11/stl.h

Lines 360 to 366 in 98bd78f

	PYBIND11_TYPE_CASTER(Type,
	io_name("collections.abc.Sequence", "list") + const_name("[")
	+ value_conv::name + const_name("]"));
	};
	template <typename Type, typename Alloc>
	structtype_caster<std::vector<Type, Alloc>> : list_caster<std::vector<Type, Alloc>, Type> {};

I will go through the other errors tomorrow.

[timohl]

I have added a marker -m stubgen to enable testing of stubgen on all test modules (off by default right now).
In order to enable this test, you can use cmake with:

cmake -S . -B build -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON -DPYBIND11_PYTEST_ARGS="-m stubgen"

Also, I went through some of the stubgen errors:

• The following stubgen errors were straigt forward to fix by changing the name attribute of the type_caster:

error log

ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.lvalue_nested : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.lvalue_pair : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.lvalue_tuple : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.nodefer_none_void : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.rvalue_nested : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.rvalue_pair : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.rvalue_tuple : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes : Can't find/import 'ConstRefCasted' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_const_ptr : Can't find/import 'ConstRefCasted' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_const_ref : Can't find/import 'ConstRefCasted' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_const_ref_wrap : Can't find/import 'ConstRefCasted' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_move : Can't find/import 'ConstRefCasted' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_ptr : Can't find/import 'ConstRefCasted' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_ref : Can't find/import 'ConstRefCasted' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.builtin_casters.takes_ref_wrap : Can't find/import 'ConstRefCasted' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_lv_array : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_lv_map : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_lv_nested : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_lv_vector : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_ptr_vector : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_rv_array : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_rv_map : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_rv_nested : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.cast_rv_vector : Can't find/import 'RValueCaster' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.copy_only : Can't find/import 'CopyOnlyInt' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.copy_optional : Can't find/import 'CopyOnlyInt' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.copy_tuple : Can't find/import 'CopyOnlyInt' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_copy_nested : Can't find/import 'MoveOnlyInt' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_copy_nested : Can't find/import 'MoveOrCopyInt' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_copy_nested : Can't find/import 'CopyOnlyInt' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_only : Can't find/import 'MoveOnlyInt' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_optional : Can't find/import 'MoveOnlyInt' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_optional_tuple : Can't find/import 'MoveOrCopyInt' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_optional_tuple : Can't find/import 'MoveOnlyInt' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_optional_tuple : Can't find/import 'CopyOnlyInt' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_or_copy : Can't find/import 'MoveOrCopyInt' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_or_copy_optional : Can't find/import 'MoveOrCopyInt' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_pair : Can't find/import 'MoveOnlyInt' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_pair : Can't find/import 'MoveOrCopyInt' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_tuple : Can't find/import 'MoveOnlyInt' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.copy_move_policies.move_tuple : Can't find/import 'MoveOrCopyInt' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.ArgInspector.f : Can't find/import 'ArgInspector1' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.ArgInspector.f : Can't find/import 'ArgAlwaysConverts' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.ArgInspector.g : Can't find/import 'ArgInspector1' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.ArgInspector.g : Can't find/import 'ArgInspector2' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.ArgInspector.g : Can't find/import 'ArgAlwaysConverts' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.ArgInspector.h : Can't find/import 'ArgInspector2' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.ArgInspector.h : Can't find/import 'ArgAlwaysConverts' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.arg_inspect_func : Can't find/import 'ArgInspector2' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.arg_inspect_func : Can't find/import 'ArgInspector1' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.arg_inspect_func : Can't find/import 'ArgAlwaysConverts' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.custom_caster_destroy : Can't find/import 'DestructionTester' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.custom_caster_destroy_const : Can't find/import 'DestructionTester' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.custom_caster_no_destroy : Can't find/import 'DestructionTester' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.custom_type_casters.other_lib_type : Can't find/import 'MyType'

• Also the following stubgen errors should be fixed by fix(types): type hints from future python versions #5693.

error log

ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.class_.ProtectedB.get_self : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.class_.ProtectedB.void_foo : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.class_.pointers_equal : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.class_.read_foo : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.opaque_types.get_void_ptr_value : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.opaque_types.return_void_ptr : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_destructor : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_destructor_2 : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_destructor_3 : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_explicit_nullptr_dtor : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_capsule_with_name_and_destructor : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_renamed_capsule_with_destructor : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.return_renamed_capsule_with_destructor_2 : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stubgen_error.identity_capsule : Can't find/import 'types.CapsuleType' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.annotate_never : Can't find/import 'typing.Never' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.annotate_type_is : Can't find/import 'typing.TypeIs' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.check_type_is : Can't find/import 'typing.TypeIs' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.weakref_from_handle : Can't find/import 'weakref' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.weakref_from_handle_and_function : Can't find/import 'weakref' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.weakref_from_object : Can't find/import 'weakref' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.weakref_from_object_and_function : Can't find/import 'weakref'

• The following stubgen errors I did not look at or I did not see an easy fix:

error log

ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.callbacks : Can't find/import 'pybind11_tests.callbacks.custom_function' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.callbacks : Can't find/import 'pybind11_tests.callbacks.custom_function2' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.callbacks.func_accepting_func_accepting_base : Invalid expression 'test_submodule_callbacks(module_&)::AbstractBase' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.class_ : Invalid identifier '' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.local_bindings.get_mixed_gl : Invalid expression 'LocalBase<5>' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.local_bindings.get_mixed_lg : Invalid expression 'LocalBase<4>' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.local_bindings.load_external1 : Invalid expression 'LocalBase<6>' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.local_bindings.load_external2 : Invalid expression 'LocalBase<7>' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.methods_and_attributes.exercise_is_setter.Field.int_value. : Invalid expression 'pybind11_tests::exercise_is_setter::FieldBase' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.native_enum.pass_some_proto_enum : Invalid expression 'test_native_enum::some_proto_enum' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.native_enum.return_some_proto_enum : Invalid expression 'test_native_enum::some_proto_enum' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_array.index_using_ellipsis : Invalid expression 'detail::accessor' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_array_array : Can't find/import 'ArrayStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_complex_array : Can't find/import 'ComplexStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_enum_array : Can't find/import 'EnumStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_rec_nested : Can't find/import 'NestedStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_rec_partial : Can't find/import 'PartialStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_rec_partial_nested : Can't find/import 'PartialNestedStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.create_string_array : Can't find/import 'StringStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.f_nested : Can't find/import 'NestedStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_array_array : Can't find/import 'ArrayStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_complex_array : Can't find/import 'ComplexStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_enum_array : Can't find/import 'EnumStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_rec_nested : Can't find/import 'NestedStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.numpy_dtypes.print_string_array : Can't find/import 'StringStruct' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.test_list_slicing : Invalid expression 'detail::accessor' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.pytypes.test_list_slicing_default : Invalid expression 'detail::accessor' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.smart_ptr.ContainerUsingPrivateESFT.ptr. : Invalid expression 'test_submodule_smart_ptr(module_&)::PrivateESFT' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.smart_ptr.ContainerUsingPrivateESFT.ptr. : Invalid expression 'test_submodule_smart_ptr(module_&)::PrivateESFT' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.smart_ptr.return_std_shared_ptr_example_drvd : Invalid expression 'holder_caster_traits_test::example_drvd' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.smart_ptr.return_std_unique_ptr_example_drvd : Invalid expression 'holder_caster_traits_test::example_drvd' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.func_with_string_or_vector_string_arg_overload : Invalid expression 'std::vector' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.func_with_string_or_vector_string_arg_overload : Invalid expression ':allocator >' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.func_with_string_or_vector_string_arg_overload : Invalid expression ':allocator > > >' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.tpl_constr_optional : Can't find/import 'TplCtorClass' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.tpl_ctor_map : Can't find/import 'TplCtorClass' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.tpl_ctor_set : Can't find/import 'TplCtorClass' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.stl.tpl_ctor_vector : Can't find/import 'TplCtorClass' ERROR pybind11_stubgen:error_handlers.py:77 In pybind11_tests.virtual_functions.Adder.__call__ : Invalid expression 'AdderBase::Data'

The error I described in my previous comment from func_with_string_or_vector_string_arg_overload stems from:

pybind11/tests/test_stl.cpp

Line 63 in 03b4a9e

PYBIND11_MAKE_OPAQUE(std::vector<std::string, std::allocator<std::string>>)

String vectors are made opaque here but are not bound to a Python type (see "Making opaque types" docs

[timohl]

Stubgen tests are now just enabled on the Ubuntu jobs (could be discussed where to roll out stubgen testing in the end).

I merged the fixes from #5693 into this (probably have to do a rebase later to fix the git history, but I wanted to filter out the errors fixed by this).

Also some test binding fixes in f9e4686, but the change to numpy dtype tests breaks now (gotta understand this test more)

[timohl]

Just a quick update on this:
I am pretty busy right now with finishing my master thesis, so I won't have much time to work on this until mid next month.

One question though:
Would it make sense to have this as a smaller PR adding stubgen/mypy tests that are disabled by default but can be manually activated with something like -DPYBIND11_PYTEST_ARGS="-m stubgen" and not including pybind11-stubgen and mypy to the test requirements yet.
That way, people could already use this for development.
Fixing/handling all stubgen/mypy errors in the test code, as well as integration into the CI workflows, could then maybe go into a subsequent PR?

[rwgk]

Just a quick update on this: I am pretty busy right now with finishing my master thesis, so I won't have much time to work on this until mid next month.

One question though: Would it make sense to have this as a smaller PR adding stubgen/mypy tests that are disabled by default but can be manually activated with something like -DPYBIND11_PYTEST_ARGS="-m stubgen" and not including pybind11-stubgen and mypy to the test requirements yet. That way, people could already use this for development. Fixing/handling all stubgen/mypy errors in the test code, as well as integration into the CI workflows, could then maybe go into a subsequent PR?

That sounds good to me, although maybe that'll be best a couple days after the pybind11 v3.0.0 release? @henryiii

[InvincibleRMC]

Now that pybind11 3.0.0 is out and if you have the free time. I would love the -m stubgen feature. It makes testing new typing stuff much easier.