Update mypy-protobuf

[JelleZijlstra]

See #10909 (comment)

I had to edit some sed commands so they worked for me on Linux.

The script also set the version of stubs/protobuf/METADATA.toml to 4.21.0; I changed it back.

I tried updating PROTOBUF_VERSION and PYTHON_PROTOBUF_VERSION to the latest (24.4/4.24.4), but the script failed; apparently they no longer distribute the Python library in the way the script expects.

[AlexWaygood]

Thank you!

[JelleZijlstra]

cc @nipunn1313

[Avasam]

How about the sync_tensorflow_protobuf_stubs.sh script? I'm also unsure how to regenerate s2clientprotocol stubs

[JelleZijlstra]

How about the sync_tensorflow_protobuf_stubs.sh script?

$ scripts/sync_tensorflow_protobuf_stubs.sh
++ dirname scripts/sync_tensorflow_protobuf_stubs.sh
+ cd scripts
+ cd ../stubs/tensorflow
+++ dirname scripts/sync_tensorflow_protobuf_stubs.sh
++ realpath scripts/..
realpath: scripts/..: No such file or directory
+ REPO_ROOT=

Don't have time to figure this out right now, sorry.

[Avasam]

The script also set the version of stubs/protobuf/METADATA.toml to 4.21.0; I changed it back.

Should the extra_description field still be updated to mention3.5.0 instead of 3.4.0 ?

[JelleZijlstra]

@Avasam I get this locally now when running the TF script:

+ protoc --mypy_out relax_strict_optional_primitives:/home/jelle/repos/typeshed/stubs/tensorflow tensorflow/compiler/xla/autotune_results.proto tensorflow/compiler/xla/xla_data.proto tensorflow/compiler/xla/xla.proto tensorflow/compiler/xla/service/hlo_execution_profile_data.proto tensorflow/compiler/xla/service/hlo_profile_printer_data.proto tensorflow/compiler/xla/service/hlo.proto tensorflow/compiler/xla/service/metrics.proto tensorflow/compiler/xla/service/test_compilation_environment.proto tensorflow/core/example/example_parser_configuration.proto tensorflow/core/example/example.proto tensorflow/core/example/feature.proto tensorflow/core/framework/allocation_description.proto tensorflow/core/framework/api_def.proto tensorflow/core/framework/attr_value.proto tensorflow/core/framework/cost_graph.proto tensorflow/core/framework/dataset_metadata.proto tensorflow/core/framework/dataset_options.proto tensorflow/core/framework/dataset.proto tensorflow/core/framework/device_attributes.proto tensorflow/core/framework/full_type.proto tensorflow/core/framework/function.proto tensorflow/core/framework/graph.proto tensorflow/core/framework/graph_transfer_info.proto tensorflow/core/framework/kernel_def.proto tensorflow/core/framework/log_memory.proto tensorflow/core/framework/model.proto tensorflow/core/framework/node_def.proto tensorflow/core/framework/op_def.proto tensorflow/core/framework/optimized_function_graph.proto tensorflow/core/framework/reader_base.proto tensorflow/core/framework/resource_handle.proto tensorflow/core/framework/step_stats.proto tensorflow/core/framework/summary.proto tensorflow/core/framework/tensor_description.proto tensorflow/core/framework/tensor.proto tensorflow/core/framework/tensor_shape.proto tensorflow/core/framework/tensor_slice.proto tensorflow/core/framework/types.proto tensorflow/core/framework/variable.proto tensorflow/core/framework/versions.proto tensorflow/core/protobuf/autotuning.proto tensorflow/core/protobuf/bfc_memory_map.proto tensorflow/core/protobuf/cluster.proto tensorflow/core/protobuf/composite_tensor_variant.proto tensorflow/core/protobuf/config.proto tensorflow/core/protobuf/control_flow.proto tensorflow/core/protobuf/conv_autotuning.proto tensorflow/core/protobuf/core_platform_payloads.proto tensorflow/core/protobuf/critical_section.proto tensorflow/core/protobuf/data_service.proto tensorflow/core/protobuf/debug_event.proto tensorflow/core/protobuf/debug.proto tensorflow/core/protobuf/device_filters.proto tensorflow/core/protobuf/device_properties.proto tensorflow/core/protobuf/eager_service.proto tensorflow/core/protobuf/error_codes.proto tensorflow/core/protobuf/fingerprint.proto tensorflow/core/protobuf/graph_debug_info.proto tensorflow/core/protobuf/master.proto tensorflow/core/protobuf/master_service.proto tensorflow/core/protobuf/meta_graph.proto tensorflow/core/protobuf/named_tensor.proto tensorflow/core/protobuf/queue_runner.proto tensorflow/core/protobuf/remote_tensor_handle.proto tensorflow/core/protobuf/replay_log.proto tensorflow/core/protobuf/rewriter_config.proto tensorflow/core/protobuf/rpc_options.proto tensorflow/core/protobuf/saved_model.proto tensorflow/core/protobuf/saved_object_graph.proto tensorflow/core/protobuf/saver.proto tensorflow/core/protobuf/service_config.proto tensorflow/core/protobuf/snapshot.proto tensorflow/core/protobuf/struct.proto tensorflow/core/protobuf/tensor_bundle.proto tensorflow/core/protobuf/tensorflow_server.proto tensorflow/core/protobuf/trackable_object_graph.proto tensorflow/core/protobuf/transport_options.proto tensorflow/core/protobuf/verifier_config.proto tensorflow/core/protobuf/worker.proto tensorflow/core/protobuf/worker_service.proto tensorflow/core/protobuf/tpu/compilation_result.proto tensorflow/core/protobuf/tpu/compile_metadata.proto tensorflow/core/protobuf/tpu/dynamic_padding.proto tensorflow/core/protobuf/tpu/optimization_parameters.proto tensorflow/core/protobuf/tpu/topology.proto tensorflow/core/protobuf/tpu/tpu_embedding_configuration.proto tensorflow/core/util/event.proto tensorflow/core/util/example_proto_fast_parsing_test.proto tensorflow/core/util/memmapped_file_system.proto tensorflow/core/util/saved_tensor_slice.proto tensorflow/core/util/test_log.proto tensorflow/python/keras/protobuf/projector_config.proto tensorflow/python/keras/protobuf/saved_metadata.proto tensorflow/python/keras/protobuf/versions.proto tensorflow/tsl/protobuf/autotuning.proto tensorflow/tsl/protobuf/bfc_memory_map.proto tensorflow/tsl/protobuf/coordination_config.proto tensorflow/tsl/protobuf/coordination_service.proto tensorflow/tsl/protobuf/distributed_runtime_payloads.proto tensorflow/tsl/protobuf/dnn.proto tensorflow/tsl/protobuf/error_codes.proto tensorflow/tsl/protobuf/histogram.proto tensorflow/tsl/protobuf/rpc_options.proto tensorflow/tsl/protobuf/test_log.proto
google/protobuf/any.proto: File not found.
google/protobuf/duration.proto: File not found.
google/protobuf/wrappers.proto: File not found.
tensorflow/tsl/protobuf/dnn.proto: Import "google/protobuf/wrappers.proto" was not found or had errors.
tensorflow/tsl/protobuf/dnn.proto:139:3: "google.protobuf.UInt64Value" is not defined.
tensorflow/tsl/protobuf/autotuning.proto: Import "google/protobuf/any.proto" was not found or had errors.
tensorflow/tsl/protobuf/autotuning.proto: Import "google/protobuf/duration.proto" was not found or had errors.
tensorflow/tsl/protobuf/autotuning.proto: Import "tensorflow/tsl/protobuf/dnn.proto" was not found or had errors.
tensorflow/tsl/protobuf/autotuning.proto:55:7: "stream_executor.dnn.AlgorithmProto" is not defined.
tensorflow/tsl/protobuf/autotuning.proto:77:3: "google.protobuf.Duration" is not defined.
tensorflow/tsl/protobuf/autotuning.proto:85:5: "stream_executor.dnn.AlgorithmProto" is not defined.
tensorflow/tsl/protobuf/autotuning.proto:92:3: "google.protobuf.Any" is not defined.
tensorflow/compiler/xla/autotune_results.proto: Import "tensorflow/tsl/protobuf/autotuning.proto" was not found or had errors.
tensorflow/compiler/xla/autotune_results.proto:42:5: "tensorflow.AutotuneResult" is not defined.

Do you know what's up with that? I indeed can't find a any.proto file in the TF repo.

[Avasam]

protoc --mypy_out relax_strict_optional_primitives

Not sure, I just ran it fresh locally and it seems to be just fine

[hmc-cs-mdrissi]

I indeed can't find a any.proto file in the TF repo.

any.proto is similar to standard library protobuf file (or maybe like protobuf-dev in analogy to python-dev). It's not part of tensorflow. It's sign that your protobuf install includes compiler, but not protobuf related headers. Fix is something like apt install libprotobuf-dev.

One option to make the script more robust is maybe to instead use python -m grpc_tools.protoc and add grpcio-tools to requirements. grpcio-tools is pip package that contains protobuf compiler + related standard protobuf files like any.proto. I'm unsure there's currently documented way of how to get protoc here. pip install grpcio-tools==X would effectively pin well defined protoc version as each grpcio-tools version submodules a specific protoc version.

[Avasam]

If #10912 is merged first, this becomes:

Suggested change

	isort "$REPO_ROOT/stubs/tensorflow/tensorflow"
	ruff check "$REPO_ROOT/stubs/tensorflow/tensorflow" --fix-only

[nipunn1313]

lgtm

[github-actions]

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉