Making pb2 auto-gen script also copy over .proto files.

[dhermes]

@tseaver As you'll notice, this is quite nasty. I'm open to suggestions for making this better. Some simple options:

• We are content that this is "good enough"
• I become competent using bash
• I rewrite this script in Python with judicious use of some system calls

FWIW, this is in advance of v1beta3 for datastore, which will re-use some of these components (e.g. timestamp) so we need this pipeline to be a bit more polished.

[tseaver]

It seems a little weird that this is a makefile, given that the logic is all imperative. It seems like there should be a way to use pattern-based dependency rules to do the compilation / copying.

[dhermes]

I agree, I hate it. Should I just take a stab at re-writing this in Python (where I can actually use a for loop)?

---

I must say I don't know much about the protoc tool.

• There is a way to specify --proto_path to look for dependencies, but we end up having lots of paths
• There is no way in protoc to accomplish the goal of this PR (copying over the .proto files while also renaming them with an _ in front)
• There is a flag that may be useful that is in the updated protoc but not in the one in the Debian package repo

--dependency_out=FILE Write a dependency output file in the format
expected by make. This writes the transitive
set of input file paths to FILE

• Since the generated files all live in the google/... file structure and the libraries don't use a namespace package, we have to move them, so again we have to manually locate the path
• It seems that the following is supported

protoc --proto_path=src --python_out=build/gen src/foo.proto src/bar/baz.proto

which produces build/gen/foo_pb2.py and build/gen/bar/baz_pb2.py

[dhermes]

@tseaver PTAL. This is (hopefully) much easier to read and for now can hold off implementing in Python (and maybe for always).

[tseaver]

Rather than treating the makefile like a bash script, could we make the generated _pb2.py files in the right locations be actual make targets, and then write pattern-based rules for producing them (in the two or three steps needed) from the original .proto files?

[dhermes]

Want to take a stab at it?

[dhermes]

@tseaver can we punt on the Makefile optimizations and try to handle it if/when the service specific protos get their own repo?

The point of this PR was to prepare for building v1beta3 in a more systematic way but it is blocking other PRs.

If you'd like I could also pull the relevant content apart from PRs that use this as a diff base

[tseaver]

@dhermes If datastore v1beta3 is the driver, I'm not sure why we're mucking about with generating protobuf stuff: wouldn't we be better off just importing from googleapis-common-protos?

[dhermes]

That will only contain "common" protos (see #1353 for which protos that means). This doesn't help for service specific protos (Bigtable, datastore and soon pub/sub and logging)

[tseaver]

Weird. The googleapis repository actually contains the datastore v1beta3 protos. It doesn't have any of the code for making the Python releases, however.

[dhermes]

@tbetbetbe never told me how they generated the code but the point of common is the set of protos that are utilities used by services but aren't defined within a specific service (e.g. google.api.duration)

[tseaver]

If we could get the googleapis team to either a) make PyPI releases for the service-specific protobuf wrappers (`googleapis-datastore-protos', etc.), or b) publish how they are generating the one they made already, that would be helpful. I'm currently hacking on how to use their Makefile to generate the Python code.

[dhermes]

@tseaver I filed #1384 to track this and cc-ed the lead of the team in charge PyPI package for common. Can we merge this and discuss the path forward in the issue?

[tseaver]

Yup. LGTM

[dhermes]

FWIW I've suggested your a) in recent conversations with Tim.