[BEAM-9044] Protobuf options to Schema options

[alexvanboxel]

[BEAM-9044] Protobuf options to Schema options

Convert protobuf options to Row schema options. It gets the options from a proto message descriptor and converts them into beam schema options on the Schema. It does the same from proto field descriptors, converting them into beam field options.

The converter creates typed options as the proto options are also fully typed. For the convertion it uses the ProtoDynamicSchema to resolve the types.

---

Lang	SDK	Apex	Dataflow	Flink	Gearpump	Samza	Spark
Go		---	---		---	---
Java
Python		---			---	---
XLang	---	---	---		---	---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website
Non-portable
Portable	---		---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

[alexvanboxel]

Run Java_Examples_Dataflow PreCommit

[alexvanboxel]

Run Java_Examples_Dataflow PreCommit

[alexvanboxel]

@reuvenlax friendly ping

[reuvenlax]

We want to switch this metadata to use options. However you are translating proto option names directly to field names, which means there's no guarantee that these other options won't conflict. Maybe prefix all of these options with something to prevent potential conflict?

[alexvanboxel]

We want to switch this metadata to use options. However you are translating proto option names directly to field names, which means there's no guarantee that these other options won't conflict. Maybe prefix all of these options with something to prevent potential conflict?

What about beam:option:proto:, that would make an proto option vptech.data.v1.description into beam:option:proto:vptech.data.v1.description. Making it a URI format

beam:option : fixed prefix for options, proto : for the extension (will be avro for avro), vptech.data.v1 package of the extension and desciption the proto extension field.

[TheNeuralBit]

I think Reuven was referring to the metadata that's added in withMetaData, since that's the "special" proto metadata

[alexvanboxel]

Is it, I don't think so... because I told Reuven I would do metadata after this. Still I think it's a good idea... and with your comment I want to revise the URI:

• beam:option:proto:field:vptech.data.v1.description for a proto field option
• beam:option:proto:message:vptech.data.v1.description for a proto message option
• beam:option:proto:meta:number for a proto field number
• beam:option:proto:meta:type_name for a proto type name

WDYT?

[reuvenlax]

I don't want to force all Beam option names to have a prefix - I think that's the wrong approach. I'm suggesting that we create well-known prefixes for these proto options (when translated to Beam options), so we can distinguish them from other options. The prefixes you suggested sound fine to me.

[alexvanboxel]

Now all the options are prefix. I moved the removal of the metadata to another ticket, because it's more involved:

[BEAM-9604] BIP-1: Remove schema metadata usage for Protobuf extension

[reuvenlax]

As mentioned above, I think we should add a well-known prefix here.

[alexvanboxel]

Run Java PreCommit

[alexvanboxel]

Run Java PreCommit

[alexvanboxel]

@reuvenlax if you don't have remarks (removing metadata will be in another PR) I can make a private build of master. I got a pipeline full of schema aware stuff to test this on.

[alexvanboxel]

Run Java PreCommit

[alexvanboxel]

I've decided to add the removal of metadata usage in the proto implementation as an additional commit, this makes the PR bigger but it also makes it complete. The option based storing of proto number and message name makes it much cleaner then the removed metadata based one.

[alexvanboxel]

@reuvenlax friendly reminder for reviewing the changes. It would be great if this would make the release (2.21) that will be cut in 6 days.

[reuvenlax]

LGTM

[alexvanboxel]

Thanks, now I can start working on the next parts and think about documentation. Hooray!