Add support for protobuf version 4.29.x

[ilanKeshet]

Describe the enhancement requested

Hi all,
First of all let me start by saying Thank You 🙏

Please add support for protobuf major version 4 preferably 4.29.1

At December 2023 Protobuf java library added support for the Editions feature which resulted in major version change starting at version 4.26

I've noticed that a PR upgrading to version 4 was previously rejected #1308.

The error below is a result of attempting to run version 1.15.0 with protobuf version 4.29.1 .
the specific failure presented is the result of the change linked below.
protocolbuffers/protobuf@65c65c2#diff-2228551d02c6661809ca7103db9512eef4c2d01f35556d42316543d92a89edef:~:text=Edition%20getEdition()%20%7B

java.lang.NoSuchMethodError: com.google.protobuf.Descriptors$FileDescriptor.getSyntax()Lcom/google/protobuf/Descriptors$FileDescriptor$Syntax;
	at org.apache.parquet.proto.ProtoWriteSupport$MessageWriter.writeAllFields(ProtoWriteSupport.java:449)
	at org.apache.parquet.proto.ProtoWriteSupport$MessageWriter.writeTopLevelMessage(ProtoWriteSupport.java:423)
	at org.apache.parquet.proto.ProtoWriteSupport.write(ProtoWriteSupport.java:139)
	at org.apache.parquet.proto.ProtoWriteSupport.write(ProtoWriteSupport.java:74)
	at org.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:152)
	at org.apache.parquet.hadoop.ParquetWriter.write(ParquetWriter.java:425

Many many thanks
🙏

Component(s)

Protobuf

[wgtmac]

@Fokko Is it safe to bump the major version of protobuf?

[ilanKeshet]

According to discussion on #1308 (comment) the parquet-java project would have to be upgraded to Java-11 before we could support protobuf version 4.x

[code4dc]

Is there a ticket out there to update to Java 11?

[uwemaurer]

We were able to upgrade to protobuf 4.32.1 and only had to replace ProtoWriteSupport.java with a patched version, using the patch below:

@@ -445,10 +445,14 @@
 
 private void writeAllFields(MessageOrBuilder pb) {
 Descriptor messageDescriptor = pb.getDescriptorForType();
- Descriptors.FileDescriptor.Syntax syntax =
- messageDescriptor.getFile().getSyntax();
+ String syntax =
+ messageDescriptor.getFile().toProto().getSyntax();
+ if ("editions".equals(syntax)) {
+ throw new UnsupportedOperationException("protocol buffers 'editions' not supported");
+ }
+ boolean isProto2 = !"proto3".equals(syntax);
 
- if (Descriptors.FileDescriptor.Syntax.PROTO2.equals(syntax)) {
+ if (isProto2) {
 // Returns changed fields with values. Map is ordered by id.
 Map<FieldDescriptor, Object> changedPbFields = pb.getAllFields();
 
@@ -464,7 +468,7 @@
 int fieldIndex = fieldDescriptor.getIndex();
 fieldWriters[fieldIndex].writeField(entry.getValue());
 }
- } else if (Descriptors.FileDescriptor.Syntax.PROTO3.equals(syntax)) {
+ } else {
 List<FieldDescriptor> fieldDescriptors = messageDescriptor.getFields();
 for (FieldDescriptor fieldDescriptor : fieldDescriptors) {
 FieldDescriptor.Type type = fieldDescriptor.getType();
@@ -797,4 +801,3 @@
 throw new InvalidRecordException(exceptionMsg);
 }
 }

This code should actually also work with the protobuf 3 library. The problem was that in protobuf 4 the getFile().getSyntax() method is not there anymore, but getFile().toProto().getSyntax() works (returns a String instead of enum)

[code4dc]

@uwemaurer Can you create a PR for this? Maybe they'll agree to merge your patch for the next release.

[Fokko]

@uwemaurer That's great! Can you create a PR so you'll be credited for the work? We can easily test to make sure that it still works against Protobuf 3 by first doing the patch and then bumping protobuf in a separate PR.

[uwemaurer]

yes I will prepare a PR