[SPARK-55062][Protobuf] Support proto2 extensions in protobuf functions

[dichlorodiphen]

What changes were proposed in this pull request?

This PR adds support for proto2 extensions to from_protobuf and to_protobuf (when file descriptor set is provided, as Java classes do not contain enough information to support extensions).

This is done by building an ExtensionRegistry and a map from descriptor name to its extensions. The registry is used during construction of the DynamicMessage to provide the Protobuf library with visibility of the extensions. The index is plumbed through the various helper classes for use in schema conversion and serde.

This new functionality is gated behind the Spark config property spark.sql.function.protobufExtensions.enabled.

Why are the changes needed?

Proto2 extensions are a valid, if somewhat uncommon, feature of Protobuf, and it therefore makes sense to incorporate them into the schema when provided so as to not confuse the user.

Does this PR introduce any user-facing change?

Yes. Previously, extension fields would be dropped by both from_protobuf and to_protobuf. Now, they are retained. This can be demonstrated with the minimal example below. See the unit tests for more examples.

messagePerson {
int32id=1;
extensions 100 to 200;
}
extendPerson {
int32age=100;
}

How was this patch tested?

Unit tests were added for the new behavior, including basic behavior, extending nested messages, and extensions defined in separate files.

Was this patch authored or co-authored using generative AI tooling?

Initial draft authored with Claude Code.

Generated-by: claude-4.5-opus

[github-actions]

JIRA Issue Information

=== Improvement SPARK-55062 ===
Summary: from_protobuf and to_protobuf do not support proto2 extensions
Assignee: None
Status: Open
Affected: ["4.1.1"]

---

This comment was automatically generated by GitHub Actions

[anishshri-db]

lgtm pending green CI