Add protobuf support for lambdas by gstvg · Pull Request #22362 · apache/datafusion

gstvg · 2026-05-19T11:52:24Z

Which issue does this PR close?

Part of #21172

Rationale for this change

Protobuf support wasn't implemented in main lambda PR to not make it even bigger

What changes are included in this PR?

Protobuf encoding and decoding (~1000 LOC in generated files, ~210 impl, ~400 tests)

Are these changes tested?

Unit tests, similar to the existing ones for scalar functions

Are there any user-facing changes?

Proto ExprType has new variants

…da_protobuf_new

LiaCastaneda · 2026-05-21T10:08:48Z

- return Err(Error::General(
- "Proto serialization error: Lambda not implemented".to_string(),
- ));
+ Expr::HigherOrderFunction(HigherOrderFunction { func, args }) => {


super nit: instead of appending this, can we move this next to the rest of functions? (Scalar, Aggregate etc)

Much easier to cross check the impls now 5688f3f thanks
(also did this to others similar matchs)

LiaCastaneda · 2026-05-21T10:14:49Z

+ HigherOrderUDFExprNode higher_order_udf_expr = 37;
+ Lambda lambda = 38;
+ LambdaVariable lambda_variable = 39;


super nit as well: Can we move this next to the other UDFs?

Isn't better being ordered by field id? so the next one who add a node can spot the next available field id right away

yeah makes sense

LiaCastaneda · 2026-05-21T10:22:17Z

+ expr_id,
+ expr_type: Some(protobuf::physical_expr_node::ExprType::LambdaVariable(
+ PhysicalLambdaVariableExprNode {
+ index: var.index() as u32,


for now we don't use this so it is always 0 no?

This is used now after lambda capture got merged. The variable below has index 1, for example:
datafusion/datafusion/sqllogictest/test_files/array/array_transform.slt
Line 102 in 50d74a7
01)ProjectionExec: expr=[array_transform(make_array(number@0), (v) -> CAST(v@1 AS Float64) + 3) as array_transform(make_array(t.number),(v) -> v + Float64(3))]

LiaCastaneda · 2026-05-21T10:25:41Z

+ protobuf::PhysicalHigherOrderUdfNode {
+ name: expr.name().to_string(),
+ args: serialize_physical_exprs(expr.args(), codec, proto_converter)?,
+ fun_definition: (!buf.is_empty()).then_some(buf),
+ },


Shouldn't we encode the return_field instead of resolving it from the schema when decoding (like ScalarUDF)?
Not really sure what would be the exact disadvantage of keeping it at is, but wondering if it would be cleaner to make the return type explicit.

I removed the HigherOrderFunctionExpr::new, the only constructor that receives return_field as arg, from the lambda PR, due to possible type mismatch, so it could be discussed on it's own PR when needed. See #21679 (review) and
FYI, the return field might not match the function output after the new arguments, but you don't have the schema here so you cant check that and I see ScalarFunctionExpr have the same problem
from https://github.com/apache/datafusion/pull/21679/changes#r3109735988 (quoting directly since the github link anchor doesn't work well)

LiaCastaneda · 2026-05-21T10:55:18Z

+ let serialize_name = extract_function_name(&expr);
+ let deserialize_name = extract_function_name(&deserialize);
+
+ assert_eq!(serialize_name, deserialize_name);


should we assert directly on expr and deserialize instead? since the Hofs implement PartialEq I think it should work
assert_eq!(expr, deserialized_expr);

Yes, 9a3f30c thanks
This was based on test_expression_serialization_roundtrip which checks only names but asserting directly on expr is definitively better

…da_protobuf_new

LiaCastaneda · 2026-07-09T08:00:58Z

Not sure why I thought this PR was already merged, not sure who to ping -- @comphead @alamb, are you able to take a look whenever you have time? 🙇 Half of the diff is the pbjson/prost generated files.

comphead · 2026-07-09T15:04:09Z

Not sure why I thought this PR was already merged, not sure who to ping -- @comphead @alamb, are you able to take a look whenever you have time? 🙇 Half of the diff is the pbjson/prost generated files.

Thanks @LiaCastaneda

@gstvg if you have anytime to resolve the conflicts

github-actions · 2026-07-10T05:40:42Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

 Cloning apache/main
Building datafusion-physical-expr v54.0.0 (current)
Built [ 29.106s] (current)
Parsing datafusion-physical-expr v54.0.0 (current)
Parsed [ 0.046s] (current)
Building datafusion-physical-expr v54.0.0 (baseline)
Built [ 28.385s] (baseline)
Parsing datafusion-physical-expr v54.0.0 (baseline)
Parsed [ 0.046s] (baseline)
Checking datafusion-physical-expr v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.395s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 58.988s] datafusion-physical-expr
Building datafusion-proto v54.0.0 (current)
Built [ 58.633s] (current)
Parsing datafusion-proto v54.0.0 (current)
Parsed [ 0.021s] (current)
Building datafusion-proto v54.0.0 (baseline)
Built [ 57.335s] (baseline)
Parsing datafusion-proto v54.0.0 (baseline)
Parsed [ 0.019s] (baseline)
Checking datafusion-proto v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 0.248s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 118.667s] datafusion-proto
Building datafusion-proto-models v54.0.0 (current)
Built [ 23.477s] (current)
Parsing datafusion-proto-models v54.0.0 (current)
Parsed [ 0.127s] (current)
Building datafusion-proto-models v54.0.0 (baseline)
Built [ 23.326s] (baseline)
Parsing datafusion-proto-models v54.0.0 (baseline)
Parsed [ 0.124s] (baseline)
Checking datafusion-proto-models v54.0.0 -> v54.0.0 (no change; assume patch)
Checked [ 1.677s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip
--- failure enum_variant_added: enum variant added on exhaustive enum ---
Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/enum_variant_added.ron
Failed in:
variant ExprType:HigherOrderUdfExpr in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:806
variant ExprType:Lambda in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:808
variant ExprType:LambdaVariable in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:810
variant ExprType:HigherOrderUdfExpr in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:806
variant ExprType:Lambda in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:808
variant ExprType:LambdaVariable in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:810
variant ExprType:HigherOrderUdf in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1565
variant ExprType:Lambda in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1567
variant ExprType:LambdaVariable in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1569
variant ExprType:HigherOrderUdf in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1565
variant ExprType:Lambda in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1567
variant ExprType:LambdaVariable in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1569
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 50.080s] datafusion-proto-models

…da_protobuf_new

gstvg · 2026-07-10T14:51:50Z

Thanks @LiaCastaneda @comphead, conflicts are resolved

comphead

Thanks @gstvg

## Which issue does this PR close? Part of apache#21172 ## Rationale for this change Protobuf support wasn't implemented in main lambda PR to not make it even bigger ## What changes are included in this PR? Protobuf encoding and decoding (~1000 LOC in generated files, ~210 impl, ~400 tests) ## Are these changes tested? Unit tests, similar to the existing ones for scalar functions ## Are there any user-facing changes? Proto `ExprType` has new variants

gstvg added 4 commits May 10, 2026 19:23

add lambda protobuf support
0ed7bc9

add tests
8436e64

Merge branch 'main' ofhttps://github.com/apache/datafusioninto lamb…
150b1c8
…da_protobuf_new

revert submodule bump
6b4ec0e

github-actionsBot added the proto Related to proto crate label May 19, 2026

This was referenced May 19, 2026
[EPIC] Full lambda support #21172
Open
Release DataFusion 54.0.0 (Apr 2026 / May 2026) #21080
Closed

LiaCastaneda approved these changes May 21, 2026
View reviewed changes

gstvg added 4 commits May 22, 2026 18:55

move higher-order udf match arms next to scalar udf
5688f3f

assert whole expr equals instead of only name in roudtrip test
9a3f30c

Merge branch 'main' ofhttps://github.com/apache/datafusioninto lamb…
1cf4ed7
…da_protobuf_new

use new SessionContext methods
6ac0278

github-actionsBot added the auto detected api change Auto detected API change label Jul 10, 2026

gstvg force-pushed the lambda_protobuf_new branch from 574ef1e to 17ad991Compare July 10, 2026 05:41

Merge branch 'main' ofhttps://github.com/apache/datafusioninto lamb…
27769b9
…da_protobuf_new

gstvg force-pushed the lambda_protobuf_new branch from 17ad991 to 27769b9Compare July 10, 2026 05:44

use new proto convention
051e819

github-actionsBot added the physical-expr Changes to the physical-expr crates label Jul 10, 2026

gstvg added 2 commits July 10, 2026 07:11

fix clippy
87a911e

Merge branch 'main' into lambda_protobuf_new
b43105b

comphead approved these changes Jul 10, 2026
View reviewed changes

comphead added this pull request to the merge queue Jul 10, 2026

Merged via the queue into apache:main with commit 7ac784fJul 10, 2026
40 checks passed

Tristan1900 mentioned this pull request Jul 23, 2026
[branch-54] Cherry-pick apache/datafusion#22362 DataDog/datafusion#158
Merged

Uh oh!

Conversation

gstvg commented May 19, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LiaCastaneda commented Jul 9, 2026

Uh oh!

comphead commented Jul 9, 2026

Uh oh!

github-actionsBot commented Jul 10, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gstvg commented Jul 10, 2026

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gstvg commented May 19, 2026•
edited
Loading

github-actionsBot commented Jul 10, 2026•
edited
Loading