ARROW-9950: [Rust] [DataFusion] Made UDFs usable without a registry by jorgecarleitao · Pull Request #8144 · apache/arrow

jorgecarleitao · 2020-09-09T03:39:05Z

This functionality is relevant for the DataFrame API only.

Sometimes a UDF declaration happens during planning, and it is expressive when the user can use it directly, without first register it in the execution context's registry and accessing the registry to plan it.

This PR proposes that, given a UDF pow: ScalarUDF named "pow", users can (logically) plan it directly:

// plan a calllet expr = pow.call(vec![col("a"), col("b")]);

or register it (as before)

// register it
ctx.register_udf(pow);

or plan it from the registry (as before):

// access it from the registry via its name, when `pow` is not in scopelet pow = df.registry().udf("pow")?;// plan a calllet expr = pow.call(vec![col("a"), col("b")]);

I changed the signature of the registry from .udf(name, args) -> Expr to .udf(name) -> ScalarUDF, so that the API to call UDFs is the same regardless of whether we take it from the registry or use it directly call(args). IMO it also makes it a bit more expressive that we are calling the udf.

github-actions · 2020-09-09T03:46:41Z

https://issues.apache.org/jira/browse/ARROW-9950

andygrove

LGTM

alamb · 2020-09-10T11:44:37Z

+ let pow = df.registry().udf("pow")?;
+ // equivalent to expr
+ let expr1 = pow.call(vec![col("a"), col("b")]);
+


Suggested change
assert_eq!(expr, expr1);

alamb · 2020-09-10T11:45:14Z

- /// Constructs a logical expression with a call to the udf.
- fn udf(&self, name: &str, args: Vec<Expr>) -> Result<Expr>;
+ /// Returns a reference to the udf named `name`.
+ fn udf(&self, name: &str) -> Result<&ScalarUDF>;


jorgecarleitao · 2020-09-09T03:39:05Z

This functionality is relevant for the DataFrame API only.

Sometimes a UDF declaration happens during planning, and it is expressive when the user can use it directly, without first register it in the execution context's registry and accessing the registry to plan it.

This PR proposes that, given a UDF pow: ScalarUDF named "pow", users can (logically) plan it directly:

// plan a calllet expr = pow.call(vec![col("a"), col("b")]);

or register it (as before)

// register it
ctx.register_udf(pow);

or plan it from the registry (as before):

// access it from the registry via its name, when `pow` is not in scopelet pow = df.registry().udf("pow")?;// plan a calllet expr = pow.call(vec![col("a"), col("b")]);

I changed the signature of the registry from .udf(name, args) -> Expr to .udf(name) -> ScalarUDF, so that the API to call UDFs is the same regardless of whether we take it from the registry or use it directly call(args). IMO it also makes it a bit more expressive that we are calling the udf.

github-actions · 2020-09-09T03:46:41Z

https://issues.apache.org/jira/browse/ARROW-9950

andygrove

LGTM

alamb · 2020-09-10T11:44:37Z

+ let pow = df.registry().udf("pow")?;
+ // equivalent to expr
+ let expr1 = pow.call(vec![col("a"), col("b")]);
+


Suggested change
assert_eq!(expr, expr1);

alamb · 2020-09-10T11:45:14Z

- /// Constructs a logical expression with a call to the udf.
- fn udf(&self, name: &str, args: Vec<Expr>) -> Result<Expr>;
+ /// Returns a reference to the udf named `name`.
+ fn udf(&self, name: &str) -> Result<&ScalarUDF>;


jorgecarleitao · 2020-09-09T03:39:05Z

This functionality is relevant for the DataFrame API only.

Sometimes a UDF declaration happens during planning, and it is expressive when the user can use it directly, without first register it in the execution context's registry and accessing the registry to plan it.

This PR proposes that, given a UDF pow: ScalarUDF named "pow", users can (logically) plan it directly:

// plan a calllet expr = pow.call(vec![col("a"), col("b")]);

or register it (as before)

// register it
ctx.register_udf(pow);

or plan it from the registry (as before):

// access it from the registry via its name, when `pow` is not in scopelet pow = df.registry().udf("pow")?;// plan a calllet expr = pow.call(vec![col("a"), col("b")]);

I changed the signature of the registry from .udf(name, args) -> Expr to .udf(name) -> ScalarUDF, so that the API to call UDFs is the same regardless of whether we take it from the registry or use it directly call(args). IMO it also makes it a bit more expressive that we are calling the udf.

github-actions · 2020-09-09T03:46:41Z

https://issues.apache.org/jira/browse/ARROW-9950

andygrove

LGTM

alamb · 2020-09-10T11:44:37Z

+ let pow = df.registry().udf("pow")?;
+ // equivalent to expr
+ let expr1 = pow.call(vec![col("a"), col("b")]);
+


Suggested change
assert_eq!(expr, expr1);

alamb · 2020-09-10T11:45:14Z

- /// Constructs a logical expression with a call to the udf.
- fn udf(&self, name: &str, args: Vec<Expr>) -> Result<Expr>;
+ /// Returns a reference to the udf named `name`.
+ fn udf(&self, name: &str) -> Result<&ScalarUDF>;


jorgecarleitao · 2020-09-09T03:39:05Z

This functionality is relevant for the DataFrame API only.

Sometimes a UDF declaration happens during planning, and it is expressive when the user can use it directly, without first register it in the execution context's registry and accessing the registry to plan it.

This PR proposes that, given a UDF pow: ScalarUDF named "pow", users can (logically) plan it directly:

// plan a calllet expr = pow.call(vec![col("a"), col("b")]);

or register it (as before)

// register it
ctx.register_udf(pow);

or plan it from the registry (as before):

// access it from the registry via its name, when `pow` is not in scopelet pow = df.registry().udf("pow")?;// plan a calllet expr = pow.call(vec![col("a"), col("b")]);

I changed the signature of the registry from .udf(name, args) -> Expr to .udf(name) -> ScalarUDF, so that the API to call UDFs is the same regardless of whether we take it from the registry or use it directly call(args). IMO it also makes it a bit more expressive that we are calling the udf.

github-actions · 2020-09-09T03:46:41Z

https://issues.apache.org/jira/browse/ARROW-9950

andygrove

LGTM

alamb · 2020-09-10T11:44:37Z

+ let pow = df.registry().udf("pow")?;
+ // equivalent to expr
+ let expr1 = pow.call(vec![col("a"), col("b")]);
+


Suggested change
assert_eq!(expr, expr1);

alamb · 2020-09-10T11:45:14Z

- /// Constructs a logical expression with a call to the udf.
- fn udf(&self, name: &str, args: Vec<Expr>) -> Result<Expr>;
+ /// Returns a reference to the udf named `name`.
+ fn udf(&self, name: &str) -> Result<&ScalarUDF>;


jorgecarleitao · 2020-09-09T03:39:05Z

This functionality is relevant for the DataFrame API only.

Sometimes a UDF declaration happens during planning, and it is expressive when the user can use it directly, without first register it in the execution context's registry and accessing the registry to plan it.

This PR proposes that, given a UDF pow: ScalarUDF named "pow", users can (logically) plan it directly:

// plan a calllet expr = pow.call(vec![col("a"), col("b")]);

or register it (as before)

// register it
ctx.register_udf(pow);

or plan it from the registry (as before):

// access it from the registry via its name, when `pow` is not in scopelet pow = df.registry().udf("pow")?;// plan a calllet expr = pow.call(vec![col("a"), col("b")]);

I changed the signature of the registry from .udf(name, args) -> Expr to .udf(name) -> ScalarUDF, so that the API to call UDFs is the same regardless of whether we take it from the registry or use it directly call(args). IMO it also makes it a bit more expressive that we are calling the udf.

github-actions · 2020-09-09T03:46:41Z

https://issues.apache.org/jira/browse/ARROW-9950

andygrove

LGTM

alamb · 2020-09-10T11:44:37Z

+ let pow = df.registry().udf("pow")?;
+ // equivalent to expr
+ let expr1 = pow.call(vec![col("a"), col("b")]);
+


Suggested change
assert_eq!(expr, expr1);

alamb · 2020-09-10T11:45:14Z

- /// Constructs a logical expression with a call to the udf.
- fn udf(&self, name: &str, args: Vec<Expr>) -> Result<Expr>;
+ /// Returns a reference to the udf named `name`.
+ fn udf(&self, name: &str) -> Result<&ScalarUDF>;


jorgecarleitao · 2020-09-09T03:39:05Z

This functionality is relevant for the DataFrame API only.

Sometimes a UDF declaration happens during planning, and it is expressive when the user can use it directly, without first register it in the execution context's registry and accessing the registry to plan it.

This PR proposes that, given a UDF pow: ScalarUDF named "pow", users can (logically) plan it directly:

// plan a calllet expr = pow.call(vec![col("a"), col("b")]);

or register it (as before)

// register it
ctx.register_udf(pow);

or plan it from the registry (as before):

// access it from the registry via its name, when `pow` is not in scopelet pow = df.registry().udf("pow")?;// plan a calllet expr = pow.call(vec![col("a"), col("b")]);

I changed the signature of the registry from .udf(name, args) -> Expr to .udf(name) -> ScalarUDF, so that the API to call UDFs is the same regardless of whether we take it from the registry or use it directly call(args). IMO it also makes it a bit more expressive that we are calling the udf.

github-actions · 2020-09-09T03:46:41Z

https://issues.apache.org/jira/browse/ARROW-9950

andygrove

LGTM

alamb · 2020-09-10T11:44:37Z

+ let pow = df.registry().udf("pow")?;
+ // equivalent to expr
+ let expr1 = pow.call(vec![col("a"), col("b")]);
+


Suggested change
assert_eq!(expr, expr1);

alamb · 2020-09-10T11:45:14Z

- /// Constructs a logical expression with a call to the udf.
- fn udf(&self, name: &str, args: Vec<Expr>) -> Result<Expr>;
+ /// Returns a reference to the udf named `name`.
+ fn udf(&self, name: &str) -> Result<&ScalarUDF>;


jorgecarleitao · 2020-09-09T03:39:05Z

This functionality is relevant for the DataFrame API only.

Sometimes a UDF declaration happens during planning, and it is expressive when the user can use it directly, without first register it in the execution context's registry and accessing the registry to plan it.

This PR proposes that, given a UDF pow: ScalarUDF named "pow", users can (logically) plan it directly:

// plan a calllet expr = pow.call(vec![col("a"), col("b")]);

or register it (as before)

// register it
ctx.register_udf(pow);

or plan it from the registry (as before):

// access it from the registry via its name, when `pow` is not in scopelet pow = df.registry().udf("pow")?;// plan a calllet expr = pow.call(vec![col("a"), col("b")]);

I changed the signature of the registry from .udf(name, args) -> Expr to .udf(name) -> ScalarUDF, so that the API to call UDFs is the same regardless of whether we take it from the registry or use it directly call(args). IMO it also makes it a bit more expressive that we are calling the udf.

github-actions · 2020-09-09T03:46:41Z

https://issues.apache.org/jira/browse/ARROW-9950

andygrove

LGTM

alamb · 2020-09-10T11:44:37Z

+ let pow = df.registry().udf("pow")?;
+ // equivalent to expr
+ let expr1 = pow.call(vec![col("a"), col("b")]);
+


Suggested change
assert_eq!(expr, expr1);

alamb · 2020-09-10T11:45:14Z

- /// Constructs a logical expression with a call to the udf.
- fn udf(&self, name: &str, args: Vec<Expr>) -> Result<Expr>;
+ /// Returns a reference to the udf named `name`.
+ fn udf(&self, name: &str) -> Result<&ScalarUDF>;


jorgecarleitao · 2020-09-09T03:39:05Z

This functionality is relevant for the DataFrame API only.

Sometimes a UDF declaration happens during planning, and it is expressive when the user can use it directly, without first register it in the execution context's registry and accessing the registry to plan it.

This PR proposes that, given a UDF pow: ScalarUDF named "pow", users can (logically) plan it directly:

// plan a calllet expr = pow.call(vec![col("a"), col("b")]);

or register it (as before)

// register it
ctx.register_udf(pow);

or plan it from the registry (as before):

// access it from the registry via its name, when `pow` is not in scopelet pow = df.registry().udf("pow")?;// plan a calllet expr = pow.call(vec![col("a"), col("b")]);

I changed the signature of the registry from .udf(name, args) -> Expr to .udf(name) -> ScalarUDF, so that the API to call UDFs is the same regardless of whether we take it from the registry or use it directly call(args). IMO it also makes it a bit more expressive that we are calling the udf.

github-actions · 2020-09-09T03:46:41Z

https://issues.apache.org/jira/browse/ARROW-9950

andygrove

LGTM

alamb · 2020-09-10T11:44:37Z

+ let pow = df.registry().udf("pow")?;
+ // equivalent to expr
+ let expr1 = pow.call(vec![col("a"), col("b")]);
+


Suggested change
assert_eq!(expr, expr1);

alamb · 2020-09-10T11:45:14Z

- /// Constructs a logical expression with a call to the udf.
- fn udf(&self, name: &str, args: Vec<Expr>) -> Result<Expr>;
+ /// Returns a reference to the udf named `name`.
+ fn udf(&self, name: &str) -> Result<&ScalarUDF>;


Made UDFs be plannable without access to the registry.
c0ac23d

andygrove added Component: Rust Component: Rust - DataFusion labels Sep 9, 2020

andygrove approved these changes Sep 10, 2020
View reviewed changes

alamb approved these changes Sep 10, 2020
View reviewed changes

andygrove closed this in 2726a71Sep 12, 2020

jorgecarleitao deleted the udf_usage branch October 28, 2020 04:17

asfimport mentioned this pull request Sep 12, 2020
[Rust] [DataFusion] Allow UDF usage without registry #25977
Closed

Made UDFs be plannable without access to the registry.
c0ac23d

andygrove added Component: Rust Component: Rust - DataFusion labels Sep 9, 2020

andygrove approved these changes Sep 10, 2020
View reviewed changes

alamb approved these changes Sep 10, 2020
View reviewed changes

andygrove closed this in 2726a71Sep 12, 2020

jorgecarleitao deleted the udf_usage branch October 28, 2020 04:17

asfimport mentioned this pull request Sep 12, 2020
[Rust] [DataFusion] Allow UDF usage without registry #25977
Closed

Made UDFs be plannable without access to the registry.
c0ac23d

andygrove added Component: Rust Component: Rust - DataFusion labels Sep 9, 2020

andygrove approved these changes Sep 10, 2020
View reviewed changes

alamb approved these changes Sep 10, 2020
View reviewed changes

andygrove closed this in 2726a71Sep 12, 2020

jorgecarleitao deleted the udf_usage branch October 28, 2020 04:17

asfimport mentioned this pull request Sep 12, 2020
[Rust] [DataFusion] Allow UDF usage without registry #25977
Closed

Made UDFs be plannable without access to the registry.
c0ac23d

andygrove added Component: Rust Component: Rust - DataFusion labels Sep 9, 2020

andygrove approved these changes Sep 10, 2020
View reviewed changes

alamb approved these changes Sep 10, 2020
View reviewed changes

andygrove closed this in 2726a71Sep 12, 2020

jorgecarleitao deleted the udf_usage branch October 28, 2020 04:17

asfimport mentioned this pull request Sep 12, 2020
[Rust] [DataFusion] Allow UDF usage without registry #25977
Closed

Made UDFs be plannable without access to the registry.
c0ac23d

andygrove added Component: Rust Component: Rust - DataFusion labels Sep 9, 2020

andygrove approved these changes Sep 10, 2020
View reviewed changes

alamb approved these changes Sep 10, 2020
View reviewed changes

andygrove closed this in 2726a71Sep 12, 2020

jorgecarleitao deleted the udf_usage branch October 28, 2020 04:17

asfimport mentioned this pull request Sep 12, 2020
[Rust] [DataFusion] Allow UDF usage without registry #25977
Closed

Made UDFs be plannable without access to the registry.
c0ac23d

andygrove added Component: Rust Component: Rust - DataFusion labels Sep 9, 2020

andygrove approved these changes Sep 10, 2020
View reviewed changes

alamb approved these changes Sep 10, 2020
View reviewed changes

andygrove closed this in 2726a71Sep 12, 2020

jorgecarleitao deleted the udf_usage branch October 28, 2020 04:17

asfimport mentioned this pull request Sep 12, 2020
[Rust] [DataFusion] Allow UDF usage without registry #25977
Closed

Made UDFs be plannable without access to the registry.
c0ac23d

andygrove added Component: Rust Component: Rust - DataFusion labels Sep 9, 2020

andygrove approved these changes Sep 10, 2020
View reviewed changes

alamb approved these changes Sep 10, 2020
View reviewed changes

andygrove closed this in 2726a71Sep 12, 2020

jorgecarleitao deleted the udf_usage branch October 28, 2020 04:17

asfimport mentioned this pull request Sep 12, 2020
[Rust] [DataFusion] Allow UDF usage without registry #25977
Closed

Made UDFs be plannable without access to the registry.
c0ac23d

andygrove added Component: Rust Component: Rust - DataFusion labels Sep 9, 2020

andygrove approved these changes Sep 10, 2020
View reviewed changes

alamb approved these changes Sep 10, 2020
View reviewed changes

andygrove closed this in 2726a71Sep 12, 2020

jorgecarleitao deleted the udf_usage branch October 28, 2020 04:17

asfimport mentioned this pull request Sep 12, 2020
[Rust] [DataFusion] Allow UDF usage without registry #25977
Closed

Uh oh!

Conversation

jorgecarleitao commented Sep 9, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actionsBot commented Sep 9, 2020

Uh oh!

andygrove left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

jorgecarleitao commented Sep 9, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actionsBot commented Sep 9, 2020

Uh oh!

andygrove left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

jorgecarleitao commented Sep 9, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actionsBot commented Sep 9, 2020

Uh oh!

andygrove left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

jorgecarleitao commented Sep 9, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actionsBot commented Sep 9, 2020

Uh oh!

andygrove left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jorgecarleitao commented Sep 9, 2020•
edited
Loading

jorgecarleitao commented Sep 9, 2020•
edited
Loading

jorgecarleitao commented Sep 9, 2020•
edited
Loading

jorgecarleitao commented Sep 9, 2020•
edited
Loading

jorgecarleitao commented Sep 9, 2020•
edited
Loading