Allow Logical expression ScalarVariable to represent an extension type or metadata

[batmnnn]

Which issue does this PR close?

• ClosesLogical expression ScalarVariable cannot represent an extension type or metadata #18230

Rationale for this change

Add richer variable metadata by switching Expr::ScalarVariable to store an Arrow Field, allowing planners to retain nullability and metadata when handling @var expressions.

What changes are included in this PR?

This PR updates ScalarVariable to use FieldRef so it can represent extension types and metadata in logical expressions.

Are these changes tested?

Yes

Are there any user-facing changes?

When planning queries with variables (e.g., @foo), the resulting logical expressions carry full field metadata instead of only a data type. This can affect downstream components that inspect nullability or custom metadata.

[batmnnn]

@paleolimbot can you review this sire

[paleolimbot]

Thank you!

@alamb kindly added a helper to make the DataType -> FieldRef cleaner, which I think will be useful here.

I think this also needs at least one test to ensure that the metadata and nullability of a variable are inferred correctly by Expr::to_field()!

I don't have the power to make the CI run but perhaps a committer who comes across this can hit the button 🙂

[alamb]

@batmnnn thank you for this PR. Do you have time to address @paleolimbot 's comments on this PR?

[batmnnn]

@alamb sorry for the delay got caught up in exams, just addressed them.

[paleolimbot]

Thank you!

This needs a committer to run CI...provided it's green, this looks good to me!

[alamb]

Thank you!

This needs a committer to run CI...provided it's green, this looks good to me!

I am so sorry I missed this -- I will check it out tomorrow

[paleolimbot]

Apologies for missing this on the last round...in addition to bringing the trait into scope to fix the CI, this also could use a quick test to ensure that creating a variable with a field returns the correct information. You should be able to pretty much copy the placeholder test:

datafusion/datafusion/expr/src/expr_schema.rs

Lines 920 to 941 in e4f2b49

	#[test]
	fntest_expr_placeholder(){
	let schema = MockExprSchema::new();
	letmut placeholder_meta = HashMap::new();
	placeholder_meta.insert("bar".to_string(),"buzz".to_string());
	let placeholder_meta = FieldMetadata::from(placeholder_meta);
	let expr = Expr::Placeholder(Placeholder::new_with_field(
	"".to_string(),
	Some(
	Field::new("",DataType::Utf8,true)
	.with_metadata(placeholder_meta.to_hashmap())
	.into(),
	),
	));
	assert_eq!(
	expr.data_type_and_nullable(&schema).unwrap(),
	(DataType::Utf8,true)
	);
	assert_eq!(placeholder_meta, expr.metadata(&schema).unwrap());

[batmnnn]

updated

[paleolimbot]

Thank you for writing the test!

This seems reasonable to me although it needs CI to run to make sure 🙂. @alamb Can you hit the button to run CI here?

[paleolimbot]

@batmnnn I see

error[E0599]: no method named `into_nullable_field_ref` found for enum `arrow::datatypes::DataType` in the current scope
--> datafusion/sql/src/expr/identifier.rs:48:38
|
48 | .map(|ty| ty.into_nullable_field_ref())
| ^^^^^^^^^^^^^^^^^^^^^^^
|
::: /__w/datafusion/datafusion/datafusion/common/src/datatype.rs:51:8
|
51 | fn into_nullable_field_ref(self) -> FieldRef;
| ----------------------- the method is available for `arrow::datatypes::DataType` here
|
= help: items from traits can only be used if the trait is in scope
help: trait `DataTypeExt` which provides `into_nullable_field_ref` is implemented but not in scope; perhaps you want to import it
|
18 + use datafusion_common::datatype::DataTypeExt;
|
help: there is a method `into_nullable_field` with a similar name
|
48 - .map(|ty| ty.into_nullable_field_ref())
48 + .map(|ty| ty.into_nullable_field())
|
error[E0599]: no method named `into_nullable_field_ref` found for enum `arrow::datatypes::DataType` in the current scope
--> datafusion/sql/src/expr/identifier.rs:125:38
|
125 | .map(|ty| ty.into_nullable_field_ref())
| ^^^^^^^^^^^^^^^^^^^^^^^
|
::: /__w/datafusion/datafusion/datafusion/common/src/datatype.rs:51:8
|
51 | fn into_nullable_field_ref(self) -> FieldRef;
| ----------------------- the method is available for `arrow::datatypes::DataType` here
|
= help: items from traits can only be used if the trait is in scope
help: trait `DataTypeExt` which provides `into_nullable_field_ref` is implemented but not in scope; perhaps you want to import it
|
18 + use datafusion_common::datatype::DataTypeExt;
|
help: there is a method `into_nullable_field` with a similar name
|
125 - .map(|ty| ty.into_nullable_field_ref())
125 + .map(|ty| ty.into_nullable_field())
|

I think the suggestion use datafusion_common::datatype::DataTypeExt; will get you there 🙂 . It is also worth running cargo test locally (or cargo test -p datafusion-expr for a smaller subset of those tests) to ensure crate compiles and the tests run!

[batmnnn]

@paleolimbot sorry my bad, can you check now :)

[batmnnn]

@alamb@paleolimbot

[alamb]

I pushed some commits to this PR to help get the CI fixed.

[alamb]

Makes sense to me -- thanks @batmnnn and @paleolimbot

I think it would be nice to add a note to the upgrading https://datafusion.apache.org/library-user-guide/upgrading.html

That helps people upgrade -- namely

[paleolimbot]

I think it would be nice to add a note to the upgrading https://datafusion.apache.org/library-user-guide/upgrading.html

I can add this to #18136 since it's related (two DataTypes became FieldRefs)

[paleolimbot]

Thank you for the edits!

[alamb]

Thanks again @batmnnn and @paleolimbot

[batmnnn]

thank you @paleolimbot and @alamb for all the guidance and help through this