[SPARK-44059] Add analyzer support of named arguments for built-in functions - #41864
[SPARK-44059] Add analyzer support of named arguments for built-in functions#41864learningchess2003 wants to merge 33 commits into
Conversation
anchovYu
left a comment
There was a problem hiding this comment.
Thanks for the effort 👍!
| "message" : [ | ||
| "Call to function <functionName> is invalid because it includes multiple argument assignments to the same name <parameterName>." | ||
| ], | ||
| "sqlState" : "42704K" |
There was a problem hiding this comment.
I feel that some of my previous comments are naturally collapsed so you can't see them without expanding. I'll copy those to this new PR.
The error code should be 5 characters according to https://github.com/apache/spark/tree/master/common/utils/src/main/resources/error.
| * @param default The default value of the argument. If the default is none, then that means the | ||
| * argument is required. If no argument is provided, an exception is thrown. | ||
| */ | ||
| case class NamedArgument(name: String, |
There was a problem hiding this comment.
| case class NamedArgument(name: String, | |
| case class NamedArgument( | |
| name: String, |
| SELECT mask(c1, replaceArg) from values('abcd-EFGH-8765-4321', 123) as t(c1, replaceArg); | ||
| SELECT mask('abcd-EFGH-8765-4321', 'A', 'w', ''); | ||
| SELECT mask('abcd-EFGH-8765-4321', 'A', 'abc'); No newline at end of file | ||
| SELECT mask('abcd-EFGH-8765-4321', 'A', 'abc'); |
There was a problem hiding this comment.
Could you revert the changes of this file?
|
@anchovYu Comments addressed! |
| }, | ||
| "UNRECOGNIZED_PARAMETER_NAME" : { | ||
| "message" : [ | ||
| "Cannot invoke function <functionName> because the function call included a named argument reference for the argument named <argumentName>, but this function does not include any signature containing an argument with this name." |
There was a problem hiding this comment.
Could you help users and propose function parameters sorted by levenshtein distance? See for instance:
There was a problem hiding this comment.
Sounds good! Added the feature.
|
@MaxGekk Comments addressed! |
dtenedor
left a comment
There was a problem hiding this comment.
I took another review pass, thanks again for working on this!
…ompilationErrors.scala Co-authored-by: Daniel Tenedorio <daniel.tenedorio@databricks.com>
Co-authored-by: Daniel Tenedorio <daniel.tenedorio@databricks.com>
…essions/aggregate/CountMinSketchAgg.scala Co-authored-by: Daniel Tenedorio <daniel.tenedorio@databricks.com>
…essions/maskExpressions.scala Co-authored-by: Daniel Tenedorio <daniel.tenedorio@databricks.com>
|
@dtenedor Addressed comments, let me know what you think! |
anchovYu
left a comment
There was a problem hiding this comment.
Could you update the PR description with the doc and the previous parser PR linked? It can provide more context for other reviewers who are interested. Thanks!
| }, | ||
| "NAMED_ARGUMENTS_NOT_SUPPORTED" : { | ||
| "message" : [ | ||
| "Named arguments are not supported for function <functionName>; please retry the query with positional arguments to the function call instead or enable the feature using the SQL config \"spark.sql.allowNamedFunctionArguments\"." |
There was a problem hiding this comment.
I don't think this error is triggered when the config is false right? It is only because the function itself doesn't implement the named argument trait.
There was a problem hiding this comment.
Actually, you are right. There is a separate exception for this.
…s/logical/SupportsNamedArguments.scala Co-authored-by: Xinyi <xyyu15@gmail.com>
…essions/maskExpressions.scala Co-authored-by: Xinyi <xyyu15@gmail.com>
Co-authored-by: Xinyi <xyyu15@gmail.com>
| expressions: Seq[Expression]) : Seq[Expression] = { | ||
| val rearrangedExpressions = if (!builder.functionSignatures.isEmpty) { | ||
| val functionSignatures = builder.functionSignatures.get | ||
| if (functionSignatures.length != 1) { |
There was a problem hiding this comment.
If we don't have a plan to support function overload soon, shall we restrict it at compile time? def functionSignatures: Option[Seq[FunctionSignature]] = None can be def functionSignatures: Option[FunctionSignature] = None
There was a problem hiding this comment.
Good question, I imagine we probably would not get around to prioritizing overloads soon with this framework. But if we did, would it become necessary to update all the places where these function signatures are defined, from an Option[FunctionSignature] to an Option[Seq[FunctionSignature]]? In contrast, if we go with the latter syntax of a Seq of just one function signature, we wouldn't have to update all those sites later.
However, I imagine such an update would be a fairly mechanical change, and in the interim, we get better compiler enforcement. It seems like this suggestion is better for now to switch the API to return just an Option[FunctionSignature] for now.
| assert(sketch == reference) | ||
| } | ||
|
|
||
| test("count-min sketch with named-arguments") { |
There was a problem hiding this comment.
nit: we can remove this end-to-end test if named-function-arguments.sql includes count_min_sketch
| { | ||
| "errorClass" : "WRONG_NUM_ARGS.WITHOUT_SUGGESTION", | ||
| "sqlState" : "42605", | ||
| "errorClass" : "DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE", |
There was a problem hiding this comment.
I feel the previous error is better. Shall we tune the named-parameter framework so that it always checks the num args first?
There was a problem hiding this comment.
Agree. We shouldn't change the error messages for function invocations that do not use named arguments.
| val parameters: Seq[NamedArgument] = functionSignature.parameters | ||
| val firstOptionalParamIndex: Int = parameters.indexWhere(_.default.isDefined) | ||
| if (firstOptionalParamIndex != -1 && | ||
| parameters.drop(firstOptionalParamIndex).exists(_.default.isEmpty)) { |
There was a problem hiding this comment.
nit: parameters.dropWhile(_.default.isEmpty).exists(_.default.isEmpty) is simpler
| functionName, functionSignature) | ||
| } | ||
|
|
||
| val firstNamedArgIdx: Int = args.indexWhere(_.isInstanceOf[NamedArgumentExpression]) |
There was a problem hiding this comment.
nit: val (posArgs, namedArgs) = args.span(!_.isInstanceOf[NamedArgumentExpression])
| // The following loop checks for the following: | ||
| // 1. Unrecognized parameter names | ||
| // 2. Duplicate routine parameter assignments | ||
| val allParameterNames: Seq[String] = parameters.map(_.name) |
There was a problem hiding this comment.
to confirm: the parameter name is case-sensitive. Is it also true in other systems or the SQL standard? cc @srielau
| import org.apache.spark.sql.catalyst.expressions.{Expression, NamedArgumentExpression} | ||
| import org.apache.spark.sql.errors.QueryCompilationErrors | ||
|
|
||
| object SupportsNamedArguments { |
There was a problem hiding this comment.
Do we have this object for ease of test? It looks to me that we can move the new trait Builder[T] to a new file and define the defaultRearrange function there.
| functionName: String, functionSignature: FunctionSignature) : Throwable = { | ||
| val errorMessage = s"Function $functionName has an unexpected required argument for" + | ||
| s" the provided function signature $functionSignature. All required arguments should" + | ||
| s" come before optional arguments." |
| def multipleFunctionSignatures(functionName: String, | ||
| functionSignatures: Seq[FunctionSignature]): Throwable = { | ||
| var errorMessage = s"Function $functionName cannot have multiple method signatures." + | ||
| s" The function signatures found were: \n" |
| def positionalAndNamedArgumentDoubleReference( | ||
| functionName: String, parameterName: String) : Throwable = { | ||
| val errorClass = | ||
| "DUPLICATE_ROUTINE_PARAMETER_ASSIGNMENT.POSITIONAL_AND_NAMED_ARGUMENT_DOUBLE_REFERENCE" |
There was a problem hiding this comment.
It is not long. Is it possible to make it shorter?
There was a problem hiding this comment.
maybe DUPLICATE_ROUTINE_PARAMETER_ASSIGNMENT.BOTH_POSITIONAL_AND_NAMED?
| var candidatesString = "" | ||
| recommendations.foreach(candidatesString += _ + " ") |
| "inputType" : "\"INT\"", | ||
| "paramIndex" : "1", | ||
| "requiredType" : "(\"ARRAY\" or \"MAP\")", | ||
| "sqlExpr" : "\"explode(1)\"" |
| "sqlExpr" : "\"explode(1)\"" | ||
| }, | ||
| "queryContext" : [ { | ||
| "objectType" : "", |
There was a problem hiding this comment.
Can we add more named argument tests in this file (table-valued-functions.sql) when the function invocation throws exceptions? For example use named arguments for these tests:
I'd like to make sure the exceptions stay the same.
There was a problem hiding this comment.
Sure, that's fine. I fixed the tests now so the errors remain consistent.
left a comment
There was a problem hiding this comment.
This is looking good, the only remaining changes left look like following Wenchen's suggestion to change the API to return an Option[FunctionSignature] instead of an Option[Seq[FunctionSignature]] for now (since we don't plan to support overloads in the short term), and some code commenting improvements.
| expressions: Seq[Expression]) : Seq[Expression] = { | ||
| val rearrangedExpressions = if (!builder.functionSignatures.isEmpty) { | ||
| val functionSignatures = builder.functionSignatures.get | ||
| if (functionSignatures.length != 1) { |
There was a problem hiding this comment.
Good question, I imagine we probably would not get around to prioritizing overloads soon with this framework. But if we did, would it become necessary to update all the places where these function signatures are defined, from an Option[FunctionSignature] to an Option[Seq[FunctionSignature]]? In contrast, if we go with the latter syntax of a Seq of just one function signature, we wouldn't have to update all those sites later.
However, I imagine such an update would be a fairly mechanical change, and in the interim, we get better compiler enforcement. It seems like this suggestion is better for now to switch the API to return just an Option[FunctionSignature] for now.
| (name, (info, (expressions: Seq[Expression]) => builder(expressions))) | ||
| } | ||
|
|
||
| def generatorBuilder[T <: GeneratorBuilder : ClassTag]( |
There was a problem hiding this comment.
please add a comment for this method
| val funcBuilder = (expressions: Seq[Expression]) => { | ||
| assert(expressions.forall(_.resolved), "function arguments must be resolved.") | ||
| val rearrangedExpressions = FunctionRegistry.rearrangeExpressions(name, builder, expressions) | ||
| val plan = builder.build(name, rearrangedExpressions) |
There was a problem hiding this comment.
you don't need the val plan here since you can just return builder.build(...)
|
|
||
| trait ExpressionBuilder { | ||
| def build(funcName: String, expressions: Seq[Expression]): Expression | ||
| trait Builder[T] { |
There was a problem hiding this comment.
please put some kind of short general comment for this Builder class mentioning that it is an interface for describing the expected parameters for SQL functions and takes responsibility for validating provided arguments for tham.
| } | ||
|
|
||
| /** | ||
| * A trait used for scalar valued functions that defines how their expression representations |
There was a problem hiding this comment.
please change this to a complete sentence ending with a period (same for L1120 below)
| def positionalAndNamedArgumentDoubleReference( | ||
| functionName: String, parameterName: String) : Throwable = { | ||
| val errorClass = | ||
| "DUPLICATE_ROUTINE_PARAMETER_ASSIGNMENT.POSITIONAL_AND_NAMED_ARGUMENT_DOUBLE_REFERENCE" |
There was a problem hiding this comment.
maybe DUPLICATE_ROUTINE_PARAMETER_ASSIGNMENT.BOTH_POSITIONAL_AND_NAMED?
| import org.apache.spark.sql.catalyst.expressions.{Expression, NamedArgumentExpression} | ||
| import org.apache.spark.sql.errors.QueryCompilationErrors | ||
|
|
||
| trait FunctionBuilderBase[T] { |
There was a problem hiding this comment.
please add a general comment describing what this object represents (using full sentences). Same for NamedArgumentsSupport below.
| * compare a function call with provided arguments to determine if that function | ||
| * call is a match for the function signature. | ||
| * | ||
| * @return a list of function signatures |
There was a problem hiding this comment.
no need for this line since the above comment conveys that information.
| } | ||
| } | ||
|
|
||
| // Check argument list size against provided parameter list length |
There was a problem hiding this comment.
| // Check argument list size against provided parameter list length | |
| // Check the argument list size against the provided parameter list length. |
| } | ||
| ) | ||
| } | ||
| positionalArgs ++ rearrangedNamedArgs |
There was a problem hiding this comment.
Could you add an assertion that the size of returning results is equal to the parameter count in function signature? Meanwhile, build can also have similar assertion as well.
| * Each function signature includes a list of parameters to which the analyzer can | ||
| * compare a function call with provided arguments to determine if that function | ||
| * call is a match for the function signature. | ||
| * |
There was a problem hiding this comment.
Could you add comment that we only support 1) one function signature 2) required should come before optional?
| } | ||
|
|
||
| // Check argument list size against provided parameter list length | ||
| if (parameters.size < args.length) { |
There was a problem hiding this comment.
Could you add tests for this case? Is this exception only possible to hit when passing in all positional args?
There was a problem hiding this comment.
I think we already have tests for these like in the context of explode for example.
There was a problem hiding this comment.
Thanks. Could you add one for mask in the bunch of error cases in the e2e tests?
| * This method is the default routine which rearranges the arguments in positional order according | ||
| * to the function signature provided. This will also fill in any default values that exists for | ||
| * optional arguments. This method will also be invoked even if there are no named arguments in | ||
| * the argument list. |
There was a problem hiding this comment.
Could you add more details:
- keeps all positional args in the original order
- rearranges named arguments according to the order defined in function signature, fill in missing optional arguments with default values
| @@ -1,5 +1,50 @@ | |||
| -- Test for named arguments for Mask | |||
There was a problem hiding this comment.
Could you add more tests on passing in all positional args and filling in the defaults?
There was a problem hiding this comment.
I think the current mask-expression.sql test already handles this.
| private def expressionGeneratorOuter[T <: Generator : ClassTag](name: String) | ||
| : (String, (ExpressionInfo, FunctionBuilder)) = { | ||
| val (_, (info, generatorBuilder)) = expression[T](name) | ||
| val (_, (info, builder)) = expression[T](name) |
There was a problem hiding this comment.
I felt it was because the name I created had the exact same name as the variable above and I didn't want to cause confusion. Also, in other example functions, builder is the variable name used for this kind of object, so I thought it be good to clear up the ambigiuity.
| val upperCharArg = NamedArgument("upperChar", Some(Literal(Mask.MASKED_UPPERCASE))) | ||
| val lowerCharArg = NamedArgument("lowerChar", Some(Literal(Mask.MASKED_LOWERCASE))) | ||
| val digitCharArg = NamedArgument("digitChar", Some(Literal(Mask.MASKED_DIGIT))) | ||
| val otherCharArg = NamedArgument("otherChar", Some(Literal(Mask.MASKED_IGNORE))) |
There was a problem hiding this comment.
Existing constructors on mask adds a type information: Literal(Mask.MASKED_IGNORE, StringType)). Could you change it?
| } | ||
|
|
||
| // Check argument list size against provided parameter list length | ||
| if (parameters.size < args.length) { |
There was a problem hiding this comment.
Thanks. Could you add one for mask in the bunch of error cases in the e2e tests?
| "NAMED_ARGUMENTS_NOT_SUPPORTED" : { | ||
| "message" : [ | ||
| "Named arguments are not supported for function <functionName>; please retry the query with positional arguments to the function call instead." |
There was a problem hiding this comment.
| "NAMED_ARGUMENTS_NOT_SUPPORTED" : { | |
| "message" : [ | |
| "Named arguments are not supported for function <functionName>; please retry the query with positional arguments to the function call instead." | |
| "NAMED_PARAMETERS_NOT_SUPPORTED" : { | |
| "message" : [ | |
| "Named parameters are not supported for function <functionName>; please retry the query with positional arguments to the function call instead." |
| ], | ||
| "sqlState" : "4274K" | ||
| }, | ||
| "NAMED_ARGUMENTS_SUPPORT_DISABLED" : { |
There was a problem hiding this comment.
| "NAMED_ARGUMENTS_SUPPORT_DISABLED" : { | |
| "NAMED_PARAMETER_SUPPORT_DISABLED" : { |
|
|
||
| /** | ||
| * This is a base trait that is used for implementing builder classes that can be used to construct | ||
| * expressions or logical plans depending on if it is a table-valued or scala-valued function. |
There was a problem hiding this comment.
| * expressions or logical plans depending on if it is a table-valued or scala-valued function. | |
| * expressions or logical plans depending on if it is a table-valued or scalar function. |
commented
Jul 14, 2023
|
seems |
| ], | ||
| "sqlState" : "428C4" | ||
| }, | ||
| "UNRECOGNIZED_PARAMETER_NAME" : { |
There was a problem hiding this comment.
| "UNRECOGNIZED_PARAMETER_NAME" : { | |
| "UNRECOGNIZED_ARGUMENT_NAME" : { |
There was a problem hiding this comment.
I asked Serge, and the original name is right. There's probably some confusion over the technical difference between argument and parameter. Typically, parameter is what each name is referring to.
commented
Jul 15, 2023
|
Migrating to #42020 due to some problems with branches. |
What changes were proposed in this pull request?
Add analyzer support for named function arguments.
Why are the changes needed?
Part of the project needed for general named function argument support.
Does this PR introduce any user-facing change?
We added support for named arguments for the
CountMinSketchAggandMaskSQL functions.How was this patch tested?
A new suite was added for this test called NamedArgumentFunctionSuite.