[SPARK-32002][SQL]Support ExtractValue from nested ArrayStruct by AngersZhuuuu · Pull Request #28860 · apache/spark

AngersZhuuuu · 2020-06-18T14:32:15Z

What changes were proposed in this pull request?

For data nest.json

{"a": [{"b": [{"c": [1,2]}]}]}
{"a": [{"b": [{"c": [1]}, {"c": [2]}]}]}

run with

valdf:DataFrame= spark.read.json(testFile("nest-data.json"))
df.createTempView("nest_table")
sql("select a.b.c from nest_table").show()

will got error

org.apache.spark.sql.AnalysisException: cannot resolve 'nest_table.`a`.`b`['c']' due to data type mismatch: argument 2 requires integral type, however, ''c'' is of string type.; line 1 pos 7;
'Project [a#6.b[c] AS c#8|#6.b[c] AS c#8]
+- SubqueryAlias `nest_table`
+- Relationa#6 json

This is because after resolve a.b, it construct GetArrayStructField from a#
result with type ArrayType(ArrayType(StructType)), then it can't be extract through current ExtractValue method.

Support extract StructField form nested Array-Struct combination

Why are the changes needed?

Support more use case

Does this PR introduce any user-facing change?

Yes, users will be able to select from nested array and structs.

How was this patch tested?

Added UT

SparkQA · 2020-06-18T14:37:42Z

Test build #124217 has finished for PR 28860 at commit da9a3d5.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-06-18T15:28:57Z

Test build #124220 has finished for PR 28860 at commit b8f4a69.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-06-19T07:05:02Z

Test build #124252 has finished for PR 28860 at commit 400ad7d.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

AngersZhuuuu · 2020-06-19T07:13:45Z

retest this please

SparkQA · 2020-06-19T11:40:55Z

Test build #124268 has finished for PR 28860 at commit 400ad7d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2020-06-20T04:42:22Z

cc @MaxGekk @HyukjinKwon

AngersZhuuuu · 2020-06-20T06:43:30Z

Seems this way will change the schema, maybe we should add a new class to get multidimensional array form GetArrayStruct

Co-authored-by: Ruslan Dautkhanov <Tagar@users.noreply.github.com>

SparkQA · 2020-06-20T07:05:02Z

Test build #124314 has finished for PR 28860 at commit b6e92c0.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2020-06-26T09:50:25Z

retest this please

HyukjinKwon · 2020-06-26T10:40:17Z

cc @viirya too FYI

HyukjinKwon · 2020-06-26T10:41:25Z

 ordinal, fields.length, containsNull)

+ case (ExtractNestedArray(StructType(fields), containsNull, containsNullSeq),
+ NonNullLiteral(v, StringType)) =>


nit:
case (ExtractNestedArray( StructType(fields), containsNull, containsNullSeq), NonNullLiteral(v, StringType)) =>

Let's also update the documentation and table above.

HyukjinKwon · 2020-06-26T10:44:37Z


 trait ExtractValue extends Expression

+object ExtractNestedArray {


Let's add documentation here.

HyukjinKwon · 2020-06-26T10:44:53Z


+object ExtractNestedArray {
+
+ type ReturnType = Option[(DataType, Boolean, Seq[Boolean])]


Let's also add some comments for what this type means.

HyukjinKwon · 2020-06-26T10:46:13Z

+
+ def extractArrayStruct(expr: Expression): ReturnType = {
+ expr match {
+ case gas @ GetArrayStructFields(child, _, _, _, _) =>


Let's avoid arguments matching. This is actually an anti pattern - https://github.com/databricks/scala-style-guide#pattern-matching

HyukjinKwon · 2020-06-26T10:47:25Z

+ expr match {
+ case gas @ GetArrayStructFields(child, _, _, _, _) =>
+ extractArrayStruct(child) match {
+ case Some((e, deep)) => Some(e, deep + 1)


deep -> depth?

HyukjinKwon · 2020-06-26T10:50:25Z

+ extractArrayType(dataType)
+ }
+
+ def extractArrayType(dataType: DataType): ReturnType = {


Can we combine this and unapply?

HyukjinKwon · 2020-06-26T10:52:37Z

+
+ checkAnswer(sql(
+ """
+ |SELECT a.b.c FROM nest


Can we add deeper cases? Also, you can simplify the test cases, for example, as below:
valdf= spark.range(10).select(array(struct(array(struct("id")).alias("col1"))).alias("col0")) df.selectExpr("col0.col1.id")

HyukjinKwon · 2020-06-26T10:57:42Z


 trait ExtractValue extends Expression

+object ExtractNestedArray {


Let's also name it something like ExtractNestedArrayType

SparkQA · 2020-06-26T14:31:17Z

Test build #124544 has finished for PR 28860 at commit b6e92c0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

guiyanakuang · 2020-07-03T13:17:14Z

org.apache.spark.sql.execution.ProjectionOverSchema.scala. A Scala extractor that projects an expression over a given schema. I think this code also needs to have matching rules added to it. To support nested ArrayStruct. Otherwise, ParquetSchemaPruning won't work either.

AngersZhuuuu · 2020-07-04T04:14:37Z

org.apache.spark.sql.execution.ProjectionOverSchema.scala. A Scala extractor that projects an expression over a given schema. I think this code also needs to have matching rules added to it. To support nested ArrayStruct. Otherwise, ParquetSchemaPruning won't work either.

Yea, right, current way change the schema and won't work well for all case, I am thinking about add a new class to handle nested ArrayStruct type

github-actions · 2020-10-13T00:52:17Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

AngersZhuuuu added 2 commits June 18, 2020 22:25

save
4339b81

Update SQLQuerySuite.scala
da9a3d5

probot-autolabelerBot added the SQL label Jun 18, 2020

Update SQLQuerySuite.scala
b8f4a69

Update SelectedFieldSuite.scala
400ad7d

Tagar reviewed Jun 19, 2020
View reviewed changes

Comment threadsql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala Outdated

maropu changed the title ~~[SPARK-32002][SQL]Support Extract valve from nested ArrayStruct~~[SPARK-32002][SQL]Support ExtractValue from nested ArrayStructJun 20, 2020

Update sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
b6e92c0
Co-authored-by: Ruslan Dautkhanov <Tagar@users.noreply.github.com>

HyukjinKwon reviewed Jun 26, 2020
View reviewed changes

github-actionsBot added the Stale label Oct 13, 2020

github-actionsBot closed this Oct 14, 2020

HyukjinKwon mentioned this pull request Nov 24, 2020
[SPARK-32002][SQL]Support ExtractValue from nested ArrayStruct #30467
Closed


		trait ExtractValue extends Expression

		object ExtractNestedArray {


		object ExtractNestedArray {

		type ReturnType = Option[(DataType, Boolean, Seq[Boolean])]

Uh oh!

Conversation

AngersZhuuuu commented Jun 18, 2020• edited by HyukjinKwon Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

SparkQA commented Jun 18, 2020

Uh oh!

SparkQA commented Jun 18, 2020

Uh oh!

SparkQA commented Jun 19, 2020

Uh oh!

AngersZhuuuu commented Jun 19, 2020

Uh oh!

SparkQA commented Jun 19, 2020

Uh oh!

Uh oh!

gatorsmile commented Jun 20, 2020

Uh oh!

AngersZhuuuu commented Jun 20, 2020

Uh oh!

SparkQA commented Jun 20, 2020

Uh oh!

HyukjinKwon commented Jun 26, 2020

Uh oh!

HyukjinKwon commented Jun 26, 2020

Uh oh!

HyukjinKwonJun 26, 2020

Choose a reason for hiding this comment

Uh oh!

HyukjinKwonJun 26, 2020

Choose a reason for hiding this comment

Uh oh!

HyukjinKwonJun 26, 2020

Choose a reason for hiding this comment

Uh oh!

HyukjinKwonJun 26, 2020

Choose a reason for hiding this comment

Uh oh!

HyukjinKwonJun 26, 2020

Choose a reason for hiding this comment

Uh oh!

HyukjinKwonJun 26, 2020

Choose a reason for hiding this comment

Uh oh!

HyukjinKwonJun 26, 2020

Choose a reason for hiding this comment

Uh oh!

HyukjinKwonJun 26, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HyukjinKwonJun 26, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jun 26, 2020

Uh oh!

guiyanakuang commented Jul 3, 2020

Uh oh!

AngersZhuuuu commented Jul 4, 2020

Uh oh!

github-actionsBot commented Oct 13, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

AngersZhuuuu commented Jun 18, 2020•
edited by HyukjinKwon
Loading

HyukjinKwonJun 26, 2020•
edited
Loading

HyukjinKwonJun 26, 2020•
edited
Loading