[SPARK-36224][SQL] Use Void as the type name of NullType by linhongliu-db · Pull Request #33437 · apache/spark

linhongliu-db · 2021-07-20T09:21:28Z

What changes were proposed in this pull request?

Change the NullType.simpleString to "void" to set "void" as the formal type name of NullType

Why are the changes needed?

This PR is intended to address the type name discussion in PR #28833. Here are the reasons:

The type name of NullType is displayed everywhere, e.g. schema string, error message, document. Hence it's not possible to hide it from users, we have to choose a proper name
The "void" is widely used as the type name of "NULL", e.g. Hive, pgSQL
Changing to "void" can enable the round trip of toDDL/fromDDL for NullType. (i.e. make from_json(col, schema.toDDL)) work

Does this PR introduce any user-facing change?

Yes, the type name of "NULL" is changed from "null" to "void". for example:

scala> sql("select null as a, 1 as b").schema.catalogString
res5: String = struct<a:void,b:int>

How was this patch tested?

existing test cases

SparkQA · 2021-07-20T11:49:12Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45835/

SparkQA · 2021-07-20T12:24:50Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45835/

SparkQA · 2021-07-20T13:31:17Z

Test build #141320 has finished for PR 33437 at commit f805f40.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-07-21T10:19:01Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45918/

SparkQA · 2021-07-21T10:57:05Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45918/

HyukjinKwon

Looks fine to me if tests pass

HyukjinKwon · 2021-07-21T11:52:07Z

cc @cloud-fan, @dongjoon-hyun, @MaxGekk FYI

SparkQA · 2021-07-21T14:19:06Z

Test build #141400 has finished for PR 33437 at commit f45cf55.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-07-21T19:19:37Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45949/

SparkQA · 2021-07-21T20:02:57Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45949/

SparkQA · 2021-07-21T20:22:45Z

Test build #141431 has finished for PR 33437 at commit 4f0a5c9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-07-22T05:12:50Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45982/

SparkQA · 2021-07-22T05:54:18Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45982/

SparkQA · 2021-07-22T07:13:26Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45989/

SparkQA · 2021-07-22T07:55:17Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45989/

SparkQA · 2021-07-22T07:57:27Z

Test build #141463 has finished for PR 33437 at commit e3e563c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-07-22T11:35:53Z

Test build #141471 has finished for PR 33437 at commit db290d9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

linhongliu-db · 2021-07-22T14:06:31Z

cc @cloud-fan, the jenkins tests passed.
#33437 (comment)

MaxGekk

Yes, the type name of "NULL" is changed from "null" to "void". for example:

@linhongliu-db If this PR introduces behavior change, should we mention this in the SQL migration guide?

linhongliu-db · 2021-07-23T08:03:07Z

If this PR introduces behavior change, should we mention this in the SQL migration guide?

@MaxGekk We can add this to the migration guide. But I personally think it's not necessary because the change is very minor and I don't think changing the type name will break the user's existing workload. i.e. there is no migration needed.

SparkQA · 2021-07-29T09:27:24Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46332/

SparkQA · 2021-07-29T10:02:34Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46332/

SparkQA · 2021-07-29T12:53:51Z

Test build #141819 has finished for PR 33437 at commit 7ac3c18.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2021-07-29T16:34:24Z

LGTM, we can also remove this line now
https://github.com/apache/spark/blob/master/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala#L378

linhongliu-db · 2021-07-30T03:20:15Z

we can also remove this line now

@cloud-fan looks like we can also remove HiveVoidType

cloud-fan · 2021-07-30T04:07:35Z

looks like we can also remove HiveVoidType

Yea, let's remove it as well

SparkQA · 2021-07-30T04:23:50Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46382/

SparkQA · 2021-07-30T05:00:16Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46382/

SparkQA · 2021-07-30T08:48:02Z

Test build #141873 has finished for PR 33437 at commit 911b5fc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-07-30T11:30:40Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46404/

SparkQA · 2021-07-30T18:55:02Z

Test build #141895 has finished for PR 33437 at commit 4d5dcab.

This patch fails from timeout after a configured wait of 500m.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-02T04:36:06Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46447/

SparkQA · 2021-08-02T08:30:47Z

Test build #141937 has finished for PR 33437 at commit eacf7ff.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2021-08-02T15:19:29Z

The DDL type string round trip is really a good point, and it's kind of a bug that from_json(col, schema.toDDL) may not work. I'm merging this to master/3.2, thanks!

### What changes were proposed in this pull request? Change the `NullType.simpleString` to "void" to set "void" as the formal type name of `NullType` ### Why are the changes needed? This PR is intended to address the type name discussion in PR #28833. Here are the reasons: 1. The type name of NullType is displayed everywhere, e.g. schema string, error message, document. Hence it's not possible to hide it from users, we have to choose a proper name 2. The "void" is widely used as the type name of "NULL", e.g. Hive, pgSQL 3. Changing to "void" can enable the round trip of `toDDL`/`fromDDL` for NullType. (i.e. make `from_json(col, schema.toDDL)`) work ### Does this PR introduce _any_ user-facing change? Yes, the type name of "NULL" is changed from "null" to "void". for example: ``` scala> sql("select null as a, 1 as b").schema.catalogString res5: String = struct<a:void,b:int> ``` ### How was this patch tested? existing test cases Closes#33437 from linhongliu-db/SPARK-36224-void-type-name. Authored-by: Linhong Liu <linhong.liu@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit 2f70077) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

linhongliu-db changed the title ~~[SPARK-36224] Use Void as the type name of NullType~~[SPARK-36224][SQL] Use Void as the type name of NullTypeJul 20, 2021

github-actionsBot added CORE PYTHON SQL labels Jul 20, 2021

HyukjinKwon reviewed Jul 20, 2021
View reviewed changes

Comment threadpython/pyspark/sql/types.py Outdated

HyukjinKwon reviewed Jul 21, 2021
View reviewed changes

cloud-fan approved these changes Jul 21, 2021
View reviewed changes

linhongliu-db force-pushed the SPARK-36224-void-type-name branch from 4f0a5c9 to e3e563cCompare July 22, 2021 02:59

linhongliu-db mentioned this pull request Jul 23, 2021
[SPARK-36241][SQL] Support creating tables with null column #33488
Closed

MaxGekk reviewed Jul 23, 2021
View reviewed changes

use void as type name of NullType
7ac3c18

linhongliu-db force-pushed the SPARK-36224-void-type-name branch from db290d9 to 7ac3c18Compare July 29, 2021 07:41

address comments
911b5fc

address more comments
4d5dcab

trigger test
eacf7ff

cloud-fan closed this in 2f70077Aug 2, 2021

Uh oh!

Conversation

linhongliu-db commented Jul 20, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Uh oh!

SparkQA commented Jul 20, 2021

Uh oh!

SparkQA commented Jul 20, 2021

Uh oh!

SparkQA commented Jul 20, 2021

Uh oh!

SparkQA commented Jul 21, 2021

Uh oh!

SparkQA commented Jul 21, 2021

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Jul 21, 2021

Uh oh!

SparkQA commented Jul 21, 2021

Uh oh!

SparkQA commented Jul 21, 2021

Uh oh!

SparkQA commented Jul 21, 2021

Uh oh!

SparkQA commented Jul 21, 2021

Uh oh!

SparkQA commented Jul 22, 2021

Uh oh!

SparkQA commented Jul 22, 2021

Uh oh!

SparkQA commented Jul 22, 2021

Uh oh!

SparkQA commented Jul 22, 2021

Uh oh!

SparkQA commented Jul 22, 2021

Uh oh!

SparkQA commented Jul 22, 2021

Uh oh!

linhongliu-db commented Jul 22, 2021

Uh oh!

MaxGekk left a comment

Choose a reason for hiding this comment

Uh oh!

linhongliu-db commented Jul 23, 2021

Uh oh!

SparkQA commented Jul 29, 2021

Uh oh!

SparkQA commented Jul 29, 2021

Uh oh!

SparkQA commented Jul 29, 2021

Uh oh!

cloud-fan commented Jul 29, 2021

Uh oh!

linhongliu-db commented Jul 30, 2021

Uh oh!

cloud-fan commented Jul 30, 2021

Uh oh!

SparkQA commented Jul 30, 2021

Uh oh!

SparkQA commented Jul 30, 2021

Uh oh!

SparkQA commented Jul 30, 2021

Uh oh!

SparkQA commented Jul 30, 2021

Uh oh!

SparkQA commented Jul 30, 2021

Uh oh!

SparkQA commented Aug 2, 2021

Uh oh!

SparkQA commented Aug 2, 2021

Uh oh!

cloud-fan commented Aug 2, 2021

Uh oh!

linhongliu-db commented Jul 20, 2021•
edited
Loading