[SPARK-2327] [SQL] Fix nullabilities of Join/Generate/Aggregate. by ueshin · Pull Request #1266 · apache/spark

ueshin · 2014-06-30T09:35:44Z

Fix nullabilities of Join/Generate/Aggregate because:

Output attributes of opposite side of OuterJoin should be nullable.
Output attributes of generater side of Generate should be nullable if join is true and outer is true.
AttributeReference of computedAggregates of Aggregate should be the same as aggregateExpression's.

AmplabJenkins · 2014-06-30T09:40:36Z

Merged build triggered.

AmplabJenkins · 2014-06-30T09:40:42Z

Merged build started.

AmplabJenkins · 2014-06-30T09:42:15Z

Merged build finished.

AmplabJenkins · 2014-06-30T09:42:15Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16262/

AmplabJenkins · 2014-06-30T09:50:37Z

Merged build triggered.

AmplabJenkins · 2014-06-30T09:50:44Z

Merged build started.

AmplabJenkins · 2014-06-30T10:36:44Z

Merged build finished.

AmplabJenkins · 2014-06-30T10:36:44Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16264/

ueshin · 2014-06-30T10:48:58Z

I guess this needs #1268 to pass Hive tests.

AmplabJenkins · 2014-07-01T07:35:40Z

Merged build triggered.

AmplabJenkins · 2014-07-01T07:35:45Z

Merged build started.

AmplabJenkins · 2014-07-01T09:15:26Z

Merged build finished.

AmplabJenkins · 2014-07-01T09:15:27Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16280/

ueshin · 2014-07-01T09:34:16Z

Oops, passed #1268 related errors, but others failed...

AmplabJenkins · 2014-07-01T15:40:41Z

Merged build triggered.

AmplabJenkins · 2014-07-01T15:40:48Z

Merged build started.

AmplabJenkins · 2014-07-01T17:19:38Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-07-01T17:19:38Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16282/

marmbrus · 2014-07-03T23:52:01Z

This is a pretty common pattern. We already have withNullability, which you could use in these cases if we move it up as an abstract method in Attribute and add a no-op version in UnresolvedAttribute.

Ah, I see.
It is the same as withQualifiers, right?

Yeah, exactly.

marmbrus · 2014-07-03T23:54:38Z

A few minor comments.

BTW, thanks for all of these clean-up PRs fixing types and nullability. Really awesome :)

AmplabJenkins · 2014-07-04T00:55:50Z

Merged build triggered.

AmplabJenkins · 2014-07-04T00:55:59Z

Merged build started.

ueshin · 2014-07-04T01:00:38Z

@marmbrus, Thank you for your comments.
Modified to use withNullability.

And no problem, because exact types and nullability is important for my project.

AmplabJenkins · 2014-07-04T01:06:03Z

Merged build finished.

AmplabJenkins · 2014-07-04T01:06:03Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16329/

marmbrus · 2014-07-04T01:50:39Z

test this please

AmplabJenkins · 2014-07-04T01:50:50Z

Merged build triggered.

AmplabJenkins · 2014-07-04T01:50:59Z

Merged build started.

AmplabJenkins · 2014-07-04T03:33:43Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-07-04T03:33:44Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16333/

Fix nullabilities of `Join`/`Generate`/`Aggregate` because: - Output attributes of opposite side of `OuterJoin` should be nullable. - Output attributes of generater side of `Generate` should be nullable if `join` is `true` and `outer` is `true`. - `AttributeReference` of `computedAggregates` of `Aggregate` should be the same as `aggregateExpression`'s. Author: Takuya UESHIN <ueshin@happy-camper.st> Closes#1266 from ueshin/issues/SPARK-2327 and squashes the following commits: 3ace83a [Takuya UESHIN] Add withNullability to Attribute and use it to change nullabilities. df1ae53 [Takuya UESHIN] Modify nullabilize to leave attribute if not resolved. 799ce56 [Takuya UESHIN] Add nullabilization to Generate of SparkPlan. a0fc9bc [Takuya UESHIN] Fix scalastyle errors. 0e31e37 [Takuya UESHIN] Fix Aggregate resultAttribute nullabilities. 09532ec [Takuya UESHIN] Fix Generate output nullabilities. f20f196 [Takuya UESHIN] Fix Join output nullabilities. (cherry picked from commit 9d5ecf8) Signed-off-by: Michael Armbrust <michael@databricks.com>

marmbrus · 2014-07-05T18:54:21Z

Thanks! I've merge this into master and we'll back port it into 1.0 after the 1.0.1 release.

Fix nullabilities of `Join`/`Generate`/`Aggregate` because: - Output attributes of opposite side of `OuterJoin` should be nullable. - Output attributes of generater side of `Generate` should be nullable if `join` is `true` and `outer` is `true`. - `AttributeReference` of `computedAggregates` of `Aggregate` should be the same as `aggregateExpression`'s. Author: Takuya UESHIN <ueshin@happy-camper.st> Closesapache#1266 from ueshin/issues/SPARK-2327 and squashes the following commits: 3ace83a [Takuya UESHIN] Add withNullability to Attribute and use it to change nullabilities. df1ae53 [Takuya UESHIN] Modify nullabilize to leave attribute if not resolved. 799ce56 [Takuya UESHIN] Add nullabilization to Generate of SparkPlan. a0fc9bc [Takuya UESHIN] Fix scalastyle errors. 0e31e37 [Takuya UESHIN] Fix Aggregate resultAttribute nullabilities. 09532ec [Takuya UESHIN] Fix Generate output nullabilities. f20f196 [Takuya UESHIN] Fix Join output nullabilities.

ueshin added 3 commits June 30, 2014 18:30

Fix Join output nullabilities.
f20f196

Fix Generate output nullabilities.
09532ec

Fix Aggregate resultAttribute nullabilities.
0e31e37

Fix scalastyle errors.
a0fc9bc

Add nullabilization to Generate of SparkPlan.
799ce56

Modify nullabilize to leave attribute if not resolved.
df1ae53

marmbrus reviewed Jul 3, 2014
View reviewed changes

Add withNullability to Attribute and use it to change nullabilities.
3ace83a

asfgit closed this in 9d5ecf8Jul 5, 2014

ueshin deleted the issues/SPARK-2327 branch August 3, 2026 22:58

Uh oh!

Conversation

ueshin commented Jun 30, 2014

Uh oh!

AmplabJenkins commented Jun 30, 2014

Uh oh!

AmplabJenkins commented Jun 30, 2014

Uh oh!

AmplabJenkins commented Jun 30, 2014

Uh oh!

AmplabJenkins commented Jun 30, 2014

Uh oh!

AmplabJenkins commented Jun 30, 2014

Uh oh!

AmplabJenkins commented Jun 30, 2014

Uh oh!

AmplabJenkins commented Jun 30, 2014

Uh oh!

AmplabJenkins commented Jun 30, 2014

Uh oh!

ueshin commented Jun 30, 2014

Uh oh!

AmplabJenkins commented Jul 1, 2014

Uh oh!

AmplabJenkins commented Jul 1, 2014

Uh oh!

AmplabJenkins commented Jul 1, 2014

Uh oh!

AmplabJenkins commented Jul 1, 2014

Uh oh!

ueshin commented Jul 1, 2014

Uh oh!

AmplabJenkins commented Jul 1, 2014

Uh oh!

AmplabJenkins commented Jul 1, 2014

Uh oh!

AmplabJenkins commented Jul 1, 2014

Uh oh!

AmplabJenkins commented Jul 1, 2014

Uh oh!

marmbrusJul 3, 2014

Choose a reason for hiding this comment

Uh oh!

ueshinJul 4, 2014

Choose a reason for hiding this comment

Uh oh!

marmbrusJul 4, 2014

Choose a reason for hiding this comment

Uh oh!

marmbrus commented Jul 3, 2014

Uh oh!

AmplabJenkins commented Jul 4, 2014

Uh oh!

AmplabJenkins commented Jul 4, 2014

Uh oh!

ueshin commented Jul 4, 2014

Uh oh!

AmplabJenkins commented Jul 4, 2014

Uh oh!

AmplabJenkins commented Jul 4, 2014

Uh oh!

marmbrus commented Jul 4, 2014

Uh oh!

AmplabJenkins commented Jul 4, 2014

Uh oh!

AmplabJenkins commented Jul 4, 2014

Uh oh!

AmplabJenkins commented Jul 4, 2014

Uh oh!

AmplabJenkins commented Jul 4, 2014

Uh oh!

marmbrus commented Jul 5, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!