Skip to content

[SPARK-23905][SQL] Add UDF weekday - #21009

Closed
yucai wants to merge 2 commits into
apache:masterfrom
yucai:SPARK-23905
Closed

[SPARK-23905][SQL] Add UDF weekday#21009
yucai wants to merge 2 commits into
apache:masterfrom
yucai:SPARK-23905

Conversation

@yucai

@yucaiyucai commented Apr 9, 2018

Copy link
Copy Markdown

What changes were proposed in this pull request?

Add UDF weekday

How was this patch tested?

A new test

@jerryshao

jerryshao commented Apr 10, 2018

Copy link
Copy Markdown
Contributor

Jenkins, add to whitelist.

@jerryshao

jerryshao commented Apr 10, 2018

Copy link
Copy Markdown
Contributor

Jenkins, test this please.

gatorsmile
gatorsmile reviewed Apr 10, 2018
View reviewed changes

@gatorsmilegatorsmile left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@SparkQA

SparkQA commented Apr 10, 2018

Copy link
Copy Markdown

Test build #89081 has finished for PR 21009 at commit 79beb00.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • case class DayOfWeek(child: Expression) extends DayWeek
  • case class WeekDay(child: Expression) extends DayWeek
  • abstract class DayWeek extends UnaryExpression with ImplicitCastInputTypes

viirya
viirya reviewed Apr 10, 2018
View reviewed changes
Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala Outdated

override def dataType: DataType = IntegerType

@transient protected lazy val c: Calendar = {

@viiryaviiryaApr 10, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: maybe calendar instead of just c.

viirya
viirya reviewed Apr 10, 2018
View reviewed changes
Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
@transient private lazy val c = {
Calendar.getInstance(DateTimeUtils.getTimeZone("UTC"))
}
case class DayOfWeek(child: Expression) extends DayWeek {

@viiryaviiryaApr 10, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like DayOfWeek is duplicate to WeekDay, do we have plan to deprecate DayOfWeek in maybe 3.0?

@gatorsmilegatorsmileApr 10, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are not duplicate.

@yucaiyucaiApr 10, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are different, see:
WeekDay: 0 = Monday, 1 = Tuesday, … 6 = Sunday
DayOfWeek: 1 = Sunday, 2 = Monday, ..., 7 = Saturday

@yucai

yucai commented Apr 10, 2018

Copy link
Copy Markdown
Author

Jenkins, retest this please

@SparkQA

SparkQA commented Apr 10, 2018

Copy link
Copy Markdown

Test build #89137 has finished for PR 21009 at commit 2b5db56.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Apr 10, 2018

Copy link
Copy Markdown

Test build #89138 has finished for PR 21009 at commit 2b5db56.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@yucai

yucai commented Apr 11, 2018

Copy link
Copy Markdown
Author

I did the local testing, all pass.

@yucai

yucai commented Apr 11, 2018

Copy link
Copy Markdown
Author

Jenkins, retest this please

@SparkQA

SparkQA commented Apr 11, 2018

Copy link
Copy Markdown

Test build #89161 has finished for PR 21009 at commit 2b5db56.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Apr 11, 2018

Copy link
Copy Markdown

Test build #89181 has finished for PR 21009 at commit 79beb00.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • case class DayOfWeek(child: Expression) extends DayWeek
  • case class WeekDay(child: Expression) extends DayWeek
  • abstract class DayWeek extends UnaryExpression with ImplicitCastInputTypes

HyukjinKwon
HyukjinKwon reviewed Apr 12, 2018
View reviewed changes

@HyukjinKwonHyukjinKwon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

change itself seems fine.

@SparkQA

SparkQA commented Apr 12, 2018

Copy link
Copy Markdown

Test build #89229 has finished for PR 21009 at commit 363ff1c.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@gatorsmile

gatorsmile commented Apr 12, 2018

Copy link
Copy Markdown
Member

retest this please

@SparkQA

SparkQA commented Apr 12, 2018

Copy link
Copy Markdown

Test build #89235 has finished for PR 21009 at commit 363ff1c.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Apr 12, 2018

Copy link
Copy Markdown

Test build #89242 has finished for PR 21009 at commit 4068289.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@gatorsmile

gatorsmile commented Apr 13, 2018

Copy link
Copy Markdown
Member

Thanks! Merged to master.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@viiryaviiryaviirya left review comments
@HyukjinKwonHyukjinKwonHyukjinKwon left review comments
@gatorsmilegatorsmilegatorsmile left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@yucai@jerryshao@SparkQA@gatorsmile@viirya@HyukjinKwon