Skip to content

[SPARK-31476][SQL] Add an ExpressionInfo entry for EXTRACT - #28251

Closed
maropu wants to merge 2 commits into
apache:masterfrom
maropu:AddExtractExpr
Closed

[SPARK-31476][SQL] Add an ExpressionInfo entry for EXTRACT#28251
maropu wants to merge 2 commits into
apache:masterfrom
maropu:AddExtractExpr

Conversation

@maropu

@maropumaropu commented Apr 18, 2020

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This PR intends to add an ExpressionInfo entry for EXTRACT for better documentations.
This PR comes from the comment in #21479 (comment)

Why are the changes needed?

To make SQL documentations complete.

Does this PR introduce any user-facing change?

Yes, this PR updates the Spark SQL, Built-in Functions page.

How was this patch tested?

Run the example tests.

dongjoon-hyun
dongjoon-hyun approved these changes Apr 18, 2020
View reviewed changes

@dongjoon-hyundongjoon-hyun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM.

@SparkQA

SparkQA commented Apr 18, 2020

Copy link
Copy Markdown

Test build #121436 has finished for PR 28251 at commit 933c89f.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • case class Extract(field: Expression, source: Expression, child: Expression)

@dongjoon-hyun

dongjoon-hyun commented Apr 18, 2020

Copy link
Copy Markdown
Member

Could you regenerate date.sql, too?

sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedException: postgreSQL/date.sql
Expected "struct<[date_part('EPOCH',] DATE '1970-01-01'):...", but got "struct<[extract('EPOCH' ```

@maropu

maropu commented Apr 18, 2020

Copy link
Copy Markdown
MemberAuthor

Ah, yes. I'm updating now. Thanks for the check, anyway.

@SparkQA

SparkQA commented Apr 18, 2020

Copy link
Copy Markdown

Test build #121444 has finished for PR 28251 at commit 5579736.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@maropu

maropu commented Apr 18, 2020

Copy link
Copy Markdown
MemberAuthor

retest this please

@SparkQA

SparkQA commented Apr 18, 2020

Copy link
Copy Markdown

Test build #121446 has finished for PR 28251 at commit 5579736.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dongjoon-hyun

dongjoon-hyun commented Apr 18, 2020

Copy link
Copy Markdown
Member

cc @gatorsmile

@dongjoon-hyun

dongjoon-hyun commented Apr 18, 2020

Copy link
Copy Markdown
Member

Merged to master/3.0. Thank you, @maropu .

dongjoon-hyun pushed a commit that referenced this pull request Apr 18, 2020
@maropu@dongjoon-hyun
[SPARK-31476][SQL] Add an ExpressionInfo entry for EXTRACT
43b5116
### What changes were proposed in this pull request?
This PR intends to add an ExpressionInfo entry for EXTRACT for better documentations.
This PR comes from the comment in #21479 (comment)
### Why are the changes needed?
To make SQL documentations complete.
### Does this PR introduce any user-facing change?
Yes, this PR updates the `Spark SQL, Built-in Functions` page.
### How was this patch tested?
Run the example tests.
Closes#28251 from maropu/AddExtractExpr.
Authored-by: Takeshi Yamamuro <yamamuro@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit 74aed8c)
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
@maropu

maropu commented Apr 18, 2020

Copy link
Copy Markdown
MemberAuthor

Thanks for the quick response, @dongjoon-hyun ! cc: @cloud-fan

cloud-fan
cloud-fan reviewed Apr 20, 2020
View reviewed changes
Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala
expression[MakeTimestamp]("make_timestamp"),
expression[MakeInterval]("make_interval"),
expression[DatePart]("date_part"),
expression[Extract]("extract"),

@cloud-fancloud-fanApr 20, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one side effect is now we support extract(field, source) other than extract(field from source). Not a big deal but better if we can avoid exposing more APIs,

@maropumaropuApr 20, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a big deal but better if we can avoid exposing more APIs

Yea, +1 .

btw, its better to add tests for the case extract(field, source)?

@cloud-fancloud-fanApr 20, 2020

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea if we decide to support it.

@maropumaropuApr 20, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, I'll do follow-up later.

@maropumaropuApr 21, 2020

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I opened #28276

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cloud-fancloud-fancloud-fan left review comments
@dongjoon-hyundongjoon-hyundongjoon-hyun approved these changes

Assignees

No one assigned

Labels

SQL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@maropu@SparkQA@dongjoon-hyun@cloud-fan