Skip to content

[SPARK-20033][SQL] support hive permanent function - #17362

Closed
cenyuhai wants to merge 4 commits into
apache:masterfrom
cenyuhai:SPARK-20033
Closed

[SPARK-20033][SQL] support hive permanent function#17362
cenyuhai wants to merge 4 commits into
apache:masterfrom
cenyuhai:SPARK-20033

Conversation

@cenyuhai

@cenyuhaicenyuhai commented Mar 20, 2017

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

support hive permanent function

How was this patch tested?

@SparkQA

SparkQA commented Mar 20, 2017

Copy link
Copy Markdown

Test build #74885 has finished for PR 17362 at commit 703d23e.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Mar 20, 2017

Copy link
Copy Markdown

Test build #74889 has finished for PR 17362 at commit a668bb5.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Mar 20, 2017

Copy link
Copy Markdown

Test build #74890 has finished for PR 17362 at commit 9325a7c.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Mar 21, 2017

Copy link
Copy Markdown

Test build #74912 has finished for PR 17362 at commit 92c077d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@gatorsmile

gatorsmile commented Mar 23, 2017

Copy link
Copy Markdown
Member

I think we already support Hive UDF. See the test case for permanent Hive UDF.

gatorsmile
gatorsmile reviewed Mar 23, 2017
View reviewed changes
Comment threadsql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala
functionResourceLoader.loadResource(resource)
} else {
val sessionState = SessionState.get()
val localPath = sessionState.add_resource(resourceType, resource.uri)

@gatorsmilegatorsmileMar 23, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you trying to create a Hive UDF via Spark, then call it through Hive?

@cenyuhaicenyuhaiMar 24, 2017

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, I just use sessionState.add_resource to download the resouces

@cenyuhai

cenyuhai commented Mar 24, 2017

Copy link
Copy Markdown
ContributorAuthor

@gatorsmile hi，spark just suport HIVE UDF which resources is in the local disk, or spark-sql --jars xxx.jar or something else. But I think spark don't support the hive permanent function which resources are in hdfs.
CREATE FUNCTION hdfs_udf AS 'xxx.udf' USING JAR 'hdfs:///user/udf/.jar';
My pr just download the hdfs resources and add to the classpath.

@wangyum

wangyum commented Mar 24, 2017

Copy link
Copy Markdown
Member

@weiqingy is doing Allow adding jars from hdfs.

@cenyuhai

cenyuhai commented Mar 24, 2017

Copy link
Copy Markdown
ContributorAuthor

Ok, I think @weiqingy 's pr will resolve this problem

@gatorsmile

gatorsmile commented Mar 24, 2017

Copy link
Copy Markdown
Member

@wangyum Thank you!

@cenyuhai Maybe you can close this PR now. Please also try the PR #17342 and see whether it resolves your issue. Thanks!

@HyukjinKwonHyukjinKwon mentioned this pull request May 4, 2017
Closed
@asfgitasfgit closed this in 4411ac7May 5, 2017
zifeif2 pushed a commit to zifeif2/spark that referenced this pull request Nov 22, 2025
@HyukjinKwon@zifeif2
[INFRA] Close stale PRs
1afe702
## What changes were proposed in this pull request?
This PR proposes to close a stale PR, several PRs suggested to be closed by a committer and obviously inappropriate PRs.
Closesapache#11119Closesapache#17853Closesapache#17732Closesapache#17456Closesapache#17410Closesapache#17314Closesapache#17362Closesapache#17542
## How was this patch tested?
N/A
Author: hyukjinkwon <gurwls223@gmail.com>
Closesapache#17855 from HyukjinKwon/close-pr.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gatorsmilegatorsmilegatorsmile left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@cenyuhai@SparkQA@gatorsmile@wangyum