[SPARK-26254][CORE] Extract Hive + Kafka dependencies from Core. by gaborgsomogyi · Pull Request #23499 · apache/spark

gaborgsomogyi · 2019-01-09T13:34:47Z

What changes were proposed in this pull request?

There are ugly provided dependencies inside core for the following:

Hive
Kafka

In this PR I've extracted them out. This PR contains the following:

Token providers are now loaded with service loader
Hive token provider moved to hive project
Kafka token provider extracted into a new project

How was this patch tested?

Existing + newly added unit tests.
Additionally tested on cluster.

SparkQA · 2019-01-09T13:48:07Z

Test build #100969 has finished for PR 23499 at commit cd7db2f.

This patch fails to build.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2019-01-09T14:03:08Z

Test build #100970 has finished for PR 23499 at commit 47ea157.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-01-09T15:54:26Z

Test build #100975 has finished for PR 23499 at commit d8188c0.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-01-09T17:45:12Z

Test build #100983 has finished for PR 23499 at commit c7a4282.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-01-09T22:05:20Z

Test build #100984 has finished for PR 23499 at commit 6e0178c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gaborgsomogyi · 2019-01-10T13:22:05Z

retest this please

SparkQA · 2019-01-10T18:26:48Z

Test build #101021 has finished for PR 23499 at commit 6e0178c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gaborgsomogyi · 2019-01-10T18:27:41Z

cc @vanzin @dongjoon-hyun @HeartSaVioR

gatorsmile · 2019-01-14T17:18:43Z

In Spark 3.0, shading all libraries that Spark depends is the direction we discussed in the mailing list. @gaborgsomogyi Thus, could you change your PR based on our direction? Thanks!

vanzin · 2019-01-14T19:04:36Z

This is a refactoring so that spark-core doesn't depend on Hive and Kafka at compile time. It's not really related to shading, and it would be distracting to mix both in this PR.

If a decision has been made about shading everything (I don't remember a vote), then there will be a lot of work that is unrelated to this PR to get there.

srowen · 2019-01-14T19:12:15Z

(Agree that shading is a big, important, but separate change to be considered)

gaborgsomogyi · 2019-01-14T23:59:24Z

@gatorsmile happy to handle this problem in the streaming/kafka area in separate PRs. I would appreciate if you can point the details and the vote to pick up the topic.

felixcheung · 2019-01-15T06:42:18Z

(I’d like to know about shading too)

vanzin · 2019-01-18T00:13:40Z

 def obtainDelegationTokens(
 hadoopConf: Configuration,
 sparkConf: SparkConf,
+ fileSystems: Set[FileSystem],


This is a bit odd, and ultimately I think that YarnSparkHadoopUtil.hadoopFSsToAccess should be used for everybody (a.k.a. moved to HadoopFSDelegationTokenProvider), but that should be a separate change.

Felt the same when changed but didn't have good solution. Good suggestion, will file a jira and resolve it.

SparkQA · 2019-01-21T18:41:09Z

Test build #101480 has finished for PR 23499 at commit a175cc1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-01-22T21:35:32Z

cc @wangyum since he is working on upgrading Hive dependency.

SparkQA · 2019-01-23T17:34:06Z

Test build #101597 has finished for PR 23499 at commit d74140d.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-01-24T20:51:34Z

Test build #101638 has finished for PR 23499 at commit ab41a9a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-01-25T09:00:03Z

Test build #101673 has finished for PR 23499 at commit 083199b.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-01-25T13:52:23Z

Test build #101676 has finished for PR 23499 at commit 7b88592.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2019-01-25T18:35:37Z

Merging to master.

## What changes were proposed in this pull request? There are ugly provided dependencies inside core for the following: * Hive * Kafka In this PR I've extracted them out. This PR contains the following: * Token providers are now loaded with service loader * Hive token provider moved to hive project * Kafka token provider extracted into a new project ## How was this patch tested? Existing + newly added unit tests. Additionally tested on cluster. Closesapache#23499 from gaborgsomogyi/SPARK-26254. Authored-by: Gabor Somogyi <gabor.g.somogyi@gmail.com> Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>

[SPARK-26254][CORE] Remove Hive + Kafka dependencies from Core.
47ea157

gaborgsomogyi force-pushed the SPARK-26254 branch from cd7db2f to 47ea157Compare January 9, 2019 13:49

Copile fix
d8188c0

gaborgsomogyi changed the title ~~[SPARK-26254][CORE] Remove Hive + Kafka dependencies from Core.~~[SPARK-26254][CORE] Extract Hive + Kafka dependencies from Core.Jan 9, 2019

Add Mima filter
c7a4282

Compile fix
6e0178c

srowen reviewed Jan 10, 2019
View reviewed changes

Comment threadpom.xml

vanzin reviewed Jan 18, 2019
View reviewed changes

Merge branch 'master' into SPARK-26254
da8ed8d

viirya reviewed Jan 21, 2019
View reviewed changes

Comment threadcore/src/main/scala/org/apache/spark/deploy/security/HadoopDelegationTokenManager.scala Outdated

Review fix.
a175cc1

HeartSaVioR mentioned this pull request Jan 21, 2019
[SPARK-26650][CORE]Yarn Client throws 'ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration' #23605
Closed

Merge remote-tracking branch 'upstream/master' into SPARK-26254
d74140d

vanzin reviewed Jan 23, 2019
View reviewed changes

Mima fix + throw provider check + hive test moved
ab41a9a

Add ref to SparkBuild.scala
083199b

Added project to MiMa
7b88592

asfgit closed this in 773efedJan 25, 2019

Uh oh!

Conversation

gaborgsomogyi commented Jan 9, 2019

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Jan 9, 2019

Uh oh!

SparkQA commented Jan 9, 2019

Uh oh!

SparkQA commented Jan 9, 2019

Uh oh!

SparkQA commented Jan 9, 2019

Uh oh!

SparkQA commented Jan 9, 2019

Uh oh!

gaborgsomogyi commented Jan 10, 2019

Uh oh!

Uh oh!

SparkQA commented Jan 10, 2019

Uh oh!

gaborgsomogyi commented Jan 10, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gatorsmile commented Jan 14, 2019

Uh oh!

vanzin commented Jan 14, 2019

Uh oh!

srowen commented Jan 14, 2019

Uh oh!

gaborgsomogyi commented Jan 14, 2019

Uh oh!

felixcheung commented Jan 15, 2019 via email

Uh oh!

Uh oh!

vanzinJan 18, 2019

Choose a reason for hiding this comment

Uh oh!

gaborgsomogyiJan 21, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SparkQA commented Jan 21, 2019

Uh oh!

dongjoon-hyun commented Jan 22, 2019

Uh oh!

SparkQA commented Jan 23, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SparkQA commented Jan 24, 2019

Uh oh!

SparkQA commented Jan 25, 2019

Uh oh!

SparkQA commented Jan 25, 2019

Uh oh!

vanzin commented Jan 25, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

gaborgsomogyi commented Jan 10, 2019•
edited
Loading