[SPARK-23639][SQL]Obtain token before init metastore client in SparkSQL CLI by yaooqinn · Pull Request #20784 · apache/spark

yaooqinn · 2018-03-09T10:28:47Z

What changes were proposed in this pull request?

In SparkSQLCLI, SessionState generates before SparkContext instantiating. When we use --proxy-user to impersonate, it's unable to initializing a metastore client to talk to the secured metastore for no kerberos ticket.

This PR use real user ugi to obtain token for owner before talking to kerberized metastore.

How was this patch tested?

Manually verified with kerberized hive metasotre / hdfs.

yaooqinn · 2018-03-09T10:32:37Z

cc @cloud-fan

SparkQA · 2018-03-09T14:26:06Z

Test build #88122 has finished for PR 20784 at commit 2011ef2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2018-03-09T18:42:48Z

Which cluster manager are you using? This should be completely unnecessary in YARN and Mesos, and standalone in general does not support kerberos.

yaooqinn · 2018-03-10T06:28:12Z

yarn @vanzin

vanzin · 2018-03-26T17:35:20Z

+ if (isSecuredAndProxy(conf)) {
+ val currentUser = UserGroupInformation.getCurrentUser
+ try {
+ SparkHadoopUtil.get.doAsRealUser {


Couldn't you just call HiveDelegationTokenProvider here instead of copy & pasting the code?

SparkQA · 2018-03-27T02:09:44Z

Test build #88612 has finished for PR 20784 at commit 4063855.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

yaooqinn

cc @vanzin

yaooqinn · 2018-03-27T02:10:02Z

+ if (isSecuredAndProxy(conf)) {
+ val currentUser = UserGroupInformation.getCurrentUser
+ try {
+ SparkHadoopUtil.get.doAsRealUser {


cloud-fan · 2018-03-27T02:32:47Z

cc @jerryshao

vanzin · 2018-03-27T02:37:48Z

 })
 }

+ private def isSecuredAndProxy(hiveConf: HiveConf): Boolean = {


Isn't this basically HiveDelegationTokenProvider.delegationTokensRequired? Doesn't it work if you just call that method? The only difference is the check for deploy mode, which should work fine in this context.

SparkQA · 2018-03-27T06:10:44Z

Test build #88613 has finished for PR 20784 at commit ba6b086.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-03-27T06:17:10Z

Test build #88615 has finished for PR 20784 at commit 5c4335b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-03-27T06:50:41Z

Test build #88617 has finished for PR 20784 at commit 1e38840.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

yaooqinn · 2018-03-27T06:55:24Z

retest this please

SparkQA · 2018-03-27T07:05:02Z

Test build #88620 has finished for PR 20784 at commit 1e38840.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

yaooqinn · 2018-03-27T08:50:50Z

retest this please

SparkQA · 2018-03-27T12:14:36Z

Test build #88624 has finished for PR 20784 at commit 1e38840.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2018-03-27T18:15:45Z

 }
 }

+ Option(new HiveDelegationTokenProvider)


What are you trying to achieve? new either returns something or throws an exception, so either you get Some(foo) here or an exception.

it will get a Some and filtered if token unneeded

vanzin · 2018-03-27T18:17:33Z

+ Option(new HiveDelegationTokenProvider)
+ .filter(_.delegationTokensRequired(sparkConf, hadoopConf))
+ .foreach(_.obtainDelegationTokens(
+ hadoopConf, sparkConf, UserGroupInformation.getCurrentUser.getCredentials))


This will not insert the tokens into the current UGI, because getCredentials returns a copy.

OK，i got it

SparkQA · 2018-03-28T05:15:30Z

Test build #88647 has finished for PR 20784 at commit cd8056c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2018-03-28T22:15:24Z

LGTM. retest this please

SparkQA · 2018-03-29T02:10:19Z

Test build #88680 has finished for PR 20784 at commit cd8056c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2018-03-29T17:46:06Z

Merging to master / 2.3.

…SQL CLI ## What changes were proposed in this pull request? In SparkSQLCLI, SessionState generates before SparkContext instantiating. When we use --proxy-user to impersonate, it's unable to initializing a metastore client to talk to the secured metastore for no kerberos ticket. This PR use real user ugi to obtain token for owner before talking to kerberized metastore. ## How was this patch tested? Manually verified with kerberized hive metasotre / hdfs. Author: Kent Yao <yaooqinn@hotmail.com> Closes#20784 from yaooqinn/SPARK-23639. (cherry picked from commit a7755fd) Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>

…SQL CLI ## What changes were proposed in this pull request? In SparkSQLCLI, SessionState generates before SparkContext instantiating. When we use --proxy-user to impersonate, it's unable to initializing a metastore client to talk to the secured metastore for no kerberos ticket. This PR use real user ugi to obtain token for owner before talking to kerberized metastore. ## How was this patch tested? Manually verified with kerberized hive metasotre / hdfs. Author: Kent Yao <yaooqinn@hotmail.com> Closesapache#20784 from yaooqinn/SPARK-23639.

…SQL CLI In SparkSQLCLI, SessionState generates before SparkContext instantiating. When we use --proxy-user to impersonate, it's unable to initializing a metastore client to talk to the secured metastore for no kerberos ticket. This PR use real user ugi to obtain token for owner before talking to kerberized metastore. Manually verified with kerberized hive metasotre / hdfs. Author: Kent Yao <yaooqinn@hotmail.com> Closesapache#20784 from yaooqinn/SPARK-23639. (cherry picked from commit a7755fd) Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com> Change-Id: I78879cd2500f911c19eccef6a1140fb996485e26

obtain token before init metastore client
2011ef2

yaooqinn mentioned this pull request Mar 24, 2018
[SPARK-23789][SQL] Shouldn't set hive.metastore.uris before invoking HiveDelegationTokenProvider #20898
Closed

vanzin reviewed Mar 26, 2018
View reviewed changes

yaooqinn added 2 commits March 27, 2018 10:06

call hive delegation token provider
4063855

import
ba6b086

yaooqinn commented Mar 27, 2018
View reviewed changes

unrelated white line
5c4335b

vanzin reviewed Mar 27, 2018
View reviewed changes

check via hive delegation token provider itself
1e38840

vanzin reviewed Mar 27, 2018
View reviewed changes

option to if else
cd8056c

asfgit closed this in a7755fdMar 29, 2018

LantaoJin mentioned this pull request May 23, 2018
[SPARK-24349][SQL] Ignore setting token if using JDBC #21396
Closed

Uh oh!

Conversation

yaooqinn commented Mar 9, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

yaooqinn commented Mar 9, 2018

Uh oh!

SparkQA commented Mar 9, 2018

Uh oh!

vanzin commented Mar 9, 2018

Uh oh!

yaooqinn commented Mar 10, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Mar 27, 2018

Uh oh!

yaooqinn left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Mar 27, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Mar 27, 2018

Uh oh!

SparkQA commented Mar 27, 2018

Uh oh!

SparkQA commented Mar 27, 2018

Uh oh!

yaooqinn commented Mar 27, 2018

Uh oh!

SparkQA commented Mar 27, 2018

Uh oh!

yaooqinn commented Mar 27, 2018

Uh oh!

SparkQA commented Mar 27, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Mar 28, 2018

Uh oh!

vanzin commented Mar 28, 2018

Uh oh!

SparkQA commented Mar 29, 2018

Uh oh!

vanzin commented Mar 29, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yaooqinn commented Mar 9, 2018•
edited
Loading

yaooqinn left a comment •
edited
Loading