[SPARK-13979][Core] Killed executor is re spawned without AWS key… by agsachin · Pull Request #14601 · apache/spark

agsachin · 2016-08-11T10:41:23Z

What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)
Summery we need to make two changes

in DataSourceStratgy.scala. use sqlContext.sparkContext.hadoopConfiguration instead of SparkHadoopUtil.get.conf
update def newConfiguration(conf: SparkConf): Configuration = {..} function to copy only s3 and swift related confs

How was this patch tested?

This patch was manually tested on local machine in standalone cluster mode.
To simulate the case of executor failure. We manually killed the executor then master started new executor with updated config params.

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
no

…s in standalone spark cluster

lresende · 2016-08-11T15:08:23Z

Jenkins test this please

lresende · 2016-08-11T15:11:37Z

@@ -107,6 +107,14 @@ class SparkHadoopUtil extends Logging {
 if (key.startsWith("spark.hadoop.")) {


the comment above does not apply for the whole loop anymore and should be moved to this if statement

lresende · 2016-08-11T15:15:24Z

Minor, the title should be [CORE] instead of [SPARK CORE]

lresende · 2016-08-11T15:17:25Z

Also, will this also fix the scenario where the user has provided the properties programmatically ?

agsachin · 2016-08-16T10:57:35Z

@lresende yes this fix works for the the scenario where the user has provided the properties programmatically

like for swift:-
sc.hadoopConfiguration.set("fs.swift2...", "")
...
and for amazon-S3:-
sc.hadoopConfiguration.set("fs.s3....", "org.apache.hadoop.fs.s3a.S3AFileSystem")

steveloughran · 2016-08-20T20:23:31Z

 }
+	// Copy any "fs.swift2d.foo=bar" properties into conf as "fs.swift2d.foo=bar"
+ else if (key.startsWith("fs.swift2d")){
+ hadoopConf.set(key, value)


What's swift2d? It's not the swift client in hadoop-openstack, which is fs.swift

swift2d is used when u are using Stocator --> https://github.com/SparkTC/stocator. now I have updated for hadoop-openstack also

have updated for hadoop-openstack package properties

added comment with jira number

steveloughran · 2016-08-21T10:01:46Z

I'd like to propose that the list of filesystem properties to propagate is actually defined as a list in a spark property, default could be "fs.s3a, fs.s3n, fs.s3, fs.swift, fs.wasb". This will allow people to extend it (google GCFS, Allyun OSS HADOOP-12756, etc, without having to patch spark or wait for a new release. And they get the option to turn off the feature whatsoever. By using the full fs.prefix you could also let people add other properties to carry over.

One thing I'm confused by though: why do spawned executors get the properties, but not respawned ones?

steveloughran · 2016-09-13T08:58:20Z

+ // Copy any "fs.s3.foo=bar" or "fs.s3a.foo=bar" or "fs.s3n.foo=bar" properties into conf 
+ else if (key.startsWith("fs.s3")){
+ hadoopConf.set(key, value)
+ }


s3 is the AWS EMR filesystem, but an obsolete one on ASF Hadoop. I would recommend the list of
s3, s3n, s3a, swift, adl, wasb, oss, gs
(edited 10/oct, set list in sync with what I believe is current set)

Actually, I would copy everything under fs.s3a, fs.s3, etc. Why? there's a lot more than passwords: for s3a we include: proxy info and passwords, a list of alternate s3 auth mechanisms (e.g. declaring using IAM), etc, etc.

@steveloughran can u help me with some default setting for adl, wasb, oss, gs to test or just syntax for them so that i can decide on filter conditions

steveloughran · 2016-09-13T09:03:48Z

Could an automated test be done here. propagation can be tested with a a function run on the
executor (such as a map) which fails if the required properties are missing

(set some properties in the conf)
restart the executor
run a map which fails if things weren't copied

Seq("s3n", s3a.aws", ...).parallelize.map { name -> (check for fs.s3a properties) or fail }

lresende · 2016-10-06T19:29:36Z

@agsachin Are you planning to address these updates on this PR ? It would be good to have this as part of Spark as it affects multiple usage scenarios in cloud platforms and other cases as well.

HyukjinKwon · 2017-02-09T14:01:28Z

(gentle ping @agsachin)

agsachin · 2017-02-15T11:06:31Z

@lresende @steveloughran @HyukjinKwon can we go ahead with key.startsWith("fs."). so that we don't need to check so many conditions.

steveloughran · 2017-02-25T16:09:11Z

spark.hadoop.fs.* would work.

The (not yet shipped in ASF code) Azure Data Lake FS has, for reasons I don't know and have only just noticed, adopted "dfs.adl" as their prefix. That's going to be fixed before it gets out the door, so fs.* will work there too.

agsachin · 2017-02-25T18:53:19Z

@steveloughran I have updated the pull request to fs.*

steveloughran · 2017-02-25T21:29:07Z

It's good to have some tests
2.-I note that appendS3AndSparkHadoopConfigurations() has a weakness in how it propagates env vars: no propagation of the session environment AWS_SESSION_TOKEN variable to "fs.s3a.session.token", same for region. That's probably significant enough to have its own JIRA though

update issue 2 is fixed

jiangxb1987 · 2017-06-21T14:31:28Z

@agsachin Are you still working on this? And if it is so, would you please update the description and provide some snapshot to demo the behaviors before & after the changes? It would also be great if some test cases could be provided(I hope it won't be too hard).

jiangxb1987 · 2017-06-21T14:33:43Z

add to whitelist

jiangxb1987 · 2017-06-21T14:33:55Z

test this please

steveloughran · 2017-06-21T19:03:07Z

Testing should not be too hard. Here's my untested attempt

valsconf=newSparkConf(false)
sconf.set("fs.example.value", "true")
valconf=newConfiguration(false)
newSparkHadoopUtil().appendS3AndSparkHadoopConfigurations(sconf, conf)
assert(conf.getBoolean("fs.example.value", false))

HyukjinKwon · 2017-07-24T03:06:08Z

gentle ping @agsachin.

agsachin · 2017-07-24T03:42:04Z

Thanks @jiangxb2987 will add this test case by tomorrow and will be update the pr with results

steveloughran · 2017-08-06T18:21:31Z

I know this hasn't been updated, but it is still important. I can take it on if all it needs is a test case

jiangxb1987 · 2017-08-07T10:19:13Z

@steveloughran Please feel free to reopen this PR. Thanks!

## What changes were proposed in this pull request? This PR proposes to close stale PRs, mostly the same instances with apache#18017Closesapache#14085 - [SPARK-16408][SQL] SparkSQL Added file get Exception: is a directory … Closesapache#14239 - [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism to accelerate shuffle stage. Closesapache#14567 - [SPARK-16992][PYSPARK] Python Pep8 formatting and import reorganisation Closesapache#14579 - [SPARK-16921][PYSPARK] RDD/DataFrame persist()/cache() should return Python context managers Closesapache#14601 - [SPARK-13979][Core] Killed executor is re spawned without AWS key… Closesapache#14830 - [SPARK-16992][PYSPARK][DOCS] import sort and autopep8 on Pyspark examples Closesapache#14963 - [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in lint-python Closesapache#15227 - [SPARK-17655][SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage Closesapache#15240 - [SPARK-17556] [CORE] [SQL] Executor side broadcast for broadcast joins Closesapache#15405 - [SPARK-15917][CORE] Added support for number of executors in Standalone [WIP] Closesapache#16099 - [SPARK-18665][SQL] set statement state to "ERROR" after user cancel job Closesapache#16445 - [SPARK-19043][SQL]Make SparkSQLSessionManager more configurable Closesapache#16618 - [SPARK-14409][ML][WIP] Add RankingEvaluator Closesapache#16766 - [SPARK-19426][SQL] Custom coalesce for Dataset Closesapache#16832 - [SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columns Closesapache#17052 - [SPARK-19690][SS] Join a streaming DataFrame with a batch DataFrame which has an aggregation may not work Closesapache#17267 - [SPARK-19926][PYSPARK] Make pyspark exception more user-friendly Closesapache#17371 - [SPARK-19903][PYSPARK][SS] window operator miss the `watermark` metadata of time column Closesapache#17401 - [SPARK-18364][YARN] Expose metrics for YarnShuffleService Closesapache#17519 - [SPARK-15352][Doc] follow-up: add configuration docs for topology-aware block replication Closesapache#17530 - [SPARK-5158] Access kerberized HDFS from Spark standalone Closesapache#17854 - [SPARK-20564][Deploy] Reduce massive executor failures when executor count is large (>2000) Closesapache#17979 - [SPARK-19320][MESOS][WIP]allow specifying a hard limit on number of gpus required in each spark executor when running on mesos Closesapache#18127 - [SPARK-6628][SQL][Branch-2.1] Fix ClassCastException when executing sql statement 'insert into' on hbase table Closesapache#18236 - [SPARK-21015] Check field name is not null and empty in GenericRowWit… Closesapache#18269 - [SPARK-21056][SQL] Use at most one spark job to list files in InMemoryFileIndex Closesapache#18328 - [SPARK-21121][SQL] Support changing storage level via the spark.sql.inMemoryColumnarStorage.level variable Closesapache#18354 - [SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Pool Limit - Class Splitting Closesapache#18383 - [SPARK-21167][SS] Set kafka clientId while fetch messages Closesapache#18414 - [SPARK-21169] [core] Make sure to update application status to RUNNING if executors are accepted and RUNNING after recovery Closesapache#18432 - resolve com.esotericsoftware.kryo.KryoException Closesapache#18490 - [SPARK-21269][Core][WIP] Fix FetchFailedException when enable maxReqSizeShuffleToMem and KryoSerializer Closesapache#18585 - SPARK-21359 Closesapache#18609 - Spark SQL merge small files to big files Update InsertIntoHiveTable.scala Added: Closesapache#18308 - [SPARK-21099][Spark Core] INFO Log Message Using Incorrect Executor I… Closesapache#18599 - [SPARK-21372] spark writes one log file even I set the number of spark_rotate_log to 0 Closesapache#18619 - [SPARK-21397][BUILD]Maven shade plugin adding dependency-reduced-pom.xml to … Closesapache#18667 - Fix the simpleString used in error messages Closesapache#18782 - Branch 2.1 Added: Closesapache#17694 - [SPARK-12717][PYSPARK] Resolving race condition with pyspark broadcasts when using multiple threads Added: Closesapache#16456 - [SPARK-18994] clean up the local directories for application in future by annother thread Closesapache#18683 - [SPARK-21474][CORE] Make number of parallel fetches from a reducer configurable Closesapache#18690 - [SPARK-21334][CORE] Add metrics reporting service to External Shuffle Server Added: Closesapache#18827 - Merge pull request 1 from apache/master ## How was this patch tested? N/A Author: hyukjinkwon <gurwls223@gmail.com> Closesapache#18780 from HyukjinKwon/close-prs.

[SPARK-13979][Spark Core]Killed executor is respawned without AWS key…
9efc140
…s in standalone spark cluster

lresende reviewed Aug 11, 2016
View reviewed changes

agsachin changed the title ~~[SPARK-13979][Spark Core][WIP]Killed executor is re spawned without AWS key…~~[SPARK-13979][Core][WIP]Killed executor is re spawned without AWS key…Aug 16, 2016

corrected the position of comment
89e7c97

agsachin changed the title ~~[SPARK-13979][Core][WIP]Killed executor is re spawned without AWS key…~~[SPARK-13979][Core] Killed executor is re spawned without AWS key…Aug 17, 2016

steveloughran reviewed Aug 20, 2016
View reviewed changes

agsachin added 2 commits August 21, 2016 10:05

Update SparkHadoopUtil.scala
ab6ab9b
have updated for hadoop-openstack package properties

Update SparkHadoopUtil.scala
9765dc9
added comment with jira number

steveloughran reviewed Sep 13, 2016
View reviewed changes

added fs.* instead of filesystem specific
dee4e6d

HyukjinKwon mentioned this pull request Jul 31, 2017
[INFRA] Close stale PRs #18780
Closed

asfgit closed this in 3a45c7fAug 5, 2017

		@@ -107,6 +107,14 @@ class SparkHadoopUtil extends Logging {
		if (key.startsWith("spark.hadoop.")) {

Uh oh!

Conversation

agsachin commented Aug 11, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

lresende commented Aug 11, 2016

Uh oh!

lresendeAug 11, 2016

Choose a reason for hiding this comment

Uh oh!

agsachinAug 16, 2016

Choose a reason for hiding this comment

Uh oh!

lresende commented Aug 11, 2016

Uh oh!

lresende commented Aug 11, 2016

Uh oh!

agsachin commented Aug 16, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steveloughranAug 20, 2016

Choose a reason for hiding this comment

Uh oh!

agsachinAug 21, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

steveloughran commented Aug 21, 2016

Uh oh!

steveloughranSep 13, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

steveloughranSep 13, 2016

Choose a reason for hiding this comment

Uh oh!

agsachinFeb 15, 2017

Choose a reason for hiding this comment

Uh oh!

steveloughran commented Sep 13, 2016

Uh oh!

lresende commented Oct 6, 2016

Uh oh!

HyukjinKwon commented Feb 9, 2017

Uh oh!

agsachin commented Feb 15, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steveloughran commented Feb 25, 2017

Uh oh!

agsachin commented Feb 25, 2017

Uh oh!

steveloughran commented Feb 25, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jiangxb1987 commented Jun 21, 2017

Uh oh!

jiangxb1987 commented Jun 21, 2017

Uh oh!

jiangxb1987 commented Jun 21, 2017

Uh oh!

steveloughran commented Jun 21, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HyukjinKwon commented Jul 24, 2017

Uh oh!

agsachin commented Jul 24, 2017

Uh oh!

steveloughran commented Aug 6, 2017

Uh oh!

jiangxb1987 commented Aug 7, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

agsachin commented Aug 11, 2016•
edited
Loading

agsachin commented Aug 16, 2016•
edited
Loading

agsachinAug 21, 2016•
edited
Loading

steveloughranSep 13, 2016•
edited
Loading

agsachin commented Feb 15, 2017•
edited
Loading

steveloughran commented Feb 25, 2017•
edited
Loading

steveloughran commented Jun 21, 2017•
edited
Loading