[SPARK-21372] spark writes one log file even I set the number of spark_rotate_log to 0 by liu-zhaokun · Pull Request #18599 · apache/spark

liu-zhaokun · 2017-07-11T08:00:25Z

https://issues.apache.org/jira/browse/SPARK-21372
Spark writes one log file even I set the number of spark_rotate_log to 0.Meanwhile,the number of spark_rotate_log also can set to a string type and other type.So I think we should make some changes.

…k_rotate_log to 0

AmplabJenkins · 2017-07-11T08:02:10Z

Can one of the admins verify this patch?

jiangxb1987 · 2017-07-11T09:12:26Z

Did you see any failure in production environment caused by this issue?

srowen · 2017-07-11T09:18:54Z

 fi
+
+ expr $num + 0 &>/dev/null
+ if [ $? -ne 0 ] || [ $num -eq 0 ]; then


Are you trying to verify if num is a number? I get it but you would already get a bash error in $num -eq 0 if so and we don't otherwise validate script input that way. Isn't this forcing num to be nonzero, when I thought the issue was handling the case of 0 correctly?

@srowen
Yes, we should avoid this scenario that num isn't number.If num isn't a number,num will be set to 5.

liu-zhaokun · 2017-07-11T09:35:10Z

@jiangxb1987
When I want to change the number of log file,I test the scenario that num=0,and spark writes one log file.When I test the scenario that set num to Invaild symbol，the logic is wrong.So I think we should modify it to guarantee this feature.

jiangxb1987 · 2017-07-11T13:34:34Z

AFAIK the function spark_rotate_log is never used with num param set, unless you set log param manually, in that case I think you are responsible for the data validation, it would be interesting to see how this issue is triggered, could you share your script please? Thank you!

liu-zhaokun · 2017-07-11T15:06:57Z

@jiangxb1987
I want to change the number of log file ,so I passed num param manually by modify code in this file.In fact,this bug is there，even I don't trigger it,right?Shouldn't we modify it?

srowen · 2017-07-11T15:26:35Z

@liu-zhaokun I don't understand what bug you are referring to. We'd need to understand how an unmodified script displays some incorrect behavior.

liu-zhaokun · 2017-07-11T15:41:07Z

@srowen
If I set num to 0,there shouldn't create a logfile or we should make some deal for 0，and If I set num some invaild symbol,the function doesn't work normally,right？I am sorry for my poor English.

srowen · 2017-07-11T15:47:28Z

How would you set this with normal operation?
If you change the script to set a value that wasn't expected at that point given how the script works, of course it may fail. That's not a bug.

liu-zhaokun · 2017-07-12T01:39:20Z

@@ -78,6 +78,12 @@ spark_rotate_log ()
 if [ -n "$2" ]; then
 	num=$2


@srowen
There provide a param,num, to set the number of logfile,but it wasn't used in line 179.Since it doesn't work,can I remove the param "num" and use it just as a variable？

liu-zhaokun · 2017-07-12T01:41:09Z

@srowen
I think spark provides the param,so I can pass it in the script.

srowen · 2017-07-12T07:59:28Z

Let's close this. You aren't providing a scenario where an input causes a problem, and I think are still saying the error happens when you modify the script.

liu-zhaokun · 2017-07-12T08:07:44Z

@srowen
The param "num" which defined in function spark_rotate_log() isn't used,if users don't modify the script,it will never be invoked,so could I create a new PR to delete the param and replace it with a fixed number 5?

## What changes were proposed in this pull request? This PR proposes to close stale PRs, mostly the same instances with apache#18017Closesapache#14085 - [SPARK-16408][SQL] SparkSQL Added file get Exception: is a directory … Closesapache#14239 - [SPARK-16593] [CORE] [WIP] Provide a pre-fetch mechanism to accelerate shuffle stage. Closesapache#14567 - [SPARK-16992][PYSPARK] Python Pep8 formatting and import reorganisation Closesapache#14579 - [SPARK-16921][PYSPARK] RDD/DataFrame persist()/cache() should return Python context managers Closesapache#14601 - [SPARK-13979][Core] Killed executor is re spawned without AWS key… Closesapache#14830 - [SPARK-16992][PYSPARK][DOCS] import sort and autopep8 on Pyspark examples Closesapache#14963 - [SPARK-16992][PYSPARK] Virtualenv for Pylint and pep8 in lint-python Closesapache#15227 - [SPARK-17655][SQL]Remove unused variables declarations and definations in a WholeStageCodeGened stage Closesapache#15240 - [SPARK-17556] [CORE] [SQL] Executor side broadcast for broadcast joins Closesapache#15405 - [SPARK-15917][CORE] Added support for number of executors in Standalone [WIP] Closesapache#16099 - [SPARK-18665][SQL] set statement state to "ERROR" after user cancel job Closesapache#16445 - [SPARK-19043][SQL]Make SparkSQLSessionManager more configurable Closesapache#16618 - [SPARK-14409][ML][WIP] Add RankingEvaluator Closesapache#16766 - [SPARK-19426][SQL] Custom coalesce for Dataset Closesapache#16832 - [SPARK-19490][SQL] ignore case sensitivity when filtering hive partition columns Closesapache#17052 - [SPARK-19690][SS] Join a streaming DataFrame with a batch DataFrame which has an aggregation may not work Closesapache#17267 - [SPARK-19926][PYSPARK] Make pyspark exception more user-friendly Closesapache#17371 - [SPARK-19903][PYSPARK][SS] window operator miss the `watermark` metadata of time column Closesapache#17401 - [SPARK-18364][YARN] Expose metrics for YarnShuffleService Closesapache#17519 - [SPARK-15352][Doc] follow-up: add configuration docs for topology-aware block replication Closesapache#17530 - [SPARK-5158] Access kerberized HDFS from Spark standalone Closesapache#17854 - [SPARK-20564][Deploy] Reduce massive executor failures when executor count is large (>2000) Closesapache#17979 - [SPARK-19320][MESOS][WIP]allow specifying a hard limit on number of gpus required in each spark executor when running on mesos Closesapache#18127 - [SPARK-6628][SQL][Branch-2.1] Fix ClassCastException when executing sql statement 'insert into' on hbase table Closesapache#18236 - [SPARK-21015] Check field name is not null and empty in GenericRowWit… Closesapache#18269 - [SPARK-21056][SQL] Use at most one spark job to list files in InMemoryFileIndex Closesapache#18328 - [SPARK-21121][SQL] Support changing storage level via the spark.sql.inMemoryColumnarStorage.level variable Closesapache#18354 - [SPARK-18016][SQL][CATALYST][BRANCH-2.1] Code Generation: Constant Pool Limit - Class Splitting Closesapache#18383 - [SPARK-21167][SS] Set kafka clientId while fetch messages Closesapache#18414 - [SPARK-21169] [core] Make sure to update application status to RUNNING if executors are accepted and RUNNING after recovery Closesapache#18432 - resolve com.esotericsoftware.kryo.KryoException Closesapache#18490 - [SPARK-21269][Core][WIP] Fix FetchFailedException when enable maxReqSizeShuffleToMem and KryoSerializer Closesapache#18585 - SPARK-21359 Closesapache#18609 - Spark SQL merge small files to big files Update InsertIntoHiveTable.scala Added: Closesapache#18308 - [SPARK-21099][Spark Core] INFO Log Message Using Incorrect Executor I… Closesapache#18599 - [SPARK-21372] spark writes one log file even I set the number of spark_rotate_log to 0 Closesapache#18619 - [SPARK-21397][BUILD]Maven shade plugin adding dependency-reduced-pom.xml to … Closesapache#18667 - Fix the simpleString used in error messages Closesapache#18782 - Branch 2.1 Added: Closesapache#17694 - [SPARK-12717][PYSPARK] Resolving race condition with pyspark broadcasts when using multiple threads Added: Closesapache#16456 - [SPARK-18994] clean up the local directories for application in future by annother thread Closesapache#18683 - [SPARK-21474][CORE] Make number of parallel fetches from a reducer configurable Closesapache#18690 - [SPARK-21334][CORE] Add metrics reporting service to External Shuffle Server Added: Closesapache#18827 - Merge pull request 1 from apache/master ## How was this patch tested? N/A Author: hyukjinkwon <gurwls223@gmail.com> Closesapache#18780 from HyukjinKwon/close-prs.

[SPARK-21372] spark writes one log file even I set the number of spar…
7c71bc7
…k_rotate_log to 0

srowen reviewed Jul 11, 2017
View reviewed changes

liu-zhaokun commented Jul 12, 2017
View reviewed changes

srowen mentioned this pull request Jul 31, 2017
[INFRA] Close stale PRs #18780
Closed

asfgit closed this in 3a45c7fAug 5, 2017

		@@ -78,6 +78,12 @@ spark_rotate_log ()
		if [ -n "$2" ]; then
		num=$2

Uh oh!

Conversation

liu-zhaokun commented Jul 11, 2017

Uh oh!

AmplabJenkins commented Jul 11, 2017

Uh oh!

jiangxb1987 commented Jul 11, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

srowenJul 11, 2017

Choose a reason for hiding this comment

Uh oh!

liu-zhaokunJul 11, 2017

Choose a reason for hiding this comment

Uh oh!

liu-zhaokun commented Jul 11, 2017

Uh oh!

jiangxb1987 commented Jul 11, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liu-zhaokun commented Jul 11, 2017

Uh oh!

srowen commented Jul 11, 2017

Uh oh!

liu-zhaokun commented Jul 11, 2017

Uh oh!

srowen commented Jul 11, 2017

Uh oh!

liu-zhaokunJul 12, 2017

Choose a reason for hiding this comment

Uh oh!

liu-zhaokun commented Jul 12, 2017

Uh oh!

srowen commented Jul 12, 2017

Uh oh!

liu-zhaokun commented Jul 12, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jiangxb1987 commented Jul 11, 2017•
edited
Loading

jiangxb1987 commented Jul 11, 2017•
edited
Loading