Skip to content

[SPARK-27088][SQL] Add a configuration to set log level for each batch at RuleExecutor - #24136

Closed
chakravarthiT wants to merge 2 commits into
apache:masterfrom
chakravarthiT:logBatches
Closed

[SPARK-27088][SQL] Add a configuration to set log level for each batch at RuleExecutor#24136
chakravarthiT wants to merge 2 commits into
apache:masterfrom
chakravarthiT:logBatches

Conversation

@chakravarthiT

@chakravarthiTchakravarthiT commented Mar 18, 2019

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Similar to #22406 , which has made log level for plan changes by each rule configurable ,this PR is to make log level for plan changes by each batch configurable,and I have reused the same configuration: "spark.sql.optimizer.planChangeLog.level".

Config proposed in this PR ,
spark.sql.optimizer.planChangeLog.batches - enable plan change logging only for a set of specified batches, separated by commas.

How was this patch tested?

Added UT , also tested manually and attached screenshots below.

1)Setting spark.sql.optimizer.planChangeLog.leve to warn.

settingLogLevelToWarn

2)setting spark.sql.optimizer.planChangeLog.batches to Resolution and Subquery.
settingBatchestoLog

  1. plan change logging enabled only for a set of specified batches(Resolution and Subquery)
    batchloggingOp

@chakravarthiT

chakravarthiT commented Mar 18, 2019

Copy link
Copy Markdown
ContributorAuthor

@gengliangwang@HyukjinKwon Please review.

@HyukjinKwon

HyukjinKwon commented Mar 19, 2019

Copy link
Copy Markdown
Member

cc @maryannxue

maropu
maropu reviewed Mar 19, 2019
View reviewed changes
Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala Outdated
maropu
maropu reviewed Mar 19, 2019
View reviewed changes
Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/rules/RuleExecutor.scala Outdated
maropu
maropu reviewed Mar 19, 2019
View reviewed changes
Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/rules/RuleExecutor.scala Outdated
maropu
maropu reviewed Mar 19, 2019
View reviewed changes
Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/rules/RuleExecutor.scala Outdated
maropu
maropu reviewed Mar 19, 2019
View reviewed changes
Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala Outdated
maryannxue
maryannxue approved these changes Mar 20, 2019
View reviewed changes

@maryannxuemaryannxue left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, except some minor suggestions.

Comment threadsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizerLoggingSuite.scala Outdated
Comment threadsql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OptimizerLoggingSuite.scala Outdated
@gatorsmile

gatorsmile commented Mar 21, 2019

Copy link
Copy Markdown
Member

ok to test

@SparkQA

SparkQA commented Mar 21, 2019

Copy link
Copy Markdown

Test build #103756 has finished for PR 24136 at commit de7aae0.

  • This patch fails due to an unknown error code, -9.
  • This patch merges cleanly.
  • This patch adds no public classes.

@maropu

maropu commented Mar 21, 2019

Copy link
Copy Markdown
Member

retest this please

maropu
maropu reviewed Mar 21, 2019
View reviewed changes
Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala Outdated
@SparkQA

SparkQA commented Mar 21, 2019

Copy link
Copy Markdown

Test build #103760 has finished for PR 24136 at commit de7aae0.

  • This patch fails to build.
  • This patch merges cleanly.
  • This patch adds no public classes.

@chakravarthiT

chakravarthiT commented Mar 21, 2019

Copy link
Copy Markdown
ContributorAuthor

looks like non-relevant failure.

@maropu

maropu commented Mar 22, 2019

Copy link
Copy Markdown
Member

retest this please

@SparkQA

SparkQA commented Mar 22, 2019

Copy link
Copy Markdown

Test build #103799 has finished for PR 24136 at commit de7aae0.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Mar 22, 2019

Copy link
Copy Markdown

Test build #103801 has finished for PR 24136 at commit f68d21b.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
  • class ArrowStreamPandasSerializer(ArrowStreamSerializer):
  • class ArrowStreamPandasUDFSerializer(ArrowStreamPandasSerializer):
  • class IdentifierImpl implements Identifier
  • class CatalogNotFoundException(message: String, cause: Throwable)
  • trait LookupCatalog
  • class FractionTimestampFormatter(zoneId: ZoneId)

@sandeep-kattasandeep-katta mentioned this pull request Mar 22, 2019
Closed
@HyukjinKwonHyukjinKwon changed the title [SPARK-27088][SQL] Apply conf "spark.sql.optimizer.planChangeLog.leve…[SPARK-27088][SQL] Add a configuration to set log level for each batch at RuleExecutorMar 25, 2019
HyukjinKwon
HyukjinKwon reviewed Mar 25, 2019
View reviewed changes
Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/rules/RuleExecutor.scala Outdated
@SparkQA

SparkQA commented Mar 26, 2019

Copy link
Copy Markdown

Test build #103991 has finished for PR 24136 at commit 59b45d1.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@chakravarthiT

chakravarthiT commented Mar 27, 2019•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@maropu@HyukjinKwon handled review comments. and now that UT is passed,as message will be evaluated only when specific log level is set.

gatorsmile
gatorsmile reviewed Mar 27, 2019
View reviewed changes
Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/rules/RuleExecutor.scala
} else {
logTrace(s"Batch ${batch.name} has no effect.")
}
planChangeLogger.logBatch(batch.name, batchStartPlan, curPlan)

@gatorsmilegatorsmileMar 27, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do the same thing for line 148?

@chakravarthiTchakravarthiTMar 27, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The config "spark.sql.optimizer.planChangeLog.level", is used for logging plan changes after applying rule or batch . As per Line 148,it is logging when there is no change in plan,which conflicts the parameter(spark.sql.optimizer.planChangeLog.level). Its not required to log this.

@chakravarthiTchakravarthiTApr 9, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gatorsmile please review

@HyukjinKwonHyukjinKwonApr 11, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please address them in a followup if there are further comments.

@gatorsmile

gatorsmile commented Mar 27, 2019

Copy link
Copy Markdown
Member

LGTM except the above comment.

@HyukjinKwon

HyukjinKwon commented Apr 10, 2019

Copy link
Copy Markdown
Member

retest this please

@SparkQA

SparkQA commented Apr 10, 2019

Copy link
Copy Markdown

Test build #104485 has finished for PR 24136 at commit 59b45d1.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@HyukjinKwon

HyukjinKwon commented Apr 11, 2019

Copy link
Copy Markdown
Member

Merged to master.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gengliangwanggengliangwanggengliangwang left review comments
@HyukjinKwonHyukjinKwonHyukjinKwon approved these changes
@gatorsmilegatorsmilegatorsmile left review comments
@maropumaropumaropu approved these changes
+1 more reviewer
@maryannxuemaryannxuemaryannxue approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@chakravarthiT@HyukjinKwon@gatorsmile@SparkQA@maropu@gengliangwang@maryannxue