Uh oh!
There was an error while loading. Please reload this page.
[SPARK-27482][SQL][WEBUI] Show estimated BroadcastHashJoinExec numOutputRows statistics info on SparkSQL UI page - #24666
[SPARK-27482][SQL][WEBUI] Show estimated BroadcastHashJoinExec numOutputRows statistics info on SparkSQL UI page#24666pengbo wants to merge 4 commits into
Conversation
pengbo
commented
May 21, 2019
@cloud-fan , Can you please have a look? |
cloud-fan
commented
May 22, 2019
ok to test |
SparkQA
commented
May 22, 2019
Test build #105683 has finished for PR 24666 at commit
|
wangyum
commented
May 23, 2019
retest this please |
SparkQA
commented
May 23, 2019
Test build #105719 has finished for PR 24666 at commit
|
…anned by SparkStrategies
pengbo
commented
May 23, 2019
retest this please |
SparkQA
commented
May 23, 2019
Test build #105727 has finished for PR 24666 at commit
|
SparkQA
commented
May 24, 2019
Test build #105745 has finished for PR 24666 at commit
|
| val accumulatorId: Long, | ||
| val metricType: String) | ||
| val metricType: String, | ||
| val stats: Long = -1) |
There was a problem hiding this comment.
not all metric has its corresponding statistics (e.g. peakMemory), and not all statistics are long type. We should think of a better place to carry the statistics.
There was a problem hiding this comment.
or we can put a val stats: Option[Statistics] = None
There was a problem hiding this comment.
@cloud-fan Thanks for your comments.
The idea is that each SQL metric can have a statistic value (-1 means not available/initialized). I set the statistic type to Long is because SQL Metric's value is always Long type as well. class SQLMetric(val metricType: String, initValue: Long = 0L)
Put Option[Statistics] in SQLMetricInfo doesn't sound quite right though. It means that all SQL metrics have an attribute including rowCount, size & column stats.
Let me know your feedback, thanks in advance.
| } | ||
| } | ||
| def stringStats(value: Long): String = { |
There was a problem hiding this comment.
we should handle stats in stringValue, different metrics may need to look at different stats and display different things.
There was a problem hiding this comment.
one example: we can also display the difference between real row count and estimated row count, e.g. 10X, 0.01X, etc. Something like row count: 4, est: 40 (10X)
@pengbo The title is a bit confusing...I think we should make it more clear, e.g. |
| SQLMetrics.createMetric( | ||
| sparkContext, | ||
| "number of output rows", | ||
| logicalPlan.map(_.stats.rowCount.map(_.toLong).getOrElse(-1L)).getOrElse(-1L))) |
There was a problem hiding this comment.
IIRC, for file sources, usually there is only sizeInBytes stats in logical plan level. So the estimated numOutputRows for logical plan should be empty for file sources.
What is the scenario of this PR?
There was a problem hiding this comment.
for file source table, there will be row count stats if CBO is enabled.
pengbo
commented
Jun 2, 2019
okay, thanks |
What changes were proposed in this pull request?
Currently, the SparkSQL UI page shows only actual metric info in each SparkPlan node. However with
statisticsinfo may help us understand how the plan is designed and the reason why it runs slowly. This PR is to shownumOutputRowsmetric'sstatisticinfo ofBroadcastHashJoinExecnode on SparkSQL UI page when it's available.The main changes:
statsfield inSQLMetricand passing it toSQLPlanMetricto show on UI page when it's availablenumOutputRowswith rowCountstatsinlogicalPlanofBroadcastHashJoinExec, thanks to [SPARK-27747][SQL] add a logical plan link in the physical plan #24626How was this patch tested?
Regarding unit test has been added, manual UI test has been tested