Uh oh!
There was an error while loading. Please reload this page.
[SPARK-24863][SS] Report Kafka offset lag as a custom metrics - #21819
[SPARK-24863][SS] Report Kafka offset lag as a custom metrics#21819arunmahadevan wants to merge 2 commits into
Conversation
SparkQA
commented
Jul 19, 2018
Test build #93296 has finished for PR 21819 at commit
|
c1fc3ca to
7129c3fComparearunmahadevan
commented
Aug 10, 2018
@HeartSaVioR@HyukjinKwon@jose-torres@tdas would you mind taking a look? |
SparkQA
commented
Aug 10, 2018
Test build #94582 has finished for PR 21819 at commit
|
| * Write per-topic partition lag as json string | ||
| */ | ||
| def partitionLags(latestOffsets: Map[TopicPartition, Long], | ||
| processedOffsets: Map[TopicPartition, Long]): String = { |
There was a problem hiding this comment.
nit:
defpartitionLags(
latestOffsets: Map[TopicPartition, Long],
processedOffsets: Map[TopicPartition, Long]):String= {per https://github.com/databricks/scala-style-guide#spacing-and-indentation
Please feel free to address it with other comments later.
There was a problem hiding this comment.
addressed. would it be possible to add this to scala style checks ?
| */ | ||
| def partitionLags(latestOffsets: Map[TopicPartition, Long], | ||
| processedOffsets: Map[TopicPartition, Long]): String = { | ||
| val result = new HashMap[String, HashMap[Int, Long]]() |
There was a problem hiding this comment.
nit: HashMap.empty[String, HashMap[Int, Long]]
There was a problem hiding this comment.
Had followed the style in other parts of the class. Addressed and refactored the other places as well.
SparkQA
commented
Aug 13, 2018
Test build #94699 has finished for PR 21819 at commit
|
HyukjinKwon
commented
Aug 14, 2018
retest this please |
HyukjinKwon
commented
Aug 14, 2018
cc @koeninger as well |
SparkQA
commented
Aug 14, 2018
Test build #94726 has finished for PR 21819 at commit
|
arunmahadevan
commented
Aug 16, 2018
@HyukjinKwon , can you take it forward? Appreciate your effort and thanks in advance. |
HyukjinKwon
commented
Aug 17, 2018
Let me leave this in few days in case someone has more comments on this. |
HyukjinKwon
commented
Aug 18, 2018
Merged to master. |
## What changes were proposed in this pull request? Revert SPARK-24863 (#21819) and SPARK-24748 (#21721) as per discussion in #21721. We will revisit them when the data source v2 APIs are out. ## How was this patch tested? Jenkins Closes#22334 from zsxwing/revert-SPARK-24863-SPARK-24748. Authored-by: Shixiong Zhu <zsxwing@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
What changes were proposed in this pull request?
This builds on top of SPARK-24748 to report 'offset lag' as a custom metrics for Kafka structured streaming source.
This lag is the difference between the latest offsets in Kafka the time the metrics is reported (just after a micro-batch completes) and the latest offset Spark has processed. It can be 0 (or close to 0) if spark keeps up with the rate at which messages are ingested into Kafka topics in steady state. This measures how far behind the spark source has fallen behind (per partition) and can aid in tuning the application.
How was this patch tested?
Existing and new unit tests
Please review http://spark.apache.org/contributing.html before opening a pull request.