Uh oh!
There was an error while loading. Please reload this page.
[SPARK-5210] Support group event log when app is long-running - #9246
[SPARK-5210] Support group event log when app is long-running#9246XuTingjun wants to merge 2 commits into
Conversation
SparkQA
commented
Oct 23, 2015
Test build #44216 has finished for PR 9246 at commit
|
SparkQA
commented
Oct 23, 2015
Test build #44219 has finished for PR 9246 at commit
|
markhamstra
commented
Nov 1, 2015
@XuTingjun you speak of an "acceptable resolution", but you haven't adequately described the problem you are trying to resolve. Yes, the event log can get long, but I'm not seeing why that is inherently a problem or why you can't "acceptably resolve" your problem by post-processing the event log while leaving that log as a single, unified stream of events. |
andrewor14
commented
Dec 15, 2015
@XuTingjun I think this is something good to fix. I've noticed that uncompressed event logs can amount up to 15GB for a 5 minute application. However, I think a lot of functionality is already implemented so we should reuse existing code where possible. In particular, have you looked at |
andrewor14
commented
Dec 15, 2015
By the way, since this patch has been opened many months ago it's now mostly stale. If you plan to work on this, would you mind closing this patch and re-opening one that uses the |
satlal
commented
May 13, 2016
@XuTingjun revisiting this thread. Since this patch seems to be abandoned, were you able to work around the large log files issue for long running streaming jobs? |
eladamitpxi
commented
Jun 26, 2016
+1 large / infinitely growing event log files are quite a problem for long running streaming jobs |
For long-running Spark applications (e.g. running for days / weeks), the Spark event log may grow to be very large.
I think group event log by job is an acceptable resolution.
StageSubmitted/ StageCompleted/ TaskResubmit/ TaskStart/TaskEnd/TaskGettingResult/ JobStart/JobEndevents into meta file, and put other events into part file. The event log shows like below:2.To HistoryServer, every part file will be treated as an application, and it will replay meta file after replay part file. Below is the display of group app on HistoryServer web:
