[ZEPPELIN-1293] Re-create Livy session if it's lost by mispecto · Pull Request #1447 · apache/zeppelin

mispecto · 2016-09-21T17:50:16Z

What is this PR for?

This patch restarts Livy session when it's lost.

What type of PR is it?

Improvement/Bug Fix

Todos

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-1293

How should this be tested?

Configure livy.server.session.timeout parameter in Livy
Start new Livy session through Zeppelin
Wait for Livy session to expire
See that new Livy session is started when running something in Zeppelin

Questions:

Does the licenses files need update? - No
Is there breaking changes for older versions? - No
Does this needs documentation? - No

zjffdu · 2016-09-23T03:28:26Z

@spektom Thanks for the contribution, I am working on a PR about integration test of livy interpreter, could you mind to wait for that PR and then add integration test to that ?

mispecto · 2016-09-23T04:37:31Z

Of course, it would be great.
On Fri, Sep 23, 2016 at 06:28 Jeff Zhang notifications@github.com wrote:

@spektom https://github.com/spektom Thanks for the contribution, I am
working on a PR about integration test of livy interpreter, could you mind
to wait for that PR and then add integration test to that ?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1447 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAJpO0VBwVJ9NayJbPq3h9j1bRji_4cuks5qs0dkgaJpZM4KDF0H
.

gss2002 · 2016-10-14T02:38:39Z

@spektom and @zjffdu is it possible to add a null check also. I've been doing some debugging over the past few days and certain situations can cause nulls to be returned and in theory if a null is being returned the session is dead.

So instead of this:
if (json.matches("^(")?Session ('[0-9]' )?not found(.?"?)$")) {
throw new LivyNoSessionException();

This:
boolean clearSession = false;
if (json != null) {
if (json.matches("^(")?Session ('[0-9]' )?not found(.?"?)$")) {
clearSession = true;
}
} else {
clearSession = true;
}
if (clearSession) {
throw new LivyNoSessionException();
}

gss2002 · 2016-10-14T03:54:44Z

Also I have some concerns on using Exceptions to as goto's as there is definite performance issues that occur by using exceptions... http://javarevisited.blogspot.com/2013/03/0-exception-handling-best-practices-in-Java-Programming.html. Specifically since code inside of the try catch cannot be optimized.

mispecto · 2016-10-14T06:26:09Z

@gss2002 you shouldn't be concerned about performance as this code only runs when command is executed.

gss2002 · 2016-10-14T16:14:22Z

@spektom this fix is good. Did some extensive load testing with it this AM and it solves the session expiration issues. Thanks for the contribution

felixcheung · 2016-10-14T19:41:09Z

+ throw new LivyException("Error executing command in Livy", e);
+ }
+ if (json == null || json.matches("^(\")?Session (\'[0-9]\' )?not found(.?\"?)$")) {
+ throw new LivyNoSessionException();


Could we add some logging when this happens?

purechoc

now, "Session not found" pattern matched,
session id between 1 and 10.

purechoc · 2016-10-19T07:33:16Z

+ } catch (Exception e) {
+ throw new LivyException("Error executing command in Livy", e);
+ }
+ if (json == null || json.matches("^(\")?Session (\'[0-9]\' )?not found(.?\"?)$")) {


need to be change like this, isn't it?
json.matches("^(")?Session ('[0-9]' )?not found(.?"?)$")

gss2002 · 2016-10-19T18:26:17Z

@spektom / @zjffdu @purechoc there is definitely an additional condition. Not sure if it's because the ConcurrentHashMaps are not being used correctly. But the exception doesn't get caught completely or correctly at times with the fix proposed here..

ERROR [2016-10-19 14:19:57,638]({pool-2-thread-11} LivyHelper.java[executeHTTP]:378) - Error with 404 StatusCode: "Session '9' not found."
ERROR [2016-10-19 14:19:57,638]({pool-2-thread-11} LivyHelper.java[interpretInput]:229) - error in interpretInput
org.apache.zeppelin.livy.LivyHelper$LivyNoSessionException: Session not found, Livy server would have restarted, or lost session.
at org.apache.zeppelin.livy.LivyHelper.executeCommand(LivyHelper.java:312)
at org.apache.zeppelin.livy.LivyHelper.interpret(LivyHelper.java:241)
at org.apache.zeppelin.livy.LivyHelper.interpretInput(LivyHelper.java:189)
at org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:106)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:390)
at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
INFO [2016-10-19 14:19:57,639]({pool-2-thread-11} SchedulerFactory.java[jobFinished]:137) - Job remoteInterpretJob_1476901197622 finished by scheduler org.apache.zeppelin.livy.LivySparkInterpreter37814848
INFO [2016-10-19 14:19:57,819]({pool-2-thread-34} SchedulerFactory.java[jobStarted]:131) - Job remoteInterpretJob_1476901197819 started by scheduler org.apache.zeppelin.livy.LivySparkInterpreter37814848
ERROR [2016-10-19 14:19:57,835]({pool-2-thread-34} LivyHelper.java[executeHTTP]:378) - Error with 404 StatusCode: "Session '9' not found."
ERROR [2016-10-19 14:19:57,835]({pool-2-thread-34} LivyHelper.java[interpretInput]:229) - error in interpretInput
org.apache.zeppelin.livy.LivyHelper$LivyNoSessionException: Session not found, Livy server would have restarted, or lost session.
at org.apache.zeppelin.livy.LivyHelper.executeCommand(LivyHelper.java:312)
at org.apache.zeppelin.livy.LivyHelper.interpret(LivyHelper.java:241)
at org.apache.zeppelin.livy.LivyHelper.interpretInput(LivyHelper.java:189)
at org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:106)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:390)
at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
INFO [2016-10-19 14:19:57,836]({pool-2-thread-34} SchedulerFactory.java[jobFinished]:137) - Job remoteInterpretJob_1476901197819 finished by scheduler org.apache.zeppelin.livy.LivySparkInterpreter37814848

mispecto · 2016-10-19T18:37:22Z

@gss2002 are you using latest code from the branch? I can't find org.apache.zeppelin.livy.LivyHelper.interpretInput(LivyHelper.java:189) invocation at org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:106) as it's shown in your stack trace.

gss2002 · 2016-10-20T03:00:40Z

@spektom I think what happens here is this code fires.. which has nothing to do with the fix here..

in LivySparkInterpreter:
return livyHelper.interpretInput(line, interpreterContext, userSessionMap, out,
sessionId2AppIdMap.get(sessionId), sessionId2WebUIMap.get(sessionId), displayAppInfo);

That gets called before the NoSessionException occurs.. And then in
LivyHelper --> public InterpreterResult interpretInput grabs the exception and handles it. I guess the question is can we do a rootcause on this and rethrow?

} catch (Exception e) {
LOGGER.error("error in interpretInput", e);
return new InterpreterResult(Code.ERROR, e.getMessage());
}

gss2002 · 2016-10-20T03:16:18Z

@spektom just tested against my build by catching the exception and rethrowing.. It definitely solves the issue.

} catch (LivyNoSessionException e) {
throw e;
} catch (Exception e) {
LOGGER.error("error in interpretInput", e);
return new InterpreterResult(Code.ERROR, e.getMessage());
}

}

public InterpreterResult interpret(String stringLines,

The code base I'm using is here with you patch and a few of @zjffdu patches and one of mine for NestedRuntimeException for 404's with KerberosTemplate: https://github.com/gss2002/zeppelin/blob/GSS_PROD_BUILD/livy/src/main/java/org/apache/zeppelin/livy/LivyHelper.java

zjffdu · 2016-10-31T09:18:20Z

Sorry for late response. @spektom Could you add some additional message of session recreation to be displayed in frontend. Because if session is recreated, we may need to rerun all the paragraphs, without knowing session recreationusers may be confused for errors like some variables are not defined.

purechoc · 2016-11-03T07:42:24Z

LGTM. it's working nicely in my environment. (zeppelin 0.7 + livy 2.1)

zjffdu · 2016-12-23T07:26:40Z

@spektom Do you want to rebase it and continue work on it ?

zjffdu · 2017-01-06T11:44:00Z

@spektom Sorry, I didn't get your reply, I create another PR for it #1861

close#83close#86close#125close#133close#139close#146close#193close#203close#246close#262close#264close#273close#291close#299close#320close#347close#389close#413close#423close#543close#560close#658close#670close#728close#765close#777close#782close#783close#812close#822close#841close#843close#878close#884close#918close#989close#1076close#1135close#1187close#1231close#1304close#1316close#1361close#1385close#1390close#1414close#1422close#1425close#1447close#1458close#1466close#1485close#1492close#1495close#1497close#1536close#1545close#1561close#1577close#1600close#1603close#1678close#1695close#1739close#1748close#1765close#1767close#1776close#1783close#1799

mispecto force-pushed the ZEPPELIN-1293 branch 2 times, most recently from 12ee62d to 5fdb6dfCompare October 14, 2016 06:26

felixcheung reviewed Oct 14, 2016
View reviewed changes

purechoc reviewed Oct 19, 2016
View reviewed changes

mispecto force-pushed the ZEPPELIN-1293 branch from 5fdb6df to d7f1da5Compare October 19, 2016 07:42

mispecto force-pushed the ZEPPELIN-1293 branch from d7f1da5 to 39c5561Compare October 20, 2016 03:07

[ZEPPELIN-1293] Re-create Livy session if it's lost
522e62d

mispecto force-pushed the ZEPPELIN-1293 branch from 39c5561 to 522e62dCompare November 1, 2016 11:27

asfgit closed this in c38a0a0May 9, 2018

Uh oh!

Conversation

mispecto commented Sep 21, 2016

What is this PR for?

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

Questions:

Uh oh!

zjffdu commented Sep 23, 2016

Uh oh!

mispecto commented Sep 23, 2016

Uh oh!

gss2002 commented Oct 14, 2016

Uh oh!

gss2002 commented Oct 14, 2016

Uh oh!

mispecto commented Oct 14, 2016

Uh oh!

gss2002 commented Oct 14, 2016

Uh oh!

felixcheungOct 14, 2016

Choose a reason for hiding this comment

Uh oh!

purechoc left a comment

Choose a reason for hiding this comment

Uh oh!

purechocOct 19, 2016

Choose a reason for hiding this comment

Uh oh!

gss2002 commented Oct 19, 2016

Uh oh!

mispecto commented Oct 19, 2016

Uh oh!

gss2002 commented Oct 20, 2016

Uh oh!

gss2002 commented Oct 20, 2016

Uh oh!

zjffdu commented Oct 31, 2016

Uh oh!

purechoc commented Nov 3, 2016

Uh oh!

zjffdu commented Dec 23, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zjffdu commented Jan 6, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

zjffdu commented Dec 23, 2016•
edited
Loading