[ZEPPELIN-1258] Add Spark packages support to Livy interpreter

[mfelgamal]

What is this PR for?

Adding extra libraries to livy interpreter which isn't exist by default.

What type of PR is it?

[ Improvement ]

Todos

• [Test case ] - Task

What is the Jira issue?

• [ZEPPELIN-1258]

How should this be tested?

• Create new livy interpreter or modify the default.
• Set livy.spark.jars.packages to list of maven coordinates of jars. The format for the coordinates should be groupId:artifactId:version.

Screenshots (if appropriate)

Questions:

• Does the licenses files need update? no
• Is there breaking changes for older versions? no
• Does this needs documentation? yes

[zjffdu]

I would not recommend user to do that to add extra library. Because most of time zeppelin user don't know the cluster details like which/where extra library are located in the cluster machines. I would suggest livy to support dynamic library loading as native spark interpreter.

[jimdowling]

@zjffdu - Livy doesn't support dynamic library loading right now. So, I don't think there's a ZeppelinContext available in Livy. Would this be acceptable until dynamic library loading is added to Livy and a Zeppelin Context becomes available? As of now, Livy is not that much use if you cannot add libraries to your Spark app.

[zjffdu]

@jimdowling Right, there's no ZeppelinContext in livy right now. Would allowing user to specify jars/packages when creating livy session more general ? (like --jars/--packages in spark-submit)

[jimdowling]

Do you mean by modifying the UI on notebook creation or using a param to selecting the interpreter like
'%livy --jars [path]

[zjffdu]

No, I mean adding interpreter property to allow user to specify jars/packages.

[mfelgamal]

I think that the livy itself doesn't support add (--jars/--packages property in spark-submit)

[zjffdu]

You can use --conf spark.jars=<jars> and --conf spark.jars.packages=<packages> instead

[mfelgamal]

@zjffdu I added spark.jars.packages property in the recent commit, and it works correctly.

[mfelgamal]

@zjffdu do you have any further comments ?

[felixcheung]

this wouldn't be the "Default" right?

[mfelgamal]

True, it's just example.

[felixcheung]

Let's change it to Example?

[mfelgamal]

@felixcheung done.

[felixcheung]

could you update the title of this PR and JIRA ZEPPELIN-1258, if we are changing how we would approach this?

[mfelgamal]

@felixcheung I suggests this title "Adding packages to livy interpreter", Is it Ok ?

[felixcheung]

how about "Add Spark packages support to Livy interpreter"?

[mfelgamal]

@felixcheung done.

[mfelgamal]

@felixcheung do you have any further comments ?

[felixcheung]

There is a test failure?

[mfelgamal]

@felixcheung : This PR does not change the code. It is just a change in the configurations. So, it should not affect the tests

[felixcheung]

That's a fair point.

I'll merge if there is no more discussion.