ZEPPELIN-142 add SparkSqlCompleter

[nevillelyh]

No description provided.

[felixcheung]

isn't this going to create a new instance of SparkSqlCompleter each time code completion is invoked on a text block?

[nevillelyh]

Moved SparkSqlCompleter construction to open()

[nevillelyh]

Looks like this failed for spark-sql < 1.3. @felixcheung What's the recommended way of dealing with multiple Spark API versions?

[nberserk]

when this completion invoked? I wonder.

[nevillelyh]

@nberserk you invoke it with ctrl-. in a %sql note.

[Leemoonsoo]

Thanks for the great contribution.

According to message from CI server

[ERROR] /home/travis/build/apache/incubator-zeppelin/spark/src/main/scala/org/apache/zeppelin/spark/utils/SparkSqlCompleter.scala:8: error: object StructType is not a member of package org.apache.spark.sql.types
[INFO] import org.apache.spark.sql.types.{StructType, StructField}
[INFO] ^
[ERROR] /home/travis/build/apache/incubator-zeppelin/spark/src/main/scala/org/apache/zeppelin/spark/utils/SparkSqlCompleter.scala:15: error: value implicits is not a member of org.apache.spark.sql.SQLContext
[INFO] import sqlContext.implicits._
[INFO] ^
[ERROR] /home/travis/build/apache/incubator-zeppelin/spark/src/main/scala/org/apache/zeppelin/spark/utils/SparkSqlCompleter.scala:20: error: value tables is not a member of org.apache.spark.sql.SQLContext
[INFO] override def get(): Seq[String] = sqlContext.tables().select('tableName).collect().map(_.getString(0))
[INFO] ^
[ERROR] /home/travis/build/apache/incubator-zeppelin/spark/src/main/scala/org/apache/zeppelin/spark/utils/SparkSqlCompleter.scala:40: error: not found: type StructField
[INFO] def fieldNames(prefix: Seq[String], fields: Array[StructField]): Seq[String] = {
[INFO] ^
[ERROR] /home/travis/build/apache/incubator-zeppelin/spark/src/main/scala/org/apache/zeppelin/spark/utils/SparkSqlCompleter.scala:41: error: type mismatch;
[ERROR] found : fields.type (with underlying type Array[<error>])
[INFO] required: ?{def flatMap: ?}
[INFO] Note that implicit conversions are not applicable because they are ambiguous:
[INFO] both method booleanArrayOps in object Predef of type (xs: Array[Boolean])scala.collection.mutable.ArrayOps[Boolean]
[INFO] and method byteArrayOps in object Predef of type (xs: Array[Byte])scala.collection.mutable.ArrayOps[Byte]
[INFO] are possible conversion functions from fields.type to ?{def flatMap: ?}
[INFO] fields.flatMap { f =>

It looks like there're some compiling error. @nevillelyh Do you have any idea about the error?

[nevillelyh]

Yeah I'm using API introduces in Spark SQL 1.3. What's the best way to make this work? Separate src/man/java directories for each spark version + Maven profiles?

[Leemoonsoo]

Can we go with reflection?

for example, api of getting progress in spark vary between versions. And it is handled using reflections, like https://github.com/apache/incubator-zeppelin/blob/branch-0.5/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java#L653

What do you think?

[nevillelyh]

It might be hard and unreadable since I rely on types StructField and StructType being available in the compiler.

[Leemoonsoo]

Hmm. I'm worried about separating src directory per spark version will make things too complicated.
And in the future, Zeppelin might have ability to work with user provided spark dependency. (like spark has package for user provided hadoop). It'll be difficult to support such case when codes are separated per spark version.

What do you think? Which one do you think better?

[nevillelyh]

Yeah make sense. I'll try to refactor it and remove the pattern matching parts.

[corneadoug]

@nevillelyh
I'm going around old PRs, is there any news on this issue?
Or do you want to close it?

[nevillelyh]

Sorry haven't worked on this in a while and probably won't have time to. Closing it for now.