Add an Elasticsearch interpreter

[bbonnin]

Elasticsearch Interpreter

Interpreter for querying ElasticSearch .
Supported requests are "get document by id" , "search documents" , "delete by id" , "count documents" and "index / update a document".
Supported versions of Elasticsearch : >= 2.1

[felixcheung]

maybe filter the returned list base on what's in s? like if the user is typing s it should return search?

[felixcheung]

it looks like you have taken out completion? I think it's still worthwhile to have some of it..

[felixcheung]

very cool - I don't have es to test it but only a few comments.

[bzz]

Would you be willing to remove that information from the comments, please?

It is not strictly documented yet, but Zeppelin so far, as many other ASF projects (Hadoop, Zookeeper, Avro, etc), do not encourage use of @author tags.

We definitely want and keep contributors credits, but we use git, JIRA and mailing list history, so nothing will be lost.

[bbonnin]

I have removed the author tag.

[bzz]

👍 for docs and tests!

I might be wrong, but because all interpreters right now are part of the Zeppelin releases, in my understanding we also need to document the licence of the dependencies (and it's transitive dependencies) in the zeppelin-distribution/src/bin_license/LICENSE

@bbonnin would you care of doing it in this PR please? Here is how the list of dependencies looks like now

 +- org.elasticsearch:elasticsearch:jar:2.1.0:compile
| +- org.apache.lucene:lucene-core:jar:5.3.1:compile
| +- org.apache.lucene:lucene-backward-codecs:jar:5.3.1:compile
| +- org.apache.lucene:lucene-analyzers-common:jar:5.3.1:compile
| +- org.apache.lucene:lucene-queries:jar:5.3.1:compile
| +- org.apache.lucene:lucene-memory:jar:5.3.1:compile
| +- org.apache.lucene:lucene-highlighter:jar:5.3.1:compile
| +- org.apache.lucene:lucene-queryparser:jar:5.3.1:compile
| | \- org.apache.lucene:lucene-sandbox:jar:5.3.1:compile
| +- org.apache.lucene:lucene-suggest:jar:5.3.1:compile
| | \- org.apache.lucene:lucene-misc:jar:5.3.1:compile
| +- org.apache.lucene:lucene-join:jar:5.3.1:compile
| | \- org.apache.lucene:lucene-grouping:jar:5.3.1:compile
| +- org.apache.lucene:lucene-spatial:jar:5.3.1:compile
| | +- org.apache.lucene:lucene-spatial3d:jar:5.3.1:compile
| | \- com.spatial4j:spatial4j:jar:0.4.1:compile
| +- com.carrotsearch:hppc:jar:0.7.1:compile
| +- joda-time:joda-time:jar:2.8.2:compile
| +- org.joda:joda-convert:jar:1.2:compile
| +- com.fasterxml.jackson.core:jackson-core:jar:2.6.2:compile
| +- com.fasterxml.jackson.dataformat:jackson-dataformat-smile:jar:2.6.2:compile
| +- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.6.2:compile
| | \- org.yaml:snakeyaml:jar:1.15:compile
| +- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.6.2:compile
| +- io.netty:netty:jar:3.10.5.Final:compile
| +- com.ning:compress-lzf:jar:1.0.2:compile
| +- com.tdunning:t-digest:jar:3.0:compile
| +- org.hdrhistogram:HdrHistogram:jar:2.1.6:compile
| +- commons-cli:commons-cli:jar:1.3.1:compile
| \- com.twitter:jsr166e:jar:1.1.0:compile
+- com.google.guava:guava:jar:18.0:compile
+- org.slf4j:slf4j-api:jar:1.7.10:compile

[jeffsteinmetz]

I love the idea of making elasticsearch a first class cititen to Zeppelin.

I was curious however, as you build out the query language, eventually it will get to the point that you'll want to add more features to keep in parity with elasticsearch's existing "extensive" query dsl. I've built some scala libraries that wrap ES for usage in API, spark utilities and other business logic. Eventually the wrapper even starts to try to mimic some of the ES query language.
As a long time user of Elasticsearch, elasticsearch-hadoop (spark integration), beta testing elasticsearch-hadoop, using ES from spark, etc, my one comment is the construction of another DSL (domain specific language) via an interpreter.

Curious if there has been any considerations made as to why you might not just pass through existing ES json using the existing query language?
Imagine if we abstracted another query language on top of Spark SQL %sql that wasn't SQL?

The ease of use is certainly welcome though. So I was curious about the long term plans for the DSL in the interpreter.

Using the native spark integration provided by Elasticsearch Hadoop after an
import org.elasticsearch.spark._
you have 2 options to load the results into an RDD, either the original json:

val myRDD: RDD[(String, String)] = sc.esJsonRDD("someindex/doctype", query)

or a Map

val myRDD: Map[(String, AnyRef)] = sc.esRDD("someindex/doctype", query)

Elasticsearch Hadoop (spark) can also create a dataframe using the elasticsearch spark sql context.

Curious if there has been any thought around how the elasticsearch interpreter / pipeline could be a bit more spark, ES native and ES hadoop native friendly.

As a side note, I import this library, https://github.com/sksamuel/elastic4s when I want to simplify my Elastic Search query experience ( this DSL misses out on a few functional builder patterns and recent features, but does cover about 80% of most use cases):

[felixcheung]

@jeffsteinmetz has a good point about ES Query DSL. I'm not sure we should require Spark for ES though. Thought?

[jeffsteinmetz]

Things also get interesting when you return Json with nested objects and try to treat them like a flat relational database response (shown in the example pie charts and sql like table response examples in this PR).

I believe the idea behind this PR is to return the raw Elasticsearch Json, and potentially graph it or put it into tables? Admittedly, I've only passed over this PR with a quick scan of your source. Happy to dig into it more over time.

ES-hadoop (native spark) has been working toward getting a variety of json use cases working with dataframes.

It is a problem being worked on in es-hadoop, that's fairly complex. I.e. how would it handle this?

{
"foo" : [5,6], "nested": { "bar" : [{"date":"2015-01-01", "scores":[1,2]}, {"date":"2015-01-01", "scores":[3,4]}], "what": "now" }
}

[jeffsteinmetz]

Regarding the DSL, I see you can pass any Json Elastic query to the elasticsearch interpreters /search command? That may address my question about not wanting to re-invent the ES query DSL.

How do people feel about letting users delete elasticsearch indexed documents via a notebook?
I see the cassandra interpreter allows DELETE, so not a concern conceptually for Zeppelin?

[bbonnin]

Thanks to all for your comments!

About the DSL, my goal was not reinvent the wheel, but just provide a simple way to access Elasticsearch.
I wanted to be closer to the Java API that provides classes such as GetRequest, CountRequest, SearchRequest, DeleteRequest, ...
So, I have just reused these concepts.
The query you can provide for some commands (search, for example) is exactly what you can provide to Elasticsearch, nothing more, nothing less.

The main problem is how to handle the json documents returned by Elasticsearch.
I have decided to flatten the json document.
If you have:

{
"field1": "value1",
"field2": {
"field3":"value3",
"field4":[1,3]
}
}

You will get a something like this:

field1	field2.field3	field2.field4[0]	field2.field4[1]
value1	value3	1	2

I will fix the code in this way, are you ok ?

Bruno

[jeffsteinmetz]

Nice!. That makes sense regarding field flattening. thank you for the follow up.

Still curious about how the community feels about Zeppelin being a CRUD tool (mainly allowing Deletes).
I would feel better if my users couldn't delete things so easily by default and it had to be enabled by an admin. Call me paranoid.

[jeffsteinmetz]

One more thing I thought of as a use case - how would it handle Shield security?

[bbonnin]

@bzz For the licence file, is there an easy way to complete this file ? Somethig like "mvn license:put-the third-party-licences-in-a-file" ?

[bbonnin]

For shield, we have to deploy the interpreter with the shield jar. For the configuration, just add the property "shield.user" in the interpreter configuration panel.

[felixcheung]

@jeffsteinmetz re: CRUD - in general we seem to be pretty open about it. For example, in the Hive interpreter a user could drop a table. Hive would have access control. ES has access control too, couldn't we rely on that? https://www.elastic.co/guide/en/found/current/access-control.html#_managing_access_control?q=access cob

@bbonnin could we add your approach to flatten JSON and shield information to the doc?

[bzz]

@bzz For the licence file, is there an easy way to complete this file ? Somethig like "mvn license:put-the third-party-licences-in-a-file" ?

@bbonnin sorry but I'm not aware of such automated solution.

There might be, but at least for my changesets I did that manually, reviewing each licence not to be apache-un-compatible

[felixcheung]

otherwise looks good, thanks!

[bbonnin]

In the last commit, I have added the possibility to search by just providingthe content of a quey_string. I think it's simpler and based on Lucene query syntax.
For example:

search /logs (status:404 AND request.method:POST)

[bzz]

@bbonnin Great! Merging latest master in should fix the CI.
Also, can you please make sure that there are no duplicates introduced in the LICENCE file.

[bbonnin]

@bzz I have checked LICENCE file, and I have seen that no duplicates, except for some components that have differents versions (for example, there are 2 versions of joda-time, I have added one).
What is the right way in such case, as they are not direct dependencies of my pom.xml, but dependencies of dependencies (so I would not like to change these components) ?

[bzz]

This is a good question boiling down how the build system resolve such conflicts at the build time, but that's our of scope of this PR I suppose.

As for LICENCE issue - I think we need to list transitive dependencies too, but if library does not change licence in a new version - it should be fine to have one line per dependency group\artifact.

[bbonnin]

For LICENCE issue, I have removed the library references that already exist with the same grouId/artifactId, if they have a different version and the same licence.

[bzz]

Looks great to me, will merge if there are no more discussions.
Thank you for awesome contribution with test & docs \w screenshots and LICENCEs!