Hbase Interpreter

[vrajat]

Support for Hbase Shell. All the commands documented here https://wiki.apache.org/hadoop/Hbase/Shell is supported.

Requirements:

• Hbase Shell should be installed on the same machine. To be more specific, the following dir. should be available: https://github.com/apache/hbase/tree/master/hbase-shell/src/main/ruby
• Hbase Shell should be able to connect to the Hbase cluster from terminal. This makes sure that the client is configured properly.

The interpreter takes 3 config parameters:

1. hbase.home: Root dir. where hbase is installed. Default is /usr/lib/hbase/
2. hbase.ruby.sources: Dir where shell ruby code is installed. Path is relative to hbase.home. Default: lib/ruby
3. hbase.irb.load: (Testing only) Default is true. Whether to load irb in the interpreter.

[Leemoonsoo]

Hi @vrajat. Thanks for contribution.

Could you make hbase related dependency configurable through maven property?
And could you check CI build log? https://travis-ci.org/apache/incubator-zeppelin/builds/60494237

[vrajat]

Sure. Will figure out how to make the dependency configurable from other modules. Will take a look at tests too.

[vrajat]

I'll need your advice on how to proceed further. I tested hbase-1.0 profile. Looks like hbase project started to distribute separate jars after 0.94.X version. So the interpreter does not compile with the default versions.
I picked up the dependencies from hbase-shell (1.0) module

[vrajat]

Hello Lee,
Any advice on how to solve the version mismatch ? The default version of hbase is really old.

[Leemoonsoo]

Understood the problem.

Currently hbase.version is written in the pom.xml and CI uses it. But hbase interpreter should built with 1.0 version. Right??

I think one possible solution is hbase interpreter uses it's own property name for configuring hbase dependency and set default value of the property 1.0 version.

[vrajat]

OK. I'll add a new property.
On Wed, 20 May 2015 at 16:36 Lee moon soo notifications@github.com wrote:

Understood the problem.

Currently hbase.version is written in the pom.xml and CI uses it. But
hbase interpreter should built with 1.0 version. Right??

I think one possible solution is hbase interpreter uses it's own property
name for configuring hbase dependency and set default value of the property
1.0 version.

—
Reply to this email directly or view it on GitHub
#55 (comment)
.

[vrajat]

Fascinating. I am unable to reproduce the build error on my box. The error is:
Non-resolvable parent POM: Failure to find org.apache.zeppelin:zeppelin:pom:0.5.0-SNAPSHOT

[vrajat]

Oh wow! Tests passed. Given the difficulty I had in getting the PR to compile, one would think I wrote code and created a PR. Rest assured that I have tested it out with a live HBase cluster :)

[Leemoonsoo]

@vrajat Looks great

[vrajat]

@Leemoonsoo Any blockers for merging this PR ? Has been open for quite some time.

[jongyoul]

Please fix the indentations for xml. Zeppelin uses two spaces.

[jongyoul]

@vrajat I just have two questions. First, why do you use a ruby script? Although I'm not familiar with hbase, I've known that hbase-client library includes the way how to connect hbase-server directly. Please describe why you use a ruby not using hbase-client directly. And second, is there any way to run this interpreter without using local hbase? Basically, Zeppelin wants to use service or servers without installing another packages when we use it.

[vrajat]

@jongyoul This is an interpreter for hbase shell. Hbase shell is written ruby. It provides a lot of utility function using the hbase client. Please check the links in my description.

This interpreter works with hbase clusters. At Qubole, we connect to hbase clusters only. The expectation is to setup the hbase configuration on the same machine and hbase shell where zeppelin is running. Since we run zeppelin on HBase master, this is a no-op. This is the preferred way to install it. Run zeppelin on hbase master. We've had very good success with this configuration. Though running zeppelin on another machine also works.

[martin-g]

The version should be updated to 0.6.0-... before merge.

[jongyoul]

@vrajat Okay, Do you mean it can make user use all of hbase shell command on Zeppelin? That makes sense. And in my second question, Is there a way to download the files related by hbase-shell? I think most of users doesn't want to install full hbase with their mac for using hbase feature in Zeppelin.

[martin-g]

Any reason why these member fields are not private?
As far as I can see they are not used in the tests.

[martin-g]

You may want to release the resources here with #terminate(): https://github.com/jruby/jruby/blob/2187665d26d227d8f0b0882c762c7b7485efe642/core/src/main/java/org/jruby/embed/ScriptingContainer.java#L1853

[vrajat]

@jongyoul You can install hbase jars and binaries using brew. Then edit configuration to point to a running hbase cluster.

@martin-g I'll address all your comments. Thanks for taking the time to review the code.

[jongyoul]

@vrajat Yes, it's pretty easy to install through brew, but someone want us to help run hbase interpreter in Zeppelin. So I think we try to avoid install additional packages except Zeppelin. That's my opinion.

[vrajat]

I am reluctant to sign up for installing hbase shell as part of Zeppelin.
There are well documented instructions for all operating systems. I will be
duplicating good work done by others.
From experience in using Zeppelin with HBase for the past two months, I
feel this is unnecessary.
Also, do you follow a similar process for hive or Cassandra or similar
integrations ? I would like to see examples to see how it's done.
On Thu, 6 Aug 2015 at 18:13 Jongyoul Lee notifications@github.com wrote:

@vrajathttps://github.com/vrajat Yes, it's pretty easy to install
through brew, but someone want us to help run hbase interpreter in
Zeppelin. So I think we try to avoid install additional packages except
Zeppelin. That's my opinion.

—
Reply to this email directly or view it on GitHub
#55 (comment)
.

[jongyoul]

I don't know the exact meaning of 'similar process' but in case of Hive, Cassandra, Flink and Spark, Its interpreter doesn't need extra packages for using it within Zeppelin. Users don't care anything except installing Zeppelin when they use theses interpreters. What do you think of it? If users can also use Hbase interpreter without installing Hbase, It gives a wonderful experience for Zeppelin users. Regardless of this issue, I think your contribution looks great and will enrich Zeppelin community.

[vrajat]

I agree that will make for a better user experience. Just that HBase is a beast :) Let me get this out. If there is enough demand, then I can work on it.

[Leemoonsoo]

Apologies for letting this PR keep opened.

Profiles and Properties for specific interpreter is now moved to interpreter's pom.xml.
It'll be helpful rebase this branch and move Profiles/Properties into hbase/pom.xml.

[vrajat]

I have addressed most review comments. Does travis run unit tests ?

[bzz]

Yes it does (and looks like CI is failing), but somehow I do not see Travis feedback for this PR any more.
Can you try closing and re-opening the PR? This should trigger CI again

[vrajat]

Hmm. Travis failed because it lost connection to maven central.

INFO: Retrying request to {s}->https://repo.maven.apache.org:443
[WARNING] Could not transfer metadata org.scala-tools:maven-scala-plugin/maven-metadata.xml from/to central (https://repo.maven.apache.org/maven2): Connection reset

[vrajat]

I'll start a new PR.

[vrajat]

Please review #278