HBase Shell Interpreter by vrajat · Pull Request #278 · apache/zeppelin

vrajat · 2015-09-03T18:15:39Z

Support for Hbase Shell. All the commands documented here https://wiki.apache.org/hadoop/Hbase/Shell is supported.

Requirements:

Hbase Shell should be installed on the same machine. To be more specific, the following dir. should be available: https://github.com/apache/hbase/tree/master/hbase-shell/src/main/ruby
Hbase Shell should be able to connect to the Hbase cluster from terminal. This makes sure that the client is configured properly.
The interpreter takes 3 config parameters:

hbase.home: Root dir. where hbase is installed. Default is /usr/lib/hbase/
hbase.ruby.sources: Dir where shell ruby code is installed. Path is relative to hbase.home. Default: lib/ruby
hbase.irb.load: (Testing only) Default is true. Whether to load irb in the interpreter.

vrajat · 2015-09-03T23:30:44Z

Unit tests passed.

felixcheung · 2015-09-06T16:49:21Z

if hbase.home hbase.ruby.sources are required for this interpreter would it make sense to have explicit checks for the hbase shell script file here?
That would make the error (in the case it is not presence) more explicit and more clear.

I already have a check. Refer to line 80.

To clarify, I'd suggest adding a check in code, not just logging the value - generally that is done in interpret() method instead of as one could return the error response to display to the user there.

Sorry. Completely missing the point here. I check if there is no hbase directory and I am throwing an InterpreterException. I am doing more than just logging the value. Can you point to examples ? Isnt it better to throw the exception in open method if the parameter is wrong ?

ok got it, sorry I think I was looking at line 84.
Could you use something like Paths or FilenameUtils instead of joining strings and requiring one has the path separator?

Done
On Mon, Jan 11, 2016 at 6:45 AM Felix Cheung notifications@github.com
wrote:
In hbase/src/main/java/org/apache/zeppelin/hbase/HbaseInterpreter.java
#278 (comment)
:
}
public HbaseInterpreter(Properties property) {
super(property);
}
@OverRide
public void open() {
String hbase_home = getProperty("hbase.home");
String ruby_src = getProperty("hbase.ruby.sources");
String abs_ruby_src = hbase_home + ruby_src;
logger.info("Home:" + hbase_home);
logger.info("Ruby Src:" + ruby_src);
Properties props = System.getProperties();
ok got it, sorry I think I was looking at line 84.
Could you use something like Paths
http://docs.oracle.com/javase/7/docs/api/java/nio/file/Paths.html or
FilenameUtils
https://commons.apache.org/proper/commons-io/javadocs/api-1.4/org/apache/commons/io/FilenameUtils.html
instead of joining strings and requiring one has the path separator?
—
Reply to this email directly or view it on GitHub
https://github.com/apache/incubator-zeppelin/pull/278/files#r49285311.

felixcheung · 2015-09-06T16:50:43Z

Could you elaborate on what this does?

hbase.irb.load: (Testing only) Default is true. Whether to load irb in the interpreter.

vrajat · 2015-09-07T20:08:23Z

@felixcheung I have addressed your comments. Please take a look

vrajat · 2015-09-10T20:22:23Z

The failure is a false positive. I made only cosmetic changes. Looks like it failed in downloading npm stuff.

felixcheung · 2015-09-12T00:42:11Z

this should rethrow the exception too? I assume the interpreter will not work if this fails

This block is used for testing only. I'll add a comment.

bzz · 2016-01-05T10:04:35Z

@vrajat do you plan to update this PR\add some docs for new interpreter?
Or shall we consider closing it as it diverged from master and CI fails, so no way to merge it in the current state.

vrajat · 2016-01-05T11:55:40Z

I'll put some effort into clean up the PR.
On Tue, 5 Jan 2016 at 15:34, Alexander notifications@github.com wrote:

@vrajat https://github.com/vrajat do you plan to update this PR\add
some docs for new interpreter?
Or shall we consider closing it as it diverged from master and CI fails,
so no way to merge it in the current state.
—
Reply to this email directly or view it on GitHub
#278 (comment)
.

vrajat · 2016-01-21T04:22:13Z

@felixcheung Done

felixcheung · 2016-01-21T05:00:38Z

thanks, trying to test this now.
btw, this doc style update has just been committed - it would be great if you could check and conform to that, if not we could come back to that shortly.

felixcheung · 2016-01-21T05:11:08Z

also, I realize when testing this, you should add to the list in ZeppelinConfiguration.java as documented in here.
Sorry I didn't catch that earlier.

felixcheung · 2016-01-21T06:02:33Z

@vrajat I am having problem getting this to work. It's likely this is my setup but I wonder if you have any idea to fix this

HBase bin release from Apache Mirror hbase/hbase-1.0.2/
HBase running, hbase shell working
Built distribution for Zeppelin
set JAVA_HOME, start Zeppelin, enable hbase, set hbase.home in property

When I ran this

%hbase
help

I got this

(NameError) undefined local variable or method `help' for main:Object

and in the log

 INFO [2016-01-21 05:40:07,314] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:129) - Job remoteInterpretJob_1453354807312 started by scheduler org.apache.zeppelin.hbase.HbaseInterpreter599771689
INFO [2016-01-21 05:40:07,494] ({pool-2-thread-2} HbaseInterpreter.java[open]:86) - Home:/opt/hbase-1.0.2
INFO [2016-01-21 05:40:07,494] ({pool-2-thread-2} HbaseInterpreter.java[open]:87) - Ruby Src:lib/ruby
INFO [2016-01-21 05:40:07,495] ({pool-2-thread-2} HbaseInterpreter.java[interpret]:110) - help
ERROR [2016-01-21 05:40:11,026] ({pool-2-thread-2} HbaseInterpreter.java[interpret]:117) - Can not run 'help'
org.jruby.embed.EvalFailedException: (NameError) undefined local variable or method `help' for main:Object
at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:136)
at org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1263)
at org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1256)
at org.apache.zeppelin.hbase.HbaseInterpreter.interpret(HbaseInterpreter.java:112)
at org.apache.zeppelin.interpreter.ClassloaderInterpreter.interpret(ClassloaderInterpreter.java:57)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:93)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:305)
at org.apache.zeppelin.scheduler.Job.run(Job.java:172)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.jruby.exceptions.RaiseException: (NameError) undefined local variable or method `help' for main:Object

vrajat · 2016-01-21T06:04:49Z

Surprising. It seems like the hbase ruby scripts arent loaded. I used the
interpreter two days back. I'll try to reproduce the problem.

On Thu, Jan 21, 2016 at 11:32 AM Felix Cheung notifications@github.com
wrote:

@vrajat https://github.com/vrajat I am having problem getting this to
work. It's like this is my setup but I wonder if you have any idea to fix
this
HBase bin release from Apache Mirror hbase/hbase-1.0.2/
HBase running, hbase shell working
Built distribution for Zeppelin
set JAVA_HOME, start Zeppelin, enable hbase, set hbase.home in
property
When I ran this
%hbase
help
I got this
(NameError) undefined local variable or method `help' for main:Object
and in the log
INFO [2016-01-21 05:40:07,314]({pool-2-thread-2} SchedulerFactory.java[jobStarted]:129) - Job remoteInterpretJob_1453354807312 started by scheduler org.apache.zeppelin.hbase.HbaseInterpreter599771689
INFO [2016-01-21 05:40:07,494]({pool-2-thread-2} HbaseInterpreter.java[open]:86) - Home:/opt/hbase-1.0.2
INFO [2016-01-21 05:40:07,494]({pool-2-thread-2} HbaseInterpreter.java[open]:87) - Ruby Src:lib/ruby
INFO [2016-01-21 05:40:07,495]({pool-2-thread-2} HbaseInterpreter.java[interpret]:110) - help
ERROR [2016-01-21 05:40:11,026]({pool-2-thread-2} HbaseInterpreter.java[interpret]:117) - Can not run 'help'
org.jruby.embed.EvalFailedException: (NameError) undefined local variable or method help' for main:Object at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:136) at org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1263) at org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1256) at org.apache.zeppelin.hbase.HbaseInterpreter.interpret(HbaseInterpreter.java:112) at org.apache.zeppelin.interpreter.ClassloaderInterpreter.interpret(ClassloaderInterpreter.java:57) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:93) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:305) at org.apache.zeppelin.scheduler.Job.run(Job.java:172) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.jruby.exceptions.RaiseException: (NameError) undefined local variable or methodhelp' for main:Object
—
Reply to this email directly or view it on GitHub
#278 (comment)
.

felixcheung · 2016-01-22T00:26:09Z

hbase shell is calling ${HBASE_HOME}/bin/hirb.rb - maybe this needs to do the same?

AhyoungRyu · 2016-01-22T03:15:36Z

@vrajat
Hi! It's really good to add HBase interpreter!!

I saw a documentation hbase.mdyou added. But currently, there is no link for this page. So you have to add a link to below this line. Then you can see your documentation page in the navigation tab : )

Plus, a few days ago, there is a PR #648 related to fixing documentation styles. This PR was working on deleting a numbering in the each sections. So why don't you remove a numbering?

Thank You ! : )

felixcheung · 2016-01-29T19:41:05Z

@vrajat ?

vrajat · 2016-01-30T01:59:38Z

Sorry. I've been busy with work. I'll try to get it next week. I'll look at
the bug. It must have been caused by one of the clean up commits.
With respect to docs, I will appreciate it if it's taken up later. This PR
has been going on for a year. I really want to get this done and move on.
On Sat, 30 Jan 2016 at 01:11, Felix Cheung notifications@github.com wrote:

@vrajat https://github.com/vrajat ?
—
Reply to this email directly or view it on GitHub
#278 (comment)
.

felixcheung · 2016-01-30T03:35:39Z

Sure - let's get it working, and here. I can help with updating doc if necessary.

vrajat · 2016-01-31T12:22:07Z

Ah ha! Now I remember why I had my own copy of hirb.rb. I couldnt get this piece of code in the original hirb.rb to work. So I had deleted it and it had no bad consequences.

35a36,41
> require 'pathname'
>
> # Add the directory names in hbase.jruby.sources commandline option
> # to the ruby load path so I can load up my HBase ruby modules
> sources = java.lang.System.getProperty('hbase.ruby.sources')
> $LOAD_PATH.unshift Pathname.new(sources)

felixcheung · 2016-02-01T07:00:09Z

@vrajat I see this error when I run it

 INFO [2016-02-01 06:55:28,887] ({pool-1-thread-3} HbaseInterpreter.java[open]:88) - Home:/opt/hbase-1.0.2/
INFO [2016-02-01 06:55:28,887] ({pool-1-thread-3} HbaseInterpreter.java[open]:89) - Ruby Src:lib/ruby
INFO [2016-02-01 06:55:28,887] ({pool-1-thread-3} HbaseInterpreter.java[open]:100) - Absolute Ruby Source:/opt/hbase-1.0.2/lib/ruby
ERROR [2016-02-01 06:55:28,904] ({pool-2-thread-2} Job.java[run]:184) - Job failed
org.apache.zeppelin.interpreter.InterpreterException: org.jruby.embed.EvalFailedException: (TypeError) can't convert Pathname into String
at org.apache.zeppelin.interpreter.ClassloaderInterpreter.open(ClassloaderInterpreter.java:76)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:68)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:92)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:299)

But the path seems correct:

 ls /opt/hbase-1.0.2/lib/ruby
hbase hbase.rb irb shell shell.rb

Also there is a typo in zeppelin-zengine/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java

felixcheung · 2016-02-01T07:00:53Z

this should be org.apache.zeppelin.hbase.HbaseInterpreter

vrajat · 2016-02-01T07:06:28Z

@felixcheung I've got past that error. I am hitting

java.lang.VerifyError: class org.apache.hadoop.hbase.protobuf.generated.HBaseProtos$ColumnFamilySchema overrides final method getUnknownFields.()Lcom/google/protobuf/UnknownFieldSet;
at java.lang.ClassLoader.defineClass1(Native Method)

This usually implies a mismatch between the jars used in the hbase client and the hbase server. I am looking into it. The same code & dependencies works in the Qubole fork. So I am eliminating what could be different about this branch.

felixcheung · 2016-02-01T07:12:28Z

cool, let me know when I should try again.

vrajat · 2016-02-01T13:23:28Z

@felixcheung Please take a look now. I have fixed all bugs and I can use it on my HBase setup (HBase 1.0.0 and Hadoop 2.6.0)

felixcheung · 2016-02-02T05:55:43Z

Great! It works (tested with hbase-1.0.2).
Merging if no more discussion (I will help fix the doc format).

vrajat · 2016-02-02T05:56:27Z

w00t!

On Tue, Feb 2, 2016 at 11:25 AM Felix Cheung notifications@github.com
wrote:

Great! It works (tested with hbase-1.0.2).
Merging if no more discussion (I will help fix the doc format).
—
Reply to this email directly or view it on GitHub
#278 (comment)
.

Leemoonsoo · 2016-02-02T08:07:02Z

Cool!

fhygh · 2022-05-24T06:22:49Z

@felixcheung hello, i saw you using hbase-1.0.2 works fine, but i using hbase 2.4.9 got the same error:

INFO [2022-05-24 14:20:49,513] ({FIFOScheduler-org.apache.zeppelin.hbase.HbaseInterpreter679627819-Worker-1} HbaseInterpreter.java[open]:80) - Home:/data/ygh/hbase-2.4.9
INFO [2022-05-24 14:20:49,514] ({FIFOScheduler-org.apache.zeppelin.hbase.HbaseInterpreter679627819-Worker-1} HbaseInterpreter.java[open]:81) - Ruby Src:lib/ruby
INFO [2022-05-24 14:20:49,514] ({FIFOScheduler-org.apache.zeppelin.hbase.HbaseInterpreter679627819-Worker-1} HbaseInterpreter.java[open]:89) - Absolute Ruby Source:/data/ygh/hbase-2.4.9/lib/ruby
INFO [2022-05-24 14:20:50,252] ({FIFOScheduler-org.apache.zeppelin.hbase.HbaseInterpreter679627819-Worker-1} HbaseInterpreter.java[interpret]:115) -
help
ERROR [2022-05-24 14:20:50,262] ({FIFOScheduler-org.apache.zeppelin.hbase.HbaseInterpreter679627819-Worker-1} HbaseInterpreter.java[interpret]:122) - Can not run '
help'
org.jruby.embed.EvalFailedException: (NameError) undefined local variable or method help' for main:Object at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:136) at org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1263) at org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1256) at org.apache.zeppelin.hbase.HbaseInterpreter.interpret(HbaseInterpreter.java:117) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:110) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:860) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:752) at org.apache.zeppelin.scheduler.Job.run(Job.java:172) at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:132) at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.jruby.exceptions.RaiseException: (NameError) undefined local variable or method help' for main:Object
INFO [2022-05-24 14:20:50,268] ({FIFOScheduler-org.apache.zeppelin.hbase.HbaseInterpreter679627819-Worker-1} AbstractScheduler.java[runJob]:154) - Job paragraph_1653298534725_1664890236 finished by scheduler org.apache.zeppelin.hbase.HbaseInterpreter679627819 with status ERROR

Below are the verisons :

Apache Zeppelin - 0.10.1
Hbase-2.4.9

do you have any idea to fix this?

Rajat Venkatesh added 17 commits April 29, 2015 10:10

Hbase Interpreter
bdba5b2

Fix test setup
03e1b7a

Add a profile for Hbase-1.0
cd41ec2

Add a new config for hbase version for interpreter
63a2d89

More fixes for version config vars
032a2a4

More fixes - sigh
b2ff198

Specify relative path
b9a407c

Merge remote branch 'upstream/master' into incubator_hbase
bc03f79

Fix version to add incubating string
b7b06cd

Add an explicit dependency for mapreduce-client-core
9a23a21

Merge master
afd50bc

Missed a couple of resolved conflicts
b8b52ba

Fix tab in pom.xml
feacd89

Explicitly add private keyword. Improve javadoc
c160787

Terminate jruby
19f137e

Fix javadoc in unit tests
b598b70

Add more info in javadoc
255d0d9

vrajat mentioned this pull request Sep 3, 2015
Hbase Interpreter #55
Closed

felixcheung reviewed Sep 6, 2015
View reviewed changes

Review comments
ca64931

felixcheung reviewed Sep 12, 2015
View reviewed changes

Merge from master
72306b7

Rajat Venkatesh added 6 commits January 31, 2016 12:38

Merge from master
49b5b27

Load hirb
b4ad18d

Address minor review comments
39dddb8

Fix hirb path
57ebad8

Set hbase.ruby.sources in system properties
69755ad

Log absolute path of ruby source
9924784

Do not use ruby 1.9 and do not bother setting paths
3efc892

felixcheung reviewed Feb 1, 2016
View reviewed changes

Fix typo in ZeppelinConfiguration
c17db3b

Fix Protobuf version number
caf87f3

asfgit closed this in 1940388Feb 3, 2016

Uh oh!

Conversation

vrajat commented Sep 3, 2015

Uh oh!

vrajat commented Sep 3, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

felixcheung commented Sep 6, 2015

Uh oh!

vrajat commented Sep 7, 2015

Uh oh!

vrajat commented Sep 10, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bzz commented Jan 5, 2016

Uh oh!

vrajat commented Jan 5, 2016

Uh oh!

vrajat commented Jan 21, 2016

Uh oh!

felixcheung commented Jan 21, 2016

Uh oh!

felixcheung commented Jan 21, 2016

Uh oh!

felixcheung commented Jan 21, 2016

Uh oh!

vrajat commented Jan 21, 2016

Uh oh!

felixcheung commented Jan 22, 2016

Uh oh!

AhyoungRyu commented Jan 22, 2016

Uh oh!

felixcheung commented Jan 29, 2016

Uh oh!

vrajat commented Jan 30, 2016

Uh oh!

felixcheung commented Jan 30, 2016

Uh oh!

vrajat commented Jan 31, 2016

Uh oh!

felixcheung commented Feb 1, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vrajat commented Feb 1, 2016

Uh oh!

felixcheung commented Feb 1, 2016

Uh oh!

vrajat commented Feb 1, 2016

Uh oh!

felixcheung commented Feb 2, 2016

Uh oh!

vrajat commented Feb 2, 2016

Uh oh!

Leemoonsoo commented Feb 2, 2016

Uh oh!

fhygh commented May 24, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

fhygh commented May 24, 2022•
edited
Loading