[ZEPPELIN-1233] Support Redis interpreter

[darionyaphet]

What is this PR for?

Support Redis interpreter

What type of PR is it?

Feature

Todos

• - Support Redis Cluster

What is the Jira issue?

• https://issues.apache.org/jira/browse/ZEPPELIN-1233

How should this be tested?

Outline the steps to test the PR here.

Screenshots (if appropriate)

Questions:

• Does the licenses files need update?
• Is there breaking changes for older versions?
• Does this needs documentation?

[anthonycorbacho]

Thank you for bringing Redis Interpreter! this is awesome!

I did a quick review on your code please check it and let me know what you think.

[anthonycorbacho]

As a redis user, i am expecting to use other redis command like hkeys, range and stuff.
Maybe its a good idea to add more here or come up with a better approach ?

[darionyaphet]

Actually I'm not sure is there any good way to running redis command so I just simply create GET and SET :)

[darionyaphet]

maybe I can use redis.clients.jedis.Protocol.Command to replacement RedisCommand

[anthonycorbacho]

yes, i think its better to use this enum
i was thinking about doing something like:

private final HashSet<String> redisCommands;
public static HashSet<String> getRedisCommands() {
HashSet<String> hashSetOfRedisCommands = new HashSet<String>();
for (Command c : Command.values()) {
hashSetOfRedisCommands.add(c.name());
}
return hashSetOfRedisCommands;
}
public RedisInterpreter(Properties property) {
super(property);
redisCommands = getRedisCommands();
}
// then use this like redisCommands.contains(tokens[1]);

what do you think?

[darionyaphet]

Maybe we don't need a redis commands set to check whether the input command is illegal because there is a default case in switch . Is it a better one?

String[] tokens = splitTokens(input);
String keyword = tokens[0];
switch (Protocol.Command.valueOf(keyword)) {
case GET:
result = getProcess();
break;
case SET:
result = setProcess();
break;
default:
result = new InterpreterResult(ERROR, "key word not found");
break;
}

[anthonycorbacho]

Yes it look better :)

[anthonycorbacho]

I think here we should provide a better Redis client creation, as far as i know, this will assume that you are running redis locally, i think we can exploit more the interpreter setting creation, here an example with JDBC.
So user will be able to set host, port and credential.

Also how about using JedisPool instead of Jedis?

It will look like something like this

private static final int MAX_CONNECTION = 142;
private static final int MAX_IDLE = 10;
private static final int MIN_IDLE = 142;
private static final long MAX_WAIT = TimeUnit.SECONDS.toMillis(30);
private JedisPool jedisPool;
public void open() {
jedisPool = new JedisPool(getPoolConfiguration(), property.getCacheHost(), property.getCachePort());
}
private JedisPoolConfig getPoolConfiguration() {
JedisPoolConfig config = new JedisPoolConfig();
config.setMaxTotal(MAX_CONNECTION);
config.setMaxIdle(MAX_IDLE);
config.setMinIdle(MIN_IDLE);
config.setMaxWaitMillis(MAX_WAIT);
return config;
}
// Utils method :)
private Jedis borrow() {
return jedisPool.getResource();
}
// Then you can use it like
private InterpreterResult getProcess(String key) {
Jedis jedis = borrow();
Pipeline pipe = jedis.pipelined();
Response<String> response = pipe.get(key);
pipe.sync();
jedis.close();
String result = response.get();
return new InterpreterResult(SUCCESS, result);
}

[darionyaphet]

I read Jedis source code and find a strange case . The returnResource will be deprecated after 3.0 and the resource shoule be close after usage (don't return into source pool ) . Now I'm using 2.9 . I'm not sure it's a good idea .

[darionyaphet]

Maybe using zeppelin to access Redis will not very frequently so create jedis client after it close is OK .

[anthonycorbacho]

I think its better to use a Pool for this job, you only need jedis instance when you actually run the jobs, that is whay i am asking for changing from Redis to JedisPool, and in the doc i didnt find the notice about removing returnResource.

[anthonycorbacho]

ASF Header is missing here.

[anthonycorbacho]

Could you also add the class header with a description of this interpreter?
This will help futur contributors.

[anthonycorbacho]

Can we change this field for private static final Logger LOG = LoggerFactory.getLogger(RedisInterpreter.class);?

this will help to have one logger instance :)

[felixcheung]

Hi - thank you for your contribution. Could you see the list of steps here:
https://zeppelin.apache.org/docs/0.6.1/development/writingzeppelininterpreter.html#contributing-a-new-interpreter-to-zeppelin-releases

Notably, could you add:

• tests
• documentation
• list licenses for dependencies

[anthonycorbacho]

@darionyaphet do you need some help for the change?

[darionyaphet]

Hi @anthonycorbacho thank you for you great help and I'm still working on it .

I have implement a lot of command include GEO ,HASH, HYPERLOGLOG, LIST, KEYS, Sets ,SORTED SET and Strings and have updated .

[anthonycorbacho]

Humm, that is a very massive switch case, i think we can come up with a more elegant way to to this kind of things,

since it mainly take the same argument String[] token, i guess we can go with the design pattern Strategy Pattern.

This idea here is to extract the logic to smaller classes and register then to a map.

eg:

// In the constructor
Map<Protocol.Command, Command> redisCommand = Maps.newHashMap<>();
// register command and class
redisCommand.put(GEOADD, GeoAdd.createInstance());
... // add other command
// create a java package org.apache.zeppelin.redis.cmd and add every implementation of redis cmd.
// but first you need to create a interface
// eg:
public interface Cmd {
public boolean execute(String[] tokens);
}
// use this interface to implement the actual cmd
public class MyRedisCmd implements Cmd {
public static MyRedisCmd createInstance() {
return new MyredisCmd();
}
@Override
public boolean execute(String[] tokens) {
// do the work here
}
}
// Once you have implemented the redis cmd, you can use this like this:
redisCommand.get(Protocol.Command.valueOf(keyword)).execute(tokens);

let me know what you think

[darionyaphet]

@anthonycorbacho@kavinkumarks thanks for you reply with great suggest 👍 . Using Strategy Pattern to execute Redis Command will take me make a lot of classes , each Redis Command should have a class to express it . I'm not sure it's a good idea maybe the long switch list is more clearly to execute the simple redis commands in a single Java class .

what do you think :)

PS : I will append some test case about Redis Command later .

[HeartSaVioR]

AFAIK, what @anthonycorbacho suggested seems more like a command pattern. (since it is about the action - what to do, not the algorithm - how to do)
Anyway +1 to apply what @anthonycorbacho suggested. Having more classes is not worse than having too long single class, (and also too long switch statement) especially they can be organized by package. We can even use static inner class to reduce the file count if we really want.

Btw, to be clear, I was the other one who want to make the Redis interpreter but at the design phase I was stuck on converting Redis command statement to Jedis method call.
(I was struggling to find a way to not having static mapping like this, but that was not easy as I thought.)
I will come up with clearer way sooner or later if possible.

[anthonycorbacho]

Personally, I would prefer to have classes (will make test more easy) than a massive switch case. the main concern i have about big switch case tis that it is really hard to maintain and make the parent class huge. by de coupling the logic into smaller classes will improve the maintainability and make testability more easy.

[darionyaphet]

@anthonycorbacho@HeartSaVioR It's seems a good idea to split execute Redis Command into classes and this command classes will reused in Redis Cluster Interpreter. I have create a abstract class RedisCommand as a basic class :

public abstract class RedisCommand {
private JedisPool clientPool;
public RedisCommand(JedisPool clientPool) {
this.clientPool = clientPool;
}
private Jedis borrowClient() {
return clientPool.getResource();
}
private void releaseClient(Jedis client) {
client.close();
}
public InterpreterResult execute(String[] tokens) {
Jedis client = borrowClient();
try {
return execute(client, tokens);
} catch (JedisException exception) {
return new InterpreterResult(ERROR, exception.getMessage());
} finally {
releaseClient(client);
}
}
/**
* @param tokens
* @return
* @throws Exception
*/
protected abstract InterpreterResult execute(Jedis client, String[] tokens);
}

I think it's OK and each redis command could extend this one .

[kavinkumarks]

I too checked the changes and they look good.The comment from @anthonycorbacho about usage of "Strategy pattern" for selecting the relevant redis command instance and executing them is the better way to implement it.

Also, could we add some test cases for this?

Thanks,
Kavin
MailTo: kavin.kumar@imaginea.com

[HeartSaVioR]

Every command methods have a chance to leak Jedis instance when command is failing and throwing a kind of JedisException.
You need to replace your borrow-call-return pattern with try-with-resource thanks to Zeppelin requires Java 1.7.

[anthonycorbacho]

@HeartSaVioR thank you for pointing that out, you are right we better handle the potential JedisException exception to avoid leak.

[darionyaphet]

Hi @anthonycorbacho@HeartSaVioR I have update RedisInterpreter with some redis commands (include APPEND ,BITCOUNT ,BITFIELD ,BITOP, BITPOS, DECR) .

Please help review this one . If anything is OK , I will clean the commented code and completion other redis commands . Thanks ~~ :)

[anthonycorbacho]

Great to see this interpreter improving day by day!!
@HeartSaVioR what do you think?

[anthonycorbacho]

Better to remove comment here if there are not intentions of using the code below.

[darionyaphet]

Hi I still working on Key commands . I will clean the unnecessary code after finished this PR :)

[anthonycorbacho]

What do you think about adding a LOG.error here? something like
LOG.error("Couldn't execute redis command {}", token[0], e);

[HeartSaVioR]

+1

[anthonycorbacho]

Need a new line here :)

[anthonycorbacho]

Need a new line here :)

[anthonycorbacho]

Maybe we can provide a better error to the user like
Keyword " + tokens + " is not supportted?

What do you think?

[darionyaphet]

Yep It's a good idea 👍

[HeartSaVioR]

I left review comments but overall direction looks good unless we find brilliant idea to reduce verbose.

Please note that I couldn't check actual call of Jedis method since they're too many. I would like to see all tests instead, but tests don't necessary need to have files for each command.

Some of Jedis methods can return null (for example, if there's no matching result) which should be handled properly. Some of implementations of RedisCommands might throw NPE in some cases since you manipulate the result without null check. I'm not sure interpreter result type NULL is for these case.
@anthonycorbacho Could you clarify above this?
May be better to have generic return type for RedisCommand, and let RedisInterpreter handles null and also type-aware result.

[HeartSaVioR]

version mismatch

[HeartSaVioR]

It can make unneeded confusing. jedis.version would be better.

[HeartSaVioR]

Since Redis interpreter doesn't need to be super fast at all, testOnBorrow() can relieve the connection issue with ignorable overhead.

[HeartSaVioR]

Wrong explanation here.

[HeartSaVioR]

You can use try-with-resource:

try (Jedis client = clientPool.getResource()) {
...
}

No need to have borrowClient() and releaseClient(), and also no need to explicitly handling closing resource.

[HeartSaVioR]

+1

[HeartSaVioR]

I'd prefer wrapping it with String.valueOf() or call toString() if it's same, but this is simpler so OK to me.

[HeartSaVioR]

And I also prefer returning it without having local variable (this style occurs all of RedisCommand implementations), but this way makes it easier to set a breakpoint. I'd like to see other's opinions here.

[darionyaphet]

toString maybe better and reasonable . @HeartSaVioR

[HeartSaVioR]

Please extract this logic to method of utility class or method of RedisCommand. This is duplicated from many places. The logics which manipulate keys/parameters are all candidates to extract.

[anthonycorbacho]

@HeartSaVioR yeah i would love to have a class that manipulate redis cmd and handle null, ussually i am not a huge fan of returning null, i would prefer to return an EMPTY type instead.

[darionyaphet]

Aha. A lot of files conflict and I can't resolve that . Maybe I will close this one and reopen another .

[HeartSaVioR]

Follow this if rebase with master is hard due to lots of conflicts:

1. create a new branch which is based on current master
2. cherry-pick your commits (write down your last commit id)
3. change branch to PR's branch
4. git reset --hard last commit id
5. git push --force

last commit id might be replaced with branch name you created, but I didn't try it.

[darionyaphet]

Add a Screenshot