[ZEPPELIN-1824] Add MetaData exploration to JDBC Interpreter

[pmccaffrey6]

What is this PR for?

Zeppelin currently has little functionality for data source exploration. This PR exists to build a small feature for the JDBC interpreter that would allow users to explore metadata for databases and database objects.

With this PR, the JDBC interpreter accepts the "explore" keyword. When run in isolation, this fetches metadata about the database as a whole (tables, views etc...). When the explore keyword is followed by the name of a table or view, this fetches metadata about that table or view (column names, data types etc...).

A video of this feature in action can be found here (https://s3.amazonaws.com/screenshots-mockups/embedvid.html).

What type of PR is it?

Improvement | Feature

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-1824

How should this be tested?

1. Run explore in a %jdbc paragraph to get a list of tables, views, system tables, global and local temporary tables, aliases and synonyms.

2. Run explore followed by the name of a database table or view in order to get a list of column names, data types etc...

Additionally, this PR adds two new unit tests to JDBCInterpreterTest which test fetching of database metadata as well as table and view metadata.

Screenshots (if appropriate)

https://s3.amazonaws.com/screenshots-mockups/embedvid.html

Questions:

• Does the licenses files need update? No
• Is there breaking changes for older versions? No
• Does this needs documentation? Yes. This would benefit from a small addition to the JDBC Interpreter documentation.

[FireArrow]

LGTM. Test failure seems unrelated.

[DrIgor]

As I remember, org.apache.commons.lang3.exception.ExceptionUtils#getStackTrace does the same thing

[DrIgor]

Unused variable

[DrIgor]

It's better to use logger, not to write to System.err

[bzz]

Huge 👍 for using Logger here, as in the rest of the project

[DrIgor]

connection can't be null here because we return at line 533 in case of null

[pmccaffrey6]

Hey guys,
Thanks so much for the review! I removed the unused variable (great catch).

As for the other issues, I think they make a ton of sense. I wrote the getMetaData() method so that it pairs closely with the executeSql() method and so these issues are present in both.

Perhaps a good option would be to open a second PR to refactor the JDBC interpreter and make these changes in both getMetaData() and executeSql() methods? What do you think?

[pmccaffrey6]

The failed test looks to be from testSparkSQLInterpreter from LivyInterpreterIT.java. More specifically, it comes from line 227 (and 226) in that file:

InterpreterResult result = sqlInterpreter.interpret("show tables", context);
assertEquals(InterpreterResult.Code.SUCCESS, result.code());

sqlInterpreter being an instance of LivySparkSQLInterpreter, it seems unrelated.

[pmccaffrey6]

I went ahead and changed parsing of the "explore" keyword to use split(" +") instead of substring so that it wasn't tied to any particular word length etc.. and the METADATA_KEYWORD can be set to any single case-insensitive word.

As an additional detail, if you look up a table that doesn't exist you'll just get an empty table in the paragraph as opposed to an explicit error message. I experimented a bit with testing for an empty result set using resultSet.next() and resultSet.first() so that I could output an error message like "Database object not found".

However, it seems that when getting metadata, since you're not using Statement, this kind of thing causes problems because re-setting cursors on the resultset using methods like resultSet.beforeFirst() isn't supported for all data sources. I don't want to use Statement because I want to have this abstracted away from SQL and simply use the JDBC api to get metadata. So, in the interest of making this as general of a feature as possible, and therefore not using Statement, as well as not having a good way to test for resultSet emptiness that is adequately vendor-agnostic, there currently isn't testing for empty result sets, they just return as empty tables.

I don't imagine this is an issue really but if anyone has a suggestion of a good way to do this, I'd be very interested to hear it!

[bzz]

Is javax.swing really required here?

[bzz]

Thank you @pmccaffrey6 , it looks great to me, modulo few things noted above.

Could you please double check that all feedback from reviews was addressed?
Also small documentation update might be in order there, to highlight this feature for the users.

[pmccaffrey6]

Hey @bzz,
Great catch! Sorry about that. I removed the unnecessary import. The error seems unrelated. It comes from:

1470 21:03:56,259 ERROR org.apache.zeppelin.AbstractZeppelinIT:136 - Exception in ZeppelinIT while testSparkInterpreterDependencyLoading 1471 org.openqa.selenium.TimeoutException: Timed out after 60 seconds waiting for org.apache.zeppelin.AbstractZeppelinIT$1@7918cba6
... Stack Trace ...
1507 Caused by: org.openqa.selenium.NoSuchElementException: Unable to locate element: {"method":"xpath","selector":"(//div[@ng-controller=\"ParagraphCtrl\"])[1]//div[contains(@class, 'control')]//span[1][contains(.,'FINISHED')]"}

As for the other excellent points that @DrIgor noted (including the great point about using logger), those issues are present both in getMetaData() as well as executeSql() methods. Would you prefer that I make those changes to the getMetaData() method in this PR or refactor both getMetaData() and executeSql() in a separate PR?

As always, thanks for your review!