[ZEPPELIN-2679] JDBC. precode for session by tinkoff-dwh · Pull Request #2442 · apache/zeppelin

tinkoff-dwh · 2017-06-27T04:20:59Z

What is this PR for?

We have a precode for interpreter (all interpreters) which executes once after start interpreter. With it we can create shared code that will be available in the paragraph, create tables in the database and etc. This precode for jdbc only which executes before run paragraph in same session (sql). In this precode we can declare session variables that will be available when you run paragraph (any sql which will be executed before the paragraph). For exabple set search_path='public,#{user}'

What type of PR is it?

Feature

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-2679

How should this be tested?

(jdbc interpreter, postgres)

set property default.sessionPrecodeset search_path='test_session_precode'
run paragraph show search_path

Questions:

Does the licenses files need update? no
Is there breaking changes for older versions? no
Does this needs documentation? no

felixcheung

is this session vs. interpreter lifetime?
i guess it's a bit confusion which/when is which. could you add to the description and also add to .md documentation?

also I guess I missed the other precode changes - time to catch up

tinkoff-dwh · 2017-06-29T04:31:47Z

@felixcheung
session == database session. added documentation

zjffdu · 2017-06-29T07:27:23Z

@tinkoff-dwh What do you mean database session ? From the code, precode is executed per statement. And what is the usage scenario for this ? Could you give an example ? Thanks

tinkoff-dwh · 2017-06-29T07:46:05Z

@zjffdu
yep, per statement.
for exp.
There is jdbc interpreter per user (postgres).
There are many users and each user has scheme to work.

set search_path='#{user}' (default.sessionPrecode)
run paragraph select ...
the user will work with the correct scheme

or set another variables and values will be available to code in the paragraph (SET TIME ZONE, custom variables, etc...). User == analitic (usually) and he doesn't have to write this in the code of paragraph

zjffdu · 2017-07-01T13:56:49Z

I am not sure whether this is a good idea to put custom initialization code in interpreter setting. Because I think one goal of notebook is to allow user to reproduce it easily without change note. So any non-trivial thing should be put to note itself. Otherwise user don't know how to reproduce the note result in another zeppelin instance.

tinkoff-dwh · 2017-07-03T10:20:38Z

@zjffdu
another instance? user == analitic. It is real task from production. The user does't configure interpreter, it is work of administrator.

zjffdu · 2017-07-03T11:41:23Z

I am thinking that every interpreter could have one special interpreter for initializing interpreter setting and custom enviroment requirement for his note. And do the initialization at the first paragraph. e.g.

%spark.init
set property_1=value_1
set property_2=value_2

So that the note could be shared to other people, and other people can reproduce the note result easily. My point is that note should be self-contained thing that could be reproduced when it is shared to other people

However it is still very immature idea.

tinkoff-dwh · 2017-07-03T15:17:29Z

it is JDBC interpretator, set variables in first paragraph will not work because we have connections pool and the paragraphs will be executed in different statemnets. What problem with shared notes. Фny user can execute and everyone will have their precode. set search_path='#{user} #{user} will be replaced with the username from the context

tinkoff-dwh · 2017-07-11T05:18:28Z

Ready to review

tinkoff-dwh · 2017-07-17T05:46:56Z

Ready to review

tinkoff-dwh · 2017-07-24T06:30:51Z

Ready to review

tinkoff-dwh · 2017-07-31T07:40:41Z

Ready to review

tinkoff-dwh · 2017-08-04T07:11:05Z

Ready to review

tinkoff-dwh · 2017-08-10T05:40:01Z

Ready to review

dwhsys · 2017-09-11T09:54:07Z

Can someone review please?

tinkoff-dwh · 2017-11-15T06:39:23Z

Ready to review

tinkoff-dwh · 2017-11-29T17:29:04Z

Ready to review

mebelousov · 2018-03-13T08:54:32Z

@zjffdu @felixcheung

Every day our users run about 1000 queries in the Greenplum (open source Postgres-based DB).
There is great need for binding the query to the user and/or to the note.

The precode set application_name=#{noteId}; allows quickly find the note.
This precode must be set by Zeppelin administrator, business users will not write any precode in each note for each jdbc-interpreter.

I believe the precode in JDBC interpreter is necessary feature for entreprise users.

zjffdu · 2018-03-13T09:08:19Z

Mostly LGTM, one concern is about the name session which might confuse users. How about changing it to statement ?
Besides, I am curious to know how do you use jdbc interpreter. Which mode do you use ? (Shared/Scoped/Isolated) . Because jdbc interpreter use connection pool, that's why I ask this question, just want to know how do you schedule the connection across users

mebelousov · 2018-03-13T09:52:31Z

@zjffdu, you are right, sessionPrecode is renamed to statementPrecode.

Since 0.8.0 I will recommend to run JDBC interpreter per note in isolated mode due to Interpreter Lifecycle Manager: the table results accumulate in ResourcePool and JVMs grow in size, but ILM will drop non-active JVMs.

zjffdu · 2018-03-13T12:03:52Z

 }

+ @Test
+ public void testSessionPrecode() throws SQLException, IOException {


session-> statement

zjffdu · 2018-03-13T12:04:01Z

+ }
+
+ @Test
+ public void testIncorrectSessionPrecode() throws SQLException, IOException {


zjffdu · 2018-03-13T12:04:07Z

+ }
+
+ @Test
+ public void testSessionPrecodeWithAnotherPrefix() throws SQLException, IOException {


# Conflicts: # jdbc/src/test/java/org/apache/zeppelin/jdbc/JDBCInterpreterTest.java

zjffdu · 2018-03-16T10:01:05Z

@tinkoff-dwh I just realised that zeppelin already support hooks for interpreter. I think you don't need to do it specifically for jdbc interpreter, instead we could introduce one general property like zeppelin.interpreter.pre_code which could execute pre code before running the paragraph text. What do you think ?

mebelousov · 2018-03-26T08:02:29Z

@zjffdu
The hooks as pre-code were discussed previously
We want precode execute before pre-hook for easy finding the note. Meanwhile the hooks doesn't realized in JDBC interpreter.

The JDBC interpreter allow to split query and precode must run before each statement in paragraph. Therefore the single way for precode for all interpreters looks unrealizable.

The rename to zeppelin.interpreter.precode is good idea. Do this?

zjffdu · 2018-03-26T08:52:38Z

Sorry @mebelousov I don't get why the general hooks could not satisfy you. The hook can run for each paragraph. Could you explain a little bit more ? Maybe an example would be very helpful.

mebelousov · 2018-04-03T12:48:48Z

@zjffdu
hooks are user defined code.
We want to have opportunity to log the user sessions, to bind queries to notes. It can be possible only by force execution of special query before user queries, that is in precode.

kapustor · 2018-04-06T09:39:30Z

+1
Must-have feature for enterprise usage. Pre-code may be a resource-management code, or a security code, etc. - users must not have possibility to change it, only admins.

anermakov · 2018-04-06T09:50:16Z

+1 Must have

aspekt112 · 2018-04-06T09:52:37Z

+1

Sanikadze · 2018-04-06T19:42:46Z

+1

zjffdu · 2018-04-09T06:31:11Z

@tinkoff-dwh Could you rebase the PR ?

# Conflicts: # jdbc/src/test/java/org/apache/zeppelin/jdbc/JDBCInterpreterTest.java

zjffdu · 2018-04-10T05:08:10Z

@tinkoff-dwh @mebelousov Could you retrigger the build ? Travis seems fail

mebelousov · 2018-04-10T07:43:49Z

@zjffdu I have retriggered several times. Travis fails with the same error on all jobs.

zjffdu · 2018-04-10T08:10:19Z

@mebelousov Could you clean the travis cache and for a dummy force push to trigger the build ?

zjffdu · 2018-04-11T07:50:51Z

ping @tinkoff-dwh @mebelousov

mebelousov · 2018-04-11T07:54:32Z

Cache is clean now. I try to retrigger from both accounts. The result is the same.

zjffdu · 2018-04-11T08:08:38Z

Could you create another PR to trigger the build ?

### What is this PR for? We have a precode for interpreter (all interpreters) which executes once after start interpreter. With it we can create shared code that will be available in the paragraph, create tables in the database and etc. This precode for jdbc only which executes before run paragraph in same session (sql). In this precode we can declare session variables that will be available when you run paragraph (any sql which will be executed before the paragraph). For example `set application_name='#{user};` Previous discussion in #2442 ### What type of PR is it? Feature ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2679 ### How should this be tested? (jdbc interpreter, postgres) 1. set property **default.statementPrecode** `set application_name='#{user}';` 2. run paragraph `select current_setting('application_name');` ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: tinkoff-dwh <tinkoff.dwh@gmail.com> Author: mebelousov <mebelousov@ya.ru> Author: Tinkoff DWH <tinkoff.dwh@gmail.com> Author: Renjith Kamath <rkamath@apache.org> Author: isys.mreshetov <m.reshetov@i-sys.ru> Closes#2920 from mebelousov/ZEPPELIN-2679 and squashes the following commits: d610805 [mebelousov] Revert "ZEPPELIN-3406 Travis build failure due to npm error" 11cbc44 [Renjith Kamath] ZEPPELIN-3406 Travis build failure due to npm error 11f89a5 [mebelousov] ZEPPELIN-2679 Small fix in docs e9d36d2 [mebelousov] Merge remote-tracking branch 'apache/master' into ZEPPELIN-2679 00aeabf [tinkoff-dwh] ZEPPELIN-2679 fix checkstyle f7529ca [tinkoff-dwh] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2679 07b41c5 [tinkoff-dwh] ZEPPELIN-2679 rename test methods cd0d6ed [tinkoff-dwh] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2679 7bba0db [tinkoff-dwh] ZEPPELIN-2679 rename session to statement 719d0a2 [isys.mreshetov] ZEPPELIN-2679 docs ae7a0d6 [Tinkoff DWH] [ZEPPELIN-2679] tests bd3d809 [Tinkoff DWH] [ZEPPELIN-2679] jdbc. precode for session

We have a precode for interpreter (all interpreters) which executes once after start interpreter. With it we can create shared code that will be available in the paragraph, create tables in the database and etc. This precode for jdbc only which executes before run paragraph in same session (sql). In this precode we can declare session variables that will be available when you run paragraph (any sql which will be executed before the paragraph). For example `set application_name='#{user};` Previous discussion in #2442 Feature https://issues.apache.org/jira/browse/ZEPPELIN-2679 (jdbc interpreter, postgres) 1. set property **default.statementPrecode** `set application_name='#{user}';` 2. run paragraph `select current_setting('application_name');` * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: tinkoff-dwh <tinkoff.dwh@gmail.com> Author: mebelousov <mebelousov@ya.ru> Author: Tinkoff DWH <tinkoff.dwh@gmail.com> Author: Renjith Kamath <rkamath@apache.org> Author: isys.mreshetov <m.reshetov@i-sys.ru> Closes#2920 from mebelousov/ZEPPELIN-2679 and squashes the following commits: d610805 [mebelousov] Revert "ZEPPELIN-3406 Travis build failure due to npm error" 11cbc44 [Renjith Kamath] ZEPPELIN-3406 Travis build failure due to npm error 11f89a5 [mebelousov] ZEPPELIN-2679 Small fix in docs e9d36d2 [mebelousov] Merge remote-tracking branch 'apache/master' into ZEPPELIN-2679 00aeabf [tinkoff-dwh] ZEPPELIN-2679 fix checkstyle f7529ca [tinkoff-dwh] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2679 07b41c5 [tinkoff-dwh] ZEPPELIN-2679 rename test methods cd0d6ed [tinkoff-dwh] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2679 7bba0db [tinkoff-dwh] ZEPPELIN-2679 rename session to statement 719d0a2 [isys.mreshetov] ZEPPELIN-2679 docs ae7a0d6 [Tinkoff DWH] [ZEPPELIN-2679] tests bd3d809 [Tinkoff DWH] [ZEPPELIN-2679] jdbc. precode for session (cherry picked from commit edf3c25) Signed-off-by: Jeff Zhang <zjffdu@apache.org>

### What is this PR for? We have a precode for interpreter (all interpreters) which executes once after start interpreter. With it we can create shared code that will be available in the paragraph, create tables in the database and etc. This precode for jdbc only which executes before run paragraph in same session (sql). In this precode we can declare session variables that will be available when you run paragraph (any sql which will be executed before the paragraph). For example `set application_name='#{user};` Previous discussion in apache#2442 ### What type of PR is it? Feature ### What is the Jira issue? https://issues.apache.org/jira/browse/ZEPPELIN-2679 ### How should this be tested? (jdbc interpreter, postgres) 1. set property **default.statementPrecode** `set application_name='#{user}';` 2. run paragraph `select current_setting('application_name');` ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: tinkoff-dwh <tinkoff.dwh@gmail.com> Author: mebelousov <mebelousov@ya.ru> Author: Tinkoff DWH <tinkoff.dwh@gmail.com> Author: Renjith Kamath <rkamath@apache.org> Author: isys.mreshetov <m.reshetov@i-sys.ru> Closesapache#2920 from mebelousov/ZEPPELIN-2679 and squashes the following commits: d610805 [mebelousov] Revert "ZEPPELIN-3406 Travis build failure due to npm error" 11cbc44 [Renjith Kamath] ZEPPELIN-3406 Travis build failure due to npm error 11f89a5 [mebelousov] ZEPPELIN-2679 Small fix in docs e9d36d2 [mebelousov] Merge remote-tracking branch 'apache/master' into ZEPPELIN-2679 00aeabf [tinkoff-dwh] ZEPPELIN-2679 fix checkstyle f7529ca [tinkoff-dwh] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2679 07b41c5 [tinkoff-dwh] ZEPPELIN-2679 rename test methods cd0d6ed [tinkoff-dwh] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2679 7bba0db [tinkoff-dwh] ZEPPELIN-2679 rename session to statement 719d0a2 [isys.mreshetov] ZEPPELIN-2679 docs ae7a0d6 [Tinkoff DWH] [ZEPPELIN-2679] tests bd3d809 [Tinkoff DWH] [ZEPPELIN-2679] jdbc. precode for session

We have a precode for interpreter (all interpreters) which executes once after start interpreter. With it we can create shared code that will be available in the paragraph, create tables in the database and etc. This precode for jdbc only which executes before run paragraph in same session (sql). In this precode we can declare session variables that will be available when you run paragraph (any sql which will be executed before the paragraph). For example `set application_name='#{user};` Previous discussion in apache#2442 Feature https://issues.apache.org/jira/browse/ZEPPELIN-2679 (jdbc interpreter, postgres) 1. set property **default.statementPrecode** `set application_name='#{user}';` 2. run paragraph `select current_setting('application_name');` * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: tinkoff-dwh <tinkoff.dwh@gmail.com> Author: mebelousov <mebelousov@ya.ru> Author: Tinkoff DWH <tinkoff.dwh@gmail.com> Author: Renjith Kamath <rkamath@apache.org> Author: isys.mreshetov <m.reshetov@i-sys.ru> Closesapache#2920 from mebelousov/ZEPPELIN-2679 and squashes the following commits: d610805 [mebelousov] Revert "ZEPPELIN-3406 Travis build failure due to npm error" 11cbc44 [Renjith Kamath] ZEPPELIN-3406 Travis build failure due to npm error 11f89a5 [mebelousov] ZEPPELIN-2679 Small fix in docs e9d36d2 [mebelousov] Merge remote-tracking branch 'apache/master' into ZEPPELIN-2679 00aeabf [tinkoff-dwh] ZEPPELIN-2679 fix checkstyle f7529ca [tinkoff-dwh] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2679 07b41c5 [tinkoff-dwh] ZEPPELIN-2679 rename test methods cd0d6ed [tinkoff-dwh] Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2679 7bba0db [tinkoff-dwh] ZEPPELIN-2679 rename session to statement 719d0a2 [isys.mreshetov] ZEPPELIN-2679 docs ae7a0d6 [Tinkoff DWH] [ZEPPELIN-2679] tests bd3d809 [Tinkoff DWH] [ZEPPELIN-2679] jdbc. precode for session (cherry picked from commit edf3c25) Signed-off-by: Jeff Zhang <zjffdu@apache.org> (cherry picked from commit 1795ed4) Change-Id: If10f5f7f87628931e5528999bc72d7874efe9a78

tinkoff-dwh added 2 commits June 23, 2017 16:58

[ZEPPELIN-2679] jdbc. precode for session
bd3d809

[ZEPPELIN-2679] tests
ae7a0d6

felixcheung reviewed Jun 29, 2017
View reviewed changes

ZEPPELIN-2679 docs
719d0a2

tinkoff-dwh closed this Jun 29, 2017

tinkoff-dwh reopened this Jun 29, 2017

ZEPPELIN-2679 rename session to statement
7bba0db

Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2679
cd0d6ed

zjffdu reviewed Mar 13, 2018
View reviewed changes

ZEPPELIN-2679 rename test methods
07b41c5

tinkoff-dwh closed this Mar 13, 2018

tinkoff-dwh reopened this Mar 13, 2018

tinkoff-dwh added 2 commits March 14, 2018 10:28

Merge remote-tracking branch 'upstream/master' into ZEPPELIN-2679
f7529ca
# Conflicts: # jdbc/src/test/java/org/apache/zeppelin/jdbc/JDBCInterpreterTest.java

ZEPPELIN-2679 fix checkstyle
00aeabf

mebelousov added 2 commits April 9, 2018 09:49

Merge remote-tracking branch 'apache/master' into ZEPPELIN-2679
e9d36d2
# Conflicts: # jdbc/src/test/java/org/apache/zeppelin/jdbc/JDBCInterpreterTest.java

ZEPPELIN-2679 Small fix in docs
11f89a5

mebelousov mentioned this pull request Apr 11, 2018
[ZEPPELIN-2679] JDBC. precode for session #2920
Closed

tinkoff-dwh closed this Apr 11, 2018

Uh oh!

Conversation

tinkoff-dwh commented Jun 27, 2017

What is this PR for?

What type of PR is it?

What is the Jira issue?

How should this be tested?

Questions:

Uh oh!

felixcheung left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tinkoff-dwh commented Jun 29, 2017

Uh oh!

zjffdu commented Jun 29, 2017

Uh oh!

tinkoff-dwh commented Jun 29, 2017

Uh oh!

zjffdu commented Jul 1, 2017

Uh oh!

tinkoff-dwh commented Jul 3, 2017

Uh oh!

zjffdu commented Jul 3, 2017

Uh oh!

tinkoff-dwh commented Jul 3, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tinkoff-dwh commented Jul 11, 2017

Uh oh!

tinkoff-dwh commented Jul 17, 2017

Uh oh!

tinkoff-dwh commented Jul 24, 2017

Uh oh!

tinkoff-dwh commented Jul 31, 2017

Uh oh!

tinkoff-dwh commented Aug 4, 2017

Uh oh!

tinkoff-dwh commented Aug 10, 2017

Uh oh!

dwhsys commented Sep 11, 2017

Uh oh!

tinkoff-dwh commented Nov 15, 2017

Uh oh!

tinkoff-dwh commented Nov 29, 2017

Uh oh!

mebelousov commented Mar 13, 2018

Uh oh!

zjffdu commented Mar 13, 2018

Uh oh!

mebelousov commented Mar 13, 2018

Uh oh!

zjffduMar 13, 2018

Choose a reason for hiding this comment

Uh oh!

zjffduMar 13, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zjffduMar 13, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zjffdu commented Mar 16, 2018

Uh oh!

mebelousov commented Mar 26, 2018

Uh oh!

zjffdu commented Mar 26, 2018

Uh oh!

mebelousov commented Apr 3, 2018

Uh oh!

kapustor commented Apr 6, 2018

Uh oh!

anermakov commented Apr 6, 2018

Uh oh!

aspekt112 commented Apr 6, 2018

Uh oh!

Sanikadze commented Apr 6, 2018

Uh oh!

zjffdu commented Apr 9, 2018

felixcheung left a comment •
edited
Loading

tinkoff-dwh commented Jul 3, 2017•
edited
Loading

zjffduMar 13, 2018•
edited
Loading

zjffduMar 13, 2018•
edited
Loading

mebelousov commented Apr 11, 2018•
edited
Loading