Using HDFS to backup and restore notebook by aspen01 · Pull Request #1600 · apache/zeppelin

aspen01 · 2016-11-04T15:41:02Z

What is this PR for?

This PR supports using HDFS to backup and restore notebook.
It is similar to #1479.
However this PR just use WebHDFS API, so we don't need to care about hadoop libraries' dependency.

What type of PR is it?

Improvement

Todos

- Task

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-1515

How should this be tested?

Set the variables in zeppelin-site.xml

zeppelin.notebook.storage : org.apache.zeppelin.notebook.repo.HDFSNotebookRepo
hdfs.url : Hadoop WebHDFS URL. default: http://localhost:50070/webhdfs/v1/
hdfs.user : HDFS user. default: hdfs
hdfs.maxlength : Maximum number of lines of results fetched. default: 100000
hdfs.notebook.dir : notebook location directory in HDFS. default: /tmp

After zeppelin daemon start, check the notebook directories in hdfs.notebook.dir.

Screenshots (if appropriate)

Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? Yes

bzz · 2016-11-24T02:19:49Z

@aspen01 thank you for contribution!

Could you please double-check that

all new files, as any other file in Apache Zeppelin, have ASF's Apache 2.0 license header?
all Java files adhere project's code conventions?

As for copyright notices, could you please help to understand if your contribution contains any third-party works or is that all code, developed for the ASF?

That would determine, if the existing copyright notice from the files should be moved to root NOTICE file or not. Thanks!

khalidhuseynov · 2016-11-28T06:38:45Z

+ public synchronized void save(Note note, AuthenticationInfo subject) throws IOException {
+ super.save(note, subject);
+ if (this.enableWebHDFS)
+ uploadNoteListToHDFS(note);


what do you think about:
whether makes sense to pass user from subject instead of getting user from environment inside HDFSCommand
from this function name seems like uploading whole list instead of a single note

If this feature save a notebook to hdfs directly, I think your opinion is right. But now the feature is just backup the notebook to hdfs, so I tried to think of the two functions(connecting hdfs and save the notebook) separately.
I'll fix it. Thanks.

khalidhuseynov · 2016-11-28T06:39:47Z

+ String notebookStatus = this.hdfsCmd.runCommand(this.hdfsCmd.getFileStatus, notebook, null);
+ fileStatus = gson.fromJson(notebookStatus, OneFileStatus.class);
+ } catch (Exception e) {
+ logger.warn("Warning: ", e);


maybe some description to give here

khalidhuseynov · 2016-11-28T06:44:01Z

+ }
+ }
+ catch (Exception e) {
+ logger.error("Exception: ", e);


a bit more description would help here as well

I'll add more description. Thanks.

placeybordeaux · 2016-12-05T18:50:03Z

@@ -0,0 +1,241 @@
+/**
+ * Copyright 2015 NAVER Corp. All rights Reserved.


Is this file licensed for use in an Apache project?

It is wrong license. I'll remove.

Please make sure all the code is distributed under one of the ASF category-a licenses

khalidhuseynov · 2016-12-08T02:16:36Z

@aspen01 I wonder if any progress on this one?

zjffdu · 2016-12-11T02:45:10Z

+</property>
+
+<property>
+ <name>hdfs.url</name>


I think it is better to add prefix zeppelin.notebook, because hdfs is used in many places. Adding prefix can help user to understand this this is for notebook storage.

HDFSCommand use hdfs.url property already. So I used the same property name.
If adding prefix can help user to understand, I'll add prefix.

zjffdu · 2016-12-11T03:10:36Z

 <version>${project.version}</version>
 </dependency>

+ <dependency>


I don't think it is a good idea to make zengine to depend on one specific interpreter.

I agree with you. But To use HDFSCommand functions, I had to add dependency.
Could you give me advice to solve this?

Indeed, Zengine should not have such dependency.
@aspen01 if HDFSCommand have to be shared:
one option is to move it to zeppelin-interpreter, as both file and zeppelin-zengine already depend on it
another option to consider would be - extract a new maven sub-module out of zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/, move HDFSCommand there and make other modules depend on it.
Second option is more involved and deserves a separate issue \w discussion. First one looks more feasible

aspen01 · 2016-12-19T13:20:11Z

@bzz Thank you for advice. But the build is failed even though I fixed license and checkstyle.

bzz · 2016-12-20T01:10:31Z

I see, thank you for taking care! This looks like something failing on the TravisCI side though.

Could you please rebase this branch on latest master and force-push it here again? This will trigger CI \w latest code\fixes from master.

placeybordeaux · 2016-12-21T20:47:20Z

Looks like you're missing a hotfix commit:

1c7d8fb

bzz · 2016-12-22T03:07:44Z

Thank you @aspen01, @placeybordeaux

In order to be merged, this branch have to be rebased on top of the master, which itself already includes CI fixes, i.e 1c7d8fb.

This branch must not include any other commits, except for ones that implement "HDFS to backup and restore notebook" - edae7bc, d94eb16 need to be removed.

bzz · 2016-12-22T03:23:38Z

+ return "/" + newUrl.replaceAll("^hdfs://", "").split("/", 2)[1];
+ }
+ else
+ return newUrl;


Please, make sure all the code follows project style guide

aspen01 · 2016-12-22T03:32:25Z

@placeybordeaux Thank you.
@bzz Do you mean I have to merge these commits to master not branch-0.6?

add header for uploading file using REST API

aspen01 · 2016-12-28T10:00:34Z

I committed this branch on latest master and force-push it.

placeybordeaux · 2017-01-10T21:41:07Z

Not sure why the CI hasn't kicked in, but I tried compiling this on my local and I am getting some test failures:

Results :
Failed tests:
NotebookTest.testAbortParagraphStatusOnInterpreterRestart:760 expected:<ABORT> but was:<RUNNING>
Tests in error:
HeliumApplicationFactoryTest.testUnloadOnInterpreterUnbind:232 » ClassCast jav...
HeliumApplicationFactoryTest.testLoadRunUnloadApplication:148 » ClassCast java...
HeliumApplicationFactoryTest.testUnloadOnInterpreterRestart:299 » ClassCast ja...
HeliumApplicationFactoryTest.testUnloadOnParagraphRemove:193 » ClassCast java....
Tests run: 149, Failures: 1, Errors: 4, Skipped: 0

Not sure if these are related to the PR at all.

aspen01 · 2017-01-11T14:21:01Z

@placeybordeaux How did you build a package?

$ mvn clean package -Pspark-1.6 -Phadoop-2.6 -Pyarn -Ppyspark -Psparkr -Pscala-2.10
...
Results :
Tests run: 66, Failures: 0, Errors: 0, Skipped: 0
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 19:23 min
[INFO] Finished at: 2017-01-11T23:00:58+09:00
[INFO] Final Memory: 160M/815M
[INFO] ------------------------------------------------------------------------

placeybordeaux · 2017-01-11T23:43:02Z

I built with mvn clean package, but got the same results with mvn clean package -Pspark-1.6 -Phadoop-2.6 -Pyarn -Ppyspark -Psparkr -Pscala-2.10. It's likely this is just a problem on my side.

keithchambers · 2017-02-04T16:24:25Z

Is this ready to be merged?

EronWright · 2017-03-10T23:04:23Z

Seems overcomplicated for HDFSNotebookRepo to extend VFSNotebookRepo, and to be synchronizing the local FS with the remote FS. Just implement NotebookRepo directly.

aspen01 · 2017-05-10T13:47:38Z

@EronWright Thank you for your feedback. When I first developed the feature, I assumed that HDFS would not be used as default storage because remote FS failure could affect zeppelin usage. So extended VFSNotebookRepo to keep the default storage space on local FS and HDFS as backup storage for failover. This makes the implementation is simple, but it may have created complexity.
I'm using this feature, but I am also considering using a third-party storage such as HBase, because the latency of HDFS affects the latency of zeppelin notebooks.

close#83close#86close#125close#133close#139close#146close#193close#203close#246close#262close#264close#273close#291close#299close#320close#347close#389close#413close#423close#543close#560close#658close#670close#728close#765close#777close#782close#783close#812close#822close#841close#843close#878close#884close#918close#989close#1076close#1135close#1187close#1231close#1304close#1316close#1361close#1385close#1390close#1414close#1422close#1425close#1447close#1458close#1466close#1485close#1492close#1495close#1497close#1536close#1545close#1561close#1577close#1600close#1603close#1678close#1695close#1739close#1748close#1765close#1767close#1776close#1783close#1799

aspen01 mentioned this pull request Nov 4, 2016
Notebook: HDFS as a backend storage (Read & Write Mode) #1479
Closed
1 task

khalidhuseynov reviewed Nov 28, 2016
View reviewed changes

placeybordeaux reviewed Dec 5, 2016
View reviewed changes

zjffdu reviewed Dec 11, 2016
View reviewed changes

bzz reviewed Dec 22, 2016
View reviewed changes

aspen01 added 7 commits December 28, 2016 18:20

Using HDFS to backup and restore notebook
fd94a51

add hadoop pseudo auth in GETFILESTATUS, LISTSTATUS
c8643a1
add header for uploading file using REST API

always add hadoop pseudo authentication
fdb48f3

add document for HDFSNotebookRepo
5ce0c39

adding Apache 2.0 license header and exception message
a955a21

fix checkstyle for hdfs notebook
815a1ee

change method for getting note's id
123a074

aspen01 force-pushed the branch-0.6 branch from d94eb16 to 123a074Compare December 28, 2016 09:55

aspen01 changed the base branch from branch-0.6 to masterDecember 28, 2016 09:56

asfgit closed this in c38a0a0May 9, 2018

		@@ -0,0 +1,241 @@
		/**
		* Copyright 2015 NAVER Corp. All rights Reserved.

Uh oh!

Conversation

aspen01 commented Nov 4, 2016

What is this PR for?

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Questions:

Uh oh!

bzz commented Nov 24, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

khalidhuseynovNov 28, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

khalidhuseynov commented Dec 8, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aspen01 commented Dec 19, 2016

Uh oh!

bzz commented Dec 20, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

placeybordeaux commented Dec 21, 2016

Uh oh!

bzz commented Dec 22, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aspen01 commented Dec 22, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aspen01 commented Dec 28, 2016

Uh oh!

placeybordeaux commented Jan 10, 2017

Uh oh!

aspen01 commented Jan 11, 2017

Uh oh!

placeybordeaux commented Jan 11, 2017

Uh oh!

keithchambers commented Feb 4, 2017

Uh oh!

EronWright commented Mar 10, 2017

Uh oh!

aspen01 commented May 10, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

bzz commented Nov 24, 2016•
edited
Loading

khalidhuseynovNov 28, 2016•
edited
Loading

bzz commented Dec 20, 2016•
edited
Loading

aspen01 commented Dec 22, 2016•
edited
Loading