Skip to content

[ ZEPPELIN-1559 ] Code Editor slow performance resolve of the many Notebooks - #1533

Closed
cloverhearts wants to merge 2 commits into
apache:masterfrom
cloverhearts:ZEPPELIN-1559-Notebook-broadcast-opti
Closed

[ ZEPPELIN-1559 ] Code Editor slow performance resolve of the many Notebooks #1533
cloverhearts wants to merge 2 commits into
apache:masterfrom
cloverhearts:ZEPPELIN-1559-Notebook-broadcast-opti

Conversation

@cloverhearts

@cloverheartscloverhearts commented Oct 17, 2016

Copy link
Copy Markdown
Member

What is this PR for?

Currently, if you have a Notebook large number of,
There is a sharp decrease in performance of the Code editor.
The number and Paragraph creation of the Notebook does not have a relationship.
We are should always use the Code editor of the same performance.

I had to print a Notebook list only if there is a request.
There was quite a lot of performance improvements.

What type of PR is it?

Improvement

Todos

  • - change Notebook list render logic

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-1559
https://issues.apache.org/jira/browse/ZEPPELIN-1513

How should this be tested?

  1. Create a notebook over 1000. (tested 10,000)
    I've used script on this. (important! please, backup for notebooks directory.)
#/bin/bash
idx="0"
while [ $idx -lt 1000 ] # notebooks count 1000
do
mkdir $idx
echo """
{
\"paragraphs\": [
{
\"text\": \"fsdkljfowiejfowejfoiwefiowejfiojlkdsjfklsdjflkjsdflkjwlkjefewf\",
\"dateUpdated\": \"Oct 13, 2016 5:02:44 PM\",
\"config\": {
\"colWidth\": 12.0,
\"graph\": {
\"mode\": \"table\",
\"height\": 300.0,
\"optionOpen\": false,
\"keys\": [],
\"values\": [],
\"groups\": [],
\"scatter\": {},
\"map\": {
\"baseMapType\": \"Streets\",
\"isOnline\": true,
\"pinCols\": []
}
},
\"enabled\": true,
\"editorMode\": \"ace/mode/scala\"
},
\"settings\": {
\"params\": {},
\"forms\": {}
},
\"apps\": [],
\"jobName\": \"paragraph_1476345687682_-1459653828\",
\"id\": \"20161013-170127_68316618\",
\"dateCreated\": \"Oct 13, 2016 5:01:27 PM\",
\"status\": \"READY\",
\"progressUpdateIntervalMs\": 500
}
],
\"name\": \""${idx}"\",
\"id\": \""${idx}"\",
\"angularObjects\": {
\"2BZSC9D3G:shared_process\": [],
\"2BXGWF5TF:shared_process\": [],
\"2BYKTADMA:shared_process\": [],
\"2BYFT9HTZ:shared_process\": [],
\"2BXDTJMED:shared_process\": [],
\"2BZHPZ6NS:shared_process\": [],
\"2BYKRWYB3:shared_process\": [],
\"2BZH2UAT4:shared_process\": [],
\"2BZ38J35G:shared_process\": [],
\"2BXJ2X464:shared_process\": [],
\"2BYQ57ED9:shared_process\": [],
\"2BXDSA2SN:shared_process\": [],
\"2BWZCTD2B:shared_process\": [],
\"2BWT81MNU:shared_process\": [],
\"2BWA7ZTRD:shared_process\": [],
\"2BXW6X5KN:shared_process\": [],
\"2BVWDQPXH:shared_process\": [],
\"2BZGAF8KG:shared_process\": []
},
\"config\": {},
\"info\": {}
}
""" > $idx/note.json
idx=$[$idx+1]
done
  1. zeppelin restart or refresh notebook.
  2. Try coding in any notebook. (fast)

Screenshots (if appropriate)

before ( 5000 notebooks)

optimization_before_codeditor

after ( 5000 notebooks)

optimization_for_codeeidtor

Questions:

  • Does the licenses files need update? no
  • Is there breaking changes for older versions? no
  • Does this needs documentation? no

@cloverhearts

Copy link
Copy Markdown
MemberAuthor

@Leemoonsoo

Copy link
Copy Markdown
Member

Tested and confirmed angular watcher reduced from 5xxx -> 3xx in case of 1k notes. Great work!

<ul class="nav navbar-nav" ng-if="ticket">
<li class="dropdown" dropdown>
<a href="#" class="dropdown-toggle" dropdown-toggle>Notebook <span class="caret"></span></a>
<li id="notebook-list-button" class="dropdown notebook-list-dropdown" dropdown>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't look like that 'id' is being used

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@corneadoug
thank you!
Modification was completed. :)

@Leemoonsoo

Copy link
Copy Markdown
Member

LGTM and merge if there're no more discussions.

darionyaphet pushed a commit to darionyaphet/zeppelin that referenced this pull request Oct 27, 2016
…tebooks
### What is this PR for?
Currently, if you have a Notebook large number of,
There is a sharp decrease in performance of the Code editor.
The number and Paragraph creation of the Notebook does not have a relationship.
We are should always use the Code editor of the same performance.
I had to print a Notebook list only if there is a request.
There was quite a lot of performance improvements.
### What type of PR is it?
Improvement
### Todos
- [x] - change Notebook list render logic
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1559https://issues.apache.org/jira/browse/ZEPPELIN-1513
### How should this be tested?
1. Create a notebook over 1000. (tested 10,000)
I've used script on this. (important! please, backup for notebooks directory.)
```
#/bin/bash
idx="0"
while [ $idx -lt 1000 ] # notebooks count 1000
do
mkdir $idx
echo """
{
\"paragraphs\": [
{
\"text\": \"fsdkljfowiejfowejfoiwefiowejfiojlkdsjfklsdjflkjsdflkjwlkjefewf\",
\"dateUpdated\": \"Oct 13, 2016 5:02:44 PM\",
\"config\": {
\"colWidth\": 12.0,
\"graph\": {
\"mode\": \"table\",
\"height\": 300.0,
\"optionOpen\": false,
\"keys\": [],
\"values\": [],
\"groups\": [],
\"scatter\": {},
\"map\": {
\"baseMapType\": \"Streets\",
\"isOnline\": true,
\"pinCols\": []
}
},
\"enabled\": true,
\"editorMode\": \"ace/mode/scala\"
},
\"settings\": {
\"params\": {},
\"forms\": {}
},
\"apps\": [],
\"jobName\": \"paragraph_1476345687682_-1459653828\",
\"id\": \"20161013-170127_68316618\",
\"dateCreated\": \"Oct 13, 2016 5:01:27 PM\",
\"status\": \"READY\",
\"progressUpdateIntervalMs\": 500
}
],
\"name\": \""${idx}"\",
\"id\": \""${idx}"\",
\"angularObjects\": {
\"2BZSC9D3G:shared_process\": [],
\"2BXGWF5TF:shared_process\": [],
\"2BYKTADMA:shared_process\": [],
\"2BYFT9HTZ:shared_process\": [],
\"2BXDTJMED:shared_process\": [],
\"2BZHPZ6NS:shared_process\": [],
\"2BYKRWYB3:shared_process\": [],
\"2BZH2UAT4:shared_process\": [],
\"2BZ38J35G:shared_process\": [],
\"2BXJ2X464:shared_process\": [],
\"2BYQ57ED9:shared_process\": [],
\"2BXDSA2SN:shared_process\": [],
\"2BWZCTD2B:shared_process\": [],
\"2BWT81MNU:shared_process\": [],
\"2BWA7ZTRD:shared_process\": [],
\"2BXW6X5KN:shared_process\": [],
\"2BVWDQPXH:shared_process\": [],
\"2BZGAF8KG:shared_process\": []
},
\"config\": {},
\"info\": {}
}
""" > $idx/note.json
idx=$[$idx+1]
done
```
2. zeppelin restart or refresh notebook.
3. Try coding in any notebook. (fast)
### Screenshots (if appropriate)
#### before ( 5000 notebooks)
![optimization_before_codeditor](https://cloud.githubusercontent.com/assets/10525473/19457150/f18845a0-94ff-11e6-876d-4386dbf1e7de.gif)
#### after ( 5000 notebooks)
![optimization_for_codeeidtor](https://cloud.githubusercontent.com/assets/10525473/19457184/1f7d5036-9500-11e6-8b0c-91b301641c73.gif)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: CloverHearts <cloverheartsdev@gmail.com>
Closesapache#1533 from cloverhearts/ZEPPELIN-1559-Notebook-broadcast-opti and squashes the following commits:
bd58102 [CloverHearts] id Delete unused for navbar
14e88bb [CloverHearts] selectively modified to render the Notebook list.
pedrozatta pushed a commit to pedrozatta/zeppelin that referenced this pull request Oct 27, 2016
…tebooks
### What is this PR for?
Currently, if you have a Notebook large number of,
There is a sharp decrease in performance of the Code editor.
The number and Paragraph creation of the Notebook does not have a relationship.
We are should always use the Code editor of the same performance.
I had to print a Notebook list only if there is a request.
There was quite a lot of performance improvements.
### What type of PR is it?
Improvement
### Todos
- [x] - change Notebook list render logic
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1559https://issues.apache.org/jira/browse/ZEPPELIN-1513
### How should this be tested?
1. Create a notebook over 1000. (tested 10,000)
I've used script on this. (important! please, backup for notebooks directory.)
```
#/bin/bash
idx="0"
while [ $idx -lt 1000 ] # notebooks count 1000
do
mkdir $idx
echo """
{
\"paragraphs\": [
{
\"text\": \"fsdkljfowiejfowejfoiwefiowejfiojlkdsjfklsdjflkjsdflkjwlkjefewf\",
\"dateUpdated\": \"Oct 13, 2016 5:02:44 PM\",
\"config\": {
\"colWidth\": 12.0,
\"graph\": {
\"mode\": \"table\",
\"height\": 300.0,
\"optionOpen\": false,
\"keys\": [],
\"values\": [],
\"groups\": [],
\"scatter\": {},
\"map\": {
\"baseMapType\": \"Streets\",
\"isOnline\": true,
\"pinCols\": []
}
},
\"enabled\": true,
\"editorMode\": \"ace/mode/scala\"
},
\"settings\": {
\"params\": {},
\"forms\": {}
},
\"apps\": [],
\"jobName\": \"paragraph_1476345687682_-1459653828\",
\"id\": \"20161013-170127_68316618\",
\"dateCreated\": \"Oct 13, 2016 5:01:27 PM\",
\"status\": \"READY\",
\"progressUpdateIntervalMs\": 500
}
],
\"name\": \""${idx}"\",
\"id\": \""${idx}"\",
\"angularObjects\": {
\"2BZSC9D3G:shared_process\": [],
\"2BXGWF5TF:shared_process\": [],
\"2BYKTADMA:shared_process\": [],
\"2BYFT9HTZ:shared_process\": [],
\"2BXDTJMED:shared_process\": [],
\"2BZHPZ6NS:shared_process\": [],
\"2BYKRWYB3:shared_process\": [],
\"2BZH2UAT4:shared_process\": [],
\"2BZ38J35G:shared_process\": [],
\"2BXJ2X464:shared_process\": [],
\"2BYQ57ED9:shared_process\": [],
\"2BXDSA2SN:shared_process\": [],
\"2BWZCTD2B:shared_process\": [],
\"2BWT81MNU:shared_process\": [],
\"2BWA7ZTRD:shared_process\": [],
\"2BXW6X5KN:shared_process\": [],
\"2BVWDQPXH:shared_process\": [],
\"2BZGAF8KG:shared_process\": []
},
\"config\": {},
\"info\": {}
}
""" > $idx/note.json
idx=$[$idx+1]
done
```
2. zeppelin restart or refresh notebook.
3. Try coding in any notebook. (fast)
### Screenshots (if appropriate)
#### before ( 5000 notebooks)
![optimization_before_codeditor](https://cloud.githubusercontent.com/assets/10525473/19457150/f18845a0-94ff-11e6-876d-4386dbf1e7de.gif)
#### after ( 5000 notebooks)
![optimization_for_codeeidtor](https://cloud.githubusercontent.com/assets/10525473/19457184/1f7d5036-9500-11e6-8b0c-91b301641c73.gif)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: CloverHearts <cloverheartsdev@gmail.com>
Closesapache#1533 from cloverhearts/ZEPPELIN-1559-Notebook-broadcast-opti and squashes the following commits:
bd58102 [CloverHearts] id Delete unused for navbar
14e88bb [CloverHearts] selectively modified to render the Notebook list.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@cloverhearts@Leemoonsoo@corneadoug