eel.init takes 20seconds

[shrsulav]

Describe the problem
In Raspbian Stretch, Python 3.5 is the highest supported version. But for eel Python 3.6 is required. So, I looking at tutorials I compiled Python 3.6 for Raspbian Stretch and got eel working but the line "eel.init" is taking 20 seconds.

How can I debug this issue?

Desktop (please complete the following information):

• OS: Raspbian Stretch
• Browser Chromium

[samuelhwilliams]

eel.init runs through the provided directory and parses files to pull out Javascript function names. If you're sure that is taking 20 seconds (I'm surprised), my first guess would be that you're pointing it at a massive directory and so the parsing is taking ages. Is that possible?

Otherwise you can debug it with the normal tools, eg pdb/ipdb, by setting a trace in the Eel library code for the init function to see exactly what's going on.

[shrsulav]

@samuelhwilliams Hello Samuel,
I also had the same guess and I did find some files which were in the range of 75MB. So, I removed them and tried, but it did not help.

The python I am using is not available for Raspbian Stretch and I had to compile it from the source. I think this could have played a major part in it. I will try multiple versions of Python3 and see if there's any improvement, before I try the pdb/ipdb.

[samuelhwilliams]

A very quick (and dirty) check might be to insert a print(name) statement before or just after https://github.com/samuelhwilliams/Eel/blob/master/eel/__init__.py#L111 so that you can quickly see what files are getting checked/processed. Good to know that you've eliminated some of the large files already, but I wonder if there aren't still more. Have you tried pointing it at a separate directory that only contains the raw HTML/JS/CSS files you need and nothing else?

P.S. Eel is only officially supported on 3.6, but depending on your use-case and how much maintenance it needs, you might be able to get away with using it on 3.5 - so possibly worth giving that a shot as well? Obviously not if that's a production use-case though.

[ChrisKnott]

The reason for the whitelist of file extensions was to speed up this bit. Previously I was making a map app which had a folder with hundreds of OpenStreetMap tile images, and it was taking a long time to start up.

I also suggest Sam's debugging idea; you can also print time.perf_counter() if you don't want to watch it

[shrsulav]

@samuelhwilliams
I tried to find out which file(s) were causing the initialization to be slow. It turns out that there was one javascript file which was taking 12s and another javascript file which was taking 6s.

I decided not to use the 6s javascript file, and for the 12s javascript file I used the minified version.
After doing this, the initialization time has come down to 6s.

Now, that I know the minified versions of javascript file have great impact on the initialization, I will try to minify every javascript file.

[ChrisKnott]

Still too long IMO. Maybe we should be hashing the file, and caching parse results, so that it's faster if the file hasn't changed.

[samuelhwilliams]

Out of curiosity @shrsulav, how big are your JS files?

[shrsulav]

The size itself does not seem big. The unminified version is around 600kB and minified version is 220kB. The number of lines of code is high though. Around 20k lines of code.

[shrsulav]

@samuelhwilliams Earlier you had mentioned, there's a way to use eel with python 3.5. Can you provide the details on how to do that?

[t-walker-wei]

I think should add a allow prefix argument in eel.init() if need.

[Malachov]

Hello,

It's definitely an issue (solution below, so chill down....). I don't know the reason, but suddenly init took ages (order of minutes). I build static assets in Vue (works like a charm in debugging on server (hope i add examples one day)).
Build assets are very big - around 7 Mb (plotly library...)) - I don't mind big files (no networking - desktop app) .

I changed webpack optimize options and separated it into small items, then run debug and analyzed where is the problem.

It's not only about big files - if js file has around 100kb, it took about 3 seconds (don't know why...) and it seem that time is not linear with size

Problem is definitely not in parsing file structure in init folder - that's miliseconds. This line is problematic...

matches = EXPOSED_JS_FUNCTIONS.parseString(contents).asList()

The parseString from pyparsing - analysis of js code (probably finding exposed functions) (but i doubt that it's caused by version change).

Maybe it's not an issue, it just takes a time (it's not easy to regex find big files) - but still there have to be a workaround.

Use case

• i want fast eel.init but i have many not exposing js functions

Workaround

I build assets in vue

• it use webpack !!

Webpack separate my functions and 3rd party libraries (then webpack add chunk-vendors to the file name)

I suppose there is not exposed function in chunk-vendors-------.js files.

To __init__.py to def init():.....
after this loop

 for root, _, files in os.walk(root_path):
 for name in files:

around line 113
Add this lines

# From init func param or module variable
exclude_chuck_vendors = True

if exclude_chuck_vendors and name.startswith('chunk-vendors'):
 continue

This will skip js parsing for this file...

There is already similar name check, so seems ok...

Now only user code is parsed - and it's fast again...
From minutes to much less than second...

I would do merge request, but i don't know if maybe some more general solution will raise...

[t-walker-wei]

Hello,

It's definitely an issue (solution below, so chill down....). I don't know the reason, but suddenly init took ages (order of minutes). I build static assets in Vue (works like a charm in debugging on server (hope i add examples one day)).
Build assets are very big - around 7 Mb (plotly library...)) - I don't mind big files (no networking - desktop app) .

I changed webpack optimize options and separated it into small items, then run debug and analyzed where is the problem.

It's not only about big files - if js file has around 100kb, it took about 3 seconds (don't know why...) and it seem that time is not linear with size

Problem is definitely not in parsing file structure in init folder - that's miliseconds. This line is problematic...

matches = EXPOSED_JS_FUNCTIONS.parseString(contents).asList()

The parseString from pyparsing - analysis of js code (probably finding exposed functions) (but i doubt that it's caused by version change).

Maybe it's not an issue, it just takes a time (it's not easy to regex find big files) - but still there have to be a workaround.

Use case

• i want fast eel.init but i have many not exposing js functions

Workaround

I build assets in vue

• it use webpack !!

Webpack separate my functions and 3rd party libraries (then webpack add chunk-vendors to the file name)

I suppose there is not exposed function in chunk-vendors-------.js files.

To __init__.py to def init():.....
after this loop

 for root, _, files in os.walk(root_path):
 for name in files:

around line 113
Add this lines

# From init func param or module variable
exclude_chuck_vendors = True

if exclude_chuck_vendors and name.startswith('chunk-vendors'):
 continue

This will skip js parsing for this file...

There is already similar name check, so seems ok...

Now only user code is parsed - and it's fast again...
From minutes to much less than second...

I would do merge request, but i don't know if maybe some more general solution will raise...

Hi, I suggest better pass the allow prefixs.
This is what I did in init():

def init( path, allowed_extensions=['.js', '.html', '.txt', '.htm','.xhtml', '.vue'], allowed_prefixs=[''], js_result_timeout=10000): global root_path, _js_functions, _js_result_timeout root_path = _get_real_path(path) js_functions = set() for root, _, files in os.walk(root_path): for name in files: if not (any(name.endswith(ext) for ext in allowed_extensions) and any(name.startswith(prefix) for prefix in allowed_prefixs)): continue

Pass the prefixs filter when init eel

eel.init('pages-folder', allowed_prefixs=['start-with-me'])

[Malachov]

Hi... yes, that's the reason why didn't merge request i just made workaround. It's question what's good general solution and it depends on use case. In one project i have most of own js and i prefer exclude logic, in other project i use most of generated 3rd party libraries and i need only few file, i would appreciate to define list of parsed files there. It's an open question how to solve it.

From my point of view add exclude prefix pattern is most simple to implement and not affection other users.

eel.init('pages-folder', excluded_prefixes=['start-with-me'])

When i have time, i'll do merge request...