Skip to content

python.d.plugin: use separate process for initial module checking - #5552

Merged
ilyam8 merged 17 commits into
netdata:masterfrom
ilyam8:pythond_plugin_refactor
Mar 7, 2019
Merged

python.d.plugin: use separate process for initial module checking#5552
ilyam8 merged 17 commits into
netdata:masterfrom
ilyam8:pythond_plugin_refactor

Conversation

@ilyam8

@ilyam8ilyam8 commented Mar 5, 2019

Copy link
Copy Markdown
Member
Summary

This PR adds (major) changes only to python.d.plugin file.

Fixes: #5525

pyhton.d.plugin imports a lot of additional packages during initial module initialization/job creating/checking and there is no way to unimport them, even if they arn't needed. It consumes relatively a lot of ram.


Memory utilization comparing before/after the PR (one job example module, py3.7.2):

21.1 => 8.8 MiB

screenshot_20190305_111837

Component Name

collectors/python.d.plugin

Additional Information

This PR adds separate process for initial module checking.

Logic:

  • main process spawns checker process
  • checker process loads every module, loads module config, creates jobs and runs job.check() for every job, if check success it adds the job to the list.
  • checker process returns list of modules and jobs.
  • main process loads only active modules, etc.

@netdatabotnetdatabot added the area/collectors Everything related to data collection label Mar 5, 2019
@netdatanetdata deleted a comment Mar 5, 2019
@netdatanetdata deleted a comment Mar 5, 2019
@netdatanetdata deleted a comment Mar 5, 2019
@ilyam8

Copy link
Copy Markdown
MemberAuthor

ok, i tested it with on Manjaro (py 3.7.2) and on Centos6(py 2.6.6), and it appears to be working correctly

@ilyam8ilyam8 changed the title [wip] python.d.plugin: use separate process for initial module checkingpython.d.plugin: use separate process for initial module checkingMar 5, 2019
@ilyam8

Copy link
Copy Markdown
MemberAuthor

Hey @Ferroin if you could find a time and test the PR a bit it would be very nice

@netdatanetdata deleted a comment Mar 6, 2019
@netdatanetdata deleted a comment Mar 6, 2019
@ilyam8

ilyam8 commented Mar 6, 2019

Copy link
Copy Markdown
MemberAuthor

ok, i don't think we need to run a job if job.check() returns False in main. It can be error prone, instead we can treat all jobs in main as autodetection jobs, so if a job's check(), for some reason, succeed in child process and failed in main we will retry check again and again

@netdatanetdata deleted a comment Mar 6, 2019
@ilyam8

Copy link
Copy Markdown
MemberAuthor

@ktsaou if you are ok with the changes this is ready.

@netdatanetdata deleted a comment Mar 6, 2019
@ilyam8

Copy link
Copy Markdown
MemberAuthor

@cakrit please approve

@ilyam8
ilyam8 merged commit 2175673 into netdata:masterMar 7, 2019
@ilyam8

Copy link
Copy Markdown
MemberAuthor

ok, here we go, i am on a high alert ⏰

@ilyam8
ilyam8 deleted the pythond_plugin_refactor branch March 12, 2019 13:48
jackyhuang85 pushed a commit to jackyhuang85/netdata that referenced this pull request Jan 1, 2020
…tdata#5552)
##### Summary
This PR adds (major) changes only to `python.d.plugin` file.
Fixes: netdata#5525
`pyhton.d.plugin` imports a lot of additional packages during initial module initialization/job creating/checking and there is no way to unimport them, even if they arn't needed. It consumes relatively a lot of ram.
___
Memory utilization comparing before/after the PR (one job `example` module, py3.7.2):
> 21.1 => 8.8 MiB
![screenshot_20190305_111837](https://user-images.githubusercontent.com/22274335/53791147-c27a6e00-3f39-11e9-8eaf-8ac3809a3b6e.png)
##### Component Name
[`collectors/python.d.plugin`](https://github.com/netdata/netdata/blob/master/collectors/python.d.plugin/python.d.plugin.in)
##### Additional Information
This PR adds separate process for initial module checking. Logic:
- main process spawns checker process
- checker process loads every module, loads module config, creates jobs and runs job.check() for every job, if check success it adds the job to the list.
- checker process returns list of modules and jobs.
- main process loads only active modules, etc.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/collectorsEverything related to data collection

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@ilyam8@cakrit@netdatabot