Skip to content

fix: reduce memory footprint when loading TLDs - #362

Merged
nandgator merged 1 commit into
python-validators:masterfrom
nandgator:workshop
Apr 4, 2024
Merged

fix: reduce memory footprint when loading TLDs#362
nandgator merged 1 commit into
python-validators:masterfrom
nandgator:workshop

Conversation

@nandgator

Copy link
Copy Markdown
Collaborator

No description provided.

@nandgatornandgator added the maintenance PR: Alters existing source code label Apr 4, 2024
@nandgatornandgator self-assigned this Apr 4, 2024
@nandgator
nandgator merged commit f89ddc2 into python-validators:masterApr 4, 2024
bmwiedemann pushed a commit to bmwiedemann/openSUSE that referenced this pull request Apr 26, 2024
https://build.opensuse.org/request/show/1170193
by user mia + anag+factory
- Update to 0.28.1
* fix: reduce memory footprint when loading TLDs
gh#python-validators/validators#362
* fix: rfc cases in the domain validator
gh#python-validators/validators#367
* chore: documentation maintenance
gh#python-validators/validators#368
@salty-horse

Copy link
Copy Markdown
Contributor

This significantly slows down TLD lookup. Opening a file and scanning it for every email validation is very inefficient. Is the memory footprint that much of a concern?

Would you be open to changing it back to something like this, which is 10 times faster?

_iana_tld_set=Nonedef_iana_tld():
global_iana_tld_setif_iana_tld_set:
return_iana_tld_setwithPath(__file__).parent.joinpath("_tld.txt").open() astld_f:
_=next(tld_f)
_iana_tld_set= {line.strip() forlineintld_f}
return_iana_tld_set

@nandgator

Copy link
Copy Markdown
CollaboratorAuthor

Opening a file and scanning it for every email validation is very inefficient.

That's true for repeated validations.

Is the memory footprint that much of a concern?

Yes, if the file is too, large and/or, system memory is insufficient.


What about a load_iana_tld() method?

It will load and store the TLDs once. If that method isn't called, it'll lookup the file every time. Associate that method with a dataclass, instead of using global variables.

A PR is welcome.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maintenancePR: Alters existing source code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@nandgator@salty-horse