Neofuzz

Blazing fast, lightweight and customizable fuzzy and semantic text search in Python.

Introduction (Documentation)

Neofuzz is a fuzzy search library based on vectorization and approximate nearest neighbour search techniques.

New in version 0.3.0

Now you can reorder your search results using Levenshtein distance! Sometimes n-gram processes or vectorized processes don't quite order the results correctly. In these cases you can retrieve a higher number of examples from the indexed corpus, then refine those results with Levenshtein distance.

fromneofuzzimportchar_ngram_processprocess=char_ngram_process()
process.index(corpus)
process.extract("your query", limit=30, refine_levenshtein=True)

Why is Neofuzz fast?

Most fuzzy search libraries rely on optimizing the hell out of the same couple of fuzzy search algorithms (Hamming distance, Levenshtein distance). Sometimes unfortunately due to the complexity of these algorithms, no amount of optimization will get you the speed, that you want.

Neofuzz makes the realization, that you can’t go above a certain speed limit by relying on traditional algorithms, and uses text vectorization and approximate nearest neighbour search in the vector space to speed up this process.

When it comes to the dilemma of speed versus accuracy, Neofuzz goes full-on speed.

When should I choose Neofuzz?

• You need to do repeated searches in the same corpus.
• Levenshtein and Hamming distance is simply not fast enough.
• You are willing to sacrifice the quality of the results for speed.
• You don’t mind that the up-front computation to index a corpus might take time.
• You have very long strings, where other methods would be impractical.
• You want to rely on semantic content.
• You need a drop-in replacement for TheFuzz.

When should I NOT choose Neofuzz?

• The corpus changes all the time, or you only want to do one search in a corpus. (It might still give speed-up in that case though.)
• You value the quality of the results over speed.
• You don’t mind slower searches in favor of no indexing.
• You have a small corpus with short strings.

Usage

You can install Neofuzz from PyPI:

pip install neofuzz

If you want a plug-and play experience you can create a generally good quick and dirty process with the char_ngram_process() process.

fromneofuzzimportchar_ngram_process# We create a process that takes character 1 to 5-grams as features for# vectorization and uses a tf-idf weighting scheme.# We will use cosine distance for the nearest neighbour search.process=char_ngram_process(ngram_range=(1,5), metric="angular", tf_idf=True)
# We index the options that we are going to search inprocess.index(options)
# Then we can extract the ten most similar items the same way as in# thefuzzprocess.extract("fuzz", limit=10)
---------------------------------
[('fuzzer', 67),
('Januzzi', 30),
('Figliuzzi', 25),
('Fun', 20),
('Erika_Petruzzi', 20),
('zu', 20),
('Zo', 18),
('blog_BuzzMachine', 18),
('LW_Todd_Bertuzzi', 18),
('OFU', 17)]

Custom Processes

You can customize Neofuzz’s behaviour by making a custom process. Under the hood every Neofuzz Process relies on the same two components:

• A vectorizer, which turns texts into a vectorized form, and can be fully customized.
• Approximate Nearest Neighbour search, which indexes the vector space and can find neighbours of a given vector very quickly.

Words as Features

If you’re more interested in the words/semantic content of the text you can also use them as features. This can be very useful especially with longer texts, such as literary works.

fromneofuzzimportProcessfromsklearn.feature_extraction.textimportTfidfVectorizer# Vectorization with words is the default in sklearn.vectorizer=TfidfVectorizer()
# We use cosine distance because it's waay better for high-dimentional spaces.process=Process(vectorizer, metric="angular")

Dimensionality Reduction

You might find that the speed of your fuzzy search process is not sufficient. In this case it might be desirable to reduce the dimentionality of the produced vectors with some matrix decomposition method or topic model.

Here for example I use NMF (excellent topic model and incredibly fast one too) too speed up my fuzzy search pipeline.

fromneofuzzimportProcessfromsklearn.feature_extraction.textimportTfidfVectorizerfromsklearn.decompositionimportNMFfromsklearn.pipelineimportmake_pipeline# Vectorization with tokens againvectorizer=TfidfVectorizer()
# Dimensionality reduction method to 20 dimensionsnmf=NMF(n_components=20)
# Create a pipeline of the twopipeline=make_pipeline(vectorizer, nmf)
process=Process(pipeline, metric="angular")

Semantic Search/Large Language Models

With Neofuzz you can easily use semantic embeddings to your advantage, and can use both attention-based language models (Bert), just simple neural word or document embeddings (Word2Vec, Doc2Vec, FastText, etc.) or even OpenAI’s LLMs.

We recommend you try embetter, which has a lot of built-in sklearn compatible vectorizers.

pip install embetter

fromembetter.textimportSentenceEncoderfromneofuzzimportProcess# Here we will use a pretrained Bert sentence encoder as vectorizervectorizer=SentenceEncoder("all-distilroberta-v1")
# Then we make a process with the language modelprocess=Process(vectorizer, metric="angular")
# Remember that the options STILL have to be indexed even though you have a pretrained vectorizerprocess.index(options)