finalfusion-python

Introduction

finalfusion is a Python package for reading, writing and using finalfusion embeddings, but also supports other commonly used embeddings like fastText, GloVe and word2vec.

The Python package supports the same types of embeddings as the finalfusion-rust crate:

• Vocabulary:
  • No subwords
  • Subwords
• Embedding matrix:
  • Array
  • Memory-mapped
  • Quantized
• Norms
• Metadata

Installation

The finalfusion module is available on PyPi for Linux, Mac and Windows. You can use pip to install the module:

$ pip install --upgrade finalfusion

Installing from source

Building from source depends on Cython. If you install the package using pip, you don't need to explicitly install the dependency since it is specified in pyproject.toml.

$ git clone https://github.com/finalfusion/finalfusion-python
$ cd finalfusion-python
$ pip install .

If you want to build wheels from source, wheel needs to be installed. It's then possible to build wheels through:

$ python setup.py bdist_wheel

The wheels can be found in dist.

Package Usage

Basic usage

importfinalfusion# loading from different formatsw2v_embeds=finalfusion.load_word2vec("/path/to/w2v.bin")
text_embeds=finalfusion.load_text("/path/to/embeds.txt")
text_dims_embeds=finalfusion.load_text_dims("/path/to/embeds.dims.txt")
fasttext_embeds=finalfusion.load_fasttext("/path/to/fasttext.bin")
fifu_embeds=finalfusion.load_finalfusion("/path/to/embeddings.fifu")
# serialization to formats works similarlyfinalfusion.compat.write_word2vec("to_word2vec.bin", fifu_embeds)
# embedding lookupembedding=fifu_embeds["Test"]
# reading an embedding into a bufferimportnumpyasnpbuffer=np.zeros(fifu_embeds.storage.shape[1], dtype=np.float32)
fifu_embeds.embedding("Test", out=buffer)
# similarity and analogy querysim_query=fifu_embeds.word_similarity("Test")
analogy_query=fifu_embeds.analogy("A", "B", "C")
# accessing the vocab and printing the first 10 wordsvocab=fifu_embeds.vocabprint(vocab.words[:10])
# SubwordVocabs give access to the subword indexer:subword_indexer=vocab.subword_indexerprint(subword_indexer.subword_indices("Test", with_ngrams=True))
# accessing the storage and calculate its dot product with an embeddingres=embedding.dot(fifu_embeds.storage)
# printing metadataprint(fifu_embeds.metadata) 

Beyond Embeddings

# load only a vocab from a finalfusion filefromfinalfusionimportload_vocabvocab=load_vocab("/path/to/finalfusion_file.fifu")
# serialize vocab to single filevocab.write("/path/to/vocab_file.fifu.voc")
# more specific loading functions existfromfinalfusion.vocabimportload_finalfusion_bucket_vocabfifu_bucket_vocab=load_finalfusion_bucket_vocab("/path/to/vocab_file.fifu.voc")

The package supports loading and writing all finalfusion chunks this way. This is only supported by the Python package, reading will fail with e.g. the finalfusion-rust.

Scripts

finalfusion also includes a conversion script ffp-convert to convert between the supported formats.

# convert from fastText format to finalfusion
$ ffp-convert -f fasttext fasttext.bin -t finalfusion embeddings.fifu

ffp-bucket-to-explicit can be used to convert bucket embeddings to embeddings with an explicit ngram lookup.

# convert finalfusion bucket embeddings to explicit
$ ffp-bucket-to-explicit -f finalfusion embeddings.fifu explicit.fifu

ffp-select generates new embedding files based on some embeddings and a word list. Using ffp-select with embeddings with a simple vocab results in a subset of the original embeddings. With subword embeddings, vectors for unknown words in the word list are computed and added to the new embeddings. The resulting embeddings cannot provide representations for OOV words anymore. The new vocabulary covers only the words in the word list.

$ ffp-select large-embeddings.fifu subset-embeddings.fifu words.txt

Finally, the package comes with ffp-similar and ffp-analogy to do analogy and similarity queries.

# get the 5 nearest neighbours of "Tübingen"
$ echo Tübingen | ffp-similar embeddings.fifu
# get the 5 top answers for "Tübingen" is to "Stuttgart" like "Heidelberg" to...
$ echo Tübingen Stuttgart Heidelberg | ffp-analogy embeddings.fifu

Where to go from here

• documentation
• finalfrontier
• finalfusion
• pretrained embeddings