Conversation
…, similarity finder processor, and embedding mapper processor
…ng map processor to put images on the scatterplot. Can be used for image walls later.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the scaffolding and features needed to run embedding models alongside
generative LLMs, using the existing
llm.serversconfiguration.Scaffolding
embed()on the LLM client hierarchy, implemented for Ollama (/api/embed)and OpenAI-compatible servers (
/v1/embeddings). Deliberately not routedthrough LangChain because it does not support multimodal inputs yet.
that: embedding models no longer appear in the LLM prompter, and generative
models no longer appear in the embedding processors. Ollama's reported
capabilities are used where available, and else it is checked whether "embed" appears
in the model ID. Imperfect, but there does not seem to be a great alternative at the moment if we also want flexibility...
/v1/models,which is what vLLM serves. Also a bit bloated, but necessary for flexibility.
the Explorer).
New processors
text-embeddings) — one vector per item.image-embeddings,video-embeddings)— one vector per file, sharing an abstract
EmbedMediabase. Files arere-encoded smaller before being sent (Pillow for images, ffmpeg for video),
since they travel base64-encoded inside the request.
embedding-map) — an interactive HTML map of theembeddings reduced to 2D, with UMAP, PCA or t-SNE. Canvas-rendered with
pan/zoom and hover, self-contained with no external dependencies.
embedding-similarity)— scores every item against a text you give it, embedded with the same model
as the parent dataset.
All four write the same output shape, so they all work together.
Also in this PR
umap-learn(for the map processor).llm.available_modelsso itis re-indexed with the new capability data.
Tested with
An SRC server running vLLM with Qwen3-VL-Embedding-8B (note:
--runner poolingis required, otherwise the embeddings endpoint is not served at all), and a
local Ollama with mxbai-embed-large and qwen3-embedding.
Known gaps
some indication of this but it is rarely reliable and still in flux, and the
name heuristic misses embedding models that are not named for it (
bge-m3,all-minilm).which is incorrect for vision and embedding models.
embed()can be moved to an OpenAI-compatible endpoint once Ollamadoes this, which they say they are working on.