Skip to content

Support for embedding models via Ollama and OpenAI-like endpoints - #627

Open
sal-uva wants to merge 6 commits into
masterfrom
embedding-llms
Open

sal-uva wants to merge 6 commits into
masterfrom
embedding-llms

Conversation

@sal-uva

@sal-uva sal-uva commented Sep 17, 2026

Copy link
Copy Markdown
Member

Adds the scaffolding and features needed to run embedding models alongside
generative LLMs, using the existing llm.servers configuration.

Scaffolding

  • embed() on the LLM client hierarchy, implemented for Ollama (/api/embed)
    and OpenAI-compatible servers (/v1/embeddings). Deliberately not routed
    through LangChain because it does not support multimodal inputs yet.
  • Models are now classified as embedding or generative, and processors filter on
    that: embedding models no longer appear in the LLM prompter, and generative
    models no longer appear in the embedding processors. Ollama's reported
    capabilities are used where available, and else it is checked whether "embed" appears
    in the model ID. Imperfect, but there does not seem to be a great alternative at the moment if we also want flexibility...
  • OpenAI-compatible model discovery now also tries the standard /v1/models,
    which is what vLLM serves. Also a bit bloated, but necessary for flexibility.
  • Embeddings can be written back to the top dataset as annotations (hidden in
    the Explorer).

New processors

  1. Generate text embeddings (text-embeddings) — one vector per item.
  2. Generate image / video embeddings (image-embeddings, video-embeddings)
    — one vector per file, sharing an abstract EmbedMedia base. Files are
    re-encoded smaller before being sent (Pillow for images, ffmpeg for video),
    since they travel base64-encoded inside the request.
  3. Plot embeddings (embedding-map) — an interactive HTML map of the
    embeddings reduced to 2D, with UMAP, PCA or t-SNE. Canvas-rendered with
    pan/zoom and hover, self-contained with no external dependencies.
  4. Calculate cosine similarity with text embedding (embedding-similarity)
    — scores every item against a text you give it, embedded with the same model
    as the parent dataset.

All four write the same output shape, so they all work together.

Also in this PR

  • New dependency: umap-learn (for the map processor).
  • VERSION bump to 1.58 plus a migration that clears llm.available_models so it
    is re-indexed with the new capability data.
  • Tests for the client layer, the processors and the annotation writing.

Tested with

An SRC server running vLLM with Qwen3-VL-Embedding-8B (note: --runner pooling
is required, otherwise the embeddings endpoint is not served at all), and a
local Ollama with mxbai-embed-large and qwen3-embedding.

Known gaps

  • Telling embedding models from generative ones is still not robust. LiteLLM has
    some indication of this but it is rarely reliable and still in flux, and the
    name heuristic misses embedding models that are not named for it (bge-m3,
    all-minilm).
  • The "Capabilities" column on the LLM settings page now always reads "text",
    which is incorrect for vision and embedding models.
  • Ollama's embed() can be moved to an OpenAI-compatible endpoint once Ollama
    does this, which they say they are working on.

…, similarity finder processor, and embedding mapper processor
- remove models.py, integrate embedding() into OpenAI and Ollama clients. Needs some unfortunate scaffolding for vLLM though...
- abstract embed_media processor with embed_image and embed_video as child processors
- improve annotation writing
- misc changes
@sal-uva
sal-uva requested a review from dale-wahl September 17, 2026 00:58
…ng map processor to put images on the scatterplot.

Can be used for image walls later.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant