Vectorize MCP Worker (Python)

Production-Grade Hybrid RAG with Multimodal Support on Cloudflare Edge in Python.

Table of Contents

• Features
• Setup
• API Endpoints
• MCP Integration
• Project Structure
• Architecture
• Credits

Features

• Hybrid Search (Vector + BM25) with Reciprocal Rank Fusion
• Multimodal Image Processing (Llama 4 Scout)
• Cross-Encoder Reranking (bge-reranker-base)
• Recursive Chunking with 15% overlap
• One-time License System
• Interactive Dashboard at /dashboard
• MCP tool integration (via vectorize-mcp-tool package)

Setup

Prerequisites

• A Cloudflare account with Workers, Vectorize, D1, and Workers AI enabled
• uv (Python package manager)
• wrangler (Cloudflare CLI) -- needed to create cloud resources

Install uv

macOS:

brew install uv

Linux:

curl -LsSf https://astral.sh/uv/install.sh | sh

Install Wrangler (Cloudflare CLI)

Wrangler is needed to create and manage Cloudflare resources (D1 databases, Vectorize indexes, secrets). Install it globally via npm:

macOS:

brew install node # if you don't have Node.js
npm install -g wrangler

Linux (Debian/Ubuntu):

curl -fsSL https://deb.nodesource.com/setup_lts.x | sudo -E bash -
sudo apt-get install -y nodejs
npm install -g wrangler

Linux (any distro, via nvm):

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash
source~/.bashrc # or restart your shell
nvm install --lts
npm install -g wrangler

Then authenticate with your Cloudflare account:

wrangler login

This opens a browser window to authorize the CLI against your account.

Install Python Dependencies

uv init
uv tool install workers-py

Create Cloudflare Resources

The project requires three Cloudflare services: Vectorize (vector database), D1 (SQL database), and Workers AI (inference). Workers AI is enabled automatically; the other two need to be created. A fourth service, multimodal-pro-worker, is optional and enables image features.

1. Create the Vectorize Index

This creates the vector store used for semantic search with 384-dimension BGE embeddings:

wrangler vectorize create mcp-knowledge-base --dimensions=384 --metric=cosine

2. Create the D1 Database

D1 stores document metadata, BM25 keyword indexes, and license records:

wrangler d1 create mcp-knowledge-db

This outputs a database ID. Copy it and update wrangler.toml:

[[d1_databases]]
binding = "DB"database_name = "mcp-knowledge-db"database_id = "paste-your-database-id-here"

3. Apply the Database Schema

Run the schema migration against the remote D1 database:

wrangler d1 execute mcp-knowledge-db --remote --file=./schema.sql

This creates the following tables:

Table	Purpose
documents	Ingested document chunks and metadata
keywords	BM25 term-frequency index per document
doc_stats	Corpus-level statistics (total docs, avg length)
term_stats	Document-frequency per term
licenses	API license keys and quotas

4. Set the API Key Secret

Set the API key that protects write operations (/ingest/document, /license/*):

wrangler secret put API_KEY

You will be prompted to enter the secret value interactively. This is stored encrypted and never appears in wrangler.toml.

5. Deploy the Multimodal Worker (optional -- for image features)

The image endpoints (/ingest/image, /search/similar-images) require a separate worker that processes images via Llama 4 Scout. If you only need text search, skip this step.

cd multimodal-pro-worker
uv tool install workers-py # if not already installed
uv run pywrangler deploy
cd ..

This deploys the multimodal-pro-worker which the main worker calls via a Service Binding. Without it, text features work normally and image endpoints return a 501 error with a clear message.

Deploy and Debug

Build and deploy to Cloudflare's global edge network:

uv run pywrangler deploy

Stream live logs from the deployed worker:

wrangler tail --format=json

For a complete step-by-step guide, see docs/quickstart.md. For production deployment, security, and monitoring, see docs/production.md.

API Endpoints

Endpoint	Method	Description
/	GET	API documentation
/health/check	GET	Health check
/dashboard	GET	Interactive playground UI
/llms.txt	GET	AI search engine info
/stats/index	GET	Index statistics
/search/multimodal	POST	Hybrid search (documents + images)
/search/documents	POST	Documents-only search
/search/similar-images	POST	Find similar images
/ingest/document	POST	Document ingestion
/ingest/image	POST	Image ingestion (requires multimodal-pro-worker)
/get/document/:id	GET	Get document by ID
/get/image/:id	GET	Get image by ID
/list/documents	GET	List documents
/delete/document/:id	DELETE	Delete document
/delete/license/:key	DELETE	Delete license
/license/validate	POST	Validate license
/license/create	POST	Create license
/license/list	GET	List licenses
/license/revoke	POST	Revoke license
/init/reset-passphrase	POST	Set passphrase for reset endpoints
/reset/all	POST	Reset all (passphrase-gated)
/reset/documents	POST	Reset documents (passphrase-gated)
/reset/licenses	POST	Reset licenses (passphrase-gated)

Search results return snippets and metadata instead of full content. The passphrase-gated reset endpoints (/init/reset-passphrase, /reset/*) prevent accidental AI deletion.

MCP Integration

All MCP operations are performed through the vectorize-mcp-tool package, which provides both a CLI and a FastMCP stdio server. The MCP tool dispatches directly to the worker's REST endpoints -- there are no /mcp/* proxy endpoints on the worker.

Operations: search_multimodal, search_documents, ingest, ingest_image, stats, delete, get_document, get_image, list_documents, license_validate, license_create, license_list, license_revoke, delete_license, reset_all, reset_documents, reset_licenses.

Install and Use

# Installcd vectorize-mcp-tool && pip install -e .# CLI usage
vectorize-mcp --url https://your-worker.workers.dev --api-key YOUR_KEY health
vectorize-mcp --url https://your-worker.workers.dev --api-key YOUR_KEY search multimodal "your query"# MCP server for Cursor
vectorize-mcp-server # reads VECTORIZE_URL and VECTORIZE_API_KEY from env

See vectorize-mcp-tool/README.md for full documentation.

Project Structure

vectorize-mcp-worker-python/
├── src/ # Main worker source
│ ├── entry.py # HTTP routing and Worker entrypoint
│ ├── bindings/ # Cloudflare binding wrappers (FFI)
│ ├── hybrid_search.py # Vector + BM25 + RRF fusion
│ ├── ingestion.py # Document/image ingestion pipeline
│ └── ...
├── multimodal-pro-worker/ # Separate worker for image processing
│ ├── src/entry.py # Llama 4 Scout vision + OCR + embedding
│ └── wrangler.toml # Independent worker config
├── vectorize-mcp-tool/ # CLI + MCP server package
│ ├── src/vectorize_mcp_tool/ # Client, server, CLI
│ └── tests/ # MCP tool unit tests
├── tests/ # Test suite
│ ├── unit/ # Unit tests (no network)
│ ├── integration/ # Contract tests (worker <-> MCP tool sync)
│ └── e2e/ # E2E tests + benchmarks (live worker)
├── schema.sql # D1 database DDL
├── wrangler.toml.example # Template config (copy to wrangler.toml)
└── docs/ # All documentation

Testing

# Unit tests (no network, fast)
uv run pytest tests/unit/ --cov=src --cov-report=term-missing
# Integration/contract tests (no network)
uv run pytest tests/integration/
# MCP tool testscd vectorize-mcp-tool && uv run pytest tests/
# E2E tests (requires live worker)
VECTORIZE_E2E_URL=https://... VECTORIZE_E2E_API_KEY=... uv run pytest tests/e2e/ -m "not benchmark"# Benchmarks (persists results, detects regressions)
VECTORIZE_E2E_URL=https://... VECTORIZE_E2E_API_KEY=... uv run pytest tests/e2e/ -m benchmark

Contract tests in tests/integration/ verify that the worker and MCP tool stay in sync. Any endpoint change in the worker that isn't reflected in the MCP tool will fail these tests.

Architecture

See docs/ for detailed documentation:

• docs/quickstart.md -- Step-by-step first-time setup and endpoint testing
• docs/production.md -- Production deployment, security, monitoring, operations
• docs/abstraction_layers.md -- Protocol design and FFI patterns