arXiv Papers Vector Search

A semantic search demo for arXiv research papers using vector embeddings. Search for papers by meaning rather than keywords.

Technologies

• Backend: FastAPI, ChromaDB, sentence-transformers
• Frontend: React, TypeScript, Vite, Bun
• Embeddings: all-MiniLM-L6-v2 (384 dimensions)
• Package Management: uv (Python), Bun (JavaScript)
• Deployment: Docker Compose

Quick Start (Local)

# Start the application
docker compose up -d --build
# Access the frontend
open http://localhost

Usage

Generate Embeddings

Download the arXiv metadata dataset and generate embeddings locally before deployment:

# Download dataset (optional - ~4GB)# https://www.kaggle.com/datasets/Cornell-University/arxiv# Install dependencies
uv sync
# Generate embeddings (default: 1000 papers)
uv run python scripts/embed_arxiv.py
# Generate more embeddings with year filter
uv run python scripts/embed_arxiv.py --limit 10000 --year 2016
# Options:# --limit, -l Number of papers (0 = all)# --year, -y Filter papers from this year onwards# --clean, -c Clear existing embeddings first

EC2 Deployment

Prerequisites:

• Ubuntu EC2 instance (t3.medium recommended)
• 20GB+ EBS storage
• Security group: ports 22, 80, 8000 open

Deploy:

# 1. Upload project to EC2
scp -i ~/.ssh/your-key.pem -r . ubuntu@YOUR_EC2_IP:~/vector_database
# 2. SSH and run setup
ssh -i ~/.ssh/your-key.pem ubuntu@YOUR_EC2_IP
cd~/vector_database
chmod +x scripts/ec2-setup.sh
./scripts/ec2-setup.sh
# 3. Log out and back in, then start
newgrp docker # or logout/login
docker compose -f docker-compose.prod.yml up -d --build
# 4. Verify
curl http://localhost:8000/health

The setup script automatically detects your EC2 public IP and configures the .env file.

Project Structure

.
├── backend/
│ ├── api.py # FastAPI application
│ ├── chroma_db/ # Vector database (generated)
│ └── Dockerfile
├── frontend/
│ ├── src/
│ └── Dockerfile
├── scripts/
│ ├── embed_arxiv.py # Embedding generation
│ └── ec2-setup.sh # EC2 setup script
├── docker-compose.yml # Local development
└── docker-compose.prod.yml # Production deployment

API Endpoints

• GET /health - Health check with collection count

• POST /search - Search papers by query

  {"query": "neural networks", "n_results": 10}

License

MIT