OpenProvHop Deep Research

An AI-powered research assistant that performs iterative, deep research on any topic by combining web search, content analysis, and large language models.

🔬 Simple yet powerful research automation in Python

The goal is to provide the simplest implementation of a deep research agent that can refine its research direction over time and dive deep into any topic. Optimized for ease of use and understanding.

Features

• 🔄 Iterative Research: Performs deep research by iteratively generating search queries, processing results, and diving deeper based on findings
• 🎯 Intelligent Query Generation: Uses LLMs to generate targeted search queries based on research goals and previous findings
• ⚙️ Depth & Breadth Control: Configurable parameters to control research scope (breadth: 1-20, depth: 1-10)
• 🔮 Smart Follow-up: Generates follow-up questions to better understand research needs
• 📊 Web Dashboard: Beautiful web interface with real-time progress tracking and organized result tabs
• 🔗 API Server: REST API for integration with other applications
• 🚀 Concurrent Processing: Handles multiple searches and result processing in parallel
• 🤖 Multiple AI Providers: Support for NVIDIA, OpenAI, Fireworks AI, OpenRouter, and local models
• ✨ Advanced Features:
  • Semantic Re-ranking: Orders results by relevance using sentence transformers
  • Smart Deduplication: Removes near-duplicate content automatically
  • Freshness Filtering: Prioritizes recent information (configurable)
  • Provenance Tracking: Transparent source attribution with supporting snippets and confidence scores

Quick Start

1. Installation

git clone https://github.com/Finance-LLMs/deep-research-python.git
cd deep-research-python
pip install -r requirements.txt

2. Configuration

Create a .env.local file with your API keys:

# Required: Web search and scraping
FIRECRAWL_KEY="your_firecrawl_key"# AI Provider (choose one)
OPEN_ROUTER_KEY="your_openrouter_key"# free models available# NVIDIA_API_KEY="your_nvidia_api_key" # Alternative - Llama 3.1 70B, DeepSeek R1# OPENAI_KEY="your_openai_key" # Alternative - GPT-4o-mini# FIREWORKS_KEY="your_fireworks_key" # Alternative - DeepSeek R1# Optional: Advanced features
USE_RERANKING=true # Enable semantic re-ranking (default: true)
DEDUP_THRESHOLD=0.9 # Deduplication threshold (default: 0.9)
MIN_YEAR=2020 # Minimum year for documents (default: 2020)

3. Usage Options

Web Dashboard (Recommended) 🚀

python run_dashboard.py
# Open http://localhost:5000

Command Line

python -m src.run

API Server

python -m src.api
# Server starts on port 3051

AI Model Providers

The system automatically selects the best available model in this order:

1. Custom Model - if CUSTOM_MODEL and OPENAI_ENDPOINT are set
2. DeepSeek R1 (OpenRouter) - if OPEN_ROUTER_KEY is set
3. NVIDIA Llama 3.1 70B - if NVIDIA_API_KEY is set
4. DeepSeek R1 (Fireworks) - if FIREWORKS_KEY is set
5. GPT-4o-mini (OpenAI) - Fallback option

Getting API Keys

OpenRouter

• Visit openrouter.ai
• Free tier includes access to many models
• Excellent DeepSeek R1 performance

NVIDIA API

• Visit build.nvidia.com
• Free access to Llama 3.1 70B and DeepSeek R1
• Great for research applications

Local/Custom Models

OPENAI_ENDPOINT="http://localhost:1234/v1"
CUSTOM_MODEL="your_model_name"
OPENAI_KEY="your_api_key_if_needed"

Usage

Web Dashboard (Recommended) 🚀

Start the interactive web dashboard:

python run_dashboard.py
# Open http://localhost:5000

Features:

• Interactive UI: User-friendly web interface
• Real-time Progress: Live updates with visual progress bars
• Organized Results: Separate tabs for Output, Learnings, Sources, and Feedback
• Download Reports: Export as Markdown files
• Provenance Display: See supporting snippets and confidence scores for each learning

Command Line Interface

python -m src.run

You'll be prompted to enter your research query and configure parameters:

• Breadth: Number of search queries per iteration (recommended: 2-10, default: 4)
• Depth: Number of research iterations (recommended: 1-5, default: 2)
• Mode: Generate report or specific answer

API Server

python -m src.api
# Server starts on port 3051

Endpoints:

# Research with concise answer
curl -X POST http://localhost:3051/api/research \
-H "Content-Type: application/json" \
-d '{"query": "Tesla stock performance 2025", "breadth": 3, "depth": 2}'# Generate detailed report
curl -X POST http://localhost:3051/api/generate-report \
-H "Content-Type: application/json" \
-d '{"query": "AI trends 2025", "breadth": 4, "depth": 2}'

Advanced Features

Enhanced Search Quality

The system includes advanced retrieval processing for higher quality results:

• Semantic Re-ranking: Orders search results by relevance using AI embeddings
• Smart Deduplication: Automatically removes near-duplicate content (configurable threshold)
• Freshness Filtering: Prioritizes recent information while filtering outdated content

Configuration:

USE_RERANKING=true # Enable processing (default: true)
DEDUP_THRESHOLD=0.9 # Similarity threshold (default: 0.9)
MIN_YEAR=2020 # Minimum document year (default: 2020)

Provenance Tracking

Every research finding includes transparent source attribution:

• Source URLs: Direct links to original documents
• Supporting Snippets: Exact 1-2 sentence excerpts supporting each learning
• Confidence Scores: Similarity scores showing reliability (0-100%)
• Matched Terms: Key terms found in supporting evidence

Example Output:

### Learning #1**Finding:** Python 3.12 introduces improved error messages
**Source:** "Python 3.12 now provides more detailed error messages..."
**From:**https://docs.python.org/3.12/whatsnew**Confidence:** 95%

Access Provenance Data:

fromsrc.deep_researchimportdeep_researchresult=awaitdeep_research("Your query", breadth=4, depth=2)
ifresult.learnings_with_provenance:
forprovenanceinresult.learnings_with_provenance:
print(f"Learning: {provenance['learning']}")
print(f"Source: {provenance['source_url']}")
print(f"Evidence: {provenance['supporting_snippet']}")
print(f"Confidence: {provenance['confidence_score']:.1%}")

Configuration

Variable	Description	Default	Options
Core APIs
FIRECRAWL_KEY	Firecrawl API key (required)	-	Your API key
FIRECRAWL_BASE_URL	Custom Firecrawl endpoint	https://api.firecrawl.dev	URL
OPEN_ROUTER_KEY	OpenRouter API key	-	Your API key
NVIDIA_API_KEY	NVIDIA API key	-	Your API key
OPENAI_KEY	OpenAI API key	-	Your API key
FIREWORKS_KEY	Fireworks AI API key	-	Your API key
Local Models
CUSTOM_MODEL	Custom model name	-	Model name
OPENAI_ENDPOINT	Custom endpoint URL	-	http://localhost:1234/v1
Search Quality
USE_RERANKING	Enable retrieval processing	true	true/false
DEDUP_THRESHOLD	Deduplication threshold	0.9	0.0-1.0
MIN_YEAR	Minimum document year	2020	2000-2025
Performance
FIRECRAWL_CONCURRENCY	Concurrent requests	2	1-10
CONTEXT_SIZE	Max context size	128000	Number

Docker Setup

1. Create .env.local file with your API keys
2. Build and run:

docker build -t deep-research-python .
docker run -p 3051:3051 --env-file .env.local deep-research-python

Or with Docker Compose:

docker compose up -d

How It Works

The research process follows these steps:

1. Query Analysis - Takes user query and generates follow-up questions for refinement
2. Search Generation - Creates multiple targeted SERP queries based on research goals
3. Content Retrieval - Searches web using Firecrawl API and scrapes relevant pages
4. Quality Enhancement - Applies semantic re-ranking, deduplication, and freshness filtering
5. Learning Extraction - Analyzes content to extract key insights and learnings
6. Provenance Tracking - Links each learning to supporting source snippets with confidence scores
7. Iterative Deepening - Generates new research directions and repeats if depth > 0
8. Report Generation - Compiles findings into comprehensive markdown reports

flowchart TB
Q[User Query] --> DR[Deep Research]
DR --> SQ[Generate SERP Queries]
SQ --> SR[Search & Scrape]
SR --> QE[Quality Enhancement]
QE --> LE[Extract Learnings]
LE --> PT[Track Provenance]
PT --> D{Depth > 0?}
D -->|Yes| RD[Generate Directions]
RD --> DR
D -->|No| R[Generate Report]

Loading

Performance & Rate Limits

Firecrawl Rate Limits

Configure concurrency based on your plan:

# Free tier
FIRECRAWL_CONCURRENCY=1
# Paid tier or self-hosted
FIRECRAWL_CONCURRENCY=5

Performance Tips

• Start small: Use breadth=2, depth=1 for testing
• Monitor rate limits: Watch for 429 errors and adjust concurrency
• Use faster models: NVIDIA models are generally quicker
• Self-host Firecrawl: For unlimited scraping

Troubleshooting

Common Issues

Issue	Solution
"No model found" error	Ensure at least one AI provider API key is set
Rate limit errors	Reduce FIRECRAWL_CONCURRENCY or upgrade Firecrawl plan
Empty search results	Check Firecrawl API key and connectivity
Import/dependency errors	Run pip install -r requirements.txt
Slow processing	Enable GPU, use smaller parameters, or faster model

Enable Debug Output

Modify logging in src/deep_research.py:

deflog(*args):
print(*args) # Enable all debug output

License

This project is licensed under MIT LICENSE.