RAG-Based Interactive AI — UChicago MS in Applied Data Science

GEN AI Principles · Course Project 1 · University of Chicago · Spring 2026

A Retrieval-Augmented Generation (RAG) system that answers natural-language questions about the UChicago MS in Applied Data Science (MSADS) program, grounding every response in official website content.

---

Deliverables

File	Description
src/scraper.py	Web scraper (requests + BeautifulSoup)
src/embeddings.py	Text chunking + all-mpnet-base-v2 embeddings
src/vector_store.py	ChromaDB setup + cosine retrieval
src/rag_chain.py	RAG orchestration + Responsible AI safeguards
src/evaluate.py	Retrieval Precision@4, MRR metrics
src/app.py	Streamlit chatbot UI
data/knowledge_base.json	Pre-scraped MSADS content (ready to use)
RAG_Documentation.docx	Technical documentation (9 sections, 7+ pages)
MSADS_RAG_Presentation.pptx	8-slide 10-min presentation

---

Quick Start

1. Install dependencies

pip install -r requirements.txt

2. Set your LLM API key

# For OpenAI (default)export OPENAI_API_KEY="your-key-here"# OR for Anthropic Claude — edit rag_chain.py to use the anthropic clientexport ANTHROPIC_API_KEY="your-key-here"

3a. Use pre-scraped data (fastest — no scraping required)

python src/vector_store.py --build # indexes data/knowledge_base.json into ChromaDB

3b. Re-scrape from the live website

python src/scraper.py # saves to data/raw_pages.json
python src/embeddings.py # chunks + embeds → data/chunks.json + embeddings.json
python src/vector_store.py --build # indexes into ChromaDB

4. Launch the Streamlit UI

streamlit run src/app.py

5. Run evaluation

python src/evaluate.py

---

Architecture

User Query
│
▼
Scope Filter ──── off-topic ──→ Polite redirect
│
▼
Query Encoder (all-mpnet-base-v2)
│
▼
ChromaDB Cosine Search (top-K=4)
│
▼
Context Assembly (≤3,000 chars)
│
▼
LLM (GPT-4o / Claude) temp=0.2
│
▼
PII Redaction
│
▼
Answer + Source Citations

---

Responsible AI Design

Safeguard	Implementation
Hallucination guard	System prompt requires answers from context only; says "I don't know" if insufficient
Scope filter	Pre-LLM keyword classifier rejects off-topic queries
PII redaction	Regex strips emails + phone numbers from all generated outputs
Low temperature	temp=0.2 minimises creative deviation from facts

---

Evaluation Results (20-question test set)

Metric	Score
Retrieval Precision@4	0.82
Mean Reciprocal Rank (MRR)	0.79
Answer Keyword Coverage	0.81
Questions answered correctly	18 / 20

---

Project Structure

rag_project/
├── src/
│ ├── scraper.py
│ ├── embeddings.py
│ ├── vector_store.py
│ ├── rag_chain.py
│ ├── evaluate.py
│ └── app.py
├── data/
│ ├── knowledge_base.json ← pre-scraped content
│ ├── raw_pages.json ← generated by scraper.py
│ ├── chunks.json ← generated by embeddings.py
│ └── chroma_db/ ← generated by vector_store.py
├── requirements.txt
└── README.md

---

Technology Stack

• Scraping: requests, beautifulsoup4
• Embeddings: sentence-transformers (all-mpnet-base-v2, 768-dim)
• Vector DB: chromadb (DuckDB+Parquet, HNSW cosine index)
• LLM: OpenAI GPT-4o or Anthropic Claude (configurable in rag_chain.py)
• Evaluation: ragas (optional), custom Precision@K + MRR
• UI: streamlit

---

Key Design Decisions

Chunking: 1500-char sliding window with 100-char overlap. Overlap ensures cross-boundary sentences are preserved. Boundaries snap to whitespace to avoid mid-word cuts.

Embedding model: all-mpnet-base-v2 was chosen over larger models (e.g., BGE-large, E5) because: (a) it runs on CPU without GPU, (b) 768-dim vectors are compact for fast HNSW search, and (c) it achieves competitive recall on domain-specific corpora after fine-tuning.

Top-K=4: Empirically chosen to balance context window utilisation (≤3,000 chars of the ~4,096 context budget) against retrieval breadth.

Temperature=0.2: Near-deterministic generation reduces hallucination risk while preserving natural language fluency.

---

Future Improvements

1. Scheduled re-scraping (weekly cron) to keep KB current
2. BM25 hybrid retrieval for exact-match queries (course codes like ADSP 32007)
3. Cross-encoder re-ranker after initial retrieval
4. FastAPI backend + iframe deployment on the official MSADS website
5. Multilingual support for international applicants
6. Query analytics dashboard to track most-asked topics

---

Authors

GEN AI Principles — Course Project 1
University of Chicago, Physical Sciences Division
MS in Applied Data Science Program, Spring 2026