This Proof of Concept (POC) demonstrates an AI-powered Model Risk Management (MRM) workflow that automatically identifies whether a quantitative method qualifies as a "model" based on regulatory guidelines
docker run --name mrm-postgres
-e POSTGRES_USER=mrm_user
-e POSTGRES_PASSWORD=mrm_pass
-e POSTGRES_DB=mrm_db
-p 5432:5432
-d postgres:16
docker pull postgres:14
docker run -d
--name mrm_postgres
-e POSTGRES_USER=mrm_user
-e POSTGRES_PASSWORD=mrm_pass
-e POSTGRES_DB=mrm_db
-p 5432:5432
postgres:14
docker ps | grep mrm_postgres
docker exec -it mrm_post gres psql -U mrm_user -d mrm_db
| Component | Technology | Version | Purpose |
|---|---|---|---|
| LLM Model | OpenAI GPT-4o-mini | gpt-4o-mini-2024-07-18 | Core reasoning, rule assessment, decision synthesis |
| Framework | LangGraph | latest | Stateful multi-step workflow with conditional routing |
| Output Parsing | LangChain + Pydantic | latest | Structured, type-safe LLM responses |
| API Client | OpenAI Python SDK | >=1.0.0 | LLM API communication |
| Technique | Implementation | Benefit |
|---|---|---|
| Structured Prompt Engineering | Custom prompts with explicit output schemas (SectionCollection, RuleSet, RuleAssessment, FinalDecision) | Ensures LLM returns predictable, validated JSON |
| State Machine Workflow | LangGraph StateGraph with conditional edges | Manages complex decision flow with clear branching logic |
| Multi-step Chain-of-Thought | Document → Sections → Rules → Assessment → Decision | Breaks complex classification into manageable steps |
| Clarification Loop with Guardrails | Max 2 rounds, counter in state, automatic escalation | Prevents infinite loops, ensures bounded execution |
| Context Window Optimization | Only relevant sections passed per rule assessment | Reduces token usage, improves accuracy |
| Few-shot Learning | Guidelines contain examples of GOOD/BAD descriptions | Improves classification accuracy without fine-tuning |
| Selective Re-evaluation | Only affected rules re-assessed after clarification | Reduces API calls by 60-70% in clarification rounds |
| File-based Caching | Generated rules cached to cached_rules.json | Eliminates redundant LLM calls (80% cost reduction) |
| Structured Output Validation | Pydantic models with field validation (confidence 0-1, literals) | Prevents hallucinated values, ensures data quality |