LLM Comparison WebApp

This Blazor web application allows you to compare responses from multiple Large Language Models (LLMs) across different providers including OpenAI, Google Gemini, Anthropic Claude, Amazon Bedrock, and locally hosted models via Ollama.

I wanted an easy way to iterate on instruction driven prompts for crafting agents and other specialist personas.

Uses and limitations

At the moment the system allows for a system prompt and one user prompt in order to set the persona and LLM instruction. This is ideal for building text and image driven personas

It will only take a single shot prompt at the moment.

It doesn't have a way to handle media generation.

🚀 Supported Providers & Models

The models here are just the ones I was interested in testing. Adding new models is trivial. Adding new providers is not hard but will require code modifications.

OpenAI

• GPT-4o
• GPT-3.5-Turbo

Google Gemini

• Gemini 2.5 Pro
• Gemini 2.5 Flash
• Gemini 2.5 Flash Lite
• Gemini 1.5 Pro
• Gemini 1.5 Flash

Anthropic Claude (Direct API)

• Claude 4 Opus
• Claude 4 Sonnet
• Claude 3 Opus
• Claude 3 Sonnet
• Claude 3 Haiku

Amazon Bedrock

• Anthropic Models: Claude Opus 4, Claude Sonnet 4, Claude 3.7 Sonnet, Claude 3.5 Sonnet v2, Claude 3.5 Sonnet, Claude 3.5 Haiku, Claude 3 series
• Amazon Nova: Nova Pro, Nova Lite, Nova Micro, Nova Canvas (image generation), Nova Reel (video generation)
• Cross-Region Support: US inference profiles for better availability

Ollama (Local Models)

• Llava (Vision-Language Model)
• Any other models you have installed locally

📋 Prerequisites

• .NET 8 SDK
• Docker (optional, for running local models with Ollama)
• Internet connection for cloud-based models
• API keys for the providers you want to use

⚙️ Configuration

The application uses a scalable configuration system where providers and models are defined separately. You can enable/disable providers and their models will automatically appear in the UI.

A concrete example for the Ollama provider is included in appsettings.json. For development, you should copy the LlmSettings section to appsettings.Development.json and enable the providers you want to use, adding your API keys.

Configuration Structure

Open appsettings.Development.json and configure the providers you want to use. Models are now nested under each provider.

{
"LlmSettings": {
"Providers": {
"OpenAI": {
"ApiKey": "YOUR_OPENAI_API_KEY",
"BaseUrl": "https://api.openai.com/v1",
"Enabled": true,
"Models": [
{
"Id": "gpt-4o",
"DisplayName": "GPT-4o",
"MaxTokens": 128000,
"SupportsImages": true,
"Category": "Chat"
}
]
}
}
}
}

Provider-Specific Setup

🔑 OpenAI

1. Get API key from OpenAI Platform
2. Set OpenAI.ApiKey in configuration
3. Set OpenAI.Enabled to true

🔑 Google Gemini

1. Get API key from Google AI Studio
2. Set Google.ApiKey in configuration
3. Set Google.Enabled to true

🔑 Anthropic Claude

1. Get API key from Anthropic Console
2. Set Anthropic.ApiKey in configuration
3. Set Anthropic.Enabled to true

🔑 Amazon Bedrock

1. Configure AWS credentials:

   aws configure
   # OR set environment variables:# AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION

2. Enable model access in AWS Bedrock Console
3. Set Bedrock.Region to your preferred region (e.g., "us-east-1")
4. Set Bedrock.Enabled to true

🔑 Ollama (Local Models)

1. Install and run Ollama:

   # Using Docker
   docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
   # Pull a vision model
   docker exec ollama ollama pull llava

2. Verify Ollama is running at http://localhost:11434
3. Set Ollama.Enabled to true

Adding providers

Where possible I have tried to use the Microsoft Semantic Kernel. It's early days for this project but I suspect it will mature and become a useful abstraction over all providers. Most connectors were in alpha at the time of writing this so I expect some things to be sub-optimal.

I've had to use more direct integrations for OpenAI and Bedrock.

When adding a new provider do try to use the semantic kernel first. The code is fairly straight forward right now but it might be nice to further abstract the provider/model architecture to allow for drop-in registration.

🚀 Getting Started

1. Clone and Build

   git clone <repository-url>cd LLMComparison.WebApp
   dotnet restore
   dotnet build

2. Configure Providers

   • Copy the LlmSettings section from appsettings.json to appsettings.Development.json.
   • In appsettings.Development.json, enable the providers you want to use and add your API keys.

3. Run the Application

   dotnet run

4. Access the Application

   • Open your browser and navigate to the URL shown in the console.
   • By default, the application runs on https://localhost:7084 and http://localhost:5214. You can find the exact ports in Properties/launchSettings.json.
   • The Home page will show available models from your enabled providers
   • Select a model and enter your prompt to compare responses

🔧 Adding New Models

The application uses a dynamic configuration system. To add new models, add them to the Models list under the desired provider in appsettings.Development.json. The model will automatically appear in the UI if its provider is enabled.

Example model configuration for OpenAI:

"OpenAI": {
"ApiKey": "YOUR_OPENAI_API_KEY",
"BaseUrl": "https://api.openai.com/v1",
"Enabled": true,
"Models": [
{
"Id": "gpt-4o",
"DisplayName": "GPT-4o",
"MaxTokens": 128000,
"SupportsImages": true,
"Category": "Chat"
},
{
"Id": "gpt-3.5-turbo",
"DisplayName": "GPT-3.5-Turbo",
"MaxTokens": 16000,
"SupportsImages": false,
"Category": "Chat"
}
]
}

📱 Features

• Multi-Provider Support: Compare responses across different AI providers
• Vision Models: Upload and analyze images with vision-capable models
• Real-time Comparison: Side-by-side response comparison
• Flexible Configuration: Easy provider and model management
• Local Model Support: Run models locally with Ollama
• Enterprise Ready: Support for AWS Bedrock and other enterprise solutions

📝 Default Prompts

You can create a default-prompts.json file in the project root to set up prompt templates that will appear in a dropdown in the UI. This file should not be committed to version control as it contains your personal prompt preferences.

Setup

1. Copy default-prompts.example.json to default-prompts.json
2. Customize the prompts to suit your needs
3. The first prompt in the file will be loaded as the default

Structured Prompts for Better Agent Building

The prompt system supports structured JSON prompts which are superior for building reliable AI agents. Instead of plain text instructions, you can provide detailed JSON structures that give the AI clear, unambiguous guidance.

Example: Simple Text Prompt

{
"simpleAssistant": {
"name": "General AI Assistant",
"description": "Basic conversational AI",
"systemPrompt": "You are a helpful AI assistant.",
"userPrompt": "Hello! How can I help you today?"
}
}

Example: Structured JSON Prompt (Recommended for Agents)

{
"documentAnalysis": {
"name": "Document Analysis Agent",
"systemPrompt": "You are a document validation expert. Never deviate from instructions.",
"userPrompt": {
"instructions": {
"task": "Analyze documents and extract information in JSON format",
"responseFormat": [
{
"documentValidation": {
"docType": "Document type",
"confidence": "High/Medium/Low",
"reason": "Reasoning for confidence"
}
}
],
"requirements": ["Only respond in JSON", "One object per document"]
}
}
}
}

Supported Prompt Formats

• JSON Structure: For complex agents with detailed instructions
• Plain Text: For simple conversational prompts
• XML Structure: For hierarchical task breakdown
• Markdown: For documentation and formatting-heavy tasks

The example file (default-prompts.example.json) includes examples of all these formats to help you build effective prompts for your specific use cases.

If no default-prompts.json file exists, the UI will show blank text boxes with helpful placeholder text.## 🎯 How to Use

1. Select the model you want to test from the dropdown menu.
2. (Optional) Choose a prompt template from the dropdown, or enter custom prompts directly.
3. Enter or modify the system prompt and user prompt in the text areas.
4. Optionally, attach any images or documents you want to include in the prompt.
5. Click the "Submit" button to send the prompt to the selected model.
6. The response will appear in the "Comparison Results" table below the input form.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.