FastAPIChat is a powerful application for testing and evaluating large language model based chatbots.
- FastAPIChat powers LLM Arena, serving over 10+ LLMs, for diverse and dynamic chatbot interactions
- FastAPIChat enables seamlessly rate LLM generated responses, ensuring continuous improvement based on user feedback.
- [2024/06] 🔥 We released FastAPIChat v1.0.0
More
- [2024/06] We released FastAPIChat v1.0.0 with ChatBot and ChatBot Arena
Key features:
- 💬 LLM Chats(10+ LLMs)
- ⚔️ LLM Arena
- 🗳️ Chat Feedback collection
- ⚡ FastAPI API Documentation and Authentication
FastAPIChat integrates seamlessly with Python programs and libraries and includes out-of-the-box integrations for:
Note: To begin, you'll need an API key from OpenRouter.
- Note: In Dev mode, the default config with backend is SQLite.
- Clone Repo:
git clone https://github.com/yuting1214/FastAPIChat.git
- Configure Virtual Environment (Recommended):
Details
Create Virtual Environment
# macOS/Linux # You may need to run `sudo apt-get install python3-venv` first on Debian-based OSs python3 -m venv .venv # Windows # You can also use `py -3 -m venv .venv` python -m venv .venvActivate the virtual environment:
# macOS/Linux source .venv/bin/activate # Windows .venv\Scripts\activate
- Install Dependencies:
pip install -r requirements.txt
- Update the .env file with
USER_NAME=<API_Doc_Username>
PASSWORD=min=<API_Doc_PASSWORD>
OPENROUTER_API_KEY=<Your_API_KEY>
- Run the application:
python -m backend.app.main --mode dev - Add extra envs in with your environment configurations:
DB_ENGINE=<Your_DB_config>
DB_USERNAME=<Your_DB_config>
DB_PASS=<Your_DB_config>
DB_HOST=<Your_DB_config>
DB_PORT=<Your_DB_config>
DB_NAME=<Your_DB_config>
API_BASE_URL=<Your_Host_URL>
- Run the application:
python -m backend.app.main --mode prod
- Alternatively, use Dockerfile for deployment.
- Control quotas by adjusting limits in
backend/app/core/constants.py:
TEXT_API_QUOTA_LIMIT = <int>
- To add a new LLM model, specify its details in backend/data/llm_models:
{
'llm_model_name': 'Mixtral 8x22B',
'llm_vendor': 'Mistral AI',
'llm_type': 'text',
'api_provider': 'OpenRouter',
'api_endpoint': 'mistralai/mixtral-8x22b-instruct',
}
Utilize Gradio and FastAPI Integration: Seamlessly combine Gradio for the frontend display and FastAPI for robust backend functionality, offering a smooth user experience.
Well-Crafted API Management: Our meticulously designed API ensures efficient data transportation and effective management of API quota usage, providing users with a reliable and hassle-free experience.
Integration of Leading Language Models: Integrate more than 10+ popular language models from various vendors, including OpenAI, Mistral, Meta and Anthropic, enabling users to leverage cutting-edge AI capabilities for text generation.
Comprehensive Testing and Documentation: Benefit from comprehensive testing suites and detailed documentation, empowering developers to understand and contribute to the project with ease, ensuring robustness and maintainability. Third-Party Services: Integrate with third-party APIs for enhanced functionality.
Third-Party Services Integration: Seamlessly integrates with third-party APIs for enhanced functionality.
- Project Details: Heptabase
More Project Details
FastAPIChat/
├── backend/ # Backend directory for the FastAPI application
│ ├── app/ # Main application directory
│ │ ├── __init__.py # Initialization file for the app package
│ │ ├── api/ # Directory for API related code
│ │ │ ├── __init__.py # Initialization file for the API package
│ │ │ ├── v1/ # Version 1 of the API
│ │ │ │ ├── __init__.py # Initialization file for the v1 API package
│ │ │ │ ├── endpoints/ # Directory for API endpoint definitions
│ │ │ │ │ ├── __init__.py # Initialization file for endpoints package
│ │ │ │ │ ├── text_generation.py # Endpoints for text generation
│ │ │ │ │ ├── llm_management.py # Endpoints for LLM model management
│ │ │ │ │ ├── api_usage.py # Endpoints for API usage management
│ │ │ │ │ ├── api_calldetail.py # Endpoints for API Detail management
│ │ │ │ │ ├── message.py # Endpoints for messasge management
│ │ │ │ │ ├── chat.py # Endpoints for chat management
│ │ │ │ │ ├── rating.py # Endpoints for rating management
│ │ │ │ │ ├── user.py # Endpoints for user management
│ │ │ │ │ ├── quota.py # Endpoints for quota management
│ │ │ │ │ ├── doc.py # Endpoints for OpenAPI doc management
│ │ ├── dependencies/ # Directory for dependency management
│ │ │ ├── __init__.py # Initialization file for dependencies package
│ │ │ ├── database.py # Database connection and session management
│ │ │ ├── rate_limiter.py # Rate limiting logic
│ │ ├── core/ # Core application logic
│ │ │ ├── __init__.py # Initialization file for core package
│ │ │ ├── constant.py # Constant settings
│ │ │ ├── config.py # Configuration settings
│ │ │ ├── init_setting.py # Init settings with user's input
│ │ ├── models/ # Directory for SQLAlchemy models
│ │ │ ├── __init__.py # Initialization file for models package
│ │ │ ├── user.py # User model
│ │ │ ├── quota.py # Quota model
│ │ │ ├── message.py # Message model
│ │ │ ├── api_usage.py # API usage model
│ │ │ ├── llm.py # LLM model
│ │ │ ├── chat.py # Chat model
│ │ │ ├── rating.py # rating model
│ │ ├── schemas/ # Directory for Pydantic schemas
│ │ │ ├── __init__.py # Initialization file for schemas package
│ │ │ ├── user.py # Schemas for user data
│ │ │ ├── quota.py # Schemas for quota data
│ │ │ ├── message.py # Schemas for message data
│ │ │ ├── api_usage.py # Schemas for API usage data
│ │ │ ├── llm.py # Schemas for LLM model data
│ │ │ ├── llm_message.py # Schemas for LLM generated output data
│ │ │ ├── chat.py # Schemas for chat data
│ │ │ ├── rating.py # Schemas for rating data
│ │ ├── crud/ # Directory for CRUD operations
│ │ │ ├── __init__.py # Initialization file for crud package
│ │ │ ├── user.py # CRUD for user management
│ │ │ ├── quota.py # CRUD for quota management
│ │ │ ├── message.py # CRUD for message management
│ │ │ ├── api_usage.py # CRUD for API usage management
│ │ │ ├── llm.py # CRUD for LLM model management
│ │ │ ├── llm_message.py # CRUD for LLM generated output management
│ │ │ ├── chat.py # CRUD for chat management
│ │ │ ├── rating.py # CRUD for rating management
│ │ ├── main.py # Main FastAPI application file
│ ├── data/ # Directory for data when initiating db
│ │ ├── __init__.py # Initialization file for data package
│ │ ├── llm_models.py # LLM models infomation
│ ├── security/ # New directory for authentication and authorization
│ │ ├── __init__.py # Initialization file for security package
│ │ ├── authentication.py # Authentication logic
│ │ ├── authorization.py # Authorization logic
│ ├── tests/ # Directory for test files
│ │ ├── __init__.py # Initialization file for tests package
│ │ ├── test_text_generation.py # Tests for text generation endpoints
├── frontend/ # Frontend directory for the Gradio app
│ ├── __init__.py # Initialization file for frontend package
│ ├── gradio/ # Main gradio UI directory
│ │ ├── __init__.py # Initialization file for the gradio package
│ │ ├── text/ # Directory for ChatBot UI related code
│ │ │ ├── __init__.py # Initialization file for the text package
│ │ │ ├── event_listeners.py # │ │ │ ├── text_generation.py # │ │ │ ├── text_generation_arena.py │ ├── login/ # Main login UI directory
│ │ ├── __init__.py # Initialization file for the gradio package
│ │ ├── static/ # Directory for static files
│ │ │ ├── style.css # CSS for login UI │ │ ├── templates/ # Directory for HTML templates
│ │ │ ├── base.html # HTML base template
│ │ │ ├── login.html # HTML login template ├── llm/
│ ├── __init__.py # Makes llm a Python package
│ ├── llm_text_chain.py # Module for LLM text generation integration
│ └── prompt/ # Folder for prompt handling
│ ├── __init__.py # Initializes the prompt package
│ ├── base_text_templates.py# Stores base prompt templates for text generation
│ └── examples/ # Directory for few-shot examples used by the chain
│ └── deprecated/ # Directory for deprecated prompts
│ └── vendors/ # Directory for vendor-specific LLM configurations
│ ├── __init__.py # Makes vendors a Python package
│ └── openrouter.py # Configurations and usage for OpenRouter as LLM provider
├── .env # Environment variables file
├── .gitignore # Git ignore file
├── requirements.txt # Python dependencies file
├── Dockerfile # Docker configuration
├── README.md # Project README file
Integraion of local LLM: Integrate with Ollama to enable users to run LLMs locally and battle with LLMs from API.
Integraion of RAG LLM: Enable users to compare different configuration of RAG LLM in LLM Arena.
Integration of Redis or other NoSQL Databases: Incorporate Redis or other NoSQL databases to efficiently track and store user history data, enabling personalized experiences and insights for users.
History Message Management: Allow user to fetch back the previous messages from database and continue the chat.
Rate Limiter: Integrate rate limiter for API protection.
We welcome contributions from the community! Whether it's bug fixes, feature enhancements, or documentation improvements, feel free to open a pull request.
Designed with ❤️ by Mark Chen