AI Gateway

A Rust-based API gateway for AI services with protocol conversion between OpenAI and Anthropic formats.

Prerequisites

• Rust (stable, latest version recommended)
• Cargo

Build

cargo build

For release builds:

cargo build --release

Run

cargo run

The server starts on http://0.0.0.0:8080 by default.

Configuration

Configuration is managed via TOML files in the config/ directory.

Server

[server]
host = "0.0.0.0"port = 8080

Logging

[logging]
level = "info"# debug, info, warn, errorformat = "json"# json or pretty

Upstreams

Define AI provider backends:

[[upstreams]]
name = "openai"url = "https://api.openai.com"format = "openai-chat"
[[upstreams]]
name = "anthropic"url = "https://api.anthropic.com"format = "anthropic"

Routes

Map incoming requests to upstreams:

[[routes]]
path = "/v1/chat/completions"input_format = "openai-chat"upstream = "openai"
[[routes]]
path = "/v1/messages"input_format = "anthropic"upstream = "anthropic"

See config/example.toml for all available options.

Metrics

The gateway exposes Prometheus metrics at GET /metrics. This endpoint bypasses access control and does not require authentication.

HTTP Metrics

Metric	Type	Labels	Description
http_requests_total	Counter	method, path, status	Total HTTP requests processed
http_request_duration_seconds	Histogram	method, path, status	Request latency including all middleware
http_requests_in_flight	Gauge	method, path	Currently processing requests

Upstream Metrics

Metric	Type	Labels	Description
upstream_request_duration_seconds	Histogram	upstream, provider	Time to first byte from upstream
upstream_requests_total	Counter	upstream, provider, status	Total requests to upstreams
upstream_errors_total	Counter	upstream, error_type	Upstream errors by type

Streaming Metrics

Metric	Type	Labels	Description
streaming_events_total	Counter	provider	Total SSE events streamed
streaming_bytes_total	Counter	provider	Total bytes streamed
streaming_duration_seconds	Histogram	provider	Full stream duration

Access Control Metrics

Metric	Type	Labels	Description
auth_requests_total	Counter	result	Authentication attempts (allowed/denied)
rate_limit_exceeded_total	Counter	key_name	Rate limit violations by key
quota_exceeded_total	Counter	key_name	Quota violations by key

Example Prometheus Queries

# Request rate per second
rate(http_requests_total[5m])
# 95th percentile latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
# Error rate
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))
# Upstream error rate by provider
sum by (provider) (rate(upstream_errors_total[5m]))
# Streaming throughput in bytes/sec
rate(streaming_bytes_total[5m])
# Authentication failure rate
sum(rate(auth_requests_total{result="denied"}[5m])) / sum(rate(auth_requests_total[5m]))

Example Grafana Dashboard Queries

# Panel: Request Rate
rate(http_requests_total[1m])
# Panel: Latency Heatmap
sum(rate(http_request_duration_seconds_bucket[1m])) by (le)
# Panel: In-Flight Requests
sum(http_requests_in_flight)
# Panel: Upstream Latency by Provider
histogram_quantile(0.50, sum(rate(upstream_request_duration_seconds_bucket[5m])) by (provider, le))
# Panel: Rate Limit Violations
sum by (key_name) (increase(rate_limit_exceeded_total[1h]))

Deployment

Docker

Build the Docker image:

docker build -t ai-gateway .

Run the container:

docker run -d \
-p 8080:8080 \
-v $(pwd)/config:/home/appuser/config:ro \
-e GATEWAY_SERVER_HOST=0.0.0.0 \
-e GATEWAY_UPSTREAMS_OPENAI_API_KEY=sk-... \
-e GATEWAY_UPSTREAMS_ANTHROPIC_API_KEY=sk-ant-... \
ai-gateway

The image uses Alpine Linux and is under 30MB. It runs as a non-root user for security.

Docker Compose

For local deployment:

docker-compose up -d

For deployment with Prometheus and Grafana monitoring:

docker-compose -f examples/docker-compose.monitoring.yml up -d

Access points:

• Gateway: http://localhost:8080
• Prometheus: http://localhost:9090
• Grafana: http://localhost:3000 (admin/admin)

Environment Variables

All configuration can be overridden via environment variables with the GATEWAY_ prefix:

Variable	Description	Example
GATEWAY_SERVER_HOST	Listen address	0.0.0.0
GATEWAY_SERVER_PORT	Listen port	8080
GATEWAY_LOGGING_LEVEL	Log level	info, warn, debug
GATEWAY_LOGGING_FORMAT	Log format	json, pretty
GATEWAY_ENV	Config environment	production
GATEWAY_UPSTREAMS_OPENAI_API_KEY	OpenAI API key	sk-...
GATEWAY_UPSTREAMS_ANTHROPIC_API_KEY	Anthropic API key	sk-ant-...

API keys should always be passed via environment variables, not config files.

Volume Mounting

Mount the config directory as read-only:

-v /path/to/config:/home/appuser/config:ro

The container expects config files at /home/appuser/config/. The gateway loads configuration based on GATEWAY_ENV:

• GATEWAY_ENV=production loads config/production.toml
• Default loads config/default.toml

See examples/config.production.toml for a production-ready configuration template.

Health Check

The gateway exposes a health check endpoint at GET /health:

curl http://localhost:8080/health

Response:

{
"status": "healthy",
"version": "0.1.0",
"uptime_seconds": 1234
}

This endpoint bypasses access control and is suitable for:

• Docker HEALTHCHECK
• Kubernetes liveness/readiness probes
• Load balancer health checks

Kubernetes

Basic deployment hints:

# DeploymentapiVersion: apps/v1kind: Deploymentspec:
template:
spec:
containers:
- name: ai-gatewayimage: ai-gateway:latestports:
- containerPort: 8080env:
- name: GATEWAY_SERVER_HOSTvalue: "0.0.0.0"
- name: GATEWAY_UPSTREAMS_OPENAI_API_KEYvalueFrom:
secretKeyRef:
name: ai-gateway-secretskey: openai-api-keylivenessProbe:
httpGet:
path: /healthport: 8080initialDelaySeconds: 5periodSeconds: 30readinessProbe:
httpGet:
path: /healthport: 8080initialDelaySeconds: 5periodSeconds: 10volumeMounts:
- name: configmountPath: /home/appuser/configreadOnly: truevolumes:
- name: configconfigMap:
name: ai-gateway-config

For Prometheus monitoring, create a ServiceMonitor:

apiVersion: monitoring.coreos.com/v1kind: ServiceMonitorspec:
endpoints:
- port: httppath: /metricsinterval: 15s

Store API keys in Kubernetes Secrets and non-secret configuration in ConfigMaps.

Development

# Run tests
cargo test# Run lints
cargo clippy
# Format code
cargo fmt

License

MIT