LLM Inference Gateway

An OpenAI-compatible API that sits in front of multiple LLM providers (Groq, Google Gemini, Ollama, with OpenAI and Anthropic pluggable). It:

• Routes each request to the cheapest model that meets a latency/quality target.
• Streams responses over Server-Sent Events.
• Caches semantically similar prompts (local fastembed embeddings + Redis vector search).
• Rate-limits per API key with atomic Redis counters.
• Tracks cost per user/key in Postgres and shows it on a React dashboard.

Live showcase: relay-llm-gateway.netlify.app

The gateway is self-hosted (it runs next to your own Postgres, Redis, and provider keys), so there is no public API endpoint. The link above is the landing site, including an in-page snapshot of the dashboard. To call the gateway or load live analytics, clone and run the stack (see Quick start).

The analytics dashboard that ships with the gateway:

Stack

Concern	Choice
Gateway	Python 3.12, FastAPI, uvicorn (async)
Providers	openai SDK (base_url swap) + Anthropic adapter
Durable store	Postgres (request logs, cost, model registry)
Cache / counters	Redis Stack (vector cache, rate limit, EWMA)
Embeddings	fastembed, BAAI/bge-small-en-v1.5 (local)
Landing + dashboard	TanStack Start, React 19, Motion, Recharts (two apps)
Observability	structlog (JSON), Prometheus /metrics

Quick start

cp .env.example .env          # add your GROQ / GEMINI keys
make up                       # postgres + redis-stack + gateway (runs migrations)
curl localhost:8000/healthz   # {"status":"ok"}
curl localhost:8000/readyz    # checks postgres + redis

Local dev (datastores in Docker, app with reload):

make install
docker compose up -d postgres redis
make migrate
make run

Project layout

backend/app/
  main.py            FastAPI factory + lifespan
  config.py          pydantic-settings
  routers/           chat, models, admin, analytics, health, metrics
  providers/         base (ABC), openai_compat, anthropic, registry
  services/          routing, model_registry, ratelimit, cost, usage_logger, tokenizer
  services/cache/    semantic, embeddings, index
  db/                base, models, repositories/
  core/              logging, metrics, sse
landing/             marketing site (TanStack Start) - deploys standalone
dashboard/           analytics dashboard (TanStack Start) - ships with the gateway

The frontend is two independent apps so they deploy separately:

• landing/ is the public showcase. It has no dependency on the gateway and is meant for a host like Vercel or Netlify. It includes an in-page snapshot of the dashboard so visitors can see it without a running backend.
• dashboard/ ships with the gateway (docker compose up dashboard) and reads /v1/analytics/*. It opens in demo mode with synthetic data; toggle to "live" and paste your GATEWAY_ADMIN_TOKEN to read a running gateway.

cd landing && pnpm install && pnpm dev        # http://localhost:5173
cd dashboard && pnpm install && pnpm dev      # http://localhost:5173

How a request flows

1. Authenticate the bearer API key (SHA-256 hash lookup).
2. Enforce rate limits: requests per minute, then reserve tokens per minute.
3. Route: filter enabled models by quality, latency, context window, circuit state, and provider availability, then sort by estimated cost. The result is an ordered candidate list (cheapest first, then fallbacks).
4. Check the semantic cache. On a hit, return the stored answer at zero upstream cost and record the avoided spend.
5. On a miss, call the chosen provider. If one fails before the first token, try the next candidate and record the failure against its circuit breaker.
6. Compute cost from token usage (provider-reported, or tokenizer-estimated and flagged), write one row to request_logs, and store the answer in the cache.

Endpoints

Method	Path	Auth	Purpose
POST	/v1/chat/completions	API key	OpenAI-compatible chat (stream + non-stream)
GET	/v1/models	none	List registry models
POST	/admin/keys	admin	Create an API key (raw key shown once)
GET	/v1/analytics/*	admin	overview, providers, timeseries, latency, keys
GET	/metrics	none	Prometheus metrics
GET	/healthz /readyz	none	Liveness / readiness

Testing

make install                  # uv venv + dev deps
docker compose up -d postgres redis
make migrate && make test     # 50 tests

Tests run against the compose Postgres and Redis Stack. The semantic-cache tests need RediSearch (Redis Stack); the rate-limit tests use fakeredis[lua].