Skip to content

README.md

Latest commit

 

History

History
122 lines (96 loc) · 5.47 KB

README.md

File metadata and controls

122 lines (96 loc) · 5.47 KB
Raw

LLM Inference Gateway

CI License: MIT

An OpenAI-compatible API that sits in front of multiple LLM providers (Groq, Google Gemini, Ollama, with OpenAI and Anthropic pluggable). It:

  • Routes each request to the cheapest model that meets a latency/quality target.
  • Streams responses over Server-Sent Events.
  • Caches semantically similar prompts (local fastembed embeddings + Redis vector search).
  • Rate-limits per API key with atomic Redis counters.
  • Tracks cost per user/key in Postgres and shows it on a React dashboard.

Live showcase: relay-llm-gateway.netlify.app

The gateway is self-hosted (it runs next to your own Postgres, Redis, and provider keys), so there is no public API endpoint. The link above is the landing site, including an in-page snapshot of the dashboard. To call the gateway or load live analytics, clone and run the stack (see Quick start).

Landing

The analytics dashboard that ships with the gateway:

Dashboard

Stack

Concern Choice
Gateway Python 3.12, FastAPI, uvicorn (async)
Providers openai SDK (base_url swap) + Anthropic adapter
Durable store Postgres (request logs, cost, model registry)
Cache / counters Redis Stack (vector cache, rate limit, EWMA)
Embeddings fastembed, BAAI/bge-small-en-v1.5 (local)
Landing + dashboard TanStack Start, React 19, Motion, Recharts (two apps)
Observability structlog (JSON), Prometheus /metrics

Quick start

cp .env.example .env          # add your GROQ / GEMINI keys
make up                       # postgres + redis-stack + gateway (runs migrations)
curl localhost:8000/healthz   # {"status":"ok"}
curl localhost:8000/readyz    # checks postgres + redis

Local dev (datastores in Docker, app with reload):

make install
docker compose up -d postgres redis
make migrate
make run

Project layout

backend/app/
  main.py            FastAPI factory + lifespan
  config.py          pydantic-settings
  routers/           chat, models, admin, analytics, health, metrics
  providers/         base (ABC), openai_compat, anthropic, registry
  services/          routing, model_registry, ratelimit, cost, usage_logger, tokenizer
  services/cache/    semantic, embeddings, index
  db/                base, models, repositories/
  core/              logging, metrics, sse
landing/             marketing site (TanStack Start) - deploys standalone
dashboard/           analytics dashboard (TanStack Start) - ships with the gateway

The frontend is two independent apps so they deploy separately:

  • landing/ is the public showcase. It has no dependency on the gateway and is meant for a host like Vercel or Netlify. It includes an in-page snapshot of the dashboard so visitors can see it without a running backend.
  • dashboard/ ships with the gateway (docker compose up dashboard) and reads /v1/analytics/*. It opens in demo mode with synthetic data; toggle to "live" and paste your GATEWAY_ADMIN_TOKEN to read a running gateway.
cd landing && pnpm install && pnpm dev        # http://localhost:5173
cd dashboard && pnpm install && pnpm dev      # http://localhost:5173

How a request flows

  1. Authenticate the bearer API key (SHA-256 hash lookup).
  2. Enforce rate limits: requests per minute, then reserve tokens per minute.
  3. Route: filter enabled models by quality, latency, context window, circuit state, and provider availability, then sort by estimated cost. The result is an ordered candidate list (cheapest first, then fallbacks).
  4. Check the semantic cache. On a hit, return the stored answer at zero upstream cost and record the avoided spend.
  5. On a miss, call the chosen provider. If one fails before the first token, try the next candidate and record the failure against its circuit breaker.
  6. Compute cost from token usage (provider-reported, or tokenizer-estimated and flagged), write one row to request_logs, and store the answer in the cache.

Endpoints

Method Path Auth Purpose
POST /v1/chat/completions API key OpenAI-compatible chat (stream + non-stream)
GET /v1/models none List registry models
POST /admin/keys admin Create an API key (raw key shown once)
GET /v1/analytics/* admin overview, providers, timeseries, latency, keys
GET /metrics none Prometheus metrics
GET /healthz /readyz none Liveness / readiness

Testing

make install                  # uv venv + dev deps
docker compose up -d postgres redis
make migrate && make test     # 50 tests

Tests run against the compose Postgres and Redis Stack. The semantic-cache tests need RediSearch (Redis Stack); the rate-limit tests use fakeredis[lua].