README.md

Private LLM Operator

Deploy private, self-hosted LLM inference endpoints on Kubernetes -- one CR, one cluster, fully yours.

Part of the ApeiroRA Platform Mesh ecosystem

---

What is Private LLM Operator?

Private LLM Operator is a Kubernetes operator that turns a simple custom resource into a fully provisioned, token-secured llama.cpp inference server. It is designed to run inside the ApeiroRA Platform Mesh as a Managed Service Provider (MSP), but works equally well as a standalone operator on any Kubernetes cluster.

Create an LLM endpoint in seconds:

apiVersion: llm.privatellms.msp/v1alpha1
kind: LLMInstance
metadata:
  name: my-llm
spec:
  model: gemma-3-4b-it
  replicas: 2

The operator handles everything: model download, Deployment, Service, Ingress routing via Traefik, and bearer-token authentication -- all reconciled continuously.

Key Features

• Declarative LLM provisioning -- one LLMInstance CR per inference endpoint
• Token-based API security -- APITokenRequest CRs mint bearer tokens backed by Kubernetes Secrets
• OpenAI-compatible API -- works out of the box with any OpenAI SDK client
• Multiple model support -- TinyLlama, Phi-2, Gemma 3 (1B, 4B, 12B) with GGUF quantization
• Horizontal scaling -- set spec.replicas and the operator handles the rest
• Platform Mesh integration -- sync agent, marketplace metadata, and portal UI included
• OCM delivery -- package and deploy via Open Component Model with KRO resource graphs
• OpenTelemetry tracing -- built-in OTLP export for observability

Architecture

                    Platform Mesh (KCP)                          MSP Cluster
               ┌──────────────────────────┐           ┌──────────────────────────────┐
               │                          │           │                              │
 User/Portal   │  ┌──────────────────┐    │   Sync    │  ┌────────────────────────┐  │
 ───────────►  │  │ Customer         │    │ ◄────────►│  │ Sync Agent             │  │
               │  │ Workspace        │    │   Agent   │  └────────────┬───────────┘  │
               │  │                  │    │           │               │              │
               │  │  LLMInstance CR  │    │           │  ┌────────────▼───────────┐  │
               │  │  APITokenReq CR  │    │           │  │ Private LLM Operator   │  │
               │  └──────────────────┘    │           │  │                        │  │
               │                          │           │  │  ┌─ Deployment ──────┐ │  │
               │  ┌──────────────────┐    │           │  │  │ llama.cpp server  │ │  │
               │  │ Provider         │    │           │  │  │ + model download  │ │  │
               │  │ Workspace        │    │           │  │  └──────────────────-┘ │  │
               │  │                  │    │           │  │  ┌─ Service ──────────┐│  │
               │  │  APIExport       │    │           │  │  │ ClusterIP :8000   ││  │
               │  │  ProviderMeta    │    │           │  │  └───────────────────-┘│  │
               │  │  ContentConfig   │    │           │  │  ┌─ Ingress ─────────┐│  │
               │  └──────────────────┘    │           │  │  │ /llm/<slug>       ││  │
               └──────────────────────────┘           │  │  └───────────────────-┘│  │
                                                      │  │  ┌─ Auth Middleware ──┐│  │
                                                      │  │  │ ForwardAuth + Token││  │
                                                      │  │  └───────────────────-┘│  │
                                                      │  └────────────────────────┘  │
                                                      └──────────────────────────────┘

Supported Models

Model	ID	Size	Quantization	Source
TinyLlama 1.1B Chat	tinyllama	~0.6 GB	Q4_K_M	HuggingFace
Phi-2	phi-2	~1.6 GB	Q4_0	HuggingFace
Gemma 3 1B IT	gemma-3-1b-it	~0.8 GB	Q4_K_M	HuggingFace
Gemma 3 4B IT	gemma-3-4b-it	~2.5 GB	Q4_K_M	HuggingFace
Gemma 3 12B IT	gemma-3-12b-it	~7.3 GB	Q4_K_M	HuggingFace
Qwen3 4B	qwen3-4b	~2.4 GB	Q4_K_M	HuggingFace

Quick Start

Install with Helm

helm upgrade --install private-llm \
  oci://ghcr.io/apeirora/charts/private-llm-operator \
  --namespace private-llm-system --create-namespace \
  --set PUBLIC_HOST=llm.example.com

Create an LLM Instance

kubectl apply -f - <<EOF
apiVersion: llm.privatellms.msp/v1alpha1
kind: LLMInstance
metadata:
  name: my-llm
spec:
  model: tinyllama
  replicas: 1
EOF

Get an API Token

kubectl apply -f - <<EOF
apiVersion: llm.privatellms.msp/v1alpha1
kind: APITokenRequest
metadata:
  name: my-token
spec:
  instanceName: my-llm
EOF

# Wait for the token to be provisioned
kubectl wait apitokenrequest/my-token --for=jsonpath='{.status.phase}'=Ready --timeout=60s

# Retrieve credentials
SECRET=$(kubectl get apitokenrequest my-token -o jsonpath='{.status.secretName}')
export OPENAI_API_KEY=$(kubectl get secret "$SECRET" -o jsonpath='{.data.OPENAI_API_KEY}' | base64 -d)
export OPENAI_API_URL=$(kubectl get secret "$SECRET" -o jsonpath='{.data.OPENAI_API_URL}' | base64 -d)

Call the API

curl -sS "$OPENAI_API_URL/v1/chat/completions" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Hello!"}]}'

Documentation

Guide	Description
Architecture	System design, component interactions, data flow
API Reference	CRD specifications and API details
Resource Guide	CRD overview with examples
User Guide	Platform Mesh integration and portal usage
Helm Installation	Production Helm deployment
OCM Installation	Open Component Model delivery
Local Development	Kind cluster setup for testing
Remote Deployment	Remote cluster with Flux GitOps
Release Flow	CI/CD pipeline and versioning
Versioning	SemVer and Conventional Commits policy
Contributing	How to contribute

Helm Charts

This repository ships four Helm charts, each handling a distinct layer:

Chart	Purpose	Registry
private-llm-operator	Core operator + optional Traefik + portal content server	oci://ghcr.io/apeirora/charts/private-llm-operator
private-llm-sync-agent	KCP sync agent + PublishedResource definitions	oci://ghcr.io/apeirora/charts/private-llm-sync-agent
private-llm-pm-integration	Platform Mesh metadata (APIExport, ProviderMetadata, ContentConfiguration)	oci://ghcr.io/apeirora/charts/private-llm-pm-integration
private-llm-operator-ocm	OCM Component + KRO ResourceGraphDefinition for supply-chain delivery	oci://ghcr.io/apeirora/charts/private-llm-operator-ocm

Project Structure

.
├── api/v1alpha1/           # CRD type definitions (LLMInstance, APITokenRequest)
├── cmd/main.go             # Operator entrypoint
├── internal/
│   ├── controller/         # Reconcilers for LLMInstance and APITokenRequest
│   └── auth/               # Lightweight bearer-token auth server
├── charts/
│   ├── private-llm-operator/        # Core Helm chart
│   ├── private-llm-sync-agent/      # Sync agent chart
│   ├── private-llm-pm-integration/  # Platform Mesh metadata chart
│   └── private-llm-operator-ocm/    # OCM delivery chart
├── config/                 # Kustomize manifests (CRDs, RBAC, samples)
├── docs/                   # Documentation
└── ocm/                    # OCM bootstrap manifests

Contributing

See CONTRIBUTING.md for guidelines on how to contribute to this project.

License

This project is licensed under the Apache License 2.0.

---

_{Built with care by ApeiroRA}