Machine Learning Infrastructure Service (MLIS)
This document records the major architectural decisions behind the Machine Learning Infrastructure Service. Each ADR describes the context that motivated the decision, the decision itself, alternatives considered, and the resulting consequences.
Status: Accepted
The system requires a durable store for job lifecycle metadata. Jobs must survive process crashes, container restarts, and node failures. The store must also support atomic job claiming, lease ownership, bounded retries, and multi-worker coordination.
Use Postgres as the durable job metadata store. Postgres owns all job lifecycle fields:
id status task_type
worker_uid attempt_no lease_expires_at
heartbeat_at created_at updated_at
input_uri output_uri payload_size_bytes
content_type
| Option | Pros | Cons |
|---|---|---|
| Redis queue | Simple, fast | Poor durability; complex recovery; limited query visibility |
| Kafka | Scalable event system | Poor fit for mutable job state; complex worker coordination |
Postgres provides strong consistency, transactional job claiming, rich introspection via SQL, and straightforward recovery logic without additional coordination infrastructure.
Status: Accepted
ML workloads frequently include large payloads — long prompts, feature tensors, inference results. Storing these inside the database causes DB bloat, expensive queries, and limits the ability to scale storage independently of scheduling.
Divide the system into two distinct planes:
| Plane | Storage | Contents |
|---|---|---|
| Control Plane | Postgres | Job lifecycle, scheduling metadata, artifact URIs |
| Data Plane | Artifact Store | Input payloads, inference results |
The database remains lightweight and focused on scheduling. Large payloads are handled by purpose-built storage. Artifacts become independently inspectable. This pattern mirrors architectures used in Kubeflow, Ray, and MLflow Pipelines.
Status: Accepted
Workers require access to job input payloads. The two primary approaches are: embed the payload directly in the database row, or store the payload in artifact storage and reference it by URI.
Payloads are stored in the Artifact Store and referenced by logical URI. Workers resolve the payload using a priority-ordered fallback:
1. input_uri → ArtifactStore.get_json(uri)
2. fallback → DB payload column (backward compatibility)
Example URI:
fs://jobs/<job_id>/input.json
| Option | Pros | Cons |
|---|---|---|
| Inline DB payload | Simple | Large payloads degrade DB performance; difficult to scale |
| Message queue payloads | Streaming capability | No durable artifact storage; poor reproducibility |
Payloads become durable, independently inspectable, and reproducible. The database row is reduced to metadata only.
Status: Accepted
Inference results can be arbitrarily large — generated text, embeddings, structured model outputs. Storing results inline in the database would cause unbounded row growth and query overhead.
Workers write results to the Artifact Store after execution:
jobs/<job_id>/output.json
The database stores only the reference:
output_uri TEXT
Results can be arbitrarily large without affecting DB performance. Output artifacts are reusable and can be consumed by external systems or downstream pipeline stages.
Status: Accepted
Multiple workers run concurrently. The system must guarantee that only one worker executes a given job at a time, and that jobs are automatically recovered if a worker crashes mid-execution.
Use a lease-based execution model. When a worker claims a job:
status = 'RUNNING'
worker_uid = <worker_id>
lease_expires_at = NOW() + lease_secondsThe worker renews the lease via periodic heartbeat. If the worker dies and the lease expires, the job reverts to PENDING and becomes reclaimable.
| Option | Pros | Cons |
|---|---|---|
| Distributed locks | Strong exclusivity | Difficult recovery; requires external coordination system |
| Queue-based workers | Simple model | Poor crash recovery semantics |
The lease system provides safe distributed execution and automatic crash recovery without external coordination infrastructure. Retry bounds (max_attempts) prevent infinite loops on persistently failing jobs.
Status: Accepted
After introducing artifact-backed payloads and results, API responses that return only scalar fields (status, result inline) leave clients without visibility into where data is stored or how to access it directly.
Expose full artifact metadata in job query responses:
{
"job_id": "...",
"state": "SUCCEEDED",
"task_type": "infer",
"input_uri": "fs://jobs/.../input.json",
"output_uri": "fs://jobs/.../output.json",
"payload_size_bytes": 214,
"content_type": "application/json",
"result": { ... }
}The API resolves result from output_uri when the result is not stored inline, and degrades gracefully (returns null) if artifact resolution fails.
Clients gain full visibility into artifact locations, enabling direct artifact access, result reprocessing, and downstream pipeline integration.
Status: Accepted
Different task types have different resource profiles. sleep is useful for recovery and scheduling demos, inference exercises the generic model-execution path, and gpu_demo demonstrates resource-aware routing. Optional runners such as tiny_llm can be enabled when a model-specific integration smoke test is useful, but they are not required for the default public demo. A single undifferentiated worker pool cannot route workloads to appropriate resources.
Introduce named worker groups. Each group is configured with the task types it will claim:
heavy:
task_types: [sleep, inference, tiny_llm, gpu_demo]Workers claim only jobs whose task_type matches their group configuration.
In the public quickstart, the main examples focus on sleep, inference, and gpu_demo. tiny_llm remains visible in the configuration as a disabled optional runner so the extension point is visible without making first-run success depend on model assets.
Worker groups provide workload isolation, independent concurrency tuning per group, and a clear extension point for future GPU node routing and batch pipeline workers.
Status: Accepted
Artifact-backed execution introduces multiple failure points that unit tests do not cover: artifact writing, cross-pod artifact resolution, metadata synchronization between the artifact write and the DB update, and API result resolution from output_uri.
Add end-to-end tests that validate the complete artifact lifecycle in an integrated environment:
- API writes
input.jsonand returnsinput_uri - Worker resolves payload from
input_uri(not DB payload) - Worker writes
output.jsonand recordsoutput_uri - DB stores
output_uriafter job completion GET /v1/jobs/{id}returns result resolved fromoutput_uri
The strong validation variant intentionally corrupts the DB payload before the worker runs, to confirm the worker truly depends on the Artifact Store and not the DB fallback.
Regressions in the artifact data plane are caught automatically. The destructive validation pattern provides high confidence that the artifact path is genuinely active, not silently bypassed.
The MLIS architecture follows a modern ML infrastructure pattern:
| Concern | Component |
|---|---|
| Metadata & scheduling | PostgreSQL (Control Plane) |
| Payload & result storage | Artifact Store (Data Plane) |
| Execution | Stateless worker pods |
| Coordination | Lease-based, no external lock manager |
This separation enables independent scalability of storage and compute, durable artifact persistence, and straightforward crash recovery — the prerequisites for production ML workloads.