Models & inference
Everything on the model plane is sovereign: the platform serves its own models, and there is no hosted-model path. mistral.rs serves the language, guard, reranker, speech-to-text, embedding and vision tiers; image generation runs stable-diffusion.cpp, and speech output runs in ttsd on ONNX Runtime (from the model cards).
The console’s Models & inference page (a rail row until 2026-11-01, then a sub-tab of Agents) was
removed on 2026-11-02: the owner judged it redundant. ?tab=inference and ?tab=modelCards are
retired names and open DevEx › Clusters › Nodes & servers, where the servers and what they serve
are. What follows describes the model plane itself, not a console page.
Where each tier is served
The router sends each request to the backend of the tier its model field names. The backends are
router settings (BACKEND_<TIER>) in its manifest:
| Tier | Model (from the card) | Served by |
|---|---|---|
| general | Qwen3.8-27B | the off-cluster inference server, through servefwd |
| coder | Qwen3.8-27B (the same server and model as general) | the off-cluster inference server, through servefwd |
| guard | Qwen3Guard-Gen-0.6B | the off-cluster inference server, through servefwd |
| reranker | Qwen3-Reranker-0.6B | the off-cluster inference server, through servefwd |
| stt | Voxtral | the off-cluster inference server, through servefwd |
| embedding | Qwen3-Embedding-0.6B | the cluster |
| vision | Qwen2-VL-7B-Instruct | the cluster |
| imagegen | Qwen-Image-2512 | the cluster |
A card served off-cluster is judged by the live router’s backend setting, not by whether a
Deployment exists. The off-cluster server also trains, and serves through its training window; when it cannot be reached, its
tiers answer 503 with Retry-After: there is no second server. The servers themselves are on
DevEx › Clusters › Nodes & servers.
Model cards
GET /api/models (session-gated) returns one row per serving tier, joined from three sources:
| Source | What it gives |
|---|---|
| card | the reviewed model card, embedded in the console: provenance, licence, pin, intended use, scope, evaluation evidence |
| live | the same tier projection /api/inference renders: model, quantization and context from the container spec; ready, restarts, last termination and digest from the pods; memory from metrics.k8s.io |
| bench | the latest inference bench, attached only when the measurement can be attributed to that tier |
The health verdict is the join. A tier is healthy only if it is deployed, ready, not just OOM-killed, and running what its card names. A tier that serves happily from the wrong weights is not healthy. Live memory at or above 90% of the limit is degraded (modelMemoryWarnPct).
Usage is reported only as far as it exists:
- Token use comes from the app agents’ own
/api/agent-healthreports. It is cumulative since each agent process started, and is neither a time series nor per tenant. - Per-tenant compute allocation is
ClientInstance.spec.computeUnits, a sizing budget. Billed model time is measured separately, in slot milliseconds, by the model router (below); see Plans and billing. - Admission, queue depth and token quotas are not exported by anything. They appear as
unmeasured[]entries.
The deployed plane
The page reads GET /api/models and the inference.router block of GET /api/agents. GET /api/inference remains the plane as deployed: engine and version (from the image reference; :latest is reported as :latest, not as a version), running digest, model and quantization (from --quantized-file/--model-id), context window, device, requests and limits, live memory and CPU, ready/desired, restarts and last termination.
- An empty context window means the engine default was taken. That is a finding, not a blank.
- Live usage is absent, not zero, when no metrics source is available. When metrics.k8s.io does not exist, the console reads the kubelet Summary API through the apiserver node proxy and labels the source
kubelet-summary. A 403 is unmeasured, never zero.
The model router (cmd/modelrouter)
One OpenAI-compatible front door that sends each request to the backend of the tier its model
field names. It is also the one admission queue in front of the engines: the engines decode a
bounded number of requests at a time, and the router decides which request is served next.
What a caller can rely on:
- Work waits rather than fails. When every slot is busy, a request queues. When the router
cannot serve it in time, it answers 503 with
Retry-Afterinstead of holding it indefinitely. The two messages aremodelrouter: inference queue full; retry laterandmodelrouter: no inference slot within <wait>; retry later. - A caller that leaves takes its work with it. A request that is still queued is dropped before the engine sees it, and one in flight is aborted, so the slot is freed for someone who is still waiting.
- A caller can say how long it will wait, and the router never queues work past that deadline.
- People are served ahead of background work. A request made for a person waiting in the console goes ahead of agents and schedulers, and that priority cannot be claimed without proof: the console signs those requests with a key the TIDE operator mints, and an unsigned claim is ignored. Background work is never starved, and nothing is refused for lack of priority.
- Tiers that cannot be substituted are not. A request for a tier with no backend of its own either falls back to the general tier (coder and vision) or is refused (embedding and voice), because a chat model returns no usable vectors or audio.
GET /statusreports, per backend, the queue and the wait it can promise.- It orders only traffic that goes through it.
The router’s configuration lives in the deployment manifest. Its admission and scheduling internals are not documented here: they are the platform’s own, and they change with measurement.
The ops doctor and the inference bench deliberately call the engine directly: the doctor must not depend on a hop it reports on, and the bench measures the engine itself.
Budgets come from one rate
Every model-calling agent derives its max_tokens and timeouts from one configured decode rate, INFERENCE_DECODE_TOK_S (the inference library’s DecodeRateEnv). The default is 3.4 tok/s, an estimate for CPU-only server hardware, to be measured with the inference bench. The platform decodes one request at a time, so a 2000-token answer is roughly ten minutes of pure decode. Size budgets from the rate, never from wall-clock intuition, and never from a developer workstation.