Skip to content
Models & inference

Models & inference

Everything on the model plane is sovereign: the platform serves its own models, and there is no hosted-model path. mistral.rs serves the language, guard, reranker, speech-to-text, embedding and vision tiers; image generation runs stable-diffusion.cpp, and speech output runs in ttsd on ONNX Runtime (from the model cards).

The console’s Models & inference page (a rail row until 2026-11-01, then a sub-tab of Agents) was removed on 2026-11-02: the owner judged it redundant. ?tab=inference and ?tab=modelCards are retired names and open DevEx › Clusters › Nodes & servers, where the servers and what they serve are. What follows describes the model plane itself, not a console page.

Where each tier is served

The router sends each request to the backend of the tier its model field names. The backends are router settings (BACKEND_<TIER>) in its manifest:

TierModel (from the card)Served by
generalQwen3.8-27Bthe off-cluster inference server, through servefwd
coderQwen3.8-27B (the same server and model as general)the off-cluster inference server, through servefwd
guardQwen3Guard-Gen-0.6Bthe off-cluster inference server, through servefwd
rerankerQwen3-Reranker-0.6Bthe off-cluster inference server, through servefwd
sttVoxtralthe off-cluster inference server, through servefwd
embeddingQwen3-Embedding-0.6Bthe cluster
visionQwen2-VL-7B-Instructthe cluster
imagegenQwen-Image-2512the cluster

A card served off-cluster is judged by the live router’s backend setting, not by whether a Deployment exists. The off-cluster server also trains, and serves through its training window; when it cannot be reached, its tiers answer 503 with Retry-After: there is no second server. The servers themselves are on DevEx › Clusters › Nodes & servers.

Model cards

GET /api/models (session-gated) returns one row per serving tier, joined from three sources:

SourceWhat it gives
cardthe reviewed model card, embedded in the console: provenance, licence, pin, intended use, scope, evaluation evidence
livethe same tier projection /api/inference renders: model, quantization and context from the container spec; ready, restarts, last termination and digest from the pods; memory from metrics.k8s.io
benchthe latest inference bench, attached only when the measurement can be attributed to that tier

The health verdict is the join. A tier is healthy only if it is deployed, ready, not just OOM-killed, and running what its card names. A tier that serves happily from the wrong weights is not healthy. Live memory at or above 90% of the limit is degraded (modelMemoryWarnPct).

Usage is reported only as far as it exists:

  • Token use comes from the app agents’ own /api/agent-health reports. It is cumulative since each agent process started, and is neither a time series nor per tenant.
  • Per-tenant compute allocation is ClientInstance.spec.computeUnits, a sizing budget. Billed model time is measured separately, in slot milliseconds, by the model router (below); see Plans and billing.
  • Admission, queue depth and token quotas are not exported by anything. They appear as unmeasured[] entries.

The deployed plane

The page reads GET /api/models and the inference.router block of GET /api/agents. GET /api/inference remains the plane as deployed: engine and version (from the image reference; :latest is reported as :latest, not as a version), running digest, model and quantization (from --quantized-file/--model-id), context window, device, requests and limits, live memory and CPU, ready/desired, restarts and last termination.

  • An empty context window means the engine default was taken. That is a finding, not a blank.
  • Live usage is absent, not zero, when no metrics source is available. When metrics.k8s.io does not exist, the console reads the kubelet Summary API through the apiserver node proxy and labels the source kubelet-summary. A 403 is unmeasured, never zero.

The model router (cmd/modelrouter)

One OpenAI-compatible front door that sends each request to the backend of the tier its model field names. It is also the one admission queue in front of the engines: the engines decode a bounded number of requests at a time, and the router decides which request is served next.

What a caller can rely on:

  • Work waits rather than fails. When every slot is busy, a request queues. When the router cannot serve it in time, it answers 503 with Retry-After instead of holding it indefinitely. The two messages are modelrouter: inference queue full; retry later and modelrouter: no inference slot within <wait>; retry later.
  • A caller that leaves takes its work with it. A request that is still queued is dropped before the engine sees it, and one in flight is aborted, so the slot is freed for someone who is still waiting.
  • A caller can say how long it will wait, and the router never queues work past that deadline.
  • People are served ahead of background work. A request made for a person waiting in the console goes ahead of agents and schedulers, and that priority cannot be claimed without proof: the console signs those requests with a key the TIDE operator mints, and an unsigned claim is ignored. Background work is never starved, and nothing is refused for lack of priority.
  • Tiers that cannot be substituted are not. A request for a tier with no backend of its own either falls back to the general tier (coder and vision) or is refused (embedding and voice), because a chat model returns no usable vectors or audio.
  • GET /status reports, per backend, the queue and the wait it can promise.
  • It orders only traffic that goes through it.

The router’s configuration lives in the deployment manifest. Its admission and scheduling internals are not documented here: they are the platform’s own, and they change with measurement.

The ops doctor and the inference bench deliberately call the engine directly: the doctor must not depend on a hop it reports on, and the bench measures the engine itself.

Budgets come from one rate

Every model-calling agent derives its max_tokens and timeouts from one configured decode rate, INFERENCE_DECODE_TOK_S (the inference library’s DecodeRateEnv). The default is 3.4 tok/s, an estimate for CPU-only server hardware, to be measured with the inference bench. The platform decodes one request at a time, so a 2000-token answer is roughly ten minutes of pure decode. Size budgets from the rate, never from wall-clock intuition, and never from a developer workstation.