Skip to content

Deployment model

A FACE instance has three workloads. The operators in CORE build most of them. The image is the same for the control plane and the runners. Only the role differs.

WorkloadRoleBuilt byContainer port
Control planeSERVICE_ROLE=server. It serves the cockpit’s gRPC-web and gRPC API, holds sessions and metadata, runs schedules and self-healing, and forwards every fetch to a runner. It never runs a fetch itself.runink operator, from a ControlPlane resource7100
Fetch runnerSERVICE_ROLE=runner. It runs the fetch pipeline (connectors, derivations, agents) on dedicated compute.runink operator, from a FaceInstance resource, or a static manifest (see below)7102
Frontend sleeveIt serves the Flutter web bundle with a single-page-app fallback and reverse-proxies backend calls to the control plane.Static manifest in CORE’s on-host bundle8080 in the image

TLS for browsers ends at the platform edge in front of these workloads. FACE itself is not configured with a certificate for the browser.

The custom resources

The runink operator (in CORE, grpc/operators/runink) owns two kinds in the API group runink.org/v1alpha1.

ControlPlane (short name cp) describes the control plane.

FieldEffect
imageThe FACE backend image.
replicasThe pod count. The Deployment uses the Recreate strategy (see Upgrades).
raftNodeIdThis control plane’s consensus node id, by convention 1.
tenantBecomes FACE_TENANT. When it is empty, the operator uses the one tenant that all of this control plane’s FaceInstances name. If they name more than one, FACE_TENANT is left unset and the operator logs that.
objectStoreEndpointThe object-store Service, as host:port with no scheme. It is used for OBJECTSTORE_ENDPOINT and for Litestream. The default is face-objectd:9000.
embeddingUrlBecomes EMBEDDING_URL.
managedRunnerEndpointBecomes MANAGED_RUNNER_ENDPOINT: the host:port the control plane forwards fetches to.
scheduleExecuteFetchSets SCHEDULE_EXECUTE_FETCH=true.
ambassadorAdds the optional mTLS ambassador sidecar.
probesTunes the kubelet probes (see Health and troubleshooting).
envExtra environment variables. They are appended after the operator’s own list, so a literal here overrides the operator’s value of the same name. FACE_TENANT, RUNNER_ID and the default FACE_APPFS_DIR are appended after env, so you cannot override those three with env.
nodeSelectorA hard pin to labelled nodes, on top of the operator’s default placement.

FaceInstance (short name fi) describes one tenant’s fetch runner. The operator names the runner Deployment face-runner-<instance>.

FieldEffect
tenantThe tenant this runner serves. It becomes FACE_TENANT. It must equal the control plane’s tenant, because the control plane reads the fetch results this runner writes under that tenant.
imageThe FACE backend image.
controlPlaneRefThe ControlPlane whose mesh this runner joins.
computeUnitsThe tenant’s compute budget. The operator turns it into CPU and memory requests and limits and reports them in status.allocatedCpu and status.allocatedMemory.
tiermanaged (the default) or local.
gpuRequests GPU scheduling.
embeddingUrlBecomes EMBEDDING_URL on the runner.
inference.modelThe model hint for the runner. The other inference.* fields (gpuLayers, contextSize, parallel, threads, batch) are deprecated. They are still accepted so that existing resources validate, but the operator no longer writes them anywhere.
suspendScales the runner to zero without deleting the resource.
env, nodeSelector, probes, ambassadorAs on ControlPlane.

status.phase moves through Pending, Provisioning, Ready, Degraded and Suspended.

A third kind, ClientInstance (core.runink.org/v1alpha1), is owned by CORE’s core operator. It records an instance’s app, tenant, image, users and subscription. FACE reads it in two ways:

  • The cockpit’s instance picker lists ClientInstances whose spec.app matches RUNINK_APP (default face). The pod’s ServiceAccount needs list on clientinstances.core.runink.org for this.
  • The operator mirrors a ClientInstance’s users and subscription onto the pod as AUTH_SEED_USERS, SUBSCRIPTION_TIER, SUBSCRIPTION_STATUS, SUBSCRIPTION_SEATS and TENANT_ID.
The operator is the source of truth for the pod spec. If you edit a running Deployment, the operator reconciles your change away. Change the ControlPlane or FaceInstance resource instead, usually through its env list.

What the operator puts in the control-plane pod

Containers

  • face-backend runs /app/server serve --port 7100.
  • litestream runs litestream replicate -config /app/litestream.yml. It continuously replicates the SQLite metadata database.
  • The ambassador sidecar is added when spec.ambassador enables it.

Init containers, in this order:

  1. Two ownership init containers prepare the connections volume and the raft volume. The connections directory is made owner-only (mode 0700).
  2. ensure-bucket runs /app/server ensure-bucket.
  3. litestream-restore runs litestream restore -if-db-not-exists -if-replica-exists -config /app/litestream.yml /data/face.db.

Volumes

MountKindHolds
/dataemptyDirface.db (SQLITE_PATH=/data/face.db). It is rebuilt from the Litestream replica on every pod start.
/raftnode-local hostPath, one per memberConsensus state (RAFT_DATA_DIR).
/connectionsnode-local hostPath, one per memberThe legacy connection layout (FACE_CONNECTIONS_DIR). When no object store is configured, it also holds the encrypted appfs root at /connections/appfs.

Environment. The operator sets the role, port, raft settings, inference and embedding URLs, object-store and Litestream settings, and admission defaults (FACE_ADMIT_MAX=1, FACE_ADMIT_QUEUE=32). It also references these Secrets:

SecretKeysRequired on the podCreated by the operator
face-sessionAUTH_JWT_SECRETyesyes, if it is absent. It is never rotated on reconcile.
core-envelope-kekCORE_ENVELOPE_KEKyesyes, if it is absent. It is never overwritten.
core-mesh-caMESH_CA_CERT, MESH_CA_KEYyesno. You distribute it with core mesh ca --namespace <ns>.
<controlplane>-adminemail, passwordyesyes. It holds the break-glass admin as ADMIN_EMAIL and ADMIN_PASSWORD.
objectd-key-face-app, objectd-key-face-litestream, objectd-key-face-vacuumaccess-key, secret-keyyes, once they existby CORE’s provisioner. Until they exist, the pod keeps the shared objectstore-creds pair.
face-oauthgoogle-client-id, allowed-emailsoptionalno
licenseLICENSE, LICENSE_PUBLIC_KEYoptionalno. You mint it with CORE’s license tool.

A required reference to a missing Secret stops the pod before FACE runs. Kubernetes reports it as CreateContainerConfigError and names the Secret. That is deliberate: the cause shows up in kubectl describe pod and does not have to be found in an application log.

The runner pod is built the same way. The differences: it runs as SERVICE_ROLE=runner on port 7102, and RUNNER_ID is the runner Deployment’s name, so the runner’s own state survives restarts. It does not set RAFT_NODE_ID, because runners are not consensus voters.

The static on-host manifests

CORE’s on-host bundle (grpc/infrastructure/apps/onhost/40-face.yaml, applied with kubectl apply -k infrastructure/apps/onhost from CORE’s grpc/ directory) declares two FACE workloads that no operator reconciles:

  • face-frontend, the sleeve. BACKEND_ADDR points it at the control-plane Service.
  • face-runner, a static fetch runner (SERVICE_ROLE=runner, port 7102). The backend’s default MANAGED_RUNNER_ENDPOINT is face-runner:7102, and it points at this runner.

The control plane is not in that file. The runink operator owns it, so edit the ControlPlane resource and not the manifest.

The frontend sleeve

The sleeve (flutter/sleeve/) is a small static Go server in a distroless image. It serves /web and forwards a request to BACKEND_ADDR when the request is one of these:

  • a gRPC-web request;
  • a request under /auth/, /api/ or /webrtc/;
  • a dotted gRPC service path.

The release web build bakes in no backend host. The cockpit takes its origin from the page it was loaded from, so one bundle works behind any hostname.

VariableDefaultPurpose
PORT8080The listen port.
WEB_DIR/webThe Flutter bundle directory.
BACKEND_ADDRthe control-plane ServiceWhere backend calls are forwarded.

The edge may split gRPC-web off to the control plane before a request reaches the sleeve. In that case the sleeve only serves the static bundle.