Prerequisites and ordering

FACE has four boot-time requirements. The order they are satisfied in matters. Each one must be present in the target namespace before you roll an image that requires it. If the image goes first, the pod crashloops instead of starting. A crashloop is the designed outcome, because an operator sees it within the first minute. Still, it is an outage you can avoid by landing the deployment side first.

Requirement	Supplied as	Where it comes from
Session signing key	`AUTH_JWT_SECRET`	The `face-session` Secret. The `runink` operator creates it if it is absent.
Shared mesh CA	`MESH_CA_CERT`, `MESH_CA_KEY`	The `core-mesh-ca` Secret, distributed with `core mesh ca --namespace <ns>`.
At-rest key encryption key (KEK)	`CORE_ENVELOPE_KEK`	The `core-envelope-kek` Secret. The operator creates it if it is absent and never overwrites it.
Inference endpoint	`INFERENCE_REMOTE_URL`	Set by the operator, or in your resource’s `env`.

The three secrets are read through the platform secret backend (SECRET_BACKEND). With the default env backend, they come from the environment. With SECRET_BACKEND=file, they come from files in SECRETS_DIR (default /var/run/secrets), and the environment is the fallback. A file may be named exactly like the variable (AUTH_JWT_SECRET) or in lower case with dashes (auth-jwt-secret).

Distribute the mesh CA

core mesh ca --namespace <ns>
kubectl -n <ns> get secret core-mesh-ca -o jsonpath='{.data.MESH_CA_CERT}' | base64 -d | openssl x509 -noout -dates

The Secret needs both keys, and the CA must be inside its validity window. An expired CA fails the same way a missing one does, and restarting the pod does not help: redistribute the CA first.

Confirm the KEK and session Secrets

kubectl -n <ns> get secret core-envelope-kek face-session

When the operator reconciles a ControlPlane, it creates both Secrets if they are missing. If you provision them yourself, the KEK must be a base64-encoded 32-byte key (openssl rand -base64 32). Escrow the KEK before any data is written. See Persistence and backup.

Confirm the inference endpoint

kubectl -n <ns> set env deploy/<instance> --list | grep INFERENCE_REMOTE_URL

Then prove the endpoint answers. See Model plane.

Roll the image

Only now roll the FACE image. See Upgrades and CD.

The fatal-at-boot checks

These checks run in runServer in the order shown. Each one ends the process with the message quoted, followed by guidance text that is not reproduced here.

1. Session signing key. This runs first, before the port is bound.

❌ refusing to serve: face has no session signing key configured (AUTH_JWT_SECRET unset)

There is deliberately no generated fallback. A generated key would differ per replica, so users would be signed out whenever a request reached another pod. Outside a cluster, set AUTH_JWT_SECRET directly.

2. Mesh identity.

❌ mesh identity unavailable, refusing to start: MESH_CA_CERT: secret "MESH_CA_CERT" is not configured

The same line names MESH_CA_KEY when that key is the one missing, and carries the CA loader’s error when the CA is unusable. The mesh planes (control plane to runner, raft peers, the mTLS half of the app port) are never served in cleartext. No setting of any variable turns this check off.

3. Embedding format. The process exits if EMBEDDING_FORMAT names an unknown format:

EMBEDDING_FORMAT: corpus: unknown embedding format "<name>" (known: …)

4. Inference endpoint.

no inference endpoint configured: INFERENCE_REMOTE_URL is unset or blank.

FACE ships no inference binary and has no local fallback tier. This check only proves that a value is set. It does not prove that the plane is reachable, or that a model is loaded.

5. The appfs root. This is where sessions live, so FACE will not serve without it.

FACE appfs root: <wrapped error> … envelope: at-rest KEK unavailable: CORE_ENVELOPE_KEK: secret "CORE_ENVELOPE_KEK" is not configured
Check the Secret is projected: kubectl -n runink-system get secret core-envelope-kek

The wrapped part depends on the backend. With no object store it is appfs: refusing to mount without at-rest encryption:. With an object store it starts approot: object store configured but unavailable:. The line always starts FACE appfs root: and always ends with the KEK error.

For any other mount failure, including an object store that is configured but unavailable, the message is:

❌ FACE appfs root could not be mounted (refusing to serve without its session store): <error>

6. MetaDB (control plane only). A missing KEK is fatal here too, and the message starts with Durable metadata store:. Other MetaDB failures are logged as Durable metadata store unavailable: <error>, and the process keeps going.

7. Later services. The process also stops if one of these fails:

ConfigService: ❌ Failed to init ConfigService (refusing to serve without it — connections, runners, integrations and SQL credentials all resolve through this store)
IdentityService: ❌ Failed to init IdentityService (refusing to serve without authentication)
mounting the shared UI components: ❌ ui estate, ❌ ui datasources, ❌ ui billing, ❌ ui profile, ❌ ui access
starting the raft orchestrator while RAFT_NODE_ID is set: raft orchestrator failed to start and RAFT_NODE_ID=<n> is set, so this pod is a configured voting member
on a runner, RUNNER_ENROLL_ADDR is set but RUNNER_ENROLL_TOKEN is not (see Runners).

The object store refuses to start without credentials: objectstore: refusing to start without credentials. It also refuses if the removed variable OBJECTSTORE_USE_SSL is still set. Put the scheme in OBJECTSTORE_ENDPOINT instead (https://host:port).

In a cluster, a missing required Secret shows up before any of these log lines, as CreateContainerConfigError on the pod. The operator marks the references to face-session, core-envelope-kek and core-mesh-ca as required on purpose, so that the failure names the Secret.

Deployment model Configuration reference