Skip to content
Health and troubleshooting

Health and troubleshooting

Health endpoints

The backend serves three unauthenticated HTTP endpoints on its app port, 7100 on the control plane and 7102 on a runner. Behind the mTLS ambassador the control plane’s app port listens on 127.0.0.1 only, so it serves these three paths, and nothing else, on the health port 7101 as well. The kubelet probes that port.

PathAnswersUse it for
GET /livezAlways 200 with "status": "alive", once the process has reached its serving loop. It checks no shared dependency.Liveness. A fleet-wide failure then does not restart every pod at once.
GET /readyz200 ready or 503 not_ready, with one entry per check.Readiness. When it fails, the pod is taken out of rotation but stays alive to be inspected.
GET /auth/configThe FACE app’s sign-in configuration.The operator’s default probe path. It answers once the web handler is serving.

/readyz evaluates three checks:

  • mesh_identity fails when the rotating mesh leaf is unhealthy, or does not chain to the loaded CA. Track the mesh CA’s expiry: an expired CA stops every mesh connection.
  • raft_consensus is evaluated only on a consensus member (RAFT_NODE_ID set). It fails when the member has had no leader for longer than a 30-second grace period, or when the raft loop does not answer a status query. On a runner it is skipped.
  • basemap fails when the image ships a basemap for the twins map but the archive at FACE_MAP_BASEMAP_PMTILES is missing or unreadable. It is skipped when the image was built without one; the map then draws plain ground and says so. The boot log line starts 🗺️ NO BASEMAP.
kubectl -n face-system port-forward deploy/face-control-plane 7100:7100 &
curl -s localhost:7100/readyz | jq

Kubelet probes

The operator puts startup, readiness and liveness probes on the backend container. Tune them under spec.probes of the ControlPlane resource. The static face-runner carries its own probes in its manifest.

FieldDefault
path/auth/config. It is used by all three probes, and it is always the startup probe’s path.
readinessPath, livenessPathpath
startupSeconds900. This is the boot budget before the kubelet restarts the container. Probes run every 10 seconds.
readinessFailureThreshold, livenessFailureThreshold3, 6. Liveness runs every 30 seconds.
timeoutSeconds5
disabledfalse. It removes all three probes.

To make readiness reflect the mesh and consensus, set readinessPath: /readyz and livenessPath: /livez. Do not point the startup probe or the liveness probe at /readyz. A shared failure, such as an expired CA, would then restart every pod and destroy the state you need to diagnose it.

Symptoms

The pod never starts: CreateContainerConfigError

A required Secret or ConfigMap is missing from face-system. kubectl describe pod names it. The usual ones are tide-mesh-ca, tide-envelope-kek, face-session and the face-install-config ConfigMap. Create or distribute it (see Prerequisites and ordering). The pod starts on its own at the next retry.

The pod crashloops

Read the first lines of the previous container’s log:

kubectl -n face-system logs deploy/face-control-plane -c face-backend --previous | grep -m3 -E '❌|refusing|no inference endpoint|FACE appfs root|Legacy face.db'
Log line starts withCause and fix
❌ refusing to serve: face has no session signing key configured (AUTH_JWT_SECRET unset)The session key is missing. Provide face-session, or set AUTH_JWT_SECRET outside a cluster.
❌ mesh identity unavailable, refusing to start:The CA is missing, incomplete or expired. Check both keys and the validity dates, then redistribute with tide mesh ca --namespace face-system. Restarting does not fix an expired CA.
no inference endpoint configured: INFERENCE_REMOTE_URL is unset or blank.Set INFERENCE_REMOTE_URL. The retired names REMOTE_LLAMA_URL and LLAMA_REMOTE_URL are not read.
FACE appfs root: … at-rest KEK unavailableTIDE_ENVELOPE_KEK did not resolve. Check kubectl -n face-system get secret tide-envelope-kek.
FACE appfs root: … envelope: TIDE_ENVELOPE_KEK is not valid base64 or KEK must be 32 bytesThe key is malformed. Do not generate a new one if data already exists: restore the original from escrow.
❌ FACE appfs root could not be mounted … object store configured but unavailableThe object store is unreachable, or the credentials are wrong. objectstore: refusing to send object-store traffic … means the endpoint needs https://.
objectstore: OBJECTSTORE_USE_SSL is setRemove the variable, and put the scheme in OBJECTSTORE_ENDPOINT.
raft orchestrator failed to start and RAFT_NODE_ID=<n> is setConsensus state could not be opened. Often the previous pod still held its lock; the next restart usually clears it. If this pod is not meant to vote, unset RAFT_NODE_ID.
❌ RUNNER_ENROLL_ADDR is set but RUNNER_ENROLL_TOKEN is emptyA self-hosted runner is missing its token (see Runners).
SQLITE_PATH="…" must be an absolute path or PORT="…" is not a port numberThe image entrypoint rejected its input. This applies only when the image runs through /app/entrypoint, outside the operator.

Running, but panels are empty or say nothing is there

FACE never fills a panel with data it did not read. An empty panel is either an empty estate or a stated failure. Work through these in order:

  1. No data source. On Why, the domain graph answers connect a data source first, and posture answers no data sources to assess. A fetch with no connection selected fails with No data source was selected. Add a connection in Settings › Integrations › Systems (see Data sources), include it on Ask, and ask a question.
  2. The fetch never reached a runner. The fetch reports Analysis failed: dedicated runner unreachable — fetch does not run on the control plane:. Check the runner’s readiness and MANAGED_RUNNER_ENDPOINT.
  3. The model plane is not answering. The boot log shows the configured URL, but calls fail. Call ModelService/Generate as shown in Model plane. A FailedPrecondition that names INFERENCE_REMOTE_URL means one call path has no endpoint.
  4. Briefs produce nothing. Without SCHEDULE_EXECUTE_FETCH=true, each trigger logs 📅 scheduled trigger (dry-run): … — set SCHEDULE_EXECUTE_FETCH=true to run the real fetch.
  5. The instance picker says “Instance discovery is unavailable”. The pod could not list ClientInstances. Its ServiceAccount needs list on clientinstances.core.runink.org.

Sign-in fails

MessageCause
invalid or expired session tokenThe session was revoked or has expired, or the bearer token was empty. On runners: the runner does not share the control plane’s appfs root.
Password sign-in is disabled — use Google sign-inAUTH_GOOGLE_ONLY=true. Sign in with Google.
SSO is not available on this instanceSSO is configured, but AUTH_ALLOWED_EMAILS is empty. The log line is sso_login_refused_no_allowlist.
This account is not authorized for this instanceThe account is not in AUTH_ALLOWED_EMAILS. The operator projects the list from the face-oauth Secret (allowed-emails). The pod must restart to pick it up.
OIDC not configured (set OIDC_ISSUER and OIDC_CLIENT_ID)SSO was attempted with no issuer or client id set.
A Google token is rejected for its audienceThe FACE app’s google_client_id (in face-frontend-config) must equal OIDC_CLIENT_ID, or set OIDC_AUDIENCE.
The admin cannot log in at allLook for break-glass admin not seeded: ADMIN_EMAIL and/or ADMIN_PASSWORD unset at boot. The operator takes both from the face-control-plane-admin Secret.

Grounding finds nothing

Retrieval is a word match, not a meaning match (see Grounding). A passage is found only when it shares words with the question. Check that the documents were ingested: Why lists the documents behind the business rules. Then ask with the words the documents use. No model or embedding setting affects retrieval.

A change “does not stick”, or a roll changes nothing

  • A config change is reverted. You edited the live Deployment, and the operator reconciled it back. Change the ControlPlane resource’s env instead.
  • A value in env is ignored. Some names are appended after env. TIDE_PLATFORM_ADMIN and FACE_TENANT always win.
  • Same image digest after the roll. The build did not reach the registry, or the commit produced an identical image. Pin the digest to be sure what is running.
  • RAFT_DATA_DIR unset — raft state is MEMORY-ONLY. This is fine for local development. In a cluster, the operator sets it to /raft.